The out-of-tree build puts absolute source-root paths on the command line
(-I${MAGICSRC}, -I${srcdir}, the source file itself), which are part of
ccache's key -- so same-tree rebuilds hit (proven) but different checkouts /
build dirs do not share cache.
New opt-in configure --enable-ccache-prefix-map adds
-ffile-prefix-map=${MAGICSRC}=.
to CFLAGS (requires ccache; the flag is probed for GCC-8+/clang support and
skipped with a warning otherwise). This relativizes the source paths embedded
in the compiled output (__FILE__, debug info) so a cached object is valid
regardless of which tree produced it.
Off by default: it makes debug paths relative, so gdb then needs
`set substitute-path . <srcdir>`. It is only for cross-tree *correctness* and
must be paired with CCACHE_BASEDIR=<build top> in the environment for cross-tree
*hits* (that half normalizes the command-line paths in the hash; this half keeps
the cached output tree-neutral). Same-tree caching needs neither.
Verified: default/--disable-ccache carry no -ffile-prefix-map;
--enable-ccache-prefix-map (probe "yes") adds it, and with --disable-ccache it
warns and is ignored. Full build rc=0 (375 files); the installed tclmagic.so
contains zero absolute /work/magic paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New configure option --disable-magic-builddate omits the build date from the
binary. It is wired via an AC_SUBST'd MAGIC_BUILDDATE_DEFS (not AC_DEFINE), so
it reaches only buildinfo.o rather than @DEFS@ / every command line:
* enabled (default): buildinfo.o gets -DMAGIC_BUILDDATE="<date>" and defs.mak
computes the date (SOURCE_DATE_EPOCH-aware);
* disabled: buildinfo.o gets -DMAGIC_NO_BUILDDATE instead, defs.mak skips the
date computation entirely (no parse-time `date` spawn), and buildinfo.c
reports an empty MagicCompileTime.
This gives a fully reproducible build whose one volatile input (the date) is
gone; what remains in buildinfo.o is commit/version/revision, which do not
change second-to-second -- the intended pairing with ccache for day-to-day
development.
Verified both configs: default bakes the date + commit; --disable-magic-builddate
compiles buildinfo.o with -DMAGIC_NO_BUILDDATE (no date), commit still present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MAGIC_BUILDDATE now derives from SOURCE_DATE_EPOCH (UTC) when it is set,
falling back to the current local time otherwise. This makes the date baked
into buildinfo.o reproducible for the appimage/npm tarballs (cf. npm/pack.sh,
which already normalizes mtimes).
Also add `2>/dev/null` to the MAGIC_COMMIT git call so a source tarball or a
non-work-tree ${MAGICSRC} yields an empty commit rather than a git error.
Only buildinfo.o consumes these (see the prior commit), so the shell still
runs at defs.mak parse time but affects just that one unit's define.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MAGIC_VERSION / MAGIC_REVISION / MAGIC_COMMIT / MAGIC_BUILDDATE were on the
global DFLAGS, so every object's compile command line carried them. Because
the command line is part of ccache/sccache's hash key and MAGIC_BUILDDATE
changes every second, that meant a cache miss on every unit on every build;
it also smeared the baked-in date/commit across "whenever each file last
recompiled".
Move the four values into a single translation unit:
* utils/magic_buildinfo.h -- extern MagicVersion/MagicRevision/MagicCommit/
MagicCompileTime (MagicCommit is new; the other three moved here from
utils/magic.h, which now includes this header).
* utils/buildinfo.c -- the ONLY unit compiled with the version defines
(target-specific `buildinfo.o: DFLAGS += ${DFLAGS_MAGICVERSION}` in
utils/Makefile.in); it defines the four globals. ${DFLAGS_MAGICVERSION}
is removed from the global DFLAGS/DFLAGS_NOSTUB.
Consumers updated to read the runtime symbol instead of the compile-time
macro:
* magicTop.c / tclmagic.c no longer define the globals (the MAGIC_WRAPPER
duplicate-symbol guard is gone -- buildinfo.o owns them for every variant,
including WASM, which links both mains).
* Tcl_PkgProvide/PkgRequire in router/ext2spice/lef/ext2sim/plot/tclmagic
now pass MagicVersion.
* extflat: EFVersion (a static-initialized copy of MAGIC_VERSION) is dropped;
EFread.c compares the .ext version against MagicVersion directly -- the same
value ExtCell.c already *writes* into .ext files, so read and write are now
consistent.
Verified: only buildinfo.o carries -DMAGIC_* (hash/DBio/windCmdSZ carry none);
Tcl (375 files) and --without-tcl builds rc=0, version/commit/date embedded in
tclmagic.so, magicTop.o no longer defines MagicVersion; in-tree source clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every converted module sets srcdir=@srcdir@ and VPATH=@srcdir@
before including defs.mak, so the full srcdir derivation is now reached
only by readline/ (never converted) and the top-level Makefile. Neither
compiles magic sources through the pattern rules, so neither needs VPATH.
Drop the MODULE_SUBPATH variable and the VPATH derivation; keep just a
one-line srcdir fallback so CPPFLAGS (-I${srcdir}) stays well-formed for
those two. BUILD_TOP is retained (GENINC still needs it).
Verified rc=0: out-of-tree Tcl (375) + in-tree Tcl (375, header present) +
out-of-tree --without-tcl (readline/libreadline.a + magic/magic built
through the fallback).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enabler for the Makefile -> Makefile.in migration. No module is
converted yet; behaviour is identical.
* scripts/defs.mak.in: dual-mode. A module converted to Makefile.in will
set `srcdir` (and VPATH) from @srcdir@ before including defs.mak; only
derive them (the $(abspath)/$(patsubst) srcdir computation) when it
did not -- `ifeq ($(origin srcdir),undefined)`. BUILD_TOP (needed by
GENINC) is computed either way.
* Makefile.in: route every subdir recursion through a single helper,
$(call submake,<dir>,<goal>)
which uses the generated build Makefile if config.status made one there
(a converted Makefile.in) and otherwise falls back to the source
Makefile with -f. Today nothing is converted, so out-of-tree always
takes the -f path and in-tree uses the (identical) static Makefile
natively. clean/ keeps its own no-mkdir variant. readline/ is
intentionally never converted and always takes -f.
* .gitignore: document the migration policy -- each converted module adds
its generated "<dir>/Makefile" here in the same commit so an in-tree
build can never commit a generated Makefile. (None converted yet.)
Verified in-tree and out-of-tree: `configure && make && make install`
each produce 376 files with the header in the right place (database/
database.h in-tree, include/database/database.h relocated), 27/28 docs,
source tree clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously the generated database/database.h was written to the build
top and exposed with -I${MAGICDIR}. Out-of-tree that placed the raw
build directory on the compiler's include search path.
Introduce ${GENINC}, the single include dir for generated headers:
* in-tree (build top == source top): ${MAGICDIR}, so the header stays
at $(builddir)/database/database.h and -I${GENINC} is exactly the old
-I${MAGICDIR} -- behaviour is unchanged.
* relocated (out-of-tree): ${MAGICDIR}/include, so the header is written
to $(builddir)/include/database/database.h and only that clean
include/ dir is on the search path; the raw build top never is.
* scripts/defs.mak.in: derive GENINC from BUILD_TOP vs MAGICSRC; add
DATABASE_H = ${GENINC}/database/database.h; CPPFLAGS uses -I${GENINC}.
* Makefile.in: the makedbh rule targets ${DATABASE_H} (mkdir its dir);
every prereq and the distclean use ${DATABASE_H}.
* database/Makefile: the fallback delegate targets ${DATABASE_H}.
Verified: relocated `make modules` builds all 33 module libraries with
the header at build/include/database/database.h and no raw build-top on
any -I line; in-tree the header stays at database/database.h and the
-I set is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The non-Tcl ("standard") configuration links the bundled GNU readline,
and that path was broken two ways:
* headers: readline's own headers use <readline/*.h> includes, so the
readline source's parent must be on the include path. Nothing added
it, so every module including readline (commands, textio, ...) failed
with "readline/rlstdc.h: No such file". Add -I${MAGICSRC}/readline to
CPPFLAGS for the bundled build only (@MAKE_READLINE@ = 1).
* library: readline/Makefile ran "../readline/configure" from the
build-readline dir -- a source-relative path that, out-of-tree, points
into the (empty) build tree. Use ${MAGICSRC}/readline/readline/
configure so the out-of-source readline build finds its sources.
Verified: `mkdir b; cd b; ../configure --without-tcl && make` now
completes (rc=0) and produces the native magic binary; previously it
died at commands/CmdFI.o (headers) then at the readline library build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When latex and dvips are available, the default no-arg "make" now
regenerates the PostScript documentation from its .tex sources into the
build tree; otherwise the pre-built PostScript shipped in the source tree
is used as-is at install time. Documentation is always available either
way -- installing latex simply refreshes it from source.
configure (scripts/configure.in, regenerated scripts/configure):
* AC_PATH_PROG for latex + dvips; DOCS_TARGET = "docs" only when both are
found; AC_SUBST(LATEX/DVIPS/DOCS_TARGET).
* Configuration summary prints "LaTeX docs: yes/no".
scripts/defs.mak.in: export LATEX/DVIPS.
Makefile.in:
* all: ... $(DOCS_TARGET) -- pulls in "docs" only when latex was found.
* docs: best-effort -- `make -k` + a warning on failure so a document
that will not typeset (or a missing texlive package) never fails the
build; install falls back to the pre-built PostScript for those.
doc/latexfiles/Makefile:
* use ${LATEX}/${DVIPS} from defs.mak.
* the .tex include figures as ../psfigures/*.ps; point the build-tree
../psfigures at ${MAGICSRC}/doc/psfigures (no-op in-tree) so latex and
dvips find them while outputs stay in the build tree.
CI: main.yml and main-aarch64.yml install
texlive-latex-base/-recommended + texlive-fonts-recommended so the Linux
builds exercise doc regeneration (best-effort, never fatal).
README.Tcl: document latex/dvips as an optional prerequisite for docs.
.gitignore: ignore in-tree doc build artifacts (doc/latexfiles/psfiles/,
*.dvi/*.aux/*.log); the pre-built doc/psfiles/ stays tracked.
Verified out-of-tree with TeX Live 2024: configure reports "LaTeX docs:
yes"; `make` regenerates 27/28 documents into the build tree (maint1.tex
has a genuine LaTeX error and falls back), stays exit 0; `make install`
installs the 27 regenerated PostScript from the build tree and maint1
from the source pre-built copy; the source tree is not modified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make a plain `make` in a build directory build every module into the
build tree. The top Makefile becomes a configure-generated Makefile.in,
and each recursion drives a build subdirectory (created on demand) using
the module Makefile from the source tree.
Makefile -> Makefile.in (config.status generates Makefile in the build
top via AC_CONFIG_FILES([Makefile:Makefile.in])):
* Add MAGICSRC = @abs_top_srcdir@; read VERSION and re-run configure from
${MAGICSRC}.
* Every descent `(cd $$dir && ${MAKE} <goal>)` and `${MAKE} -C $(dir $@)
<goal>` becomes:
mkdir -p <builddir> && ${MAKE} -C <builddir> \
-f ${MAGICSRC}/<dir>/Makefile <goal>
so the build subdir need not pre-exist and the source Makefile is used.
Applied to modules/libs/depend pattern rules and the
tcllibrary/mains/techs/install/clean loops.
* database/database.h: generate from ${MAGICSRC}/.../database.h.in into
the build tree (mkdir -p database), via the generated build-tree
${MAGICDIR}/scripts/makedbh.
* install-dirs/install-tcl-dirs use ${SCRIPTS}/mkdirs (now source-tree);
clean skips absent build subdirs; distclean/dist/tags updated for the
split source/build layout.
scripts/defs.mak.in:
* SCRIPTS now points at the source tree (${MAGICSRC}/scripts) for the
static helpers (mkdirs, printmans, ...); the one generated helper,
makedbh, is referenced directly at ${MAGICDIR}/scripts (build tree).
* Guard VPATH so it is set only inside module subdirectories, never at
the build top -- a top-level VPATH into the source tree could let a
stale source database/database.h shadow the build-tree copy.
scripts/configure.in: AC_CONFIG_FILES([Makefile:Makefile.in]); regenerated
scripts/configure.
Verified out-of-tree: `../configure && make modules && make libs` builds
all 33 module libraries (lib*.o and lib*.a) plus the generated
database/database.h into the build tree, with no errors and the source
tree left clean. A full `make` (link the magic program, build techs)
still needs the later outlier fixes (magic/, scmos/, ...).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With the autoconf layer relocatable, teach the shared make
files to read sources from the source tree and write objects into the
build tree. The ~26 "standard" and "near-standard" module Makefiles now
build out-of-tree with no per-module edits.
scripts/defs.mak.in:
* Add MAGICSRC = @abs_top_srcdir@ (absolute source top) and derive each
module's srcdir/VPATH from CURDIR relative to the build top:
BUILD_TOP := $(abspath $(MAGICDIR))
MODULE_SUBPATH := $(subst $(BUILD_TOP),,$(CURDIR))
srcdir := $(MAGICSRC)$(MODULE_SUBPATH)
VPATH := $(srcdir)
(MAGICDIR keeps meaning the *build* top; in-tree the trees coincide.)
* CPPFLAGS: add -I${MAGICSRC} (source top, for "<module>/<hdr>.h") and
-I${srcdir} (same-dir headers), keeping -I${MAGICDIR} for the generated
database/database.h in the build top.
* MAGIC_VERSION/REVISION read ${MAGICSRC}/VERSION; MAGIC_COMMIT uses
`git -C ${MAGICSRC}` so it works from a build dir outside the work-tree.
rules.mak:
* Compile rule: `-c $*.c` -> `-c $< -o $@` so the source is found via
VPATH in srcdir and the object is written to the build cwd.
* Depend and tags rules: prefix sources with $(srcdir)/ (they are passed
to the compiler/ctags directly, which do not honor VPATH).
scripts/configure.in: AC_CONFIG_FILES([rules.mak:rules.mak]) copies the
VPATH-aware rules.mak into the build top, so a single ${MAGICDIR}
resolves both `include`s in every subdir Makefile. Regenerated configure.
Known limitation (v1): the Depend post-processing sed strips
absolute-path headers, so with an absolute srcdir most header deps are
dropped; out-of-tree incremental header-change rebuilds are weaker.
Depend is already "optimistic/optional"; relativizing is a follow-up.
Verified out-of-tree: `tiles` and `drc` (the latter #includes the
generated database/database.h) build to completion; all objects land in
the build tree; the source tree stays clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Also fixed up CC/CXX/CXXFLAGS in the same way.
Main notable concern the tclmagic.so now has reordered position of
$LDFLAGS (was start AND end of list, now start of list only),
$LDDL_FLAGS would have already provided it start of list, so it
would be duplicated set of options.
@LDDL_FLAGS@ no longer prepend $LDFLAGS which is now explicit in the
Makefile to allow specific target control.
The purpose of all this is the toplevel ./configure now has LDFLAGS
control passed down to linker command lines as configure --help
indicates.
./configure
# If you are brave, enable with your favourite editor after ./configure
sed -e 's/^#FEATURE_FLAGS /FEATURE_FLAGS /' -i defs.mak
make
make install
This is a standard autoconf variable, it was probably disabled due to
how the graphics/Makefile works and the more complex detection taking
place there.
Needed to use it to add -lpthread to some platforms (freebsd/openbsd)
to make it build out the box.
This split has the effect (with modified .o.c Makefile rule of
correctly using the $CC for .c files and not try to build .cpp
with C compiler, which fails in oa/** for example.
This can only cause trouble being a global default setting.
All project include files are qualified with a path prefix to help
disambiguate duplication of header file names to effectively provide
a namespace.
When removing it nothing seemed break. Maybe more configurations
should be tested. They were all ok.
Previously the variable resolution was passed to compiler command line
when ideally this only needs to be resolved once per build and the
entrie build uses the same value.
Compiler command line is computed at least once per file (plus Depends
pass), so this is forking at least 5 extra times during the build.
This improves things somewhat so resolution is performed less.
This may have the effect of making the build appear faster when
ccache is working. As the command line options maybe seen as a
constant.
The -L option needs to be placed before the -l it needs to affect.
This maybe important on MacOSX where a nonstandard/optional package
provides X11 support so the locations are not in system locations.
This is a series of updates to make building magic far less of a headache:
* Drop `csh`/`tcsh` dependency and detection from `./configure`.
* Rewrite makedbh in Python.
* Rewrite printmans in POSIX sh.
* Stop deleting Depend before every compile (which causes some files to recompile and thus increases recompile times significantly)
* Add Depend to CLEANS in scripts/defs.mak.in
* Turn POSIX suffix rule in magic/rules.mak to a pattern rule with proper prerequisites
linking it to Tk or X11 graphics. Added new command "display" which
returns the display type, which is good for finding out if the
display is "NULL". Added code to allow the wrapper to be defined
for NULL graphics with the Tk console, the main necessities of which
are to remove the "openwrapper" command, and to return immediately
from a number of tag callback functions.
number from the git repository, which is overall more reliable than
the version number, but mainly to support a common method across the
open source tools for providing information to builds like open_pdks
that may need to know what version of every tool was used for the
build.
tech file format "version" section. This can be used to specify the
version of magic that must be used to be compatible with the tech file.
This effectively supercedes the technology version number. (2) Changed
the behavior of "make" to set the version and revision numbers on doing
"make" instead of "configure". This allows the version to update
correctly after doing a "git pull" followed by "make" without doing
"configure" in between. (3) Fixed a couple of issues that were flagged
as compile-time warnings.
preproc.py script from another project. Had to further develop it to
get around the amazingly complicated preprocessor usage in the scmos
subdirectory. Needed to add handling of parameterized definitions;
could not figure out how to align the syntax used in scmos/extract_template
with any consistent syntax. Gave up and rewrote some of the contents of
extract_template to avoid the more ambiguous usage. All of this is to
support a completely deprecated scmos.tech. However, it does avoid both
the M4 and cpp preprocessors altogether. Also did auto-detection of
python3 in the configure script for use of the preproc.py preprocessor,
and applied the same preprocessor to the macro definitions.
distributed installations, where the immediate installation location
is different from the final installation location, but in the case
where it is not desirable to put the entire install hierarchy as a
subdirectory of DESTDIR.
the python preprocessor version, the techfiles and .magicrc file
will get the correct arguments, without affecting the use of CPP
in the configure script.
python3 preprocessor is captured in the variable MCPP, not CPP,
since CPP is meaningful to the make process. Also, the
MAGICDIR variable in the name needed to be escaped so that the
variable is not evaluated within the configure script.
Darryl L. Miles
Donn
Tim Edwards