The error was in selStretchEraseFunc2() which was calling
DBErase(). But DBErase() can call DBSrPaintArea() and thereby
start another search of the same plane while the plane is
already in a search. This is absolutely invalid. However, it
is simple just to use selStretchEraseFunc2() to collect a mask
of the types that need to be erased and pass that back to the
caller, and apply DBErase() to each type in the mask, outside
of the DBSrPaintArea() search. With this fix, the tile
allocation is no longer causing problems, and the code to add
tiles deallocated by TiJoinX and TiJoinY to the free tile list
has been uncommented again.
allocation fixes the issue with memory being grabbed constantly
and never released, but re-surfaces an error with tiles being
used after being freed. This is assumed to be a long-standing
bug that has not been found yet. Meanwhile, the error that uses
up excess memory is better than the error that causes layouts to
get completely screwed up.
1) Added a "*showmem" "wizard" command to get a dump of all memory
being used by tiles in the database.
2) Made a slight correction to the way magic detects exact overlap
of instances of the same cell. This probably does not make any
actual difference in practice.
3) Corrected an uninitialized variable in dbReComputeBboxFunc().
4) Changes DBSrCellPlaneArea() to use a static BPEnum variable, so
that it does not waste time allocating and freeing memory for
the same thing over and over again.
5) Corrected a memory leak in the tech file "extract" section that
loses memory every time the extraction style is changed.
6) Corrected the tile join routines to fix a bad memory leak in the
tile allocation and recovery---a fix which was mentioned in issue
#414 but which had not yet been implemented. This has now been
tested and confirmed to work.
This is considered external public API for the tiles module, so
while internal magic code might inline all call sites internally
for performance the DSO (tclmagic.so) should always provide the
API symbol even if it does not use it.
This affects EL7 (gcc 4.8.5) and older:
#define __GNUC_GNU_INLINE__ 1
This does not affect newer GCC from EL8 (gcc 8.5.0) and later has:
#define __GNUC_STDC_INLINE__ 1
This can be seen as:
./Magic-8.3.536-x86_64-EL7.AppImage
/tmp/.mount_Magic-FDdbKA/bin/wish: symbol lookup error: /tmp/.mount_Magic-FDdbKA/lib/magic/tcl/tclmagic.so: undefined symbol: TiFree
This fixes the oversight introduced around commit f8ef715608
tile.h is included by a large part of the project, but the definitions
in sys/mman.h are only needed by the implementation code inside tile.c
so this reduces project wide include dependencies.
Removal of unnecessary function call indirection into TileStoreFree()
since TiFree() is has multiple implementation already and we can
customize TiFree() directly as required.
Move the function definitions into the tile.h to they are visible to
compiler during code generation to consider inline potential.
This has made global 'Tile *TileStoreFreeList' more visible.
database corruption discovered recently that was uncovered by a
commit on Jan. 31 and is caused by DBMergeNMTiles0() using a
freed tile (reported in github issue #404).
the TiAlloc() and TiFree() code path.
The rationale of the changes:
Performing the one-time initialization check (first call to
mmapTileStore()) for every TiAlloc() is unnecessary if the decision code
is reworked to allow NULL pointers to exist in the computation and still
make the correct decision.
Use of 'unsigned long' for pointer arithmetic is not compatible with _WIN64
compiler model LLP64. Changed to 'pointertype'.
The computations addition/subtraction/greater-than were performed
multiple times. Seemed a convoulted method when a single operation
should be good enough.
The use of a single-linked-list with HEAD and TAIL does not make sense.
My gut is telling me that for the purpose of memory allocation a LIFO is
better as a free-and-reuse of the most recently freed item is more likely
to already be in the CPU cache and the oldest freed item is more likely
to have been evicted from CPU cache. So given the use of a custom
allocator and no support to reclaim/compact or manage fragmentation
other factor didn't carry any weight.
Technically TiAlloc() returns undefined memory and the first action of
the caller is to write new values, but the point remains that write is
more likely to cause eviction of something else from cache with the
original FIFO scheme.
Due to the free list use during TiFree() there is a write to each alloc
which will make the cache-line hotter in the cache (less likely to have
been evicted) when using LIFO scheme than FIFO scheme.
Further more the use of HEAD and TAIL had a far more complex
TileStoreFree() than was necessary even for such a list. A 4 line
method turned into 7 with multiple conditions tested when branching.
The TileStoreFreeList/TileStoreFreeList_end were public symbols which may
also impact the freedom the compiler has to optimise around them.
Using LIFO single-linked-list resulted in the removal of
TileStoreFreeList_end and associated simplification.
Use of 'static' for the methods mmapTileStore() and getTimeFromTileStore()
these are not public API so adding the 'static' give the compiler a hint
these methods maybe inlined as they are not accessile from outside this
compliation unit.
The -O3 assembly result looks quite healthy in achieving the original
goal of instruction and branch reductions with the compiler able to
inline all 3 methods into a single TiAlloc().
This commit makes the code (mostly) C99-compatible, enabling to compile
it without the -Wno-error=implicit-function-declaration flag. This
way, Magic becomes usable on arm64 architectures, specifically on Apple
computers with M1/M2 SoC.
R. Timothy Edwards
Darryl L. Miles
Alessandro De Laurenzis