[File] newlines break file tests

[Christos Zoulas]

Ok, this was a big rabbit hole. The key here is the way magic entries are sorted in apprentice_sort() 
with memcmp.  Part of the magic entry is the line number.
So there were 3 PK\003\004 entries in archive and changing the line number, 
changed the sorting order. I killed the line number now in sorting
so duplicate magic entries will produce a warning. I also merged some of the easy ones. 
Merging is not always a good solution because it
moves magic entries to the same file where that is not always ideal (CISO magic in 
console and filesystems for example). The problem is that
we don't have a good way to see if order makes a difference when we have 
duplicate magic and we need to disambiguate them or merge them.
I am leaving the duplicate magic message on for now...

Best,

christos

> On Aug 26, 2024, at 2:21 PM, Christos Zoulas <christos at zoulas.com> wrote:
> 
> They should not have a maximum size. I will take a look.
> 
> christos
> 
>> On Aug 26, 2024, at 1:19 PM, Alexandre IOOSS <erdnaxe at crans.org> wrote:
>> 
>> Hello,
>> 
>> I recently noticed that tests are currently broken on file master branch. `make check` fails on my machine with:
>> ```
>> TZ=UTC MAGIC=../magic/magic ./test -e ../tests/keyman-2.testfile ../tests/keyman-2.result
>> ../tests/keyman-2.testfile: Zip archive data, at least v1.0 to extract, compression method=deflate
>> test: ERROR: result was (len 70)
>> Zip archive data, at least v1.0 to extract, compression method=deflate
>> expected (len 29)
>> Keyman Compiled Package File
>> make[2]: *** [Makefile:791: check-local] Error 1
>> make[2]: Leaving directory '/home/erdnaxe/git/file/tests'
>> make[1]: *** [Makefile:667: check-am] Error 2
>> make[1]: Leaving directory '/home/erdnaxe/git/file/tests'
>> make: *** [Makefile:465: check-recursive] Error 1
>> ```
>> 
>> Reverting commit 3bce3e16ec645be7a012ea9a29ad858510f84d4b makes the test pass, but if you look at the commit it does not really make sense why it would break the test.
>> During debugging, I noticed a strange behavior: adding 50 newlines at line 260 in magic/Magdir/archive breaks the tests/keyman-2 test.
>> I believe this is a bug in the file magic parser, and this bug is triggered when the magic definitions file is too big.
>> 
>> Do files under `magic/Magdir/` folder have a maximum size?
>> 
>> Best regards,
>> -- 
>> Alexandre
>> -- 
>> File mailing list
>> File at astron.com
>> https://mailman.astron.com/mailman/listinfo/file
>