From jason1 at pobox.com  Mon Aug  5 18:24:18 2024
From: jason1 at pobox.com (Jason Summers)
Date: Mon, 5 Aug 2024 14:24:18 -0400
Subject: [File] Script to find issues in pattern files
Message-ID: <CAPqxkbTyAHO6sgjFLgmO_hx-eWC5DZP6D3cn+KODP1jfXrsL8w@mail.gmail.com>

Hello,

I want to announce the release of a Python script I've written that
analyzes magic pattern files, and reports some types of issues that are
likely to be errors. It's at <https://github.com/jsummers/mgchkj>.

Comments, suggestions, etc., are welcome, though it's far from perfect, and
I'm not necessarily planning to do much more with it.

-- 
Jason Summers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20240805/dbc0fb91/attachment.htm>

From erdnaxe at crans.org  Tue Aug 13 15:12:00 2024
From: erdnaxe at crans.org (Alexandre IOOSS)
Date: Tue, 13 Aug 2024 17:12:00 +0200
Subject: [File] [PATCH] Magdir/virtual QEMU replay image
Message-ID: <7c650bce-e92c-4e3d-abf6-3e82bd0d30b6@crans.org>


Hello,

QEMU can generate replay files to deterministically replay virtual 
machine execution later. I seem that file doesn't recognize QEMU replay 
files.

As QEMU is opensource, I looked at ./replay/replay.c in QEMU source tree 
and notice that there is a magic with version.

Attached, you will find a patch to add QEMU replay image format to file.

Example with a replay generated using QEMU master:
```
$ file replay.bin
replay.bin: QEMU replay, version 12 (QEMU 7.1+)
```

Thanks a lot,
Best regards,
-- 
Alexandre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: qemu-replay.diff
Type: text/x-patch
Size: 727 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20240813/8d8e16bd/attachment.bin>

From christos at zoulas.com  Sun Aug 18 16:02:22 2024
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 18 Aug 2024 12:02:22 -0400
Subject: [File] [PATCH] Magdir/virtual QEMU replay image
In-Reply-To: <7c650bce-e92c-4e3d-abf6-3e82bd0d30b6@crans.org>
References: <7c650bce-e92c-4e3d-abf6-3e82bd0d30b6@crans.org>
Message-ID: <00EC9A0B-B399-4D38-8C4A-B94C86F3DDB2@zoulas.com>

Committed, thanks!

christos

> On Aug 13, 2024, at 11:12?AM, Alexandre IOOSS <erdnaxe at crans.org> wrote:
> 
> 
> Hello,
> 
> QEMU can generate replay files to deterministically replay virtual machine execution later. I seem that file doesn't recognize QEMU replay files.
> 
> As QEMU is opensource, I looked at ./replay/replay.c in QEMU source tree and notice that there is a magic with version.
> 
> Attached, you will find a patch to add QEMU replay image format to file.
> 
> Example with a replay generated using QEMU master:
> ```
> $ file replay.bin
> replay.bin: QEMU replay, version 12 (QEMU 7.1+)
> ```
> 
> Thanks a lot,
> Best regards,
> -- 
> Alexandre<qemu-replay_diff.DEFANGED-18>-- 
> File mailing list
> File at astron.com
> https://mailman.astron.com/mailman/listinfo/file
> <sanitizer.log>


From erdnaxe at crans.org  Mon Aug 26 17:19:45 2024
From: erdnaxe at crans.org (Alexandre IOOSS)
Date: Mon, 26 Aug 2024 19:19:45 +0200
Subject: [File] newlines break file tests
Message-ID: <2f2c0033-9791-43ff-a152-dd6153773614@crans.org>

Hello,

I recently noticed that tests are currently broken on file master 
branch. `make check` fails on my machine with:
```
TZ=UTC MAGIC=../magic/magic ./test -e ../tests/keyman-2.testfile 
../tests/keyman-2.result
../tests/keyman-2.testfile: Zip archive data, at least v1.0 to extract, 
compression method=deflate
test: ERROR: result was (len 70)
Zip archive data, at least v1.0 to extract, compression method=deflate
expected (len 29)
Keyman Compiled Package File
make[2]: *** [Makefile:791: check-local] Error 1
make[2]: Leaving directory '/home/erdnaxe/git/file/tests'
make[1]: *** [Makefile:667: check-am] Error 2
make[1]: Leaving directory '/home/erdnaxe/git/file/tests'
make: *** [Makefile:465: check-recursive] Error 1
```

Reverting commit 3bce3e16ec645be7a012ea9a29ad858510f84d4b makes the test 
pass, but if you look at the commit it does not really make sense why it 
would break the test.
During debugging, I noticed a strange behavior: adding 50 newlines at 
line 260 in magic/Magdir/archive breaks the tests/keyman-2 test.
I believe this is a bug in the file magic parser, and this bug is 
triggered when the magic definitions file is too big.

Do files under `magic/Magdir/` folder have a maximum size?

Best regards,
-- 
Alexandre

From christos at zoulas.com  Mon Aug 26 18:21:51 2024
From: christos at zoulas.com (Christos Zoulas)
Date: Mon, 26 Aug 2024 14:21:51 -0400
Subject: [File] newlines break file tests
In-Reply-To: <2f2c0033-9791-43ff-a152-dd6153773614@crans.org>
References: <2f2c0033-9791-43ff-a152-dd6153773614@crans.org>
Message-ID: <7A027E1B-B966-4823-87C8-A380F0C8C35F@zoulas.com>

They should not have a maximum size. I will take a look.

christos

> On Aug 26, 2024, at 1:19?PM, Alexandre IOOSS <erdnaxe at crans.org> wrote:
> 
> Hello,
> 
> I recently noticed that tests are currently broken on file master branch. `make check` fails on my machine with:
> ```
> TZ=UTC MAGIC=../magic/magic ./test -e ../tests/keyman-2.testfile ../tests/keyman-2.result
> ../tests/keyman-2.testfile: Zip archive data, at least v1.0 to extract, compression method=deflate
> test: ERROR: result was (len 70)
> Zip archive data, at least v1.0 to extract, compression method=deflate
> expected (len 29)
> Keyman Compiled Package File
> make[2]: *** [Makefile:791: check-local] Error 1
> make[2]: Leaving directory '/home/erdnaxe/git/file/tests'
> make[1]: *** [Makefile:667: check-am] Error 2
> make[1]: Leaving directory '/home/erdnaxe/git/file/tests'
> make: *** [Makefile:465: check-recursive] Error 1
> ```
> 
> Reverting commit 3bce3e16ec645be7a012ea9a29ad858510f84d4b makes the test pass, but if you look at the commit it does not really make sense why it would break the test.
> During debugging, I noticed a strange behavior: adding 50 newlines at line 260 in magic/Magdir/archive breaks the tests/keyman-2 test.
> I believe this is a bug in the file magic parser, and this bug is triggered when the magic definitions file is too big.
> 
> Do files under `magic/Magdir/` folder have a maximum size?
> 
> Best regards,
> -- 
> Alexandre
> -- 
> File mailing list
> File at astron.com
> https://mailman.astron.com/mailman/listinfo/file


From christos at zoulas.com  Tue Aug 27 19:26:25 2024
From: christos at zoulas.com (Christos Zoulas)
Date: Tue, 27 Aug 2024 15:26:25 -0400
Subject: [File] newlines break file tests
In-Reply-To: <7A027E1B-B966-4823-87C8-A380F0C8C35F@zoulas.com>
References: <2f2c0033-9791-43ff-a152-dd6153773614@crans.org>
 <7A027E1B-B966-4823-87C8-A380F0C8C35F@zoulas.com>
Message-ID: <2EF8A585-10A3-4676-91F5-6C42DB9A12D7@zoulas.com>

Ok, this was a big rabbit hole. The key here is the way magic entries are sorted in apprentice_sort() 
with memcmp.  Part of the magic entry is the line number.
So there were 3 PK\003\004 entries in archive and changing the line number, 
changed the sorting order. I killed the line number now in sorting
so duplicate magic entries will produce a warning. I also merged some of the easy ones. 
Merging is not always a good solution because it
moves magic entries to the same file where that is not always ideal (CISO magic in 
console and filesystems for example). The problem is that
we don't have a good way to see if order makes a difference when we have 
duplicate magic and we need to disambiguate them or merge them.
I am leaving the duplicate magic message on for now...

Best,

christos

> On Aug 26, 2024, at 2:21?PM, Christos Zoulas <christos at zoulas.com> wrote:
> 
> They should not have a maximum size. I will take a look.
> 
> christos
> 
>> On Aug 26, 2024, at 1:19?PM, Alexandre IOOSS <erdnaxe at crans.org> wrote:
>> 
>> Hello,
>> 
>> I recently noticed that tests are currently broken on file master branch. `make check` fails on my machine with:
>> ```
>> TZ=UTC MAGIC=../magic/magic ./test -e ../tests/keyman-2.testfile ../tests/keyman-2.result
>> ../tests/keyman-2.testfile: Zip archive data, at least v1.0 to extract, compression method=deflate
>> test: ERROR: result was (len 70)
>> Zip archive data, at least v1.0 to extract, compression method=deflate
>> expected (len 29)
>> Keyman Compiled Package File
>> make[2]: *** [Makefile:791: check-local] Error 1
>> make[2]: Leaving directory '/home/erdnaxe/git/file/tests'
>> make[1]: *** [Makefile:667: check-am] Error 2
>> make[1]: Leaving directory '/home/erdnaxe/git/file/tests'
>> make: *** [Makefile:465: check-recursive] Error 1
>> ```
>> 
>> Reverting commit 3bce3e16ec645be7a012ea9a29ad858510f84d4b makes the test pass, but if you look at the commit it does not really make sense why it would break the test.
>> During debugging, I noticed a strange behavior: adding 50 newlines at line 260 in magic/Magdir/archive breaks the tests/keyman-2 test.
>> I believe this is a bug in the file magic parser, and this bug is triggered when the magic definitions file is too big.
>> 
>> Do files under `magic/Magdir/` folder have a maximum size?
>> 
>> Best regards,
>> -- 
>> Alexandre
>> -- 
>> File mailing list
>> File at astron.com
>> https://mailman.astron.com/mailman/listinfo/file
> 


From erdnaxe at crans.org  Thu Aug 29 16:31:21 2024
From: erdnaxe at crans.org (Alexandre IOOSS)
Date: Thu, 29 Aug 2024 18:31:21 +0200
Subject: [File] [PATCH] Magdir: remove some duplicated magic entries
Message-ID: <be9c80c3-9a5e-49d4-bfb0-0899d977b822@crans.org>

Hello,

Following the change in apprentice_sort(), we now have some "Duplicate 
magic entry" warnings to fix.

Attached you may find a patch that merges the following magic:
- Apple ][ QBoot/0Boot Image (magic/Magdir/apple): can be merged as the 
subbranches match different strings.
- Mach-o and Java (magic/Magdir/cafebabe): can be merged as the mach-o 
branch is "belong 1" and the java one is "ubelong >30".
- Alpha COFF object file (magic/Magdir/digital): can be merged as one 
branch is "leshort 0" and the other one "leshort !0".
- PowerPC COFF object file (magic/Magdir/coff): same as Alpha COFF 
object file.

Best,
-- 
Alexandre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: remove_dupl1.diff
Type: text/x-patch
Size: 5205 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20240829/1e00d802/attachment.bin>

From christos at zoulas.com  Fri Aug 30 17:07:56 2024
From: christos at zoulas.com (Christos Zoulas)
Date: Fri, 30 Aug 2024 13:07:56 -0400
Subject: [File] [PATCH] Magdir: remove some duplicated magic entries
In-Reply-To: <be9c80c3-9a5e-49d4-bfb0-0899d977b822@crans.org>
References: <be9c80c3-9a5e-49d4-bfb0-0899d977b822@crans.org>
Message-ID: <3A594126-84FC-4FA8-8564-1B6EC20D94A6@zoulas.com>



> On Aug 29, 2024, at 12:31?PM, Alexandre IOOSS <erdnaxe at crans.org> wrote:
> 
> Hello,
> 
> Following the change in apprentice_sort(), we now have some "Duplicate magic entry" warnings to fix.
> 
> Attached you may find a patch that merges the following magic:
> - Apple ][ QBoot/0Boot Image (magic/Magdir/apple): can be merged as the subbranches match different strings.
> - Mach-o and Java (magic/Magdir/cafebabe): can be merged as the mach-o branch is "belong 1" and the java one is "ubelong >30".
> - Alpha COFF object file (magic/Magdir/digital): can be merged as one branch is "leshort 0" and the other one "leshort !0".
> - PowerPC COFF object file (magic/Magdir/coff): same as Alpha COFF object file.
> 

Applied, thanks.

BTW merging duplicate entries does not fix ambiguity that depends on sorting order. For example

[magic 1]
0	string	foo
>20	string	bar	A bar file


0	string	foo
>30	string	baz	A baz file

Changed into:

[magic 2]
0	string	foo
>20	string	bar	A bar file
>30	string	baz	A baz file

Will still behave differently than:

[magic 3]
0	string	foo
>30	string	baz	A baz file
>20	string	bar	A bar file

So for a file that contains foo in offset 0 bar in offset 20 and baz in offset 30.
Using [magic 1] will either print 
"A bar file"
or 
"A baz file"
depending on the ordering the two magic entries have been found. 

Using [magic 2] it will print
"A bar file A baz file"

and using [magic 3] it will print
"A baz file A bar file"


I think that merging the entries makes sense, but we need to make sure
that the combined entries work properly.

Best,

christos






From git at mochaa.ws  Tue Aug 27 10:54:32 2024
From: git at mochaa.ws (Zephyr Lykos)
Date: Tue, 27 Aug 2024 10:54:32 -0000
Subject: [File] [PATCH v1] Add more versions of IDA database file
Message-ID: <20240827104356.168679-1-git@mochaa.ws>

Magic number cross confirmed from
https://github.com/Vector35/idb-parser-rs/blob/7ff21084/src/lib.rs#L18
---
 magic/Magdir/database | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/magic/Magdir/database b/magic/Magdir/database
index f42e5f56..fb838cec 100644
--- a/magic/Magdir/database
+++ b/magic/Magdir/database
@@ -820,7 +820,9 @@
 0	string		ZEC3	Zope Object Database Client Cache File (data)
 
 # IDA (Interactive Disassembler) database
+0	string		IDA0	IDA (Interactive Disassembler) database
 0	string		IDA1	IDA (Interactive Disassembler) database
+0	string		IDA2	IDA (Interactive Disassembler) database
 
 # Hopper (reverse engineering tool) https://www.hopperapp.com/
 0	string		hopperdb	Hopper database
-- 
2.46.0