[File] Indirect offset issue

[B Watson]

Working on some magic that uses indirect offsets, like:

>>(4.s-(-2))    ubyte    x

I found that this only works if there are at least 96 bytes in the
file beyond the location of the offset. It turns out to be this, in
softmagic.c, line 1531 (just before the switch):

         if (OFFSET_OOB(nbytes, offset + off, sizeof(*q)))
            return 0;

sizeof(*q) is 96 on my platform, which explains the problem I was having.
It looks like the intent is to check the size of whichever element in
the union is the 'active' one, but instead it's getting the size of the
largest element in the union.

The attached patch gets rid of this check, and instead adds OFFSET_OOB
checks for the correct number of bytes at each switch label. It fixes the
96-byte problem and doesn't seem to break anything else, but I'm pretty
new to file-hacking so I'd like to hear from someone more experienced.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-softmagic-indirect-offset-fix.diff.gz
Type: application/x-gzip
Size: 522 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20190213/11c77aea/attachment.bin>