[File] [PATCH] Magdir/images for TIM misidentifies dBase memo DBT+ wrong width
Jörg Jenderek
joerg.jen.der.ek at gmx.net
Sat Oct 16 17:32:17 UTC 2021
Hello,
some times ago i handled dBase files. Some unexpected results lead me
to inspection of Sony PlayStation bitmap images with file name
extension TIM.
When running running file command version 5.40 and later with -k
option on such images and related files i get an output like:
16383x1-rle.tim: TIM image, 24-Bit,
Pixel at (3,5) Size=24574x1
2x5.tim: TIM image,
Pixel at (23,29) Size=1x5
CHIMES.BIN: data
FONT0P.TIM: TIM image, 4-Bit,
Pixel at (0,0) Size=64x256,
16 CLUT Entries at (256,480)
input.tim: TIM image, 4-Bit,
Pixel at (640,96) Size=16x32,
16 CLUT Entries at (960,21)
KAGE.TIM: TIM image, 8-Bit,
Pixel at (0,0) Size=13x29,
256 CLUT Entries at (0,480)
MAP01.TIM: TIM image, 8-Bit,
Pixel at (768,256) Size=64x128,
256 CLUT Entries at (0,496)
pal4-7x256.tim: TIM image,
Pixel at (7,256) Size=31x64
RGB24.TIM: TIM image, 24-Bit,
Pixel at (0,0) Size=190x64
SIDE06.TIM: TIM image, 15-Bit,
Pixel at (0,0) Size=320x240
TADA.BIN: data
test.dbf: FoxBase+/dBase III, with memo .DBT DBF,
49 records * 232, update-date 97-5-15,
at offset 257 1st record " 1One
test.dbt: TIM image, (4702,14221)
dBase III DBT, next free block index 16,
1st item "WHAT IS XBASE"
TEX3P.TIM: TIM image, 4-Bit,
Pixel at (0,0) Size=64x256,
16 CLUT Entries at (0,480)
TOFULOGO.TIM: TIM image, 8-Bit,
Pixel at (0,0) Size=128x256,
256 CLUT Entries at (0,480)
TYPE00.TIM: TIM image, 15-Bit,
Pixel at (0,0) Size=320x240
VIRGIN.TIM: TIM image, 8-Bit,
Pixel at (480,0) Size=128x256,
256 CLUT Entries at (0,0)
Furthermore for TIM images only generic application/octet-stream is
shown by option -i and with --extension option only ??? is displayed.
For comparison reason i run the file format identification utility
TrID ( See https://mark0.net/soft-trid-e.html).
This does not misidentifies dBase memo test.dbt as TIM image. It also
identifies example pal4-7x256.tim correctly as 4-bit variant without
CLUT ( colour lookup table) by definition bitmap-tim-4-nc.trid.xml and
example 2x5.tim correctly as 8-bit variant without CLUT by
bitmap-tim-8-nc.trid.xml. But TrID also misidentifies all TIM images as
"LTAC compressed audio (v1.61)" with low priority by definition
audio-ltac-161.trid.xml like the real audio examples CHIMES.BIN and
TADA.BIN (See appended trid-v-tim.txt.gz).
It also displays correct file name extension TIM and related URL.
There i find also some links to samples, but the mentioned 4 and 8
bit examples always have a Colour Look Up Table (CLUT).
So first i add this information inside Magdir/images by additional
comment lines likes:
# URL: fileformats.archiveteam.org/wiki/TIM_(PlayStation_graphics)
# Reference: https://mrclick.zophar.net/TilEd/download/timgfx.txt
The recognition happened by lines like:
0 lelong 0x00000010 TIM image,
>4 lelong 0x8 4-Bit,
>4 lelong 0x9 8-Bit,
>4 lelong 0x2 15-Bit,
>4 lelong 0x3 24-Bit,
The first 4 bytes are used for detection, but unfortunately 3 bytes
are nil. So this pattern is not very unique enough. So more
additional test lines are needed.
At offset four 32 flag bits are stored in little endian order.
According to documentation the first 3 bits determine the used bits
per pixel (BPP) with the following value meanings:
0 4BPP
1 8BPP
2 16BPP
3 24BPP
4 Mixed Format
That can be shown by debug line like:
>4 lelong&0x00000007 x \b, BPP~%u
Unfortunately the mentioned sample files links only point to 4, 8 and
15 bit samples. So i create other samples with help of tools like
BMP2TIM. But i was not able to create a Mixed Format example nor an
example with "high" (above 32768) image dimensions. I did do that
because in the Kaitai Structure Specification for TIM images the
integers are unsigned whereas in magic lines signed integers are
used. So i expect trouble ( different results for "high" values" ),
but i do if this is true. So i just keep signed integers and mention
this fact by a comment line.
The forth bit (value 8) determines if image contains a Colour Lookup
Table (CLUT) or not. The remaining 28 bits in flag field are unused,
that means these bits are 0 or 3 upper bytes of flag field are nil.
That facts are used by TrID definitions bitmap-tim-*.trid.xml. For
dBase memo example test.dbt that value is 22d10189h. That can be
shown by debug line like:
>4 lelong x FLAGS=%x
So TrID does not misidentifies DBT example as TIM image. So i skip
this dBase III memo example by test for unused bits of flags field by
additional second line. So this now starts like:
0 lelong 0x00000010
>4 lelong&0xffFFffF0 =0 Sony PlayStation bitmap image
!:mime image/x-sony-tim
!:ext tim
>>4 lelong&0x07 0x0 4-Bit,
>>4 lelong&0x07 0x1 8-Bit,
>>4 lelong 0x2 15-Bit,
>>4 lelong 0x3 24-Bit,
>>4 lelong&0x07 0x4 Mixed-Bit,
In file command version 5.40 the describing phrase "TIM image" was
used. I think that is not a help for users. The users may think that
these are images of a person like Tim Berners-Lee. So i looked how
others name these images. ImageMagick software use phrase "PSX TIM".
XnView graphic software use the twisted phrase "TIM PSX" and TrID use
a describing text like "PSX TIM bitmap". If you are not a computer
player, you also do not know what is a PSP. That is the codename for
Sony's video game console PlayStation. I myself did not know that.
So i now use the phrase "Sony PlayStation PSX image" instead. The TIM
abbreviation is now shown by file name extension and by a user
defined mime type.
To get the correct BPP information the flags field must be unmasked
of CLUT bit. The 16BPP and 24BPP examples have no CLUT. So the
original expression can be kept.
At offset 8 a integer value is stored as 4 byte little endian. That
value is a sum. One part is the size of the following CLUT block for
examples with CLUT or the image data size for examples without CLUT.
The second part of the sum is the number (12=Ch) of prepending bytes
with containing information like dimensions and origin points. So the
reminder of that value seems to be always ending with hexadecimal
value C. For misidentified DBT example this remaining value is 3.
That information can be shown by debugging lines like:
>8 lelong x \b, 12+CLUT or data size=%8.8x
>8 ubyte&0x0F =0x0C \b, SIZE REMAINDER IS 12
So this can probably be used as third possible test.
For examples without CLUT directly after first size field comes image
data block starting with X+Y origin followed by dimensions. That was
expressed by lines like:
>4 lelong ^8
>>12 leshort x Pixel at (%d,
>>14 leshort x \b%d)
>>16 leshort x Size=%dx
>>18 leshort x \b%d
On reference site software is mentioned that can handle such images like:
XnView
ImageMagick
RECOIL
So run i command line tools to get information about images by
commands like:
nconvert -fullinfo *.tim > xnview.txt
identify. *.tim > imagemagick.txt
recoil2png -o TMP.PNG input.tim; file TMP.PNG >>recoil.txt
If i look in text output i see that image height is always the same
but the width of images reported by file command is often wrong.
When looking in documentation there is written that in field before
height not the real width width is stored. Instead the value must be
multiplied with a scale factor to get the real image width. This
factor depends on BPP value. So multiply by 4 (for 4BPP) 2 (for 8BPP)
1 (for 16BPP) and 2/3 (for 24BPP). So the branch without CLUT now
becomes like:
>4 lelong ^8
>>12 leshort x Pixel at (%d,
>>14 leshort x \b%d) Size=
>>>4 lelong 0x0
>>>>16 leshort*4 x \b%d
>>>4 lelong 0x1
>>>>16 leshort*2 x \b%d
>>>4 lelong 0x2
>>>>16 leshort x \b%d
>>>4 lelong 0x3
#>>>>16 leshort*2/3 x \b%d
>>>>16 leshort x \b2/3*%d
>>>4 lelong 0x4
>>>>16 leshort x \b%d
>>18 leshort x \bx%d
Unfortunately multiplying with a fraction like 2/3 does not work. So
i display that information in text form like for example
16383x1-rle.tim by phrase like "Size=2/3*24574x1".
Then i do also scaling of width for branch with CLUT in same way.
After applying the above mentioned modifications by patch
file-images-tim.diff then now all inspected Sony PlayStation TIM
bitmap images are described with correct BPP and dimension
information and misidentification of dBase memo test.dbt vanish. This
now looks like:
16383x1-rle.tim: Sony PlayStation PSX image, 24-Bit,
Pixel at (3,5) Size=2/3*24574x1
2x5.tim: Sony PlayStation PSX image, 8-Bit,
Pixel at (23,29) Size=2x5
CHIMES.BIN: data
FONT0P.TIM: Sony PlayStation PSX image, 4-Bit,
Pixel at (0,0) Size=256x256,
16 CLUT Entries at (256,480)
input.tim: Sony PlayStation PSX image, 4-Bit,
Pixel at (640,96) Size=64x32,
16 CLUT Entries at (960,21)
KAGE.TIM: Sony PlayStation PSX image, 8-Bit,
Pixel at (0,0) Size=26x29,
256 CLUT Entries at (0,480)
MAP01.TIM: Sony PlayStation PSX image, 8-Bit,
Pixel at (768,256) Size=128x128,
256 CLUT Entries at (0,496)
pal4-7x256.tim: Sony PlayStation PSX image, 4-Bit,
Pixel at (7,256) Size=124x64
RGB24.TIM: Sony PlayStation PSX image, 24-Bit,
Pixel at (0,0) Size=2/3*190x64
SIDE06.TIM: Sony PlayStation PSX image, 15-Bit,
Pixel at (0,0) Size=320x240
TADA.BIN: data
test.dbf: FoxBase+/dBase III, with memo .DBT DBF,
49 records * 232, update-date 97-5-15,
at offset 257 1st record " 1One
test.dbt: dBase III DBT, next free block index 16,
1st item "WHAT IS XBASE"
TEX3P.TIM: Sony PlayStation PSX image, 4-Bit,
Pixel at (0,0) Size=256x256,
16 CLUT Entries at (0,480)
TOFULOGO.TIM: Sony PlayStation PSX image, 8-Bit,
Pixel at (0,0) Size=256x256,
256 CLUT Entries at (0,480)
TYPE00.TIM: Sony PlayStation PSX image, 15-Bit,
Pixel at (0,0) Size=320x240
VIRGIN.TIM: Sony PlayStation PSX image, 8-Bit,
Pixel at (480,0) Size=256x256,
256 CLUT Entries at (0,0)
I hope my diff file can be applied in future version of file utility.
With best wishes
Jörg Jenderek
--
Jörg Jenderek
-------------- next part --------------
A non-text attachment was scrubbed...
Name: xnview.txt.gz
Type: application/x-gzip
Size: 642 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20211016/570cb399/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: recoil.txt.gz
Type: application/x-gzip
Size: 221 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20211016/570cb399/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagemagick.txt.gz
Type: application/x-gzip
Size: 402 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20211016/570cb399/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: trid-v-tim.txt.gz
Type: application/x-gzip
Size: 873 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20211016/570cb399/attachment-0003.bin>
-------------- next part --------------
--- file-master-oct/magic/Magdir/images.old 2021-10-07 17:41:22.000000000 +0200
+++ file-master-oct/magic/Magdir/images 2021-10-16 17:29:57.053109879 +0200
@@ -1937,23 +1937,73 @@
# TIM images
-0 lelong 0x00000010 TIM image,
->4 lelong 0x8 4-Bit,
->4 lelong 0x9 8-Bit,
->4 lelong 0x2 15-Bit,
->4 lelong 0x3 24-Bit,
->4 lelong &8
->>(8.l+12) leshort x Pixel at (%d,
->>(8.l+14) leshort x \b%d)
->>(8.l+16) leshort x Size=%dx
->>(8.l+18) leshort x \b%d,
->>4 lelong 0x8 16 CLUT Entries at
->>4 lelong 0x9 256 CLUT Entries at
->>12 leshort x (%d,
->>14 leshort x \b%d)
->4 lelong ^8
->>12 leshort x Pixel at (%d,
->>14 leshort x \b%d)
->>16 leshort x Size=%dx
->>18 leshort x \b%d
+# URL: http://fileformats.archiveteam.org/wiki/TIM_(PlayStation_graphics)
+# Reference: https://mrclick.zophar.net/TilEd/download/timgfx.txt
+# Update: Joerg Jenderek
+# Note: called as "PSX TIM *bpp bitmap" by bitmap-tim-*.trid.xml
+# verified as "TIM PSX" by XnView `nconvert -fullinfo *.tim` and
+# by RECOIL `recoil2png -o TMP.PNG input.tim; file TMP.PNG` and often
+# as "PSX TIM" by ImageMagick version 7.1.0-10 command `identify *.tim`
+# here signed integers are used but according to Kaitai unsigned
+0 lelong 0x00000010
+# 32 Flag bits *cttt; c~CLUT flag t~type 000~4BPP 001~8BPP 010~16BPP 011~24BPP 100~Mixed
+#>4 lelong x FLAGS=%#x
+# 12+Size of CLUT (2Ch for 4BPP; 20Ch 40Ch 60Ch 80Ch C0Ch for 8BPP) or
+# +image data size (800Ch 2000Ch 2580C for 16BPP) (02000003h for dBase memo test.dbt)
+#>8 lelong x \b, 12+CLUT or data size=%#8.8x
+# CLUT or data size remainder is 12 (Ch), but 03 for dBase memo test.dbt
+#>8 ubyte&0x0F =0x0C \b, SIZE REMAINDER IS 12
+# skip dBase III memo test.dbt with invalid flags 22D10189h
+>4 lelong&0xffFFffF0 =0 Sony PlayStation PSX image,
+# file (version 5.40) labeled the above entry as "TIM image"
+!:mime image/x-sony-tim
+!:ext tim
+#>>4 lelong&0x00000007 x \b, BPP~%u
+# 4BPP and 8BPP examples exist with CLUT or without CLUT
+>>4 lelong&0x07 0x0 4-Bit,
+>>4 lelong&0x07 0x1 8-Bit,
+# 16BPP and 24BPP examples have no CLUT
+>>4 lelong 0x2 15-Bit,
+>>4 lelong 0x3 24-Bit,
+# no example
+>>4 lelong&0x07 0x4 Mixed-Bit,
+# CLUT flag set
+>>4 lelong &8
+# 12 + size of CLUT like: 1000Ch 800Ch 400Ch 40Ch and 2FEh (KAGE.TIM)
+#>>>(8.l+8) lelong x \b, 12+CLUT SIZE=%#8.8x
+>>>(8.l+12) leshort x Pixel at (%d,
+>>>(8.l+14) leshort x \b%d) Size=
+# image width (to get actual width multiply by 4 for 4BPP and by 2 for 8BPP)
+>>>>4 lelong 0x8
+>>>>>(8.l+16) leshort*4 x \b%d
+>>>>4 lelong 0x9
+>>>>>(8.l+16) leshort*2 x \b%d
+# image height like: 32 64 128 144 160 208 256
+>>>(8.l+18) leshort x \bx%d,
+>>>4 lelong 0x8 16 CLUT Entries at
+>>>4 lelong 0x9 256 CLUT Entries at
+>>>12 leshort x (%d,
+>>>14 leshort x \b%d)
+# no Color LookUp Table (CLUT)
+>>4 lelong ^8
+# image orgin X Y
+>>>12 leshort x Pixel at (%d,
+>>>14 leshort x \b%d) Size=
+# real image width = multiply by 4 (4BPP) 2 (8BPP) 1 (16BPP) 2/3 (24BPP)
+>>>>4 lelong 0x0
+>>>>>16 leshort*4 x \b%d
+>>>>4 lelong 0x1
+>>>>>16 leshort*2 x \b%d
+>>>>4 lelong 0x2
+>>>>>16 leshort x \b%d
+>>>>4 lelong 0x3
+# GRR: NOT working
+#>>>>>16 leshort*2/3 x \b%d
+>>>>>16 leshort x \b2/3*%d
+# mixed format width not explained!
+>>>>4 lelong 0x4
+>>>>>16 leshort x \b%d
+# image height like: 64 240 256
+>>>18 leshort x \bx%d
+# TIM image data
# MDEC streams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file-images-tim.diff.sig
Type: application/octet-stream
Size: 1638 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20211016/570cb399/attachment.obj>
More information about the File
mailing list