From joerg.jen.der.ek at gmx.net Sun Dec 1 20:14:06 2019 From: joerg.jen.der.ek at gmx.net (=?UTF-8?Q?J=c3=b6rg_Jenderek?=) Date: Sun, 1 Dec 2019 21:14:06 +0100 Subject: [File] [PATCH] of Magdir/map for Garmin update *.rgn, *.exe Message-ID: <7fe94a84-23d4-1bdb-2e67-7e72200e7a2d@gmx.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, some weeks ago i send patches to recognize maps found on garmin navigation devices. After updating firmware of garmin device i run file command version 5.37 on garmin device. There exist some files with name extension exe and rgn. These are recognized only as "data". Luckily i find a file format documentation by Herbert Oppmann. So i add the PDF-URL by comment line to Magdir/map like: # Ref.: www.memotech.franken.de/FileFormats/Garmin_RGN_Format.pdf According to reference such Garmin firmware Update start with specific magic keyword. So first test line looks like: 0 string KpGr Garmin update To keep out put columns low, i display additional information if not standard. So next show header format version if not 1.0 (That is 0064h as 2 byte little endian integer) by lines: >0x4 uleshort !0x0064 >>4 uleshort/100 x \b, version %u >>4 uleshort%100 x \b.%u After the header comes some RGN records. To display information of Garmin RGN record i introduce a subroutine by lines like 0 name garmin-entry For debugging purpose the record length can be shown by line like >0 ulelong x \b, length 0x%x This is 2 for Data record, for Application often this value is 1Bh sometimes 1Dh and for Region record this value is "big". Furthermore this values can be used to jump from record to record. After the record length comes a record identifying byte. Usual values are D, A or R. So at end of sub routine show unusual values by lines like >4 default x \b; other >>4 ubyte x ID '%c' The data record (ID='D'=44h) with version content like 1.0 (hexadecimal 0064h) are handled by branch with lines >4 ubyte =0x44 >>5 uleshort !0x0064 \b; Data >>>5 uleshort/100 x \b, version %u >>>5 uleshort%100 x \b.%u The Application Record (ID='A'=41h) has a varying size, because builder name like "SQA" or "hales" are different. There standard version is 2.0 ( hexadecimal 00c8h). So show version, builder name, date and time by lines like: >4 ubyte =0x41 \b; App >>5 uleshort !0x00C8 >>>5 uleshort/100 x \b, version %u >>>5 uleshort%100 x \b.%u >>7 string x \b, build by %s >>>&1 string x %s >>>>&1 string x %s Third branch is for region record (ID='R'=52h). There first i show region id by lines starting like: >4 ubyte =0x52 \b; Region >>5 uleshort x ID=%u This id is different and characteristic for region record content. So i show first bytes of content by line like >>15 ubequad x \b, content 0x%llx... When interpreting the content as string, we get "KpGr" for recursive embedded updates, "GARMIN BITMAP" for Garmin Bitmap images (*.srf), "PK" for ZIP/JAR archive and others. I tried to inspect content by calling file command again, but i failed at that position. So at the end i inspect region content by looking for specific content magic. To use default clause i look again at version by line >0x4 uleshort x First test for region content with Garmin images *.srf by specific magic and using test lines of Magdir/images like >>0x3a search/5/s GARMIN\ BITMAP \b= !:mime image/x-garmin-exe !:ext exe >>>&0 indirect x The updates with embedded garmin bitmap images of vehicles have file name extension exe like example Vehicle_Compact-Purple_220.exe. I also add a user defined mime type. For test purpose i extract Garmin Bitmap (*.srf) and convert this by tool srftopam of NetPBM package to Netpbm PAM image (*.pam) that can be viewed/converted by ImageMagick. If update does not contain bitmap *.srf then it is a "region" with file name extension rgn. These example are handled by other branch starting with lines like: >>0x3a default x \b= !:mime application/x-garmin-rgn !:ext rgn Recursive embedded garmin updates are handled by >>>0x3a search/5/s KpGrd >>>>&0 indirect x Regions with ZIP or JAR archive content are handled by last branch by magic lines inside Magdir/archive and Magdir/zip like: >>>0x3a search/5/s PK\003\004 >>>>&0 indirect x After header handle first and second garmin RGN record by lines like >6 use garmin-entry >(0x6.l+10) ubyte x >>&0 use garmin-entry After applying the above mentioned modifications by patch file-5.37-map-rgn.diff then i get an output like: a.rgn: Garmin update; App, build by hales Oct 25 1999 11:52:30; Region ID=7, 13579 ms, content 0x48656c6c6f000000...= B1907,ANT_250_USB.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=250, content 0xb8210020ad940100...= c.rgn: Garmin update; App, build by hales Oct 25 1999 11:52:30; Region ID=6, 24680 ms, content 0x424865790e000000...= Fenix3,Wi-Fi_200.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=251, content 0xacd94e2e2135f570...= Fenix3_650.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=14, content 0x4b70477264000200...= Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=14, content 0x4ff000004ff00001...= fenix3HR,Display_370.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=249, content 0x4a14f103c80dc70c...= IndexSmartScale_310.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=1, content 0x2c1102002e535546...= nuvi25x9LanguageText_260.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=255, content 0x504b030414000200...= Zip archive data, at least v2.0 to extract nuvi25x9LanguageText_260.zip: Zip archive data, at least v2.0 to extract nuvi4692_CHN_regionfileonly__590.jar: Java archive data (JAR) nuvi4692_CHN_regionfileonly__590.rgn: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=148, content 0x504b030414000800...= Java archive data (JAR) nuvi900T_update.rgnfile__360.rgn: Garmin update; App, build by sqa Feb 23 2009 18:23:01; Region ID=80, content 0x0...= test-versions.rgn: Garmin update, version 1.2; Data, version 1.3; App, version 2.3, build by joerg Oct 25 1999 11:52:30; Region ID=7, 13579 ms, content 0x48656c6c6f000000...= test-year2018.rgn: Garmin update; App, build by joerg Mar 13 2018 12:34:56; Region ID=7, 13579 ms, content 0x4248657900000000...= Vehicle_Compact-Purple_220.exe: Garmin update; App, build by SQA Oct 25 1999 14:16:13; Region ID=78, content 0x4741524d494e2042...= Garmin Bitmap file, version 2.20, 2880x80 Vehicle_Compact-Purple_220.pam: Netpbm PAM image file Vehicle_Compact-Purple_220.srf: Garmin Bitmap file, version 2.20, 2880x80 I hope my diff file can be applied in a future version of file utility. With best wishes J?rg Jenderek - -- J?rg Jenderek -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iF0EARECAB0WIQS5/qNWKD4ASGOJGL+v8rHJQhrU1gUCXeQe8gAKCRCv8rHJQhrU 1gBgAJ9kb5to2No1ihDSpYvnpuNNjKDRwgCfeSUPiriVflesxK6dyvigF+0yc5k= =0OqQ -----END PGP SIGNATURE----- -------------- next part -------------- --- file-5.37/magic/Magdir/map.old 2019-04-30 04:02:04 +0000 +++ file-5.37/magic/Magdir/map 2019-12-01 19:57:54 +0000 @@ -316,6 +316,83 @@ # LBL:2A9h,SRT:1Dh 25h 27h,TRE:CFh 135h,TRF:5Ah,TYP:5Bh 6Eh 7Ch AEh,RGN:7Dh >>0 uleshort x \b, header length 0x%x +# URL: https://www.memotech.franken.de/FileFormats/ +# Reference: https://www.memotech.franken.de/FileFormats/Garmin_RGN_Format.pdf +# From: Joerg Jenderek +0 string KpGr Garmin update +# format version like: 0064h~1.0 +>0x4 uleshort !0x0064 +>>4 uleshort/100 x \b, version %u +>>4 uleshort%100 x \b.%u +# 1st Garmin entry +>6 use garmin-entry +# 2nd Garmin entry +>(0x6.l+10) ubyte x +>>&0 use garmin-entry +# 3rd entry +>(0x6.l+10) ubyte x +>>&(&0.l+4) ubyte x +>>>&0 use garmin-entry +# look again at version to use default clause +>0x4 uleshort x +# test for region content by looking for +# Garmin *.srf by ./images with normal builder name "SQA" or longer "hales" +# 1 space after equal sign +>>0x3a search/5/s GARMIN\ BITMAP \b= +!:mime image/x-garmin-exe +!:ext exe +>>>&0 indirect x +# if not bitmap *.srf then region; 1 space after equal sign +>>0x3a default x \b= +!:mime application/x-garmin-rgn +!:ext rgn +# recursiv embedded +>>>0x3a search/5/s KpGrd +>>>>&0 indirect x +# look for ZIP or JAR archive by ./archive and ./zip +>>>0x3a search/5/s PK\003\004 +>>>>&0 indirect x +# TODO: other garmin RGN record content like foo +#>>0x3a search/5/s bar BAR +# display information of Garmin RGN record +0 name garmin-entry +# record length: 2 for Data, for Application often 1Bh sometimes 1Dh, "big" for Region +#>0 ulelong x \b, length 0x%x +# data record (ID='D') with version content like 0064h~1.0 +>4 ubyte =0x44 +>>5 uleshort !0x0064 \b; Data +>>>5 uleshort/100 x \b, version %u +>>>5 uleshort%100 x \b.%u +# Application Record (ID='A') +>4 ubyte =0x41 \b; App +# version content like 00c8h~2.0 +>>5 uleshort !0x00C8 +>>>5 uleshort/100 x \b, version %u +>>>5 uleshort%100 x \b.%u +# builder name like: SQA sqa build hales +>>7 string x \b, build by %s +# build date like: Oct 25 1999, Oct 1 2008, Feb 23 2009, Dec 15 2009 +>>>&1 string x %s +# build time like: 11:26:12, 11:45:54, 14:16:13, 18:23:01 +>>>>&1 string x %s +# region record (ID='R') +>4 ubyte =0x52 \b; Region +# region ID:14~fw_all.bin: 78~ZIP, RGN or SRF bitmap; 148~ZIP or JAR; 249~display firmware; 251~WiFi or GCD firmware; 255~ZIP +>>5 uleshort x ID=%u +# delay in ms: like 0, 500 +>>7 ulelong !0 \b, %u ms +# region size (is record length - 10) +#>>11 ulelong x \b, length 0x%x +# region content like: +# "KpGr"~recursiv embedded,"GARMIN BITMAP"~Garmin Bitmap *.srf, "PK"~ZIP archive +#>>15 string x \b, content "%s" +>>15 ubequad x \b, content 0x%llx... +# This does NOT WORK! +#>>15 indirect x \b; contains +>4 default x \b; other +# garmin Record ID Identifies the record content like: D A R +>>4 ubyte x ID '%c' + # TOM TOM GPS watches ttbin files: # https://github.com/ryanbinns/ttwatch/tree/master/ttbin # From: Daniel Lenski -------------- next part -------------- A non-text attachment was scrubbed... Name: file-5.37-map-rgn.diff.sig Type: application/octet-stream Size: 95 bytes Desc: not available URL: From christos at zoulas.com Sun Dec 1 22:46:41 2019 From: christos at zoulas.com (Christos Zoulas) Date: Sun, 1 Dec 2019 17:46:41 -0500 Subject: [File] [PATCH] of Magdir/map for Garmin update *.rgn, *.exe In-Reply-To: <7fe94a84-23d4-1bdb-2e67-7e72200e7a2d@gmx.net> References: <7fe94a84-23d4-1bdb-2e67-7e72200e7a2d@gmx.net> Message-ID: <4F581903-C831-456A-A0CE-C4B000CFF724@zoulas.com> Committed, thanks! christos > On Dec 1, 2019, at 3:14 PM, J?rg Jenderek wrote: > > From joerg.jen.der.ek at gmx.net Sat Dec 14 14:26:07 2019 From: joerg.jen.der.ek at gmx.net (=?UTF-8?Q?J=c3=b6rg_Jenderek?=) Date: Sat, 14 Dec 2019 15:26:07 +0100 Subject: [File] [PATCH] of Magdir/macintosh,uuencode for BinHex binary; *.hqx Message-ID: <4e7fc048-daa8-c75f-bc4a-c198fc12f9e1@gmx.net> Hello, some days ago i handled some BinHex binaries. When searching on my systems and the internet for such files and similar test files and running file command version 5.37 with -k option i get unexpected output like: mac-update-40b7.hqx: news or mail, ASCII text MACDEV6B.HQX: BinHex binary text, version 4.0\012- BinHex binary text, version 4.0, ASCII text RESRC_DE.HQX: ASCII text shrinkwrap-21.hqx: BinHex binary text, version 4.0\012- BinHex binary text, version 4.0, ASCII text zap-res-forks-101.hqx: data Furthermore with --extension only ??? is displayed. First of all i get duplicate messages. One is triggered by line inside Magdir/macintosh looking like: 11 string must\ be\ converted\ with\ BinHex BinHex binary text The other message is triggered by nearly same magic inside Magdir/uuencode looking like 11 search/1 must\ be\ converted\ with\ BinHex BinHex binary text Most information about this file format ca be found on Wikipedia. So i add a comment line like # URL: https://en.wikipedia.org/wiki/BinHex According to documentations the characteristic magic for BinHex is a sentence starting like: (This file must be converted with BinHex For most inspected examples this sentence is found at offset 0 of file. But sometimes one empty line is before the sentence. This should be matched only by magic inside Magdir/uuencode. And according to RFC 1741 any text before this magic sentence is to be ignored. So there exist samples with text consisting of thousand characters before magic sentence like in mac-update-40b7.hqx found on ftp.vim.org. So search range must be raised. Furthermore another error occurs. The "string" and "search" directive used at the moment only works for text files and not binary considered files. So examples like zap-res-forks-101.hqx also found on ftp.vim.org are described by file command only as "binary" data. When looking in that file i find control-characters like ^C and a copyright symbol like ?. So this file is not ASCII text and must be handled as binary. The magic test lines now become like 0 search/1602 (This\ file\ >&0 use binhex 0 search/2652/b (This\ file\ >&0 use binhex 0 name binhex >0 string must\ be\ converted\ with\ BinHex\ BinHex binary text The first search directive is for ASCII text samples and the second is for binary branch. Now afterwards show apple type according to web site www.macdisk.com by line !:apple BNHQTEXT At the end of the sentence the version is stored. This is shown at the moment in current magic files by lines >41 string x \b, version %.3s The documentation mainly mentions version 4.0 and i only found test examples with version 4.0 and hqx extension. But according to www.faqs.org and Wikipedia for other versions different file name extensions and maybe different mime types are used, but the description are not so precisely. I have done my best. Maybe an BinHex expert should check this. So depending on version this is now described by >>&0 string 1.0 1.0 !:mime application/mac-binhex !:ext hex >>&0 string 2.0 2.0 !:mime application/mac-binhex !:ext hcx >>&0 string 4.0 4.0 !:mime application/mac-binhex40 !:ext hqx >>&0 string 5.0 5.0 !:mime application/mac-binhex40 !:ext hqx >>&0 default x >>>&0 string x %.3s !:mime application/mac-binhex !:ext hqx After applying the above mentioned modifications by patches file-5.37-uuencode-macbin.diff and file-5.37-macintosh-macbin.diff the duplicate messages vanish. Further more now cases with garbage before magic sentence and binary looking files are also recognised and i get an output like: mac-update-40b7.hqx: news or mail text BinHex binary text, version 4.0, ASCII text MACDEV6B.HQX: BinHex binary text, version 4.0, ASCII text RESRC_DE.HQX: BinHex binary text, version 4.0, ASCII text shrinkwrap-21.hqx: BinHex binary text, version 4.0, ASCII text zap-res-forks-101.hqx: BinHex binary text, version 4.0\012- data I hope my diff files can be applied in a future version of file utility. With best wishes J?rg Jenderek -- J?rg Jenderek -------------- next part -------------- --- file-5.37/magic/Magdir/uuencode.old 2019-02-22 12:06:34 +0000 +++ file-5.37/magic/Magdir/uuencode 2019-12-14 13:52:50 +0000 @@ -23,9 +23,4 @@ # Greg Roelofs, newt at uchicago.edu 0 search/1 Decode\ the\ following\ with\ bdeco bencoded News text -# BinHex is the Macintosh ASCII-encoded file format (see also "apple") -# Daniel Quinlan, quinlan at yggdrasil.com -11 search/1 must\ be\ converted\ with\ BinHex BinHex binary text ->41 search/1 x \b, version %.3s - # GRR: handle BASE64 -------------- next part -------------- --- file-5.37/magic/Magdir/macintosh.old 2019-04-19 00:42:27 +0000 +++ file-5.37/magic/Magdir/macintosh 2019-12-14 14:13:11 +0000 @@ -7,5 +7,38 @@ # Daniel Quinlan, quinlan at yggdrasil.com -11 string must\ be\ converted\ with\ BinHex BinHex binary text +# Update: Joerg Jenderek +# URL: https://en.wikipedia.org/wiki/BinHex +# Reference: http://fileformats.archiveteam.org/wiki/BinHex +# Note: only tested with version 4.0 and hqx extension +# Any text/binary before the characteristic comment sentence is to be ignored like in +# http://ftp.vim.org/pub/ftp/ftp/infomac/disk/mac-update-40b7.hqx +0 search/1602 (This\ file\ +>&0 use binhex +# http://ftp.vim.org/pub/ftp/ftp/infomac/_Disk_&_File/zap-res-forks-101.hqx +0 search/2652/b (This\ file\ +>&0 use binhex +0 name binhex +# keep splitted search string format similar like in version 5.37 +>0 string must\ be\ converted\ with\ BinHex\ BinHex binary text, version +# http://www.macdisk.com/binhexen.php3 +!:apple BNHQTEXT +# http://www.faqs.org/faqs/macintosh/comm-faq/part1/ +>>&0 string 1.0 1.0 +!:mime application/mac-binhex +!:ext hex +>>&0 string 2.0 2.0 +!:mime application/mac-binhex +!:ext hcx +# BinHex 3.0 never existed +>>&0 string 4.0 4.0 !:mime application/mac-binhex40 ->41 string x \b, version %.3s +!:ext hqx +# BinHex 5.0 also MacBinary I +>>&0 string 5.0 5.0 +!:mime application/mac-binhex40 +!:ext hqx +# this should never happen +>>&0 default x +>>>&0 string x %.3s +!:mime application/mac-binhex +!:ext hqx From christos at zoulas.com Sat Dec 14 20:40:40 2019 From: christos at zoulas.com (Christos Zoulas) Date: Sat, 14 Dec 2019 15:40:40 -0500 Subject: [File] [PATCH] of Magdir/macintosh,uuencode for BinHex binary; *.hqx In-Reply-To: <4e7fc048-daa8-c75f-bc4a-c198fc12f9e1@gmx.net> References: <4e7fc048-daa8-c75f-bc4a-c198fc12f9e1@gmx.net> Message-ID: <7B25986D-948D-4A43-8165-2833DB7931C1@zoulas.com> Committed, thanks! christos > On Dec 14, 2019, at 9:26 AM, J?rg Jenderek wrote: > > Hello, > > some days ago i handled some BinHex binaries. When searching on my > systems and the internet for such files and similar test files and > running file command version 5.37 with -k option i get unexpected > output like: > > mac-update-40b7.hqx: news or mail, ASCII text > MACDEV6B.HQX: BinHex binary text, version 4.0\012- > BinHex binary text, version 4.0, ASCII text > RESRC_DE.HQX: ASCII text > shrinkwrap-21.hqx: BinHex binary text, version 4.0\012- > BinHex binary text, version 4.0, ASCII text > zap-res-forks-101.hqx: data > > Furthermore with --extension only ??? is displayed. > First of all i get duplicate messages. One is triggered by line > inside Magdir/macintosh looking like: > 11 string must\ be\ converted\ with\ BinHex BinHex binary text > The other message is triggered by nearly same magic inside > Magdir/uuencode looking like > 11 search/1 must\ be\ converted\ with\ BinHex BinHex binary text > > Most information about this file format ca be found on Wikipedia. So > i add a comment line like > # URL: https://en.wikipedia.org/wiki/BinHex > > According to documentations the characteristic magic for BinHex is a > sentence starting like: > (This file must be converted with BinHex > > For most inspected examples this sentence is found at offset 0 of > file. But sometimes one empty line is before the sentence. This > should be matched only by magic inside Magdir/uuencode. And according > to RFC 1741 any text before this magic sentence is to be ignored. So > there exist samples with text consisting of thousand characters > before magic sentence like in mac-update-40b7.hqx found on > ftp.vim.org. So search range must be raised. > > Furthermore another error occurs. The "string" and "search" directive > used at the moment only works for text files and not binary > considered files. So examples like zap-res-forks-101.hqx also found > on ftp.vim.org are described by file command only as "binary" data. > When looking in that file i find control-characters like ^C and a > copyright symbol like ?. So this file is not ASCII text and must be > handled as binary. The magic test lines now become like > > 0 search/1602 (This\ file\ >> &0 use binhex > 0 search/2652/b (This\ file\ >> &0 use binhex > 0 name binhex >> 0 string must\ be\ converted\ with\ BinHex\ BinHex binary text > The first search directive is for ASCII text samples and the second > is for binary branch. > > Now afterwards show apple type according to web site www.macdisk.com > by line > !:apple BNHQTEXT > > At the end of the sentence the version is stored. This is shown > at the moment in current magic files by lines >> 41 string x \b, version %.3s > > The documentation mainly mentions version 4.0 and i only found test > examples with version 4.0 and hqx extension. But according to > www.faqs.org and Wikipedia for other versions different file name > extensions and maybe different mime types are used, but the > description are not so precisely. I have done my best. Maybe an > BinHex expert should check this. So depending on version this is now > described by >>> &0 string 1.0 1.0 > !:mime application/mac-binhex > !:ext hex >>> &0 string 2.0 2.0 > !:mime application/mac-binhex > !:ext hcx >>> &0 string 4.0 4.0 > !:mime application/mac-binhex40 > !:ext hqx >>> &0 string 5.0 5.0 > !:mime application/mac-binhex40 > !:ext hqx >>> &0 default x >>>> &0 string x %.3s > !:mime application/mac-binhex > !:ext hqx > > After applying the above mentioned modifications by patches > file-5.37-uuencode-macbin.diff and file-5.37-macintosh-macbin.diff > the duplicate messages vanish. Further more now cases with garbage > before magic sentence and binary looking files are also recognised > and i get an output like: > > mac-update-40b7.hqx: news or mail text > BinHex binary text, version 4.0, ASCII text > MACDEV6B.HQX: BinHex binary text, version 4.0, ASCII text > RESRC_DE.HQX: BinHex binary text, version 4.0, ASCII text > shrinkwrap-21.hqx: BinHex binary text, version 4.0, ASCII text > zap-res-forks-101.hqx: BinHex binary text, version 4.0\012- data > > I hope my diff files can be applied in a future version of > file utility. > > With best wishes > J?rg Jenderek > -- > J?rg Jenderek > > > > > > > > > -- > File mailing list > File at astron.com > https://mailman.astron.com/mailman/listinfo/file > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP URL: From christos at zoulas.com Tue Dec 17 02:19:03 2019 From: christos at zoulas.com (Christos Zoulas) Date: Mon, 16 Dec 2019 21:19:03 -0500 Subject: [File] file 5.38 is now available. Message-ID: <20191217021904.05DEB17FDB0@rebar.astron.com> Hi, A bug fix release of file is now available from: ftp://ftp.astron.com/pub/file/file-5.38.tar.gz This release fixes: - Always accept -S (no sandbox) even if we don't support sandboxing - More syscalls elided for sandboxiing - For ELF dynamic means having an interpreter not just PT_DYNAMIC - Check for large ELF session header offset - When saving and restoring a locale, keep the locale name in our own storage. - Add a flag to disable CSV file detection. - Don't pass NULL/0 to memset to appease sanitizers. - Avoid spurious prints when looks for extensions or apple strings in fsmagic. - Add builtin decompressors for xz and and bzip. - Add a limit for the number of CDF elements. - More checks for overflow in CDF. Happy Holidays, christos From mforney at mforney.org Fri Dec 20 23:49:34 2019 From: mforney at mforney.org (Michael Forney) Date: Fri, 20 Dec 2019 15:49:34 -0800 Subject: [File] [PATCH] Avoid cast of address constant to integer constant in initalizer Message-ID: <20191220234934.31950-1-mforney@mforney.org> RCAST converts an address constant to an integer, and then to pointer. An integer constant cast to a pointer type is a valid address constant, however, a pointer cast to a integer is not an integer constant: > Cast operators in an integer constant expression shall only convert > arithmetic types to integer types, except as part of an operand to the > sizeof operator. So the result of RCAST(const void *, zlibcmp) is not technically a constant expression, and might be rejected by some compilers when it is used in an initializer for an object with static storage. Instead, just use a union so no casts are necessary. --- Initializing the second member of a union requires C99 initialization syntax. I couldn't figure out if file is strictly ANSI C, but if so, the union could be replaced with two struct members instead. src/compress.c | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/src/compress.c b/src/compress.c index 33ce2bc9..569c87e8 100644 --- a/src/compress.c +++ b/src/compress.c @@ -161,7 +161,10 @@ static const char *zstd_args[] = { #define do_bzlib NULL private const struct { - const void *magic; + union { + const void *magic; + int (*func)(const unsigned char *); + } u; int maglen; const char **argv; void *unused; @@ -171,26 +174,26 @@ private const struct { #define METH_XZ 9 #define METH_LZMA 13 #define METH_ZLIB 14 - { "\037\235", 2, gzip_args, NULL }, /* 0, compressed */ + { { "\037\235" }, 2, gzip_args, NULL }, /* 0, compressed */ /* Uncompress can get stuck; so use gzip first if we have it * Idea from Damien Clark, thanks! */ - { "\037\235", 2, uncompress_args, NULL }, /* 1, compressed */ - { "\037\213", 2, gzip_args, do_zlib }, /* 2, gzipped */ - { "\037\236", 2, gzip_args, NULL }, /* 3, frozen */ - { "\037\240", 2, gzip_args, NULL }, /* 4, SCO LZH */ + { { "\037\235" }, 2, uncompress_args, NULL }, /* 1, compressed */ + { { "\037\213" }, 2, gzip_args, do_zlib }, /* 2, gzipped */ + { { "\037\236" }, 2, gzip_args, NULL }, /* 3, frozen */ + { { "\037\240" }, 2, gzip_args, NULL }, /* 4, SCO LZH */ /* the standard pack utilities do not accept standard input */ - { "\037\036", 2, gzip_args, NULL }, /* 5, packed */ - { "PK\3\4", 4, gzip_args, NULL }, /* 6, pkzipped, */ + { { "\037\036" }, 2, gzip_args, NULL }, /* 5, packed */ + { { "PK\3\4" }, 4, gzip_args, NULL }, /* 6, pkzipped, */ /* ...only first file examined */ - { "BZh", 3, bzip2_args, do_bzlib }, /* 7, bzip2-ed */ - { "LZIP", 4, lzip_args, NULL }, /* 8, lzip-ed */ - { "\3757zXZ\0", 6, xz_args, NULL }, /* 9, XZ Utils */ - { "LRZI", 4, lrzip_args, NULL }, /* 10, LRZIP */ - { "\004\"M\030",4, lz4_args, NULL }, /* 11, LZ4 */ - { "\x28\xB5\x2F\xFD", 4, zstd_args, NULL }, /* 12, zstd */ - { RCAST(const void *, lzmacmp), -13, xz_args, NULL }, /* 13, lzma */ + { { "BZh" }, 3, bzip2_args, do_bzlib }, /* 7, bzip2-ed */ + { { "LZIP" }, 4, lzip_args, NULL }, /* 8, lzip-ed */ + { { "\3757zXZ\0" }, 6, xz_args, NULL }, /* 9, XZ Utils */ + { { "LRZI" }, 4, lrzip_args, NULL }, /* 10, LRZIP */ + { { "\004\"M\030" }, 4, lz4_args, NULL }, /* 11, LZ4 */ + { { "\x28\xB5\x2F\xFD" }, 4, zstd_args, NULL }, /* 12, zstd */ + { { .func = lzmacmp }, -13, xz_args, NULL }, /* 13, lzma */ #ifdef ZLIBSUPPORT - { RCAST(const void *, zlibcmp), -2, zlib_args, NULL }, /* 14, zlib */ + { { .func = zlibcmp }, -2, zlib_args, NULL }, /* 14, zlib */ #endif }; @@ -262,10 +265,9 @@ file_zmagic(struct magic_set *ms, const struct buffer *b, const char *name) if (nbytes < CAST(size_t, abs(compr[i].maglen))) continue; if (compr[i].maglen < 0) { - zm = (RCAST(int (*)(const unsigned char *), - CCAST(void *, compr[i].magic)))(buf); + zm = compr[i].u.func(buf); } else { - zm = memcmp(buf, compr[i].magic, + zm = memcmp(buf, compr[i].u.magic, CAST(size_t, compr[i].maglen)) == 0; } -- 2.24.1 From astron.com.bwoj at manchmal.in-ulm.de Sat Dec 21 15:55:18 2019 From: astron.com.bwoj at manchmal.in-ulm.de (Christoph Biedl) Date: Sat, 21 Dec 2019 16:55:18 +0100 Subject: [File] Rework detection of Linux swap files Message-ID: <20191221155518.ch5eo3paalbikgo6@mail.manchmal.in-ulm.de> Hello, Currently, file(1) identifies Linux swap files as "Linux/i386", "Linux/ia64", or "Linux/ppc". While not completely wrong, it is misleading for other architectures since the first is as well reported for e.g. "amd64", "armhf", or "powerpc" (aka "ppc32"). Reason for this: The implementation actually probes the page size of the swap file. Which happens to match the description - but it does so for other architectures as well. For example, even ppc64 swap files with an according page size configuration (PPC_4K_PAGES) will be reported as "i386". So let's fix this. And while on it, provide full information for all page sizes, support all combinations of page size and hibernate file type, and mark the old and long-gone format that way. Besides the Linux kernel, the util-linux sources provided some information for suspend file magic and page sizes - I never encountered page sizes of 8k or 32k but it's better to be in sync. The rules are limited to swap format v1. Given the fact this has been around for more then twenty years, it's sane to assume there will not be a v2 format any time soon. Gotchas: * While the offset of the magic is relative (at the end of the first page size), the other information is absolute at 0x400. The kernel sources mention this and suggest a change in 2.5 :-) [1] * Version number, page size and bad pages count are stored in the host's native endianess. So this patch also fixes a bug here for swap files created on BE machines. Tests were done with a bunch of swap files, taken from production and hand-crafted[2], on both big- end little-endian machines: < swap-04k.be: Linux/i386 swap file (new style), version 16777216 (4K pages), size -16580608 pages, LABEL=swap-04k, UUID=1f9c3545-ba55-4671-9265-313852bd3ebd > swap-04k.be: Linux swap file, 4k page size, big endian, version 1, size 1023 pages, 0 bad pages, LABEL=swap-04k, UUID=1f9c3545-ba55-4671-9265-313852bd3ebd < swap-04k.le: Linux/i386 swap file (new style), version 1 (4K pages), size 1023 pages, LABEL=swap-04k, UUID=1411ded3-5511-48b9-aa3a-ce9ddd980ca1 > swap-04k.le: Linux swap file, 4k page size, little endian, version 1, size 1023 pages, 0 bad pages, LABEL=swap-04k, UUID=1411ded3-5511-48b9-aa3a-ce9ddd980ca1 < swap-04k-hibernate: Linux/i386 swap file (new style) with SWSUSP1 image > swap-04k-hibernate: Linux swap file, 4k page size, little endian, version 1, size 1023 pages, 0 bad pages, no label, UUID=a182dcb0-545a-4390-a8e3-f99dd70fc3dd, with SWSUSP1 image < swap-08k.be: data > swap-08k.be: Linux swap file, 8k page size, big endian, version 1, size 511 pages, 0 bad pages, LABEL=swap-08k, UUID=5c1ca23d-eade-406e-b374-8e45dfedfa6b < swap-08k.le: data > swap-08k.le: Linux swap file, 8k page size, little endian, version 1, size 511 pages, 0 bad pages, LABEL=swap-08k, UUID=908bb488-c659-4ce6-8c40-ceaa2db60353 < swap-16k.be: Linux/ia64 swap file > swap-16k.be: Linux swap file, 16k page size, big endian, version 1, size 255 pages, 0 bad pages, LABEL=swap-16k, UUID=e77317df-19d1-4d1f-9180-833e97ee18d5 < swap-16k.le: Linux/ia64 swap file > swap-16k.le: Linux swap file, 16k page size, little endian, version 1, size 255 pages, 0 bad pages, LABEL=swap-16k, UUID=ef09b166-f92f-4ab0-970d-5b1e319e3d30 < swap-32k.be: data > swap-32k.be: Linux swap file, 32k page size, big endian, version 1, size 127 pages, 0 bad pages, LABEL=swap-32k, UUID=03a96585-4d2e-4349-b7c6-2ebafc40610d < swap-32k.le: data > swap-32k.le: Linux swap file, 32k page size, little endian, version 1, size 127 pages, 0 bad pages, LABEL=swap-32k, UUID=f72f0246-9344-40c6-ade8-a249ceec73bf < swap-64k.be: Linux/ppc swap file version 16777216, size 1056964608 pages, LABEL=swap-64k, UUID=187cabb6-6478-493c-9f02-e5858b38c3c6 > swap-64k.be: Linux swap file, 64k page size, big endian, version 1, size 63 pages, 0 bad pages, LABEL=swap-64k, UUID=187cabb6-6478-493c-9f02-e5858b38c3c6 < swap-64k.le: Linux/ppc swap file version 1, size 63 pages, LABEL=swap-64k, UUID=63caa820-b8f1-4e6b-abe5-a5902dc20424 > swap-64k.le: Linux swap file, 64k page size, little endian, version 1, size 63 pages, 0 bad pages, LABEL=swap-64k, UUID=63caa820-b8f1-4e6b-abe5-a5902dc20424 < swap-i386-suspend: Linux/i386 swap file (new style) with SWSUSP1 image > swap-i386-suspend: Linux swap file, 4k page size, little endian, version 1, size 262143 pages, 0 bad pages, no label, UUID=3feffb71-ec0b-4723-9e2c-d6c872144d15, with SWSUSP1 image < swap-ppc64: Linux/i386 swap file (new style), version 16777216 (4K pages), size -16318464 pages, LABEL=swap-ppc64, UUID=5b94af95-2c8e-451c-9bb3-a0e890184540 > swap-ppc64: Linux swap file, 4k page size, big endian, version 1, size 2047 pages, 0 bad pages, LABEL=swap-ppc64, UUID=5b94af95-2c8e-451c-9bb3-a0e890184540 Cheers, Christoph [1] Kernel 2.6 was release in December 2003. [2] "mkswap --pagesize" allows creation of swap files with page sizes differing from the system value. -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Rework-detection-of-Linux-swap-files.patch Type: text/x-diff Size: 4019 bytes Desc: not available URL: From christos at zoulas.com Sat Dec 21 16:29:35 2019 From: christos at zoulas.com (Christos Zoulas) Date: Sat, 21 Dec 2019 11:29:35 -0500 Subject: [File] Rework detection of Linux swap files In-Reply-To: <20191221155518.ch5eo3paalbikgo6@mail.manchmal.in-ulm.de> References: <20191221155518.ch5eo3paalbikgo6@mail.manchmal.in-ulm.de> Message-ID: <35A986CA-7B65-40D8-958E-582F73B04FA3@zoulas.com> Committed, thanks! christos > On Dec 21, 2019, at 10:55 AM, Christoph Biedl wrote: > > Hello, > > Currently, file(1) identifies Linux swap files as "Linux/i386", > "Linux/ia64", or "Linux/ppc". While not completely wrong, it is > misleading for other architectures since the first is as well reported > for e.g. "amd64", "armhf", or "powerpc" (aka "ppc32"). > > Reason for this: The implementation actually probes the page size of the > swap file. Which happens to match the description - but it does so for > other architectures as well. For example, even ppc64 swap files with an > according page size configuration (PPC_4K_PAGES) will be reported as > "i386". > > So let's fix this. And while on it, provide full information for all > page sizes, support all combinations of page size and hibernate file > type, and mark the old and long-gone format that way. > > Besides the Linux kernel, the util-linux sources provided some > information for suspend file magic and page sizes - I never encountered > page sizes of 8k or 32k but it's better to be in sync. > > The rules are limited to swap format v1. Given the fact this has been > around for more then twenty years, it's sane to assume there will not be > a v2 format any time soon. > > Gotchas: > > * While the offset of the magic is relative (at the end of the first > page size), the other information is absolute at 0x400. The kernel > sources mention this and suggest a change in 2.5 :-) [1] > > * Version number, page size and bad pages count are stored in the host's > native endianess. So this patch also fixes a bug here for swap files > created on BE machines. > > Tests were done with a bunch of swap files, taken from production and > hand-crafted[2], on both big- end little-endian machines: > > < swap-04k.be: Linux/i386 swap file (new style), version 16777216 (4K pages), size -16580608 pages, LABEL=swap-04k, UUID=1f9c3545-ba55-4671-9265-313852bd3ebd >> swap-04k.be: Linux swap file, 4k page size, big endian, version 1, size 1023 pages, 0 bad pages, LABEL=swap-04k, UUID=1f9c3545-ba55-4671-9265-313852bd3ebd > > < swap-04k.le: Linux/i386 swap file (new style), version 1 (4K pages), size 1023 pages, LABEL=swap-04k, UUID=1411ded3-5511-48b9-aa3a-ce9ddd980ca1 >> swap-04k.le: Linux swap file, 4k page size, little endian, version 1, size 1023 pages, 0 bad pages, LABEL=swap-04k, UUID=1411ded3-5511-48b9-aa3a-ce9ddd980ca1 > > < swap-04k-hibernate: Linux/i386 swap file (new style) with SWSUSP1 image >> swap-04k-hibernate: Linux swap file, 4k page size, little endian, version 1, size 1023 pages, 0 bad pages, no label, UUID=a182dcb0-545a-4390-a8e3-f99dd70fc3dd, with SWSUSP1 image > > < swap-08k.be: data >> swap-08k.be: Linux swap file, 8k page size, big endian, version 1, size 511 pages, 0 bad pages, LABEL=swap-08k, UUID=5c1ca23d-eade-406e-b374-8e45dfedfa6b > > < swap-08k.le: data >> swap-08k.le: Linux swap file, 8k page size, little endian, version 1, size 511 pages, 0 bad pages, LABEL=swap-08k, UUID=908bb488-c659-4ce6-8c40-ceaa2db60353 > > < swap-16k.be: Linux/ia64 swap file >> swap-16k.be: Linux swap file, 16k page size, big endian, version 1, size 255 pages, 0 bad pages, LABEL=swap-16k, UUID=e77317df-19d1-4d1f-9180-833e97ee18d5 > > < swap-16k.le: Linux/ia64 swap file >> swap-16k.le: Linux swap file, 16k page size, little endian, version 1, size 255 pages, 0 bad pages, LABEL=swap-16k, UUID=ef09b166-f92f-4ab0-970d-5b1e319e3d30 > > < swap-32k.be: data >> swap-32k.be: Linux swap file, 32k page size, big endian, version 1, size 127 pages, 0 bad pages, LABEL=swap-32k, UUID=03a96585-4d2e-4349-b7c6-2ebafc40610d > > < swap-32k.le: data >> swap-32k.le: Linux swap file, 32k page size, little endian, version 1, size 127 pages, 0 bad pages, LABEL=swap-32k, UUID=f72f0246-9344-40c6-ade8-a249ceec73bf > > < swap-64k.be: Linux/ppc swap file version 16777216, size 1056964608 pages, LABEL=swap-64k, UUID=187cabb6-6478-493c-9f02-e5858b38c3c6 >> swap-64k.be: Linux swap file, 64k page size, big endian, version 1, size 63 pages, 0 bad pages, LABEL=swap-64k, UUID=187cabb6-6478-493c-9f02-e5858b38c3c6 > > < swap-64k.le: Linux/ppc swap file version 1, size 63 pages, LABEL=swap-64k, UUID=63caa820-b8f1-4e6b-abe5-a5902dc20424 >> swap-64k.le: Linux swap file, 64k page size, little endian, version 1, size 63 pages, 0 bad pages, LABEL=swap-64k, UUID=63caa820-b8f1-4e6b-abe5-a5902dc20424 > > < swap-i386-suspend: Linux/i386 swap file (new style) with SWSUSP1 image >> swap-i386-suspend: Linux swap file, 4k page size, little endian, version 1, size 262143 pages, 0 bad pages, no label, UUID=3feffb71-ec0b-4723-9e2c-d6c872144d15, with SWSUSP1 image > > < swap-ppc64: Linux/i386 swap file (new style), version 16777216 (4K pages), size -16318464 pages, LABEL=swap-ppc64, UUID=5b94af95-2c8e-451c-9bb3-a0e890184540 >> swap-ppc64: Linux swap file, 4k page size, big endian, version 1, size 2047 pages, 0 bad pages, LABEL=swap-ppc64, UUID=5b94af95-2c8e-451c-9bb3-a0e890184540 > > > Cheers, > > Christoph > > [1] Kernel 2.6 was release in December 2003. > [2] "mkswap --pagesize" allows creation of swap files with page sizes differing from the system value. > <0001-Rework-detection-of-Linux-swap-files.patch>-- > File mailing list > File at astron.com > https://mailman.astron.com/mailman/listinfo/file -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 235 bytes Desc: Message signed with OpenPGP URL: From christos at zoulas.com Sat Dec 21 17:53:06 2019 From: christos at zoulas.com (Christos Zoulas) Date: Sat, 21 Dec 2019 12:53:06 -0500 Subject: [File] [PATCH] Avoid cast of address constant to integer constant in initalizer In-Reply-To: <20191220234934.31950-1-mforney@mforney.org> from Michael Forney (Dec 20, 3:49pm) Message-ID: <20191221175306.5E1E317FDB5@rebar.astron.com> On Dec 20, 3:49pm, mforney at mforney.org (Michael Forney) wrote: -- Subject: [File] [PATCH] Avoid cast of address constant to integer constant | RCAST converts an address constant to an integer, and then to pointer. An | integer constant cast to a pointer type is a valid address constant, | however, a pointer cast to a integer is not an integer constant: | | > Cast operators in an integer constant expression shall only convert | > arithmetic types to integer types, except as part of an operand to the | > sizeof operator. | | So the result of RCAST(const void *, zlibcmp) is not technically a | constant expression, and might be rejected by some compilers when it is | used in an initializer for an object with static storage. | | Instead, just use a union so no casts are necessary. | --- | Initializing the second member of a union requires C99 initialization | syntax. I couldn't figure out if file is strictly ANSI C, but if so, | the union could be replaced with two struct members instead. Committed, thanks! I kept the C99 init, but did not make it a transparent union :-). Let's see if anyone complains... christos