From greg at rhobard.com  Thu Apr  3 15:24:57 2025
From: greg at rhobard.com (Gregory Lepore)
Date: Thu, 3 Apr 2025 11:24:57 -0400
Subject: [File] PRONOM signature conversions
Message-ID: <CAPrLS5_DnqVu5oyCGtmpYMMEZfzdGn9yiP0opkbLauzuEYSMGw@mail.gmail.com>

Greetings - for the past 5 years I have been working with the UK National
Archives to develop file format signatures for the PRONOM database. Those
signatures are very similar to those used by file. I have converted the
roughly 1,600 format signatures I've created to the 'file' format and
posted those signatures on my GitHub site at:

https://github.com/gleporeNARA/pronom-research/tree/master

The vast majority of these signatures are not currently being used by
'file'.

https://github.com/gleporeNARA/pronom-research/blob/master/formats.txt

lists an older set of formats that I have produced signatures for.

https://github.com/gleporeNARA/pronom-research/blob/master/'file'%20magic%20file%20from%20PRONOM%20sigs

is my collection of PRONOM signatures converted to 'file' format

https://github.com/gleporeNARA/pronom-research/blob/master/PRONOM%20to%20'file'%20test%20files

are the results of running file against my test corpus. As you can see,
most of the formats are not recognized.

In addition to the format signature I have sample files and a brief
writeup on each format.

I'm looking for help on how best to get these signatures into file.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20250403/b25222a0/attachment.htm>

From erdnaxe at crans.org  Sun Apr  6 17:58:37 2025
From: erdnaxe at crans.org (A. IOOSS)
Date: Sun, 6 Apr 2025 19:58:37 +0200
Subject: [File] [PATCH] Magdir/firmware ESP8266/ESP32 and NXP i.MX RT
 firmware images
Message-ID: <284f755a-b0e8-417b-8367-a48cf788bff5@crans.org>

Hello,

I'm back with a few more improvements regarding firmware detection in file.

Attached you may find a patch that:
- fixes a small printing issue on ESP-IDP signature,
- enables file to recognize ESP8266 and ESP32 flash dumps,
- enables file to recognize NXP i.MX RT firmware image dumps.

To test ESP32/NXP images, I highly recommend using MicroPython builds 
(MIT licensed).
These open-source firmware images can be easily downloaded using the 
following script: 
https://github.com/erdnaxe/firmware-samples/blob/main/micropython.sh

Thanks a lot,
Best regards,
-- 
erdnaxe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: esp_nxp.patch
Type: text/x-patch
Size: 3193 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20250406/4c9fedb1/attachment.bin>

From christos at zoulas.com  Sun Apr  6 18:37:52 2025
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 6 Apr 2025 14:37:52 -0400
Subject: [File] [PATCH] Magdir/firmware ESP8266/ESP32 and NXP i.MX RT
 firmware images
In-Reply-To: <284f755a-b0e8-417b-8367-a48cf788bff5@crans.org>
References: <284f755a-b0e8-417b-8367-a48cf788bff5@crans.org>
Message-ID: <4D8B4A51-B30A-4D8D-935E-46C16588CA32@zoulas.com>

Committed, thanks!

christos

> On Apr 6, 2025, at 1:58?PM, A. IOOSS <erdnaxe at crans.org> wrote:
> 
> Hello,
> 
> I'm back with a few more improvements regarding firmware detection in file.
> 
> Attached you may find a patch that:
> - fixes a small printing issue on ESP-IDP signature,
> - enables file to recognize ESP8266 and ESP32 flash dumps,
> - enables file to recognize NXP i.MX RT firmware image dumps.
> 
> To test ESP32/NXP images, I highly recommend using MicroPython builds (MIT licensed).
> These open-source firmware images can be easily downloaded using the following script: https://github.com/erdnaxe/firmware-samples/blob/main/micropython.sh
> 
> Thanks a lot,
> Best regards,
> -- 
> erdnaxe
> <esp_nxp.patch>-- 
> File mailing list
> File at astron.com
> https://mailman.astron.com/mailman/listinfo/file

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP
URL: <https://mailman.astron.com/pipermail/file/attachments/20250406/2fbb128c/attachment.asc>

From christos at zoulas.com  Sun Apr  6 18:43:31 2025
From: christos at zoulas.com (Christos Zoulas)
Date: Sun, 6 Apr 2025 14:43:31 -0400
Subject: [File] [PATCH] Magdir/linux S390 kernel image; Show version
In-Reply-To: <f8670d62-6218-4b26-8d3b-56a4b23d6495@linux.ibm.com>
References: <20250317135228.1812146-1-jremus@linux.ibm.com>
 <f8670d62-6218-4b26-8d3b-56a4b23d6495@linux.ibm.com>
Message-ID: <B218A05F-0281-49EF-9BB0-6D169B25A7B5@zoulas.com>

Committed, I kept the word "kernel" in the name.

christos

> On Mar 17, 2025, at 12:10?PM, Jens Remus <jremus at linux.ibm.com> wrote:
> 
> On 17.03.2025 14:52, Jens Remus wrote:
>> Linux kernel S390 boot executable images have the magic string "S390EP"
>> at 0x10008 (since v3.2). [1]  The parameter area has a pointer to the
>> kernel version at 0x10428 (since v5.3, which has been 0 since v3.2) [2]
>> Based on Vasily's implementation from his commit message of his Linux
>> kernel commit 6abe28197024 ("s390: enable detection of kernel version
>> from bzImage").
>> While at it update the identifier to "Linux kernel S390 boot executable
>> Image", to align it to the other architectures.
> 
> Marc brought up (offline), that changing the identifier from "Linux
> S390" to "Linux kernel S390 boot executable Image" might break existing
> scripts that grep for "Linux S390".  He suggested that a compromise
> might be to change it to "Linux S390 kernel boot executable Image",
> which only appends to the existing identifier.
> 
>> [1]: Linux kernel, arch/s390/boot/head.S, EP_OFFSET and EP_STRING.
>> [2]: Linux kernel, arch/s390/include/asm/setup.h, struct parmarea,
>>      kernel_version.
>> Co-authored-by: Vasily Gorbik <gor at linux.ibm.com>
>> Signed-off-by: Jens Remus <jremus at linux.ibm.com>
>> ---
>>  magic/Magdir/linux | 9 ++++++++-
>>  1 file changed, 8 insertions(+), 1 deletion(-)
>> diff --git a/magic/Magdir/linux b/magic/Magdir/linux
>> index d2e3aaa1d68c..4afcc0736707 100644
>> --- a/magic/Magdir/linux
>> +++ b/magic/Magdir/linux
>> @@ -456,7 +456,9 @@
>>  ############################################################################
>>  # Linux S390 kernel image
>>  # Created by: Jan Kaluza <jkaluza at redhat.com>
>> -8 string \x02\x00\x00\x18\x60\x00\x00\x50\x02\x00\x00\x68\x60\x00\x00\x50\x40\x40\x40\x40\x40\x40\x40\x40 Linux S390
>> +# Update: Jens Remus <jremus at linux.ibm.com> based on Vasily Gorbik <gor at linux.ibm.com>
>> +# Linux kernel: arch/s390/boot/head.S and arch/s390/include/asm/setup.h
>> +8 string \x02\x00\x00\x18\x60\x00\x00\x50\x02\x00\x00\x68\x60\x00\x00\x50\x40\x40\x40\x40\x40\x40\x40\x40 Linux kernel S390 boot executable Image
>>  >0x00010000 search/b/4096 \x00\x0a\x00\x00\x8b\xad\xcc\xcc
>>  # 64bit
>>  >>&0 string \xc1\x00\xef\xe3\xf0\x68\x00\x00 Z10 64bit kernel
>> @@ -468,6 +470,11 @@
>>  >>&0 string \x81\x00\xc8\x80\x00\x00\x00\x00 Z9-109 32bit kernel
>>  >>&0 string \x80\x00\x20\x00\x00\x00\x00\x00 Z990 32bit kernel
>>  >>&0 string \x80\x00\x00\x00\x00\x00\x00\x00 Z900 32bit kernel
>> +# Linux kernel v3.2+
>> +>0x10008		string		S390EP
>> +# Linux kernel v5.3+
>> +>>0x10428		ubequad		>0
>> +>>>(0x10428.Q)		string		>\0	\b, version %s
>>    ############################################################################
>>  # Linux ARM compressed kernel image
> 
> Regards,
> Jens
> -- 
> Jens Remus
> Linux on Z Development (D3303)
> +49-7031-16-1128 Office
> jremus at de.ibm.com <mailto:jremus at de.ibm.com>
> 
> IBM
> 
> IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Gesch?ftsf?hrung: David Faller; Sitz der Gesellschaft: B?blingen; Registergericht: Amtsgericht Stuttgart, HRB 243294
> IBM Data Privacy Statement: https://www.ibm.com/privacy/
> 
> -- 
> File mailing list
> File at astron.com <mailto:File at astron.com>
> https://mailman.astron.com/mailman/listinfo/file

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20250406/099ba6c5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP
URL: <https://mailman.astron.com/pipermail/file/attachments/20250406/099ba6c5/attachment-0001.asc>

From amansha at kth.se  Thu Apr 10 17:20:55 2025
From: amansha at kth.se (Aman Sharma)
Date: Thu, 10 Apr 2025 17:20:55 +0000
Subject: [File] ASCII text detected as data
In-Reply-To: <d4515349130b483e8c9422c3c0053936@kth.se>
References: <5915c688d96d4371a187b6ed3a01eb2d@kth.se>
 <6596ceec4a784edaa8225fb68ffc238a@kth.se>
 <b4771435-2de1-459e-819d-7f6af9dfd6c5@vad.cx>,
 <B8C0F7D3-C028-4C91-AAC9-5CF438CC2694@zoulas.com>,
 <d4515349130b483e8c9422c3c0053936@kth.se>
Message-ID: <8edd420dee814cbba8b65278b56c2c23@kth.se>

Hi Christos,


Any follow up on this? I just noticed there is a typo in the email. "Had it been some" ignore this.


Regards,
Aman Sharma

PhD Student
KTH Royal Institute of Technology
School of Electrical Engineering and Computer Science (EECS)
Department of Theoretical Computer Science (TCS)
<http://www.kth.se><https://www.kth.se/profile/amansha><https://www.kth.se/profile/amansha>
<https://www.kth.se/profile/amansha>https://algomaster99.github.io/
________________________________
From: Aman Sharma
Sent: Monday, March 17, 2025 7:26:54 PM
To: Christos Zoulas
Cc: file at astron.com
Subject: Re: [File] ASCII text detected as data


Hi,


Thanks for the response! I had overlooked as I was too fixated with the diff.

If this is the case, shouldn't the ouput of file be `Unicode text, UTF-8 text`? I verified the following with Python code as well. It does fail for encoding="ascii".


```

codecs.open("ref", encoding="utf-8", errors="strict").readlines()

```


Had it been some


Regards,
Aman Sharma

PhD Student
KTH Royal Institute of Technology
School of Electrical Engineering and Computer Science (EECS)
Department of Theoretical Computer Science (TCS)
<http://www.kth.se><https://www.kth.se/profile/amansha><https://www.kth.se/profile/amansha>
<https://www.kth.se/profile/amansha>https://algomaster99.github.io/
________________________________
From: Christos Zoulas <christos at zoulas.com>
Sent: Saturday, March 15, 2025 6:33:03 PM
To: Aman Sharma
Cc: file at astron.com
Subject: Re: [File] ASCII text detected as data

They contain non-ascii characters eg:

 Ga \xc3\xa9\xc3\xa9n stap verder

christos

> On Mar 15, 2025, at 12:16?PM, Vadim <vad.mailingfile at vad.cx> wrote:
>
> Hi,
>
> > xxd -s 0x5a30 -l 16 -g 1 reb
> > xxd -s 0x5ed0 -l 16 -g 1 ref
>
> shows "0x1e" character, that's ASCII control char "record separator".
>
> xxd highlights UTF-8 characters in red too.
>
> Cheers,
> Vadim
> --
> File mailing list
> File at astron.com
> https://mailman.astron.com/mailman/listinfo/file

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20250410/37401c9d/attachment.htm>

From jason1 at pobox.com  Sat Apr 12 12:55:55 2025
From: jason1 at pobox.com (Jason Summers)
Date: Sat, 12 Apr 2025 08:55:55 -0400
Subject: [File] PRONOM signature conversions
In-Reply-To: <CAPrLS5_DnqVu5oyCGtmpYMMEZfzdGn9yiP0opkbLauzuEYSMGw@mail.gmail.com>
References: <CAPrLS5_DnqVu5oyCGtmpYMMEZfzdGn9yiP0opkbLauzuEYSMGw@mail.gmail.com>
Message-ID: <CAPqxkbTKZ56VRXfa8cxfFv+S2tMrXs91sp3PeF8UvW4+xgRyFw@mail.gmail.com>

I don't speak for the 'file' project maintainers.

Looks like a lot of good information here. But I feel like I must be
missing something. Does the file on GitHub only include the first *line* of
each pattern?

Consider these examples. (Note that these are the complete patterns. There
are no ">" lines.)

0   string  \x46\x4f\x52\x4d    Amiga Metafile Format
0   string  \x46\x4f\x52\x4d    Extended MIDI Audio File
0   string  \x46\x4f\x52\x4d    FAXX IFF Facsimile Image
0   string  \x46\x4f\x52\x4d    IFF Amiga Contiguous BitMap

These are obviously not usable as they are.

Are your PRONOM patterns more strict than this? I know that not every
PRONOM pattern can easily be converted to file's format. But if a pattern
can't be converted properly, I'd think it should be rejected, not converted
badly.

On Thu, Apr 3, 2025 at 11:32?AM Gregory Lepore <greg at rhobard.com> wrote:

> Greetings - for the past 5 years I have been working with the UK National
> Archives to develop file format signatures for the PRONOM database. Those
> signatures are very similar to those used by file. I have converted the
> roughly 1,600 format signatures I've created to the 'file' format and
> posted those signatures on my GitHub site at:
>
> https://github.com/gleporeNARA/pronom-research/tree/master
>
> The vast majority of these signatures are not currently being used by
> 'file'.
>
> https://github.com/gleporeNARA/pronom-research/blob/master/formats.txt
>
> lists an older set of formats that I have produced signatures for.
>
>
> https://github.com/gleporeNARA/pronom-research/blob/master/'file'%20magic%20file%20from%20PRONOM%20sigs
>
> is my collection of PRONOM signatures converted to 'file' format
>
>
> https://github.com/gleporeNARA/pronom-research/blob/master/PRONOM%20to%20'file'%20test%20files
>
> are the results of running file against my test corpus. As you can see,
> most of the formats are not recognized.
>
> In addition to the format signature I have sample files and a brief
> writeup on each format.
>
> I'm looking for help on how best to get these signatures into file.
>
> Thanks!
> --
> File mailing list
> File at astron.com
> https://mailman.astron.com/mailman/listinfo/file
>


-- 
Jason Summers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20250412/a08213cb/attachment.htm>

From erdnaxe at crans.org  Sat Apr 12 15:39:18 2025
From: erdnaxe at crans.org (A. IOOSS)
Date: Sat, 12 Apr 2025 17:39:18 +0200
Subject: [File] [PATCH] Magdir/firmware Fix ESP8266/ESP32 firmware images
 strength
Message-ID: <e9d5622f-558f-442e-8a69-e3cb95cc4589@crans.org>

Hello,

It seems that 'DOS/MBR boot sector' magic has a much higher strength 
than 'ESP8266/ESP32 firmware image'. Attached you may find a patch that 
tries to address this issue.

Also I noticed that commit d407f210d35f88a68124a44a80a5c9c462eb2bb2 
(from 2014) seems incomplete has it does not print 'DOS/MBR boot sector' 
for files starting with 0xE9.

Thanks,
-- 
erdnaxe
-------------- next part --------------
A non-text attachment was scrubbed...
Name: firmware-strength.diff
Type: text/x-patch
Size: 2785 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20250412/fa9de04d/attachment.bin>

From christos at zoulas.com  Sat Apr 12 16:05:53 2025
From: christos at zoulas.com (Christos Zoulas)
Date: Sat, 12 Apr 2025 12:05:53 -0400
Subject: [File] PRONOM signature conversions
In-Reply-To: <CAPqxkbTKZ56VRXfa8cxfFv+S2tMrXs91sp3PeF8UvW4+xgRyFw@mail.gmail.com>
References: <CAPrLS5_DnqVu5oyCGtmpYMMEZfzdGn9yiP0opkbLauzuEYSMGw@mail.gmail.com>
 <CAPqxkbTKZ56VRXfa8cxfFv+S2tMrXs91sp3PeF8UvW4+xgRyFw@mail.gmail.com>
Message-ID: <59BF4478-6AE1-46A7-A53B-213FA64B5CDC@zoulas.com>



> On Apr 12, 2025, at 8:55?AM, Jason Summers <jason1 at pobox.com> wrote:
> 
> I don't speak for the 'file' project maintainers.
> 
> Looks like a lot of good information here. But I feel like I must be missing something. Does the file on GitHub only include the first *line* of each pattern?
> 
> Consider these examples. (Note that these are the complete patterns. There are no ">" lines.)
> 
> 0   string  \x46\x4f\x52\x4d    Amiga Metafile Format
> 0   string  \x46\x4f\x52\x4d    Extended MIDI Audio File
> 0   string  \x46\x4f\x52\x4d    FAXX IFF Facsimile Image
> 0   string  \x46\x4f\x52\x4d    IFF Amiga Contiguous BitMap
> 
> These are obviously not usable as they are.
> 
> Are your PRONOM patterns more strict than this? I know that not every PRONOM pattern can easily be converted to file's format. But if a pattern can't be converted properly, I'd think it should be rejected, not converted badly.
> 

Plus 
> 0   string  \x46\x4f\x52\x4d    Amiga Metafile Format
> 
is just:
0	string FORM	Amiga Metafile format

which is already there:

$ grep FORM iff
iff:0   string          FORM            IFF data

christos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20250412/341afbc4/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP
URL: <https://mailman.astron.com/pipermail/file/attachments/20250412/341afbc4/attachment.asc>

From greg at rhobard.com  Sun Apr 13 14:45:08 2025
From: greg at rhobard.com (Gregory Lepore)
Date: Sun, 13 Apr 2025 10:45:08 -0400
Subject: [File] PRONOM signature conversions
Message-ID: <ca5dda9e-fb23-4329-8405-7ab105163ff7@rhobard.com>

@jason and @christos

Thank you for looking over the signatures I submitted. Unfortunately my 
conversion code from PRONOM to file format had a bug where multiple 
SubSequences in the PRONOM XML file were not properly identified. I have 
updated the code and the output file on GitHub to properly convert these 
signatures.

0?? string? \x46\x4f\x52\x4d??? Amiga Metafile Format
is now:
# Amiga Metafile Format
0??? string??? \x46\x4f\x52\x4d??? Amiga Metafile Format
 >8??? string??? \x41\x4d\x46\x46??? Amiga Metafile Format

0?? string? \x46\x4f\x52\x4d??? Extended MIDI Audio File
is now:
# Extended MIDI Audio File
0??? string??? \x46\x4f\x52\x4d??? Extended MIDI Audio File
 >30??? string??? \x58\x4d\x49\x44\x46\x4f\x52\x4d??? Extended MIDI 
Audio File

0?? string? \x46\x4f\x52\x4d??? FAXX IFF Facsimile Image
is now:
# FAXX IFF Facsimile Image
0??? string??? \x46\x4f\x52\x4d??? FAXX IFF Facsimile Image
 >8??? string??? \x46\x41\x58\x58??? FAXX IFF Facsimile Image

0?? string? \x46\x4f\x52\x4d??? IFF Amiga Contiguous BitMap
is now:
# IFF Amiga Contiguous BitMap
0??? string??? \x46\x4f\x52\x4d??? IFF Amiga Contiguous BitMap
 >0??? string??? \x41\x42\x49\x54??? IFF Amiga Contiguous BitMap

At the moment I am using byte sequences for everything, although looking 
at other file signatures it appears they could be ASCII as well. Is 
there anywhere that the file signature format is documented? Also, is 
there a standard test corpus of files that I can use to ensure there are 
no collisions between my signatures and existing file signatures?

Please look over the signatures again and let me know what additional 
work I need to do.
https://github.com/gleporeNARA/pronom-research/tree/master
thanks.




From avidseeker at disroot.org  Mon Apr 14 18:02:12 2025
From: avidseeker at disroot.org (Avid Seeker)
Date: Mon, 14 Apr 2025 18:02:12 +0000
Subject: [File] lua file detected as javascript
Message-ID: <D96K4SBSJRT3.2SYD2BPE1QDAP@disroot.org>

$ wget 'https://raw.githubusercontent.com/ibhagwan/fzf-lua/refs/heads/main/lua/fzf-lua/defaults.lua'
&& file defaults.lua

Expected text/plain but file reads that as: `defaults.lua: JavaScript source,
Unicode text, UTF-8 text`

I reached the following as a minimal reproducible example. If the last
function is deleted, the file is detected as text/plain.

```
local path = require "fzf-lua.path"
local utils = require "fzf-lua.utils"
local actions = require "fzf-lua.actions"
local previewers = require "fzf-lua.previewer"

local M = {}

function M._default_previewer_fn()
  local winopts = M.globals.winopts
  if type(winopts) == "function" then
    winopts = winopts() or {}
    winopts.preview = type(winopts.preview) == "table" and winopts.preview or {}
    winopts.preview.default = winopts.preview.default or M.defaults.winopts.preview.default
  end
  local previewer = M.globals.default_previewer or winopts.preview.default
  -- the setup function cannot have a custom previewer as deepcopy
  -- fails with stack overflow while trying to copy the custom class
  -- the workaround is to define the previewer as a function instead
  -- https://github.com/ibhagwan/fzf-lua/issues/677
  return type(previewer) == "function" and previewer() or previewer
end

function M._preview_pager_fn()
  return vim.fn.executable("delta") == 1 and ("delta --width=$COLUMNS --%s"):format(vim.o.bg) or
      nil
end

function M._man_cmd_fn(bat_pager)
  local cmd = utils.is_darwin() and "man -P cat"
      or vim.fn.executable("mandb") == 1 and "man"
      or "man -c"
  local bat_cmd = bat_pager and (function()
    for _, bin in ipairs({ "batcat", "bat" }) do
      if vim.fn.executable(bin) == 1 then
        return string.format("%s --color=always -p -l man", bin)
      end
    end
  end)()
  local pager = bat_cmd or "col -bx"
  return string.format("%s %%s 2>/dev/null | %s", cmd, pager)
end
```

From avidseeker at disroot.org  Sun Apr 13 00:52:36 2025
From: avidseeker at disroot.org (Avid Seeker)
Date: Sun, 13 Apr 2025 00:52:36 +0000
Subject: [File] subscribe
Message-ID: <D953LWWUEVTT.1TLANG0YD3CNK@disroot.org>

subscribe

From avidseeker at disroot.org  Sun Apr 13 00:57:15 2025
From: avidseeker at disroot.org (Avid Seeker)
Date: Sun, 13 Apr 2025 00:57:15 +0000
Subject: [File] lua file detected as javascript
Message-ID: <D953PHDV4D1V.2MUFYVPRO0ADX@disroot.org>

$ wget 'https://raw.githubusercontent.com/ibhagwan/fzf-lua/refs/heads/main/lua/fzf-lua/defaults.lua' &&
 file defaults.lua

Expected text/plain but file reads that as: `defaults.lua: JavaScript source, Unicode text, UTF-8 text`

From vad.mailingfile at vad.cx  Wed Apr 16 13:13:51 2025
From: vad.mailingfile at vad.cx (Vadim)
Date: Wed, 16 Apr 2025 15:13:51 +0200
Subject: [File] lua file detected as javascript
In-Reply-To: <D96K4SBSJRT3.2SYD2BPE1QDAP@disroot.org>
References: <D96K4SBSJRT3.2SYD2BPE1QDAP@disroot.org>
Message-ID: <04f915c5-31fa-4bd1-b2ba-0a09290cc7eb@vad.cx>

Thanks. You would have seen the rule if you ran file with "-d".

The rule is too broad and only needs function[( ] to declare JS. I would 
make async mandatory to keep this rule.

$ file -d defaults.lua

*unknown*, 42: > 0 regex,=\((async )?function[( ],"JavaScript source"]
0 == 0 = 1 strength=68
[try ascmagic 1]
defaults.lua: JavaScript source, Unicode text, UTF-8 text


https://github.com/file/file/blob/f77a1092e1862c2295a21077c9e28c2614a0eede/magic/Magdir/javascript#L42 


!:ext???? js
0??? regex??? \\((async\x20)?function[(\x20]??? JavaScript source
!:strength +20
!:mime??? application/javascript
!:ext???? js


I attached 3 patches to fix this problem and add detection for similar 
Lua scripts that make use of "require". Christos, please let me know if 
these are OK for merging.


On 4/14/25 20:02, Avid Seeker wrote:
> $ wget 
> 'https://raw.githubusercontent.com/ibhagwan/fzf-lua/refs/heads/main/lua/fzf-lua/defaults.lua'
> && file defaults.lua
>
> Expected text/plain but file reads that as: `defaults.lua: JavaScript 
> source,
> Unicode text, UTF-8 text`
>
> I reached the following as a minimal reproducible example. If the last
> function is deleted, the file is detected as text/plain.
>
> ```
> local path = require "fzf-lua.path"
> local utils = require "fzf-lua.utils"
> local actions = require "fzf-lua.actions"
> local previewers = require "fzf-lua.previewer"
>
> local M = {}
>
> function M._default_previewer_fn()
> ?local winopts = M.globals.winopts
> ?if type(winopts) == "function" then
> ?? winopts = winopts() or {}
> ?? winopts.preview = type(winopts.preview) == "table" and 
> winopts.preview or {}
> ?? winopts.preview.default = winopts.preview.default or 
> M.defaults.winopts.preview.default
> ?end
> ?local previewer = M.globals.default_previewer or winopts.preview.default
> ?-- the setup function cannot have a custom previewer as deepcopy
> ?-- fails with stack overflow while trying to copy the custom class
> ?-- the workaround is to define the previewer as a function instead
> ?-- https://github.com/ibhagwan/fzf-lua/issues/677
> ?return type(previewer) == "function" and previewer() or previewer
> end
>
> function M._preview_pager_fn()
> ?return vim.fn.executable("delta") == 1 and ("delta --width=$COLUMNS 
> --%s"):format(vim.o.bg) or
> ???? nil
> end
>
> function M._man_cmd_fn(bat_pager)
> ?local cmd = utils.is_darwin() and "man -P cat"
> ???? or vim.fn.executable("mandb") == 1 and "man"
> ???? or "man -c"
> ?local bat_cmd = bat_pager and (function()
> ?? for _, bin in ipairs({ "batcat", "bat" }) do
> ???? if vim.fn.executable(bin) == 1 then
> ?????? return string.format("%s --color=always -p -l man", bin)
> ???? end
> ?? end
> ?end)()
> ?local pager = bat_cmd or "col -bx"
> ?return string.format("%s %%s 2>/dev/null | %s", cmd, pager)
> end
> ```
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Fix-false-positive-for-Lua-detected-as-Javascript.patch
Type: text/x-patch
Size: 1049 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20250416/05d972dd/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Add-file-extension-to-previous-Lua-magic-rules.patch
Type: text/x-patch
Size: 1000 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20250416/05d972dd/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-regex-magic-for-Lua-to-match-require.patch
Type: text/x-patch
Size: 1626 bytes
Desc: not available
URL: <https://mailman.astron.com/pipermail/file/attachments/20250416/05d972dd/attachment-0002.bin>

From amansha at kth.se  Sun Apr 20 20:53:31 2025
From: amansha at kth.se (Aman Sharma)
Date: Sun, 20 Apr 2025 20:53:31 -0000
Subject: [File] Java source file incorrectly identified as HTML document
Message-ID: <0e34f0a6cc60433a898399793dd67182@kth.se>

Hi,


I have two files, Reference.txt<https://github.com/user-attachments/files/19689452/Reference.txt> and Rebuild.txt<https://github.com/user-attachments/files/19689451/Rebuild.txt>. Their file type is:


$ file Reference.txt Rebuild.txt

Reference.txt: HTML document, ASCII text, with very long lines (6135)
Rebuild.txt:   Java source, ASCII text, with very long lines (6135)


Both are Java source files. However, Reference.txt is incorrectly identified as an HTML document. As suggested by Chris here<https://lists.reproducible-builds.org/pipermail/diffoscope/2025-April/002838.html>, if line 30 in Reference.txt is removed,`"  <title>(.*?)<\\/title>"`, file command correctly classifies it as Java source.


Regards,
Aman Sharma

PhD Student
KTH Royal Institute of Technology
School of Electrical Engineering and Computer Science (EECS)
Department of Theoretical Computer Science (TCS)
<http://www.kth.se><https://www.kth.se/profile/amansha><https://www.kth.se/profile/amansha>
<https://www.kth.se/profile/amansha>https://algomaster99.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.astron.com/pipermail/file/attachments/20250420/e0a789f3/attachment.htm>