Integrate from fileinfo.com list of file extensions

[rpdelaney]

Should be some kind of metadata, but maybe not in the existing metadata group

Edit: This was originally about .vcf and .vcard extensions but it sprawled into something much more ambitious.

[trapd00r]

I've added a few filetypes in my personal branch. About 2500 of them, so I didn't want to destroy the work you've done with sorting files in the master branch, and sorting all of these would be way too tedious. Just a heads up. :)

[rpdelaney]

Showing 1 changed file with 3,038 additions and 651 deletions.

Holy ... what? Where did you get all this?

[trapd00r]

I scraped data from the wikipedia article on file extensions. Wanted to send you a pm about it but there's no such feature yet on github it seems. :)

[rpdelaney]

Hahaha. Holy crap. You don't get any performance issues or anything with that? When you type env does your terminal explode?

It might be possible to organize these (relatively) rapidly using a script that prints Wikipedia's description of a filetype and gives you buttons to hit for which category to put it in. But that's still 3000 bloody mouse clicks.

In #109 I planned to make a CONTRIBUTING.md. Maybe we could stick them in there and invite people to add them to the appropriate category, but then we might as well just hyperlink to the Wikipedia article you got them from. I dunno what's best.

[trapd00r]

env 0.00s user 0.00s system 51% cpu 0.007 total
I haven't noticed any performance drops what so ever. :)

Yeah, I don't know the best way to approach this either. If the wiki description for each extension read something like audio/mp3, image/jpeg or whatever, it would be possible to do this programmaticly...

However, I've made a somewhat clean dump of the extensions and descriptions if anyone's up for sorting all of these out somehow: https://github.com/trapd00r/LS_COLORS/blob/japh/wiki_fileext.txt

[trapd00r]

I figured using libmagic would work wonders (file uses it: file foo.*). The issue, though, is that it guesses the filetype based on the first few bytes of a file, so you can't just touch all of these 3k file.extensions since they'll be empty. You'll have to actually create the files in question.

Here's the database that libmagic uses: https://github.com/threatstack/libmagic/tree/master/magic/Magdir

[trapd00r]

Give me a few days...

[rpdelaney]

Why days? I could cross-reference that super fast in sqlite. If this is a lot of work for you, stand back! I got this.

Still not sure we want to do this though. env is going to fill up my terminal buffer...

[trapd00r]

Sure, go ahead. I added everything in a dictionary: https://gist.github.com/trapd00r/554f03450ed114fee191e794c87b0215

I am not sure either, but there's no performance issues so why not, really. :D

[rpdelaney]

Great, that will be super easy to parse.

Some of these are kind of giving me lulz though. '9.PNG' => "NinePatchDrawable Image", really? But I'll probably only include those that I can cross-reference with libmagic.

I am not sure either, but there's no performance issues so why not, really. :D

I use direnv and environment variables for various purposes so I often do env | grep -i foo. I'm not going to enjoy all the extra accidental collisions with LS_COLORS, especially since each false match will scroll everything off my terminal buffer. Might just have to write a wrapper of some kind that extracts what I need with some explicit exclusion of LS_COLORS so I don't ever accidentally hit it. Edit: Now that I think about it I bet there is something that could handle this for me. I'll look around.

Anyway, the point is my use case is probably not the normal one, so if performance is really that much of a non-issue then there's little reason not to include these if we can automate the categorization.

[rpdelaney]

Also, would it be a goal to automate the scraping / categorization? That seems horrendously over-engineered but people will be updating the list on Wikipedia ...

edit: a script to build the LS_COLORS out of some kind of database (dunno what format yet, probably simple json would do it) could be useful regardless. That would enable us to do things like have names/labels for the colors themselves and then associate extensions with the named labels, etc.

[trapd00r]

Yeah, forgot to tell you but the extensions in my dict above is scraped from fileinfo.com - their descriptions were a lot better (and also more extensions). And yeah, some of them are pretty bonkers...

I'm all for automation, I'll tinker more with this tomorrow after a good nights sleep...

[trapd00r]

We could cheat and scrape from their already defined categories but not sure if every filetype is categorized. Maybe it's good enough anyway.

Edit: If you're going to scrape anything, do note that only 500 results are showed by default - you'll have to scroll down and click view full list