The Cinemateca Brasileira was ravaged by a fire last Thursday, on June 27. A core institution for preserving Brazillian cinematography, the Cinemateca was estimated to house over 2000 films and 4 tons of documents in its archives. Artists and conservationists had expressed their concerns on the frailty of the material housed in the Cinemateca, as early as April of this year, noting for instance, the risk of self-combustion of cellulose nitrate films.

According to the local fire brigade, the fire began after maintenance on the air-conditioning system. After assessing the damage, Fire captain Karina Paula Moreira announced -regrettably- that “we will only know for certain after the experts (i.e. forensics), but, probably, nothing was preserved”. A grim end, to remind us all how media preservation is NOT a thing postponable.

New Projects

YouTube finally privated pre-2017 unlisted videos last week, after a delay of a few days. But that wasn’t the end of the story, we still have a few archiving projects and tools to talk about.

Our good friend Jopik has made a TamperMonkey script for revealing the original name and uploader of privated videos in your playlists, not unlike a certain other project from a few months ago.

Jopik was also behind the filmot collection) and that’s actually where the info is being fed in from. You can install the filmot Title Restorer script from here.

Another WIP project is rebane2001’s playlist restorer which will also be able to grab thumbnails and even the videos themselves, if available. Unfortunately, this tool is undownloadable at this time so we will have to settle for the Reddit trailer for now.

Updates

Speaking of YouTube’s unlisted videos, Archive Team’s grabs have landed in the ArchiveTeam Inbox collection on the Internet Archive. In case you are out of the loop, Archive Team grabbed 200 TiB of data on 5,739,754 videos! Titles, comments and low-resolution copies of the original videos!

Last year, when Yahoo Groups shut down archivists pooled together data on different groups and published them on the Internet Archive. Now, the Save Yahoo Groups! project is looking for help in identifying fan groups.

TV Shows, literature, games… If you have had any history on Yahoo Groups prior to its shut-down, or consider yourself a fandom enthusiast, we implore you to head over to the SYG Discord Server or their fandom identification spreadsheet.

The latest version of MAME (0.234), the arcade machine emulator, focuses on 3D arcade games. You are in luck if you were in the mood for some GTI Club! Read more about it on MAME’s blog!

Thankfully, owing to the efforts of the Save Yahoo Groups Project and Archive Team the data of many groups has been preserved. If you missed out on the GYD tool, you might still be able to retrieve your groups’ data by following the steps below.

To begin, can you remember your group’s name? If yes, the following steps will go by a lot faster; but if not, you might want to make a list of potential names to go by. Was the name of your group Fireflylovers, or Firefliers, or LoversofFF? Write down all likely candidates.

For demonstration’s sake let’s search for data on NFforKids, a non-fiction writing group.

Let’s perform a metadata search, to see when NFforKids was started. Head over to the Yahoo Groups Metadata Collection page on the Internet Archive. Ignoring the no preview warning, either click on Show all files or scroll down until you see DOWNLOAD OPTIONS on the right side of the page.

Click on COMMA-SEPARATED VALUES, to reveal a list of files. Since NFforKids starts with an N, if it does exist, it will be indexed under master_N.csv. Download this CSV file to your device.

You can now open this CSV file using Excel or another spreadsheet program. Search for NFforKids to find the corresponding information row. What do you know? NFforKids was started on 11 June 2000. You can scroll accross this row to find the group’s primary language, the category of the group, if the group was public or not, and more!

If you weren’t able to find metadata on your group, it’s time to pull up that list I told you to make above. Fall back to the other candidates and try another name. If the first letter (or two) of this second name is different, you will need to download the corresponding CSV file before resuming your search.

Please note that while the Yahoo! Groups collections on the Internet Archive are thorough, they are NOT exhaustive. It is entirely possible that data on your group might have been missed. That being said the metadata collection sports a whopping 1.1 million groups. Even if you weren’t able to find your group in the first round, it is very likely that you may have misremembered the name, so keep on trying!

Once you have confirmed the name of your group, and that it has been catalogued in the Metadata Collection, you can then download the corresponding TAR file, which contains even more details. Again, if we’re looking for a group called NFforKids we’ll be looking for the first two letters from the list. That’s NF.tar for NFforKids.

If you’re on Mac or Linux, you should be able to open this .tar file to reveal a folder titled media. If you’re on Windows, you can use 7-zip to open it. This TAR file contains the same information as the CSV, plus additional details. Did the group have spam filtering, was media sharing allowed or was the group text-only? You might even find the URL for group images, although unfortunately most of those links are now dead.

Stats are fine and dandy, but what about messages or activity? If your group was restricted, tough luck, you’ll need to find a member who made a GYD copy before the shutdown. This is where our luck with NFforKids has run out, seeing as chats of the group were not public. For the final step, let’s switch to a public group whose history is visible. We’ll go with nfwritersontheirwayup. Messages in this group were visible to all subscribers, so archivists were able to grab its contents.

Raw data collections are stored in assorted, non-alphabetic, batches. To see if a group has its raw data available on the Internet Archive, simply query subject:"yahoo groups" nfwritersontheirwayup. If you get any results, your group’s raw data is most likely located here. You can double check the item description to be sure that nfwritersontheirwayup is indeed included in the batch.

Pop open the WEB ARCHIVE GZ download option from the left side of the page. Scroll down until you see nfwritersontheirwayup.bcqkJvN.warc.gz and proceed to download. To unpack this gzip you can use thegzip -d nfwritersontheirwayup.bcqkJvN.warc.gz command on Unix systems or good old 7-zip on Windows.

Last but not least, you’ll need a WARC viewer. If this is your first time with WARCs replayweb.page is very straightforward and runs right out of your browser. Simply upload the WARC contents of the group and voila, you can now navigate through the group’s chat logs.

Recovering your Yahoo! Groups from yesteryear is as simple as that. Got any questions? Or perhaps you have made some worthwhile discoveries while group hunting. Comment below!

Yahoo! Groups’ mailing lists, which are the last remaining part of Yahoo! Groups, will be shutting down in 10 days, on December 15, 2020. However, since group content is no longer accessible to the public, there is little left to archive.

Next year, volunteers will be needed to sort and organize the full group data so related groups can be uploaded to the Internet Archive together. This will make it easier to access and browse archives for multiple groups related to similar topics.

For more information about Yahoo! Groups, please see Doranwen’s blog or our Yahoo! Groups articles.

The story did not end there however. So let’s talk about what has transpired since…

Despite us even reporting 30 January as the final deadline, Yahoo continued to accept Get My Data (GMD) requests for about a week. So active efforts ceased around that time. Now was the waiting game, as it took a few more weeks for some of those GMD requests to process.

By late February, most of the volunteers had disbanded or moved onto other projects. But there was still much to be done. For one thing, people had rushed so much to grab everything that they could, that a lot of these group files were a total mess, not made any better by how Yahoo’s GMD exports worked. So the remaining volunteers stuck around to label their massive collection.

Doranwen, one of the leads on the Yahoo-Geddon (aka Save Yahoo Groups) project, frequently documented their progress during this time.

A few numbers and random other bits of info:

~2 TB of fandom data saved (that I know of, for now)
~200,000 confirmed fandom groups saved in some fashion
~2,000 Sims groups saved* …

*The only reason I know the Sims number is because I was tracking those groups on Google spreadsheets in order to find all of them and get volunteers to join them. For other fandoms it’s impossible to give any sort of number at this point (although I know there was a ton of LOTR, HP, Buffy, and Westlife, lol). Yahoo’s categorization was terrible and a group name doesn’t always give good clues as to whether it’s fandom/non-fandom. Getting that sort of data will take a good deal of time and work.

Doranwen, The end of Yahoo Groups – a few thoughts & stats

Another issue was that the collection was not actually unified. Archive Team had also archived a bunch of data, so the Yahoo-Geddon team continued to label those batch by batch for a few more months.

It truly is endless!!

Yahoo-Geddon volunteer, 14 July 2020

Yet another reason the Yahoo-Geddon team was taking so long was because of how meticulous they were. They worked to not only curate this collection for the sake of archiving, not only to trace the history of fandom, but also to be able to provide a rich dataset that researchers might want to use in the future.

-[Stage] 4.5b: Remember that we got a bunch of groups from scrounging the links of other groups for new groups to join? Some of the commands used to process that data generated “groups” that never existed (with http: stuck at the end, apostrophes or commas in them, etc.). Also one stage of the spreadsheet work ended up with a certain number of groups getting a duplicate version added to the spreadsheet with _dupe after the name.

So for this stage I send the spreadsheets to my assistant who runs a script against them to find groups with punctuation in them or _dupe at the end. A very very tiny number of very old (grandfathered from who knows which list service) groups actually legitimately have periods in their names, but in most cases groups with periods never existed either.

This process is fairly quick for each letter but varies greatly in what has to be done, as sometimes group folders are affected (and some punctuation marks Yahoo simply ignored everything from that mark onwards and treated the letters before it as a group name).

Yahoo Groups metadata processing steps, stage 4.5b

Sadly, Yahoo!, blind as ever to Yahoo-Geddon’s efforts, have decided to permanently shut down Yahoo Groups. While Yahoo Groups only retained its bare-bone features, this will be putting an end to some decade-old mailing lists…

On a related note, an interesting discovery Yahoo-Geddon made is that Yahoo actually has not deleted archives, photos and files but only removed public access.

The files are still there, from what I can tell! They’ve just blocked us from getting to them.

The monthly reminder emails with attachments are still coming in – and the attachments come from files in the files sections. Clearly those were never removed.

Which means that Yahoo could have chosen to grant us access to all of that for a full year before closing Groups entirely, but did not.

via the Save Yahoo Groups Discord server

Just goes to show that curation is the one half of archiving/preservation… If you would like to learn more or even participate in Yahoo Group dissection, check out the Save Yahoo Groups discord server: https://discord.com/invite/DyCNddf