Shutdowns

In about a week’s time, Google Drive’s Security Update for shared links will be rolling out on September 13. Google is introducing a new resource key to shared links, aiming to make link sharing more secure. This means, that in short, many links shared before September 13 will no longer function beyond that date, rendering many shared files inaccessible.

What you need to know! A few gotchas to this update:

• If any Google account has accessed a shared-link prior to September 13, they will retain full-access to said file/drive beyond that date, unless access is revoked by the owner. More on this later.
• Drive accounts can view their affected files and even opt-out of the security update altogether, if they are not a Family Link account. However this opt-out is possible not before, but after September 13, see https://support.google.com/drive/answer/10729743 for details.
• Don’t worry about your petitions getting blocked; Google Docs, Sheets and Slides are exempted from the update. To be clear, this does not mean all .docx, .xlsx and .pptx files; it only means that Google Workspace‘s native files will not be affected. Links to folders containing Docs, Sheets or Slides do not seem to be exempted either. For further details see https://support.google.com/a/answer/10685032

New Projects

On the subject of Google Drive, recall that accounts which have previously accessed a file will be able to retain access. So we at Data Horde developed a tool just for that purpose. Google-Drivac, allows you to “vaccinate” a particular Google account against a list of links.

Just log into a Google account, copy some cookies, gather a list of links you want to retain access to (either through crawling, or through maybe another account) and presto!

You can find Google-Drivac on our GitHub at https://github.com/Data-Horde/gdrivac.

In the meantime, for those hoarders among you who would prefer a hard-grab, Archive Team is working on doing just that. You can follow development on the Google Drive Warrior on #[email protected]

Last but not least, some of you might recall the panic surrounding Mediafire allegedly deleting accounts from earlier this year. While the threat had been greatly exaggerated, the scare did bring with it some innovation.

Zopolis from the Media Flare project has developed the Automatic Mediafire Link Archiver (AMLA). This is a Chrome extension which automatically records Mediafire links on websites you visit, in the background. These links are then forwarded to Ajay’s MediaFire URL Submitter public database and Archive Team who periodically grab its contents. All website activity is tracked by default, so if you’re keen on your privacy you should also restrict its permissions in the extension settings.

AMLA can be downloaded from here: https://github.com/Zopolis4/AMLA, and if you are into URL hunting be sure to visit the Media Flare Discord server.

New Projects

History buffs better follow the Institute for Digital Heritage which just launched last week. It’s a new organization aiming to turn cultural heritage into digital heritage. They help museums and cultural institutions digitize and open up their collections, with partners like FloatScans who specialize in 2D/3D scanning. Among its ranks are scholars in digital humanities, culture studies and related fields primarily from Europe and Africa.

Got any processing power to spare? Crawling@Home is a project to match text and images, en masse, from the >3 PB Common Crawl dataset. When completed it will be the world’s largest image-text pair dataset, to fuel next-generation machine learning models. Current stats are available on their homepage.

u/-Archivist has made a tutorial on how to join C@H (periodically updated) which you can find on r/datahoarder. If you have Docker installed, go ahead and execute docker pull arkseal/cah-worker && docker run --shm-size=2g -e NAME={username}-TE arkseal/cah-worker replacing {username} with a nickname of your own.

Updates

Speaking of web crawling, the Internet Archive’s crawler Heritrix3 received an extensive update last week adding in a browser-based link extractor ExtractorChrome. While now only limited to grabbing <a> and <area> links, the team plan to extend it for crawling intercepted sub-requests, possibly mimicking Javascript behavior.

Last but not least, The-Archive has updated their website. Whether you’re looking for Strategy Guides for retro gaming or Milkdrop Plugins for Winamp Wednesday, The-Archive has much to offer.

The Cinemateca Brasileira was ravaged by a fire last Thursday, on June 27. A core institution for preserving Brazillian cinematography, the Cinemateca was estimated to house over 2000 films and 4 tons of documents in its archives. Artists and conservationists had expressed their concerns on the frailty of the material housed in the Cinemateca, as early as April of this year, noting for instance, the risk of self-combustion of cellulose nitrate films.

According to the local fire brigade, the fire began after maintenance on the air-conditioning system. After assessing the damage, Fire captain Karina Paula Moreira announced -regrettably- that “we will only know for certain after the experts (i.e. forensics), but, probably, nothing was preserved”. A grim end, to remind us all how media preservation is NOT a thing postponable.

New Projects

YouTube finally privated pre-2017 unlisted videos last week, after a delay of a few days. But that wasn’t the end of the story, we still have a few archiving projects and tools to talk about.

Our good friend Jopik has made a TamperMonkey script for revealing the original name and uploader of privated videos in your playlists, not unlike a certain other project from a few months ago.

Jopik was also behind the filmot collection) and that’s actually where the info is being fed in from. You can install the filmot Title Restorer script from here.

Another WIP project is rebane2001’s playlist restorer which will also be able to grab thumbnails and even the videos themselves, if available. Unfortunately, this tool is undownloadable at this time so we will have to settle for the Reddit trailer for now.

Updates

Speaking of YouTube’s unlisted videos, Archive Team’s grabs have landed in the ArchiveTeam Inbox collection on the Internet Archive. In case you are out of the loop, Archive Team grabbed 200 TiB of data on 5,739,754 videos! Titles, comments and low-resolution copies of the original videos!

Last year, when Yahoo Groups shut down archivists pooled together data on different groups and published them on the Internet Archive. Now, the Save Yahoo Groups! project is looking for help in identifying fan groups.

TV Shows, literature, games… If you have had any history on Yahoo Groups prior to its shut-down, or consider yourself a fandom enthusiast, we implore you to head over to the SYG Discord Server or their fandom identification spreadsheet.

The latest version of MAME (0.234), the arcade machine emulator, focuses on 3D arcade games. You are in luck if you were in the mood for some GTI Club! Read more about it on MAME’s blog!

Public Interest crusader and Internet Archive partner Sherwin Siy was reported to have passed on 7 July, 2021, at the age of 40. Through the span of his career, Sherwin Siy served as Public Knowledge‘s VP of Legal Affairs, helped shape the Public Policy of Wikimedia and had a long history of cooperating with the EFF.

Sherwin Siy’s contribution to the Internet Archive was making sense of how the Archive handled TV news, from a legal perspective, as reported by Lila Bailey, his former partner in law. Suffice to say, news archives on IA such as the Third Eye news chyron collection might have never came to be, were it not for his efforts.

Updates

Bluemaxima’s Flashpoint, the webgame preservation project/community, finally has a dedicated submission website at https://fpfss.unstable.life/web. Previously, game submissions were made through their Discord server and had to await a laborious approval/rejection process. Let us hope, that this change will make archiving efforts a whole lot easier. You can read more about the how-to of it here.

Again on the subject of Flashpoint, the community has taken it upon themselves to preserve interactive “YouTube games”. As known to our longtime readers, once upon a time YouTube had an annotation system which could be used to link videos together. Some folks went out of their way to make Choose Your Adventure styled games, where depending on your choices, you would be taken to another video. But after the feature’s removal in 2019, many channels unlisted their annotation-intensive videos and now with YouTube’s plans to forcibly private old unlisted videos, these games are at risk of extinction. To help out with the Flashpoint project to preserve these video games, head over to the #youtube-games channel on the Flashpoint Discord Server.

With 11 days to go before the delisting of unlisted videos, archivists are working around the clock to hunt down unlisted videos. All the projects we discussed last week are still in motion; from Archive Team’s metadata scraping on #[email protected], to subreddit frenzies for harvesting videos linked on the filmot.com index.

Another noteworthy project is the #youtube-unlisted project on the Distributed YouTube Archive. A major bottleneck for Archive Team, and other groups, has been archiving of raw video files, as the Internet Archive and Google Cloud are not suited for a sudden influx of large video files. The DYA project, aims to mitigate this by splitting the task of storage between contributors. If a video is requested for download, the contributors who have made a copy of that video share their copy. While this might seem like a tedious process, it means that anyone with spare space can contribute to the storage, without TBs of hardware.

Finally, Omniarchive is holding a competition to collect Minecraft related unlisted videos.

What do I get out of it?
We will be giving 1 month of Discord Nitro, as well as a unique Discord role, and credit on the Omniarchive index, to the three users who submit the highest number of unique, valid videos.

How can I help?
Simply all you need to do is post as many UNLISTED Minecraft videos uploaded on or before 31st December 2011 into #unlisted-videos as you can possibly find. That’s it. If you find an unlisted Minecraft video uploaded anywhere from 2009-2011, post it! We’ll handle the rest using a few scripts to filter out any duplicates and other unwanted links. Check out this link for a detailed guide on how you can find such videos: https://docs.google.com/document/d/1HMeoH7XOpUvqBkUD1jBy5MfzgzK4X4wgUpzV2YIZ4Fs/view

HalfOfAKebab, Omniarchive Mod

Are you involved with an archiving project related to YouTube’s unlisted videos, or not? Reach out to us at [email protected] so we can give you a shoutout!

Shutdowns

Webs, the once popular website host, was expected to shut down after 20 years, at the end of June. Fortunately, Archive Team is on the case, and double fortunately it seems that sites are still up, even today on 5 July. Archive stats available here.

Microsoft’s Codeplex archive was expected to shut down some time in July. I am happy to report that Codeplex has already been fully scanned and Archive Team’s collection is available here.

Unlisted YouTube Video Scramble

With 18 days to go before YouTube forcibly privates old unlisted videos and playlists on July 23 many archiving projects are being run concurrently.

There has been much activity on certain subreddits for sniffing out links to unlisted videos. These include r/speedrun and speedrun.com collecting unlisted speedruns, r/nerdfighters backing up unlisted Vlogbrothers content, r/homestuck chasing after fan videos, besides the usual suspects from r/datahoarder.

Our fellow archivist Jopik, recently published his own unlisted video collection on filmot.com. Privately collected from various corners of the internet over the past 2-3 years, this collection contains links and metadata on several million unlisted YouTube videos. Members of Flashpoint and The Eye have taken note of the collection and have began archiving video files of interest.

Finally Archive Team is working on a major project for collecting metadata for unlisted video links on hackint#down-the-tube and a minor project for saving notable videos on hackint#youtubearchive. Some of Archive Team’s unlisted videos are being provided by Sponsorblock which continues to ask users to submit unlisted video URLs.

Forthcoming projects include Omniarchive‘s rush to grab unlisted Minecraft footage and the Distributed YouTube Archive prioritizing a queue of unlisted videos…

Other Updates

The popular modding community Nexus Mods has taken a decision to make permanent archives of all mods hosted on their site. While the decision has been celebrated by web preservationists the world over, it has also drawn ire from some modders who are upset that they have lost the freedom to delete their mods, as reported by Kotaku.

This change is intended to lay the groundwork for Nexus’ upcoming collections system, which will allow modders to combine different mods for the same game. Obviously, if the dependencies of a collection were deleted the entire collection could break, left-pad style. Community manager BigBizkit made it clear that as part of Nexus’ “noble” mission to make modding easier, it was essential that modders be able to build upon previous work.

Let me stress that even without collections in the picture, file deletions and disappearing data constitute a problem and create a development environment that cannot serve as a strong foundation for the future of our platform. 

BigBizkit, An important notice and our future plans for collections

As a compromise for those not fond of the change, modders have a month-long grace period to delete any files/mods that they do not want indefinitely preserved.

Yahoo! Answers might have shut down a couple of months ago, but a substantial portion of a former hallmark of the web was saved. Unfortunately said massive collection is not very human-readable. There are talks on hackint#yahooanswersgraveyard to build a “Yahoo! Answers Archive” search engine akin to the YouTube Community Contributions search tool. Get hyped and stay tuned!

Discoveries

Hit Save! and the e-Reader shared a number of discoveries over the past few weeks. Notably, they managed to acquire a rare 2002 Battle Road trophy card and play it on an e-Reader.

You ask what is a Battle Road trophy card? These “trophies” were awarded at official Pokémon TCG tournaments held across Japan. These cards came in several flavors, with a male/female trainer, with different pokémon in background and even what the winner had placed in the tournament they had earned the card.

Though serving no gameplay effect in the card game, these cards were readable by Nintendo’s now-defunct e-Reader GBA extension. Card Capto- (ahem) Collector Qwachansey was kind enough to send a card he had acquired to the folks at Hit Save, and now for the first time ever you can see the funny little message you got for scanning these trophy cards.

The full blog post has a lot of other interesting details on how the team built a modified e-Reader to be able to capture footage, as well as some other interesting cards they have come into possession of recently. Read about the The Ren-e-ssance for yourself here!

Long time SNES manual preservationist, author and contributor to numerous emulation projects and friend to many game preservationists Near/Byuu has passed away on June 27, 2021.

Earlier that day, they had sent out a series of Tweets to the effect of a suicide letter. Wishing to remain anonymous, the last person to have talked to Near prior to their death would soon contact Hector Martin “marcan”, believing them to have taken their own life while on the phone. Several hours later the event was confirmed by local police.

This tragic turn of events had followed episodes of harassment, which Near detailed in their final words. Their parting request was that they be remembered for their many contributions to the community, and not for they were about to undertake.

Shutdowns

Last Wednesday, on June 23, YouTube announced a decision to automatically set all Unlisted Videos uploaded prior to 2017 to private, one month later on July 23. The difference being that; unlisted videos are hidden from search results, but people with the link can access them, whereas private videos are inaccessible to other users unless the uploader gives manual approval.

While channels have the ability to opt-out, to keep their unlisted videos unlisted, and not privated, it has come to the attention of archivists that many inactive channels are unlikely to pick this option given the short timeframe presented.

In the way of archiving projects for unlisted videos, there has already been much discussion and some organization on #down-the-tube on hackint, Hacker News, r/datahoarder and the Distributed YouTube Archive. Alas, there is yet no project in motion, at this time.

The Unlisted Videos website, which was made specifically for the purpose of collecting links to unlisted videos, has been scraped or is being scraped by several groups, worth about half a million videos. There is also Jopik’s searchable collection of 4.5 million unlisted videos, which is a monument in its own right. That being said, these are only links, and the video files themselves have yet not been mirrored. So be sure to stay tuned for upcoming projects.

To spread awareness of the situation, we are doing countdown of unlisted videos on the Data Horde Twitter account.

This upcoming change from YouTube comes with a similar update to Google Drive, which will render many shared files inaccessible to users who have not accessed them prior to a certain date.

Updates

In support of the efforts to archive unlisted videos, Sponsorblock has introduced a new feature to detect, and anonymously submit links to unlisted videos, that users might be watching.

The Flash Player emulator Ruffle, is now a bit easier to install. Ruffle has finally been added as an extension to the Chrome Web Store, and you can run it from the comfort of Chromium browsers.

In other news, the mod community/archive Gamebanana suffered a major outage over the weekend. Thankfully, as it turns out, this hiccup was only the result of a billing glitch on their host’s side. The site is now up and running once more.

Can you believe our host accidentally suspended 16 of our servers due to a billing glitch, and nobody was around to fix it because it’s a Sunday. This is the biggest host blooper we’ve ever encountered in 20 years.

tom, Gamebanana Admin

Discoveries

Members of Omniarchive, a group dedicated to archiving old lost versions of Minecraft, managed to recover the elusive Alpha 1.1.1. version on June 25, 2021. The first of many Seecret updates, Alpha 1.1.1. was notable for being online for only a few hours before the Alpha 1.1.2 hotfix.

Archivist ProffApple found a tweet someone who had just downloaded the update had made over a decade ago on the day that Alpha 1.1.1. came out, September 18, 2010 to be specific. Turns out, they still had the game files lying around!

You can read more about the story on this Kotaku Article by Zack Zwiezen and this PC Gamer article by Jonathan Bolding.

Unable to settle their differences peacefully, a number of publishers have decided to take legal action against the Internet Archive for their National Emergency Library initiative.

If anything good has come of this tense situation, it’s that Internet Archive and oldgamemags.com seem to have established a ceasefire. oldgamemags admin Kiwi has stated that this is only a temporary pause on the site closure and the countdown of 5 days might resume if things don’t work out. Further details are available at: http://www.oldgamemags.com/index.php/forum/latest-news/109-site-closure?start=42

The main story we’ll be covering today is a conflict of interest which emerged between preservation communities a week or so ago.

Shutdowns

Today I am grieved to be reporting a shutdown for a community of our own, OldGameMags*, a long running website dedicated to the preservation of old game magazines and other retro goodies. The website which was formerly known as Kiwi’s World migrated to their current domain about one year ago.

* Unaffiliated with the Twitter/Tumblr account which go by the same name.

If the previous name was any indicator, the admin and sole proprietor of the website and downloadable magazines goes under the name of “Kiwi”. The website was a collection of whatever Kiwi could manage to get his hands on through his own means.

While initially offering viewing of these collections for free, asking only for donations (monetary and in the form of magazines/digitizations) to keep the website online; increasing costs of procurement, digitization equipment and keeping the hosting going lead to a switch to a payed membership model. Being his own personal project, Kiwi did all in his power to ensure the website would remain self-sufficient.

This stricter policy, however surprising as it may seem, worked to foster a community. While initially membership was a process Kiwi had to undertake manually through emails, the small group of preservation enthusiasts he was able to gather eventually convinced him to switch to a forum. Like a white dwarf, a seemingly collapsing star had become a tightly packed, sincere community.

So where’d it all go wrong? When it became apparent that portions of Kiwi’s collections were being uploaded to the Internet Archive, he was flabbergasted to say the least. Despite making these inaccessible, obscure materials accessible online also being his very own goal, the idea of these being nabbed without his permission was quite aggravating.

The Internet Archive doesn’t have the same constraints when it comes to costs associated with procurement, digitization and hosting; all of which Kiwi had to undertake himself. This is of course beyond the time and effort he had to personally invest into the management of the project. Same goal or not, it wouldn’t be natural if he didn’t take this as some form of offense.

And so, on May 26, he took the decision to activate a time-bomb on his website to go off in 2 weeks. He urged members to download whatever they could, as in his book, these people all had some share in building up that massive collection.

Since the announcement, he’s been in contact with fellow archivists who are members/affiliates of the Internet Archive, in hopes of reaching an alternative settlement.

In one of his latest forum posts he detailed 4 possible outcomes and asked forum members on what they’d prefer:

1. Making the site private in some way. Significant members would likely retain access to all content.
2. Status quo, just live with the fact that some members will just upload scans elsewhere.
3. Migrate, maybe even to archive.org, but close the site for good.
4. Add in a second tier of membership, and limit the accessibility for tier one. If all else fails default to solution #1.

Regardless of what option they choose to go with, their story of how they almost (or possibly still might) shutdown is one worth detailing. It’s a tale that really highlights how not all preservationists have access to the same means, something I personally think we don’t keep in mind too much.

As Data Horde, a group who seeks to inspire cooperation among the many archivist groups across the internet, I think we need to keep that in mind. It’s called cooperation for a reason, yes it might be easier for one particular group to host a certain article, but we must never forget the blood, tears and sweat which it might have taken to get a hold of said article in the first place.

For the settlement of this conflict or any of a similar nature, I believe this to be the first step: both sides seeing each other for what they are…

New Projects

On a more positive albeit related note, u/Ruthalas is working on making a Webcomic Scraper, for quickly collecting webcomic panels off of websites:

https://github.com/Ruthalas/not-quite-daily-scraper

This is more of an announcement for programmers who might be interested in joining in, as it’s nowhere near complete. For a more proven solution check out: https://github.com/webcomics/dosage

Shutdowns

Imgur’s Community forums shut down on May 22, following an announcement from a month ago, which served as the finest epitaph the forums deserved:

This isn’t a decision that we take lightly, and it’s one that I deliver with a heavy heart. Throughout the past five years, IC has helped Imgur to evolve, recognize areas of strength and weakness, and connect on a human level with Imgurians. More importantly, it became a tight-knit Internet home for some incredibly awesome people, leading to lifelong friendships and even tiny new humans! We sent a banana around the world, Q&A’d with interesting people, defied the “no selfies” rule (three times over), counted halfway to ten thousand, and brought civil discourse to challenging topics. It has been a marvelous run.

Sarah Schaaf, Admin of the Imgur Community Forum

Fortunately a portion of the forums survives on the WayBackMachine, feel free to check it out: https://web.archive.org/web/20200521041503/https://community.imgur.com/

Next we have Heroes wiki, a wiki for the NBC series Heroes is shutting down June 1st. The shutdown announcement came a few days ago:

I want to thank the Heroes Wiki community for everything they’ve done. From interviews conducted with cast and crew to detailed character and episode information to fan theories. We’ve created something great and I will always have the fondest of memories of our times and achievements.

Brian, Admin of Heroes Wiki https://heroeswiki.com/User:Admin/Goodbye

New Projects

Have you ever had videos deleted from a YouTube playlist and wondered what those videos were? Using a variety of techniques, it is actually possible to retrieve the name of a removed video.

Reddit user u/Shining_Force_Unity recently unveiled a tool they made, which automates the process:

Updates

Capcom recently re-released 4 April Fool’s games from past years on their website. A quiz game, a card game, a shooter and my personal favorite: 2020’s NECO DROP, a Shariki/Bubble Popper game with Street Fighter characters as cute cats!

The web-games will remain available for the rest of the year, head on over to https://www.capcom-unity.com/2020/05/21/street-fighter-april-fools-mini-games-have-returned/ yourself if you’re looking to kill a few minutes!

Shutdowns

For starters Google is preparing to shelve Google Play Music. In an effort to shift attention to YouTube Music, they’re endorsing people to transfer over their libraries, even offering a tutorial:

This came on the same day as Google’s shutdown of Neighbourly and Shoelace last week, a pair of experimental “local social media” apps, as reported by Killed by Google.

Next up Cardcast, a popular card-matching party game in the vein of Cards Against Humanity or Xyzzy mysteriously went offline on May 16. The game was best known for its ease in constructing custom decks, which would currently appear to be lost with its shutdown.

Cardcast would only make a brief announcement, to indicate this wasn’t an accident, and that they had to go with such an arrangement. Although stating that further details would be provided, there hasn’t been any further explanation thus far.

Next up, the Myst Online and Cyan forums were announced to be going offline June 16. The community is transitioning to a Discord Server, with this announcement serving as a warning for people who might want to archive their messages, you can read more about it at: https://mystonline.com/forums/viewtopic.php?f=36&t=29240

Updates

In a surprising turn of events, 8tracks returns! And what a heartwarming story it is!

TL;DR:  The 8tracks platform & brand are now owned & operated by BackBeat Inc., a new startup which acquired a portion of 8tracks’ assets during its wind-down earlier this year. 

While certain music will not be available due to licensing restrictions, most existing playlists will be available to stream, beginning today. 
If you still have the iOS app on your device, it will work out of the box.

Otherwise, we’ll be re-introducing the 8tracks mobile apps and playlist creation (and perhaps a few new surprises) in the months ahead.

Welcome Back announcement taken from https://blog.8tracks.com/2020/04/19/welcome-back-8tracks/

Besides a few changes in the business model, and currently being available only in the US, they seem to be working things out. We wish them the best of luck, seeing how brutally competitive the music streaming industry is today.

Discoveries

David Ashley, who you might remember as the sole programmer behind the Genesis port for Breach from last week, recently also released source code on a number of Gameboy Color projects: https://github.com/dashxdr/cgb