Make sure to install Beautifulsoup4 – `pip install beautifulsoup4`

In mfdl.py – Find the ‘find_direct_url’ function and replace it with this:

def find_direct_url(info_url):
rq = requests.get(info_url, headers=HTTP_HEADERS, timeout=TIMEOUT_T)
web_html = rq.text
soup = BeautifulSoup(web_html, ‘html.parser’)

# Find the download link
download_link = soup.find(‘a’, {‘class’: ‘input popsok’, ‘aria-label’: ‘Download file’})
if download_link is None:
return {“success”: 0}

direct_url = download_link[‘href’]

# Find the uploaded location
uploaded_from_tag = soup.find(‘p’, text=re.compile(‘This file was uploaded from ‘))
if uploaded_from_tag is None:
return {“url”: direct_url, “location”: “Unknown”, “success”: 1}

uploaded_from = uploaded_from_tag.text
location = uploaded_from.split(” on “)[0].replace(“This file was uploaded from “, “”)

return {“url”: direct_url, “location”: location, “success”: 1}

This uses BS4 to parse the webpage and look for the href download link we want, avoiding the manual way of just searching for the exact download_link_prefix. I would make a commit or merge, but it seems the 2 maintainers aren’t active on there anymore. Hope this helps anyone using the tool! Everything else in the tutorial (at least for downloading folders) should work fine.

Hi themadprogramer,

Thank you so much for this post. Quick query: If I get to the part of your post where you say:

“Raw data collections are stored in assorted, non-alphabetic, batches. To see if a group has its raw data available on the Internet Archive, simply query:”

… and no result appears ~ is there a workaround, or someone to contact to see when the OPEN group I have found (using the previous parts of your blog) posts might be uploaded again?

Thank you so much.
Simon

Worked perfectly, thanks!

Awesome.

That would be great even if just to read.

That would be awesome, but so far have not heard anything. Thanks. Would love seeing the yahoo groups return even if we can’t post but can read.

If you read the post on his site (the one that redirects to the discord server), the admin laments that certain egos in the community get on a soapbox and mask their own selfish desires as doing things ‘for the community’. You can see how it is then extraordinarily ironic that this admin shuts down the site and speaks for the ‘community’ about the archiving issue. He is effectively taking the ball and going home, flexing the last remnants of control in some hilarious ego power play. The fallout with Reshax, and anyone else reaching out to him in good faith, about continuing supporting the people he hosted on his forums (read: leeched off of and now apparently represents their interests) all but confirms this. There is no logic to it. But the author of this article is trying to rationalize the decision and present it in the most painfully neutral way to not step on anyone’s toes.

The author isn’t asking the most obvious question: Why the hell should the privilege of “privacy” be afforded to an admin/”community” who built their reputation and site off of decompiling and reverse engineering copyrighted and restricted materials? THAT is the most ironic part about this whole ordeal!