Unfortunately, MediaFire doesn’t have a search/discovery feature, relying entirely on search engine traffic and external linking. There’s a lot of undiscovered things on MediaFire, and Pyxia’s mf-dl tool is one of the first tools that we have for exploring it. Read on to learn how to install and use mf-dl to easily download MediaFire files and crawl undiscovered corners of the internet!
mf-dl follows the usual steps for setting up a python tool:
https://gitgud.io/Pyxia/mf-dl.git
using a git client. Alternatively download the repo and unzip it.cd
into the mf-dl
directory and run python3 -m pip install -r requirements.txt
.mfdl.py
mfdl.py
is a bulk-downloader for MediaFire links. You may have found these links yourself, copied them from your bookmarks or possibly scraped them beforehand. At any rate, mfdl.py
will download the contents and metadata for a list of links that have already been collected.
The input is a sequence of links and can be any file separated by spaces, new-lines or commas. Ideally, you might want to use a spreadsheet-friendly CSV file. For this tutorial, copy the table below into Excel, or another spreadsheet editor, and save it aslinks.csv
.
https://www.mediafire.com/file/y1s9a51a941h7b8/The_Wasteland_%2528MP3%2529.mp3/file |
https://www.mediafire.com/folder/xb49obyqfut8d/Merlin_Das_Trevas |
https://www.mediafire.com/file/ngteu63n26rhncj/readmetxt4999251.zip/file |
Next we will need an output directory to save mf-dl’s grabs. mf-dl does not have permission to create new directories, so you will have to create a new folder if the destination doesn’t already exist. For demonstration’s sake we will create/output
directory under mf-dl
.
To run mfdl.py
, execute the following command from inside your terminal and mfdl.py
will begin downloading the contents of the input links into the output directory.
> python3 mfdl.py output links.csv
Protip #1: Increasing Download Throughputmfdl.py
can download several files concurrently. By default, mfdl.py
runs 6 threads; so that means that it will initiate 6 synchronous downloads at a time. If you have a high network bandwidth, you might want to increase the number of threads to maximize your downloading speed. Or if MediaFire is upset with your frequent downloads and is throwing CAPTCHAs your way, you can decrease your thread count. Use this modified version of the mfdl.py
call to change your thread-count.
> python3 mfdl.py --threads NEWTHREADCOUNT output links.csv
Protip #2: Multiple Input Files
All arguments after the output are treated as input files. If you have links split across several files, you can simply concatenate them to the end of the command.
> python3 mfdl.py output links.csv links2.csv links3.csv
web_crawler.py
web_crawler.py
is a utility for discovering new MediaFire links. That’s right, links not files.web_crawler.py
does not download the corresponding files and we will need to later feed the outputted links into mfdl.py
.
Setting up web_crawler.py
is a bit more straightforward. Then we need a seed URL to initiate the crawl. Any site with downloadables will make for a nice link farm. In this case we’ll be using the Minecraft Pocked Edition Downloads
site https://mcpedl.com/ as our seed.
To run web_crawler.py
, execute the following. Note that web_crawler.py
will run indefinitely as new links are discovered, until its execution is interrupted.
> python3 web_crawler.py https://mcpedl.com/ links_found.txt
Protip #1: Feeding Back Links
You can feed links found using web_crawler.py
intomfdl.py
with
> python3 mfdl.py output links_found.txt
In fact, if you’re familiar with Crontab, you can schedule periodicmfdl.py
jobs to download new links as they are added to links_found.txt
. This away, you can continue to download new links, without ever stopping web_crawler.py
.
Protip #2: Depth Control
You can limitweb_crawler.py
‘s search by specifying a filter. If you want to keep your search to mcpedl.com
, ignoring out-links to facebook etc. you can --filter
https://mcpedl.com
.
> python3 web_crawler.py --filter https://mcpedl.com https://mcpedl.com/ links_found.txt
Alternatively, you can specify --regex
option if would rather filter with regular expressions instead.
Protip #3: Thread Controlweb_crawler.py
can also run on multiple threads, 6 by default. You can choose the number of maximum threads you want to use by, again, specifying the --threads
option.
> python3 web_crawler.py --threads NEWTHREADCOUNT https://mcpedl.com/ links_found.txt
Have any more questions? To learn more about MediaFire archiving, check out the MediaFlare project!
]]>