Well, my gig internet is really paying for itself today. I'm pulling in +500gb of data at some breakneck speeds. This is going to take a week...
@Clifford I understand those mixed feelings 😆
@Clifford Let me see the damage you've done to these tools
Literally a nasty one liner using wget to download the raw html, sed to delete html on rows with an href, then sed one more time to convert the remainder into a url in which I loop through to download a shit ton of files. It's still running 24 hours later.
I could have forked processes to speed it up... But I don't want to overwhelm their infra. It's internet archive. Slow and steady when's the race.
Mostly an instance for friends and family. Nothing special.