Archival Tools - If you make a thread without archiving shit I will strangle you.

[Wärring Ornac]

Just informing you that

this line of code allow archive.md to bypass the safe mode filter of tumblr. So now you can freely archive nsfw tumblr blog.

 

[XYZpdq]

Is there any way to download twitch streams after they're done or do I need to dub out a copy in OBS or something?

 

[awoo]

Is there any way to download twitch streams after they're done or do I need to dub out a copy in OBS or something?

Twitch clips, VODs, and highlights are all supported by youtube-dl. It is more powerful than you might think.

 

[Wärring Ornac]

Googleweblight is a service from google with the purpose of aiding your browsing experience... by simplifying everything to the most basic form possible

Spoiler: Example

http://googleweblight.com/?lite_url=https://kiwifarms.net/threads/archival-tools.6561/page-3

No pop-up, no fancy css, image were resized to be smaller. It looks ugly, but it serve its intended purpose of making site load faster by reducing everything. Several site like wikia just lose its functionality. So why am I posting this here?

Well it's capability to simplifying everything helps in archiving with archive.md. For example, archiving patreon comments:

Spoiler: Original

This is the normal archive

https://archive.md/4HyiD
This is the GW archive

https://archive.md/U0R72

Note that while the GW archive is messy, it retains some comment from the patreon site. The normal archive did not

 

[OG 666]

Figured this thread was the most appropriate place to post this tool. You plug in a Tumblr URL and it will give you an archive that only shows their original posts: https://studiomoh.com/fun/tumblr_originals/

It's extremely useful when you're trying to find original content by someone who reblogs 1,000 things on a daily basis.

 

[Wärring Ornac]

When archiving Google Docs (or at least the one that is mostly text), you should add /mobilebasic to the end of the link
Normal link:

http://archive.md/dnmEv
with /mobilebasic:

http://archive.md/yP205

 

[awoo]

When archiving Google Docs (or at least the one that is mostly text), you should add /mobilebasic to the end of the link
Normal link:
View attachment 502255
http://archive.md/dnmEv
with /mobilebasic:
View attachment 502256
http://archive.md/yP205

Can't you just download the document?

 

[Wärring Ornac]

Can't you just download the document?

That is always an option, this is just an additional way of archiving where people cant accuse us of tampering with the document

 

[Uncle Warren]

If you want to archive videos, streamable.com may be useful. I'm not sure if it's a reliable site but I seem to have no problem with it so far. You don't need an account to upload and the site uses a cookie to keep track of what you upload from your specific browser.

 

[AA 102]

Just on the off chance this info could be useful, I found this webm online showing how you can use inspect element on a video player to download a video without a program. I know it shows you how to download videos from some sketchy streaming site, but I figured it might be able to be applied to unusual video players/ video players without a downloader online.

 

[AF 802]

I'm going to give it a go to do an archival service. If you need something archived that might not fix in the context of archive.md or archive.org, I'd be willing to take suggestions. PM me if you want to ask.

URL is here:
http://mirror.lolcow.club/

 

[awoo]

I'm hesistant to use such a service since its longevity hasn't been proven. In comparison, archive.org will almost certainly last for many years, though they may takedown copyrighted content. archive.md is used frequently and seems like it will last, though not with the certainty of the Internet Archive.

 

[trannyfucker]

If you want to archive an entire account on a site with infinite scroll, such as ask.fm, or Twitter, forget archive.md, it's shit.

Instead, infinite scroll down to the oldest tweet/question (preferably using 'tweets with replies', rather than 'tweets'), then 'save complete web page' (in chrome) or probably other browser.

seems to work fine.

note that for 'tweets with replies' you need to be logged in. So you'll want to use a completely anonymous twitter account for this purpose, or you'll need to remove all traces of doxxing yourself.

 

[highdegger]

Instead, infinite scroll down to the oldest tweet/question (preferably using 'tweets with replies', rather than 'tweets'), then 'save complete web page' (in chrome) or probably other browser.

seems to work fine.

One problem with doing it this way is that the libraries required to correctly display the site might not be available forever. I think you can scroll down to the beginning as you say, then use "full page caputre" which printscreens the whole page with the simple click of a button. I use a Chrome-extension named "Full Screen Capture" which exports to png IIRC.

You can of course also download the HTML as you suggest and download what you can of the CSS and JS required to display the site but I'm not sure how comprehensible the result will be.

 

[Gustav Schuchardt]

Just on the off chance this info could be useful, I found this webm online showing how you can use inspect element on a video player to download a video without a program. I know it shows you how to download videos from some sketchy streaming site, but I figured it might be able to be applied to unusual video players/ video players without a downloader online.

If you want to download videos use youtube-dl

https://rg3.github.io/youtube-dl/download.html

On a Mac you just type

brew install youtube-dl

On Windows just run the exe file

On Linux you can install with

sudo pip install --upgrade youtube_dl

or with

sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl

Or maybe your distro's package manager has it.

It's a really nice tool - you can do

youtube-dl -F <youtube_url>

to see a list of formats ant then

youtube-dl -f <format> <youtube_url>

to download. Generally I just use

youtube-dl -f mp4 <youtube_url>

I.e. format=mp4 for video+audio and

youtube-dl -f mp4 <youtube_url>

I.e. format=m4a for audio only and just get the default quality. mp4 is H.264 video and AAC audio, m4a is just AAC audio and both are pretty widely supported, aka 'the GPU in your old ass phone and computer can probably decode them with low CPU usage'.

 

[AA 102]

If you want to download videos use youtube-dl

The webm I posted is meant for websites without downloaders already dude.

 

[CIA Nigger]

Ah, I get it now

Speaking of archive.md, I advised anyone who failed to archived a page with it to use via.hypothes.is
Example:
https://mastodon.xyz/@donleykogn
View attachment 402340

Now if you use via.hypothes.is : https://via.hypothes.is/https://mastodon.xyz/@donleykogn
View attachment 402341

EDIT: So apparently if you try to archive deviantart with archive.md, this is what you got:
View attachment 403248

Spoiler: Archive log

View attachment 403256

I've tried this with several blog, but the result is still the same. But it turns out that using via.hypothes.is is also a solution to this problem:
View attachment 403257

Spoiler: Archive log

View attachment 403258

I'd like to add that with Mastodon instances you will want to use this. Just about all mastodon instances tend to block archival tools, but archiving it through a proxy (like the hypothes.is trick does) to archive it works. You'll need to look in the source code to see spoilered/content warning posts though.

 

[awoo]

I remember from a long time ago one neat thing Internet Explorer supported was saving a webpage as MHTML, which is like saving the complete website but all in one file. The same effect can be achieved by zipping the folder with all the page resources for a little less convenience.

 

[CatchFire]

I found a tool that might be useful for saving ongoing live streams it's called streamlink and it works on the Linux command line and in windows power shell (If you have python installed)

Link to the GitHub project: https://github.com/streamlink/streamlink

To download a live stream from the start while it is in progress you use this command:

streamlink <URL to twitch stream or YouTube stream>  best --hls-live-restart -o out.mp4

 

[AnOminous]

I found a tool that might be useful for saving ongoing live streams it's called streamlink and it works on the Linux command line and in windows power shell (If you have python installed)

Link to the GitHub project: https://github.com/streamlink/streamlink

To download a live stream from the start while it is in progress you use this command:

streamlink <URL to twitch stream or YouTube stream>  best --hls-live-restart -o out.mp4

Looks like there's a homebrew version too for MacOS.

brew install streamlink

Also looks like it has a lot of other options, including crunchyroll (!).