Archival Tools - If you make a thread without archiving shit I will strangle you.

No Shot Bucko

No Shot!
kiwifarms.net
Anyway to download archived video tweets from archive.org ?

I been trying for hours to download tweets clips that are not longer than 1 minute and they usually at 10 to 15 seconds clips
I tried the both firefox and chrome developer tools and still no avail to find the mp4 file
Quick note: in the console mode it seems to appear blocked:csp following by a mix of random letter and numbers and following with a .m3u8 file extension.
 

BlancoMailo

True & Honest Fan
kiwifarms.net
Anyway to download archived video tweets from archive.org ?

I been trying for hours to download tweets clips that are not longer than 1 minute and they usually at 10 to 15 seconds clips
I tried the both firefox and chrome developer tools and still no avail to find the mp4 file
Quick note: in the console mode it seems to appear blocked:csp following by a mix of random letter and numbers and following with a .m3u8 file extension.
Can you link an example?
 

BlancoMailo

True & Honest Fan
kiwifarms.net
The video plays perfectly but by checking in further examination the only i can be able to find are .ts files which are bit of segments of the video.

again im clueless on how to download the actual full clip
I'm seeing the issue, it's a fucking blob url. They're a massive pain in the ass to deal with if the usual go-tos like ytdl or videodownload helper can't deal with them. Unless someone feels like writing you up something, you're probably just better off screenrecording it.
 

No Shot Bucko

No Shot!
kiwifarms.net
The video plays perfectly but by checking in further examination the only i can be able to find are .ts files which are bit of segments of the video.

again im clueless on how to download the actual full clip
thanks for trying.

also i guess i'm just too silly to forgot to do a reverse search of the deleted tweet to find out another account got the video still posted from the original author so in conclusion i manage to download it at his full glory!
 

Pepper Jack

Pepper Jack are you SERIOUS?!?!?!
True & Honest Fan
kiwifarms.net
U2Convert, which I used to archive on mobile, got Susan'ed and you can't download archives from their website anymore. Any suggestions on an alternative?

F to U2Convert
 

Ellesse_warrior

Plz dox thumb
True & Honest Fan
kiwifarms.net

Respect the Erect

kiwifarms.net
If you guys are too lazy to copy-paste links to archive.md/tweetsave/whatever, use these bookmarklets instead:
archive.md:
Code:
javascript:void(open('https://archive.md/?run=1&url='+encodeURIComponent(document.location)))
tweetsave:
Code:
javascript:void(open('https://tweetsave.com/?tweet='+encodeURIComponent(document.location)))
go1dfish (for reddit stuff):
Code:
javascript:if (window.location.hostname.match(/reddit/)) {window.location = 'https://r.go1dfish.me' + window.location.pathname + window.location.search;}

save the shit in the code tag as bookmark and click them when you open a page you'd like to archive
Wonder if I can combine this with an iOS shortcut. I was debating going the curl route and then realized that’s retarded.

*******

And for everyone else that is fully fucking retarded, telegram has bots to download YouTube videos.

GoZilla will allow you to download up to 720p and also allow you to download just the audio


AnyVideoDownloader does a good job with a lot of things that aren’t YouTube. This includes Periscope, Gab, Parler (lmao imagine using that) and any other webpage that has a video on it.


This one also does YouTube and I use it if others are down/not working


For Instagram


This one saves all pictures/videos in a post


If someone upload something to SoundCloud use this


I think that’s most of the Telegram bots I use for archiving/preserving. I’m on mobile a lot and can’t necessarily use desktop tools without remoting into a desktop.

Grep Tweet saves up to 3200 tweets, I've used it and it's pretty useful.
It still works? Because Twitter keeps blocking access to tools. I can’t get TWINT to work but maybe it’s just me
 

thrusting

iced out
True & Honest Fan
kiwifarms.net
It still works?
Nah I'm pretty sure that service died all the way back in 2016 after an API change. Twitter is cancer incarnate in general. Tried to make a throwaway account so I could easily follow various exceptional people and after 10 minutes I got locked out and asked for a phone number. Need to figure out how to use one of those Russian throwaway number services. If someone knows an easy way for non Cyrillic speakers please let me know.
 

Distended Gunt

kiwifarms.net
The video plays perfectly but by checking in further examination the only i can be able to find are .ts files which are bit of segments of the video.

again im clueless on how to download the actual full clip
Attached the video. The process for downloading these isn't super easy but doable with a bit of detective work and autism on your side.

Here's the steps I took:
  1. Load the page in your browser with the network inspector open and capturing. (F12 -> Network)
  2. Find the m3u8 playlist containing the different quality streams, it should be the first one to load (Filter by Media to make this easier)
  3. Look at the response body / payload for that request, it should contain #EXTM3U and a bunch of m3u8 files with varying qualities. For me the highest quality one was last on the list and has the URL /ext_tw_video/1303975175365558272/pu/pl/912x720/znvQsl9g6trnA11O.m3u8
  4. Take the path for the highest quality .m3u8, prepend "https://web.archive.org/web/20201002052552if_/https://video.twimg.com" to it and download the file
  5. Open that file in Notepad or whatever, prepend "https://web.archive.org/web/20201002052552if_/https://video.twimg.com" to each segment so they look like "https://web.archive.org/web/2020100...272/pu/vid/0/3000/912x720/vhHf_WB7vfYwoEaZ.ts", etc. and save the file
  6. Run "ffmpeg -protocol_whitelist file,http,https,tcp,tls,crypto -i "znvQsl9g6trnA11O.m3u8" -c copy boobies.mp4". ffmpeg will basically grab the segments and assemble them into the .mp4 for you, it's actually way higher quality than the archive.org one as their player is selecting the lowest quality
If you're doing this shit all the time then let me know and I can write a tool to do all of this. The metadata at https://web.archive.org/web/2020100.../videos/tweet/config/1303975362397908992.json is sufficient to automate this process.
 

Attachments

  • video.mp4
    661.6 KB · Views: 0

Respect the Erect

kiwifarms.net
Nah I'm pretty sure that service died all the way back in 2016 after an API change. Twitter is cancer incarnate in general. Tried to make a throwaway account so I could easily follow various exceptional people and after 10 minutes I got locked out and asked for a phone number. Need to figure out how to use one of those Russian throwaway number services. If someone knows an easy way for non Cyrillic speakers please let me know.
Use http://smspva.com/

Or, you can order a 0.99 Mint Mobile SIM card. It comes with a phone number and 7 days unlimited data (if you already have an unlocked phone)

Downside to smspva is if you’re going to use twitter you’ll likely get locked again. So you’ll have to rent a number and keep it for as long as you want to use twitter.
 

Libav Supporter

kiwifarms.net
The archiveweb.page Webrecorder extension will allow you to make local archives of extremely complex sites (Twitter, FB, etc....) and/or sites where you need to login. You can then upload these archived pages to the Webrecorder service, where others can access them.

Archive.st and etched.page are less known archiving sites. Since archive.md keeps putting captchas on their pages, this could come to prove useful.

Archivesort will allow you to archive and download youtube and bitchute videos. The upside of this service is that archived videos are searchable and they keep a copy of all of the videos on their end as well, so you don't need to expend all of your storage space and you can share the archived link with others without having to create an account/upload.
 

Ero

What tha...
kiwifarms.net
I have a question:

So I use youtube.dl and ffmpeg to download specific parts of a long video.
What I do is just use the command:
youtube-dl -f 18 https://www.youtube.com/watch?v=4dTtFBBIq3I --external-downloader ffmpeg --external-downloader-args "-ss 2:01:54 -to 2:02:41"
When I play it in VLC the audio and video are synced however if I upload it to kiwifarms they're no longer synced like in the video below (It's just a random part from IP2 streamer "Reformed Stoic", ignore him being retarded)
Is there any way to fix this?
 

Account

A nondescript anime avatar account
kiwifarms.net
People don't seem to know this but youtube-dl can be used during a livestream to download a livestream from when you start it. This is useful when someone deletes a livestream shortly after it has finished airing but you catch it while it's live. This works with basically any streaming website that youtube-dl works with. youtube-dl will not download the previous content, but on youtube you can rewind the stream (I dunno how far) so if you're really determined you can screen-record the part you missed. A shit archive is better than none at all.

Also, if youtube-dl doesn't work with a certain website, chances are you can dig around the HTML source and search for either "m3u8" or "mp4" links that will give you links to the video/livestream in question. It might be easier to save the HTML to a file on your computer and then use a good text editor like Notepad++ or Vim to search this instead of using the inspect element window in your browser. You can then plug in this direct link to youtube-dl and it should work.

Sometimes the URLs need to be modified commonly because they escape characters like (") or (/). If a URL you get from snooping around the HTML source has something of the form (\uXXXX) or (\Uxxxxxxxx) (there are four characters after "u" and eight after "U") then those are unicode characters. You can look up the replacement by just searching "Unicode character U+XXXX". If the URL has something like (\/) or (\") then you should remove the backslashes, and if you see (\\) you should remove one backslash.

As an example, when Trovo first went public youtube-dl didn't work with it correctly. During a livestream on Trovo, however, the HTML source of the stream's page had the links to the m3u8 files labeled as "PlayUrl" in some embedded JSON. Once you filtered for all entries labeled "PlayUrl" and cleaned up the surrounding text you got a URL that had escaped characters (\u002F corresponding to the forward slash in URLs) that you needed to replace before plugging it into youtube-dl to download.
 

McCarthy

kiwifarms.net
Are there any forums dedicated to people that archive? There's some good discussion about it on KF, but I would love to join a forum that's specifically dedicated to that subject alone.
 

BlancoMailo

True & Honest Fan
kiwifarms.net

stain-maker

kiwifarms.net
i wrote a little python script for archiving youtube livechat
It might suck, work in progress

you just add the youtube watch="Add these letters" at the prompt
example
for youtube url
www.youtube.com/watch?v=NkWUPyKZY34
>> url = NkWUPyKZY34
Rename File or remove the .txt extension , run with python

It creates a .html file in the directory you run the script in with the live chat, named the url hash you enterered at the prompt, also displays a text output in the cli
 

Attachments

  • cpLiveChat.py.txt
    830 bytes · Views: 14
Last edited:

Similar threads

World-class veteran video game critic and Shaggy look-alike
Replies
20
Views
4K
Unemployed Yorkshireman, fighting a war against flat earthers, trolls, masons and feminists, one ranting video at a time
Replies
29
Views
1K
Top