Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

Overview

Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

forthebadge made-with-python

windows python

screenshot

Finding and downloading deleted Tweets takes a lot of time. Thankfully, with this tool, it becomes a piece of cake! ๐ŸŽ‚

Twayback is a portmanteau of Twitter and the Wayback Machine. Enter your desired Twitter username, and let Twayback do the rest!

Requirements

  • Python 3
  • waybackpack
    • Install: pip install waybackpack or pip3 install waybackpack
    • Link to repo

Features

  • Can download some or all of a user's archived deleted Tweets.
  • Allows custom time range to narrow search for deleted Tweets archived between two dates.
  • Differentiates between accounts that are active, suspended, or don't/no longer exist.
  • Lets you know if a target handle's archived Tweets have been excluded from the Wayback Machine.

Usage

twayback -u USERNAME [OPTIONS]
Example: twayback -u jack

-u, --username        Specify target user's Twitter handle
-from, --fromdate     Narrow search for deleted Tweets *archived* on and after this date
                      (can be combined with -to)
                      (format YYMMDD)
-to, --todate         Narrow search for deleted Tweets *archived* on and before this date
                      (can be combined with -from)
                      (format YYMMDD)

Installation

For Windows only

  1. Download the latest EXE file.
  2. Launch Command Prompt in the EXE file's directory.
  3. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For Windows, Linux, and macOS

  1. Download the latest ZIP file.
  2. Extract ZIP file to a directory of your choice.
  3. Open terminal in that directory.
  4. Run the command pip install -r requirements.txt.
  5. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For more information, check out the Usage section above.

Things to keep in mind

  • Quality of the HTML files depends on how the Wayback Machine saved them. Some are better than others.
  • This tool is best for text. You might have some luck with photos. You cannot download videos.
  • By definition, if an account is suspended or no longer exists, all their Tweets would be considered deleted.
  • Custom date range is not about when Tweets were made, but rather when they were archived. For example, a Tweet from 2011 may have been archived today.

Future plans

  • GUI. This is a biggie. I don't know shit about Python, let alone GUI. But I'm hoping I can design one using Tkinter Designer. But I don't know how I can link actions to buttons and shit like that, that stuff is super foreign to me, so any help is appreciated, it would mean so much.

Plenty of thanks to jsvine for his amazing work on waybackpack. Without it, this tool cannot work nearly as well.

I hope you enjoy my little script. Please use it for good. Whatever you are, be a good one.

Comments
  • AttributeError: 'NoneType' object has no attribute 'getText'

    AttributeError: 'NoneType' object has no attribute 'getText'

    I got two errors:

    1. The first error occured after grabbing links from wayback machine UnboundLocalError: local variable 'wayback_id' referenced before assignment

    2. Second, the error occured when I typed 'text' or 'both' AttributeError: 'NoneType' object has no attribute 'getText'

    please help, thanks.

    bug good first issue 
    opened by adrn-mm 9
  • Error in the process

    Error in the process

    Hello,

    Thanks for the app which helps save a lot of time. I have tried with several users and in various ways and it does not work. I pass the screenshots of the errors.

    Thank you very much.

    image image

    opened by barripdmx 8
  • Twayback Partial Re-Write

    Twayback Partial Re-Write

    Hello,

    We recently came across your project, and thought we could contribute by re-writing some parts of the code. An effort was made to keep the logic and structure of the code the same.

    By our metrics, we have achieved a speed-up of around 20-30% for accounts with over 1000 tweets. We don't have any actual tests written for the script however, so we have attached it here for review.

    A substantial change was moving from Selenium to Playwright. Users need to run 'playwright install' to install the playwright browsers before running the script.

    Another smaller, but notable change is that due to the nature of the as_completed function, it is not possible to show any indicator of progress during the gathering of statuses, and the order in which the tweets are returned is scrambled due to the script processing website information based on what web request finishes first.

    If the style & format of the re-write is acceptable, and the changes proposed are not considered to be critical, we will proceed with creating a pull request.

    twayback2.txt

    opened by AccentuSoft 5
  • Parsing fails when encountering text in cyrillic

    Parsing fails when encountering text in cyrillic

    Hi, when I try to search for deleted tweets from an account that uses cyrillic, the process fails with the following exception:

    Parsing text...: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 4/4 [00:07<00:00, 2.00s/it] Traceback (most recent call last): File "twayback.py", line 184, in File "encodings\cp1250.py", line 19, in encode UnicodeEncodeError: 'charmap' codec can't encode character '\u0430' in position 0: character maps to [20476] Failed to execute script 'twayback' due to unhandled exception!

    Anything that could be done on my end? Thanks,

    opened by Traut89now 5
  • KeyError: 'closest' when parsing accounts with larger number of tweets.

    KeyError: 'closest' when parsing accounts with larger number of tweets.

    Hi,

    after the latest update, the tool seems to be working perfectly when scraping accounts with lower number of tweets archived, but when I tried accounts with 1000+ tweets archived, the process failed with: Traceback (most recent call last): File "C:\15tway\twayback B\twayback.py", line 112, in wayback_url = (jsonResponse['archived_snapshots']['closest']['url']) KeyError: 'closest'

    as also seen below on the screenshots.

    image

    opened by Traut89now 4
  • No deleted tweets have been found

    No deleted tweets have been found

    Hi! Does anyone else, after they pull one search of an user, get an "No deleted tweets have been found" when they try to look for another one? I even try with the same one I did first and it pulls the same thing. I even close and open again the terminal but nothing, got the same issue. Was anyone run into this?

    opened by sofiemmc 3
  • Cleaned up code a bit

    Cleaned up code a bit

    No changes to functionality. I went through and added some more comments, renamed variables to be more descriptive and combined some redundancy with list creation

    opened by humandecoded 2
  • Feature proxy

    Feature proxy

    to avoid rate limiting for large groups of tweets added the ability to use proxies with our GET requests. User will need to provide a list of proxy URLs for the script to randomly pull from

    opened by humandecoded 1
  • Twitter Rate Limiting

    Twitter Rate Limiting

    It looks like twitter is throwing up 429 (too many requests) after the first 900 or so hits. According to their API documentation they limit to around 900 hits per 15 minutes. Although we are not using the API, it seems they are putting similar limiting in place per IP.

    I've begun working on a branch that lets users plug in a list of proxies the script will rotate through to avoid 429. This does make the tool less approachable but at this point it's not working on large groups of tweets. Will keep everyone updated

    enhancement 
    opened by humandecoded 1
  • Adding more Detailed information

    Adding more Detailed information

    I am submitting this PR so that newbie cyber people & OSINT Analysts with no python experience can better understand installation and usage.

    • There is no information about using the command git clone in order to clone the repository
    • No information in regards to changing directory | cd
    • No information in regards to the command pip3 install -r requirements.txt
    • No information in regards to the command python3 twayback.py -u username (if we put ourselves in the shoes of anyone without any python knowledge, they will just write in their terminal twayback.py which won't be of any use, it's typically something I would have done in the past as when reading installation & usage instructions I would just copy paste without trying to understand. ๐Ÿคฆ

    Hope this helps a little, awesome tool & thanks for putting it out there !

    opened by C3n7ral051nt4g3ncy 1
  • UPDATE requirements.txt

    UPDATE requirements.txt

    Module aiohttp is required

    Traceback (most recent call last):
      File "/home/gdhindii/twayback/twayback.py", line 18, in <module>
        from aiohttp import ClientSession, TCPConnector
    ModuleNotFoundError: No module named 'aiohttp'
    
    opened by sam5epi0l 1
  • Proxy File List

    Proxy File List

    I am not sure this is an issue or just lack of knowledge from my part, I rather suspect it is the latter, however, I am attempting to recover tweets from an account that has a couple of thousand deleted tweets. I eventually encounter a rate limit error as is to be expected. I saw the way to handle requests like these is to use a proxy file. Now I have a .txt file that I added inside the twayback folder with a list of proxies and formatted according to the guidelines:

    url:port url:port url:port

    and so on. I keep getting error for each proxy server so Im not sure if Im doing this right and what the issue could be. It will stay stuck at 0%, switch between proxies until it eventually just gives up with a "" error.

    My request looks like this:

    python3 twayback.py --proxy-file proxyfiles.txt -u USERNAME

    Am I doing this correctly?

    opened by DirkGaston 2
  • it doesn't work anymore

    it doesn't work anymore

    once I launch the app it closes itself, it was working fine before the last update I tried downloading a pervious version and the same thing happened I tried it on 2 devices and it didn't work on any of them

    opened by unaufhaltbar 4
Releases(10-16-22)
Owner
He/Him. Good-for-nothing GitHubber.
null
A lightweight, dependency-free Python library (and command-line utility) for downloading YouTube Videos.

A lightweight, dependency-free Python library (and command-line utility) for downloading YouTube Videos.

pytube 7.9k Jan 2, 2023
Making the process of downloading youtube videos faster and more convinient.

Easy-YT Making the process of downloading youtube videos faster and more convinient. What can it do? This python script can be used to download youtub

Meynam 39 Nov 15, 2021
This script fully automates of downloading tiktok videos, editing them,compiling them and finally uploading them to youtube.

This script fully automates of downloading tiktok videos, editing them,compiling them and finally uploading them to youtube. If you wanted to create a tiktok video compiilation youtubbe channel this script is here to help you :D

Supriyo Sarkar 32 Dec 16, 2022
ImageScraper is a cross-platform tool for downloading a specified count from xkcd, Astronomy Picture of the Day and Existential Comics

ImageScraper The ImageScraper is a cross-platform tool for downloading a specified count from xkcd, Astronomy Picture of the Day and Existential Comic

1amnobody 1 Jan 25, 2022
A simple Python program which uses youtube-dl for downloading YouTube videos as mp3 files.

yt-mp3 converter This is a simple Python program which uses youtube-dl for downloading YouTube videos as mp3 files. This program is for you if you are

nostalgicnerdpenguin 1 Oct 24, 2021
This is a tool to allow downloading any links from r/Roms Magethread

ILYFPR This is a tool to allow downloading any links from r/Roms Magethread Limitations: It downloads ALL roms of the system you choose. This will be

Erase 4 Nov 4, 2021
A discord bot for downloading youtube video and audio files

disctube disctube is a discord bot for downloading video and audio files from youtube using python pytube. disclaimer i am not the best python program

razor420 3 Feb 3, 2022
Userscript qutebrowser for downloading audio / video from youtube using aria2

Yt-Downloader Userscript qutebrowser for downloading video / audio from youtube using aria2 by hint links. Requirements Rofi youtube-dl aria2 dunst In

Ara 0 Dec 11, 2021
A very fast file streaming bot used for streaming and downloading movies

FileStreamBot GIVE A STAR AND FORK ELSE NO MORE OPENSOURCE A Telegram bot to turn all media and documents files to web link . Report a Bug | Request F

Code X Mania a.k.a Adarsh Goel 190 Jan 4, 2023
Python script for downloading audio from YouTube songs/videos.

Python script for downloading audio from YouTube songs/videos. All you have to do is specify the path to your folder and then type song's/video's name and the sound will be downloaded into your folder.

Mateusz Polis 0 Oct 5, 2022
Utility for downloading works from AO3 (Archive Of Our Own)

ao3d video preview A small graphical utility for batch downloading works from AO3 (Archive Of Our Own) Features Batch downloading works to supported f

flux 24 Dec 9, 2022
A web app for downloading Facebook comments as a csv file

Facebook Comment Downloader A small web app for downloading comments from a public facebook page post. Comment downloading from https://github.com/min

WSDOT 23 Jan 4, 2023
Utility for downloading works from AO3 (Archive Of Our Own)

froyo A small graphical application for batch downloading works from Archive Of Our Own (AO3). Curate a fic repo of your own today :) Features Batch d

flux 24 Dec 9, 2022
Youtube_dl_helper - A hacky python script meant to automate the process of downloading mp3 files from YouTube using youtube-dl library

youtube_dl_helper A helper program meant to automate the process of downloading mp3 files from YouTube using youtube-dl library Dependencies In order

Guilherme Bittencourt de Borba 1 Jan 4, 2022
Gogoanime-dl - Gogoanime downloader for downloading anime.

gogoanime-dl With this script, you can download episodes of your favorite anime from Gogoanime. The current site that's developed against is https://w

null 1 Jan 6, 2022
A Python package for downloading / archiving all available episodes from a podcast RSS feed.

allcasts ?? ?? A Python package for downloading all available episodes from a podcast RSS feed. Useful for making private archives of your favourite p

Lewis Gentle 5 Nov 20, 2022
A simple python script and it's used for mp4 type video downloading from youtube.

This is a simple python script and it's used for mp4 type video downloading from youtube. also, it's used inbuilt python module pytube. Furthermore, I know we have so many apps and online websites to do the same thing so it's just an experiment to study how to do those things in python.

Yousaf K Hamza 1 Jan 10, 2022
Heroic-gogdl - GOG Downloading module for Heroic Games Launcher

heroic-gogdl GOG download module for Heroic Games Launcher Purpose This will tak

Paweล‚ Lidwin 36 Dec 23, 2022
YouPlay is a python based tool for downloading YouTube videos through its URL

YouPlay is a python based tool for downloading YouTube videos through its URL. It is capable to download videos from YouTube playlists too and can extract the audio file only from the video. It can read URLs from files and can download contents as per instruction.

Nitin Choudhury 10 Sep 15, 2022