squid-dl is a massively parallel yt-dlp-based YouTube downloader.

Related tags

Downloader squid-dl
Overview

squid-dl logo

squid-dl

Code style: black License: MIT

squid-dl is a massively parallel yt-dlp-based YouTube downloader.

Installation

Run the setup.py, which will install squid-dl and its two dependencies: linode-cli and yt-dlp.

$ python3 setup.py install

Linode Setup

If you want to use the Linode SOCKS proxy feature, be sure to configure the linode-cli first:

$ linode-cli configure --token

Follow the onscreen instructions and be sure to set sensible defaults. The default region for new Linodes that you pick here is where your proxies will live. For more information, see this page on Linode's website.

Usage

To download a playlist, first fetch its metadata with squidson:

(.venv) $ squidson 'https://www.youtube.com/playlist?list=PLhI42_YpLVHvYqvjniSp7lhAM55K4fllO'
[youtube:tab] PLhI42_YpLVHvYqvjniSp7lhAM55K4fllO: Downloading webpage
[youtube:tab] PLhI42_YpLVHvYqvjniSp7lhAM55K4fllO: Downloading API JSON with unavailable videos
[download] Downloading playlist: Programming
[youtube:tab] playlist Programming: Downloading 12 videos
[download] Downloading video 1 of 12
[youtube] Wp9XD5FKZ2c: Downloading webpage
[youtube] Wp9XD5FKZ2c: Downloading android player API JSON
...
[download] Downloading video 12 of 12
[youtube] CIpfoMKqPAg: Downloading webpage
[youtube] CIpfoMKqPAg: Downloading android player API JSON
[youtube] CIpfoMKqPAg: Downloading MPD manifest
[youtube] CIpfoMKqPAg: Downloading MPD manifest
[download] Finished downloading playlist: Programming
[INFO]: Writing JSON-ified playlist info_dict to "Programming.json"

Now that you have the playlist .json file, you can download it with squid-dl. Use the -n flag to tell squid-dl how many workers you'd like to spawn downloading videos at once (the default is 8 workers). In this 12-video playlist example, we'll spawn 12 workers:

(.venv) $ squid-dl -n 12 Programming.json

[INFO]: Starting squid-dl...
[INFO]: saving videos to "Programming" directory

...
[download] Download completed
[download] Download completed
[download] Download completed
[INFO]: Worker 80214 done...
[INFO]: Worker 80319 done...
[download] Download completed
[INFO]: Worker 80150 done...
[INFO]: Worker 80248 done...
[download] Download completed
[INFO]: Worker 80219 done...
[INFO]: Worker 80109 done...
[INFO]: Worker 80341 done...
[download] Download completed
[INFO]: Worker 80193 done...
[download] Download completed
[INFO]: Worker 80117 done...
[INFO]: All done!

For more information see the built-in help by running squid-dl -h.

Linode Proxying

With the -L option, you can run each worker through its own Linode-powered SSH-tunneled SOCKSv5 proxy! squid-dl will make an temporary SSH key in the current working directory and then get to work spinning up Linodes and downloading your videos:

(.venv) anon@fire-crotch:~/butter/youtube/btr$ squid-dl -L Mem.json

[INFO]: Creating SSH key for Linode proxying...
Generating public/private rsa key pair.
Your identification has been saved in /home/anon/fuse/butter-anon/youtube/btr/proxy
_key
Your public key has been saved in /home/anon/fuse/butter-anon/youtube/btr/proxy_key
.pub
The key fingerprint is:
SHA256:cSns+aK0l7xS+fRSXRdqmc5DXkT5pIQsDV8/Ql526vg anon@fire-crotch
The key's randomart image is:
+---[RSA 3072]----+
|          .+ oo+o|
|       .  .o*.+*o|
|        + o..+*++|
|       . =   Oo.=|
|        S.  B.o..|
|        o.. .*.  |
|      .o.+.o  E  |
|     ..o+.o .    |
|      oo.. .     |
+----[SHA256]-----+

[INFO]: Starting squid-dl...
[INFO]: saving videos to "Mems" directory
[INFO]: Created Linode 30970792.
[INFO]: Waiting for Linodes to come online...........................

[INFO]: Starting proxy on port 1337 with Linode 30970792
Warning: Permanently added '45.79.47.110' (ED25519) to the list of known hosts.
[INFO]: SOCKS validation succeeded on port 1337 from ID 30970792
[youtube] fIdfGtG7Isg: Downloading android player API JSON
[youtube] fIdfGtG7Isg: Downloading iframe API JS
[youtube] fIdfGtG7Isg: Downloading player 03869671
[youtube] fIdfGtG7Isg: Downloading web player API JSON
[youtube] fIdfGtG7Isg: Downloading MPD manifest
[youtube] fIdfGtG7Isg: Downloading MPD manifest
[youtube] fIdfGtG7Isg: Downloading initial data API JSON
[info] fIdfGtG7Isg: Downloading 1 format(s): 22
[info] Downloading video thumbnail ...
[info] Writing video thumbnail to: Plankton goes to an anime convention [fIdfGtG7Is
g].webp
[download] Destination: Plankton goes to an anime convention [fIdfGtG7Isg].mp4
[download] Download completed
[INFO]: Cleaning up worker 83832's exclusive proxy
[CLEANUP]: Deleted Linode 30970792.
[INFO]: Worker 83832 done...
[INFO]: All done!
[CLEANUP]: Deleting Linode proxies...

Bugs

If you encounter any issues running squid-dl, please create an issue on GitHub.

This software has only tested on Linux so far (patches welcome).

Contributing

Submit your patches as GitHub pull requests.

Contact

Feel free to holler at me on #squid-dl on Libera.chat.

Style notes

I am an 80 column stickler. I run all Python code in this repository through black -l 80.

You might also like...
PyQt5 simple files , youtube videos and youtube playlist downloader

PyQt5 simple files , youtube videos and youtube playlist downloader

Youtube list to mp3 - Youtube list to mp3 downloader

Youtube list to mp3 downloader Tiny script to convert a list of youtube videos t

YouTube Downloader is extremely simple program for downloading songs or playlists (in audio or video) from YouTube. Created using Python, PyTube and PySimpleGUI.
YouTube Downloader is extremely simple program for downloading songs or playlists (in audio or video) from YouTube. Created using Python, PyTube and PySimpleGUI.

YouTube Downloader YouTube Downloader is extremely simple program for downloading songs or playlists (in audio or video) from YouTube. Disclaimer It's

Terminal based YouTube player and downloader
Terminal based YouTube player and downloader

termitube NOTE: THIS REPOSITORY IS A FORK OF mps-youtube as mps-youtube has been unmaintained for almost a year now. Features Search and play audio/vi

Python based YouTube video Downloader GUI Application.
Python based YouTube video Downloader GUI Application.

Youtube video Downloader Python based Youtube video Downloader GUI Application. Installation Python Dependencies Import pytube pip install pytube Im

Youtube Downloader Telegram Bot 😉

Youtube Dl bot 😉 Prerequisite ffmpeg install dependencies pip3 install -r requirements.txt Setup Bot - Change configuration config.py File - insta

 📺 YouTube Song Downloader Bot For Telegram 🔮
📺 YouTube Song Downloader Bot For Telegram 🔮

📺 YouTube Song Downloader Bot For Telegram 🔮 Powerd By TamilBots.

Pantheon - The fastest YouTube downloader.
Pantheon - The fastest YouTube downloader.

A Youtube downloader written in Python3, using HTTP requests and an API.

Youtube playlist downloader with full metadata support
Youtube playlist downloader with full metadata support

ytrake GUI tool to embed metadata for albums on Youtube with youtube-dl. Requires youtube-dl v2021.06.06. Post-processing Album metadata: Usage ytrake

Comments
  • [BUG] String errors

    [BUG] String errors

    Strings with escape sequences and slashes tend to error when creating a .json file of a playlist. Thus, if playlist names have something like "blah / blah" in them, .json files are not created.

    Solution: Add a way for letting users specify the json file names themselves.

    opened by KajeArch 1
  • used the proper filename sanitizer

    used the proper filename sanitizer

    As reported by KajeArch on GitHub on issue #3 (https://github.com/swolegoal/squid-dl/issues/3), I was not correctly handling playlists named with characters invalid for a file path. As it turns out, I was using the wrong helper function from yt_dlp.utils.

    I changed all the calls from sanitize_path(), a function made to clean up Windows paths to the more fully-featured sanitize_filename(). If any Windows users notice any bad behavior after this change, please feel free to make a ticket at: https://github.com/swolegoal/squid-dl/issues/new/choose

    opened by swolegoal 0
  • Python error when executing

    Python error when executing

    Hi, I wanted to give squid-dl a try but I'm getting the following error:

    squid-dl -T Brogramming.json
         ⣠⣾⢿⣿⣷⣦
       ⣀⣼⢟⡱⣆⡹⠹⣿⣇            ⢠⣶⣿⠿⣦⡀
      ⠈⠛⠛⠊⠉⠁  ⠹⣿⣧⡀         ⢠⣿⠏⠁⠝⢪⣻⡦
               ⠙⢻⣿⡄       ⣴⣿⠏
                 ⣿⣷      ⢸⣿⠁
                ⢀⣽⣿⣿⣿⣿⣿⣶⣦⣼⠟
       ⣀⡀     ⢀⣴⣿⣿⣿⢿⣿⣿⣿⣿⣿⣿⣷⡀
    ⢀⣴⣿⡿⢿⣷⡄   ⣼⣿⣿⣿⣿ ⠈⠙⠻⣿⣿⣿⣿⣷
    ⣾⣿⠱⠉⠛⣿⣿⣄ ⣀⣿⣿⣿⣿⣿    ⣀⣽⣿⣿⣿  ⣠⣶⣿⣷⣦⡀
    ⠈⠁   ⠙⠿⠿⠛⠛⢻⣿⣿⣿⣿ ⣀⣴⣾⣿⣿⣿⣿⣿⣤⣾⣿⡿⠖⡯⢿⣷
              ⠈⠻⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⡿⠉⠛⠛⠉   ⠹⠟
              ⢀⣴⡾⠟⠿⣿⣿⣿⣿⠿⢿⡍
             ⣀⣾⡟        ⠈⣿⣶⣤⣤⣄⡀
        ⣰⣾⣿⣿⣿⣿⠏          ⠈⠛⠛⢿⣿⣿⣦
        ⠛⠉⠉⠉⠉      _    _   ⠈⠻⣿⣿⣇_ _
     ___ __ _ _  _(_)__| |___ __| | |
    (_-</ _` | || | / _` |___/ _` | |
    /__/\__, |\_,_|_\__,_|   \__,_|_|
           |_|
    
    
    [INFO]: Starting squid-dl...
    [INFO]: saving videos to "Brogramming" directory
    Traceback (most recent call last):
      File "/opt/homebrew/bin/squid-dl", line 4, in <module>
        __import__('pkg_resources').run_script('squid-dl==0.9', 'squid-dl')
      File "/opt/homebrew/lib/python3.9/site-packages/pkg_resources/__init__.py", line 656, in run_script
        self.require(requires)[0].run_script(script_name, ns)
      File "/opt/homebrew/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1460, in run_script
        exec(script_code, namespace, namespace)
      File "/opt/homebrew/lib/python3.9/site-packages/squid_dl-0.9-py3.9.egg/EGG-INFO/scripts/squid-dl", line 31, in <module>
      File "/opt/homebrew/lib/python3.9/site-packages/squid_dl-0.9-py3.9.egg/squid_dl/__init__.py", line 7, in main
      File "/opt/homebrew/lib/python3.9/site-packages/squid_dl-0.9-py3.9.egg/squid_dl/downloader.py", line 468, in main
      File "/opt/homebrew/lib/python3.9/site-packages/squid_dl-0.9-py3.9.egg/squid_dl/downloader.py", line 307, in resume_preprocess
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
        return Popen(process_obj)
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
        super().__init__(process_obj)
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
        prep_data = spawn.get_preparation_data(process_obj._name)
      File "/opt/homebrew/Cellar/[email protected]/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 183, in get_preparation_data
        main_mod_name = getattr(main_module.__spec__, "name", None)
    AttributeError: module '__main__' has no attribute '__spec__'
    

    I saw a few threads about a similar issues but I'm not a Python expert so I haven't dug to deep yet. Have you seen this before?

    opened by lpgauth 0
Owner
tuxlovesyou
Lover of all things Linux and Unix
tuxlovesyou
YouTube-Downloader - YouTube Video Downloader made using python

YouTube-Downloader YouTube Videos Downloder made using python.

Shivam 1 Jan 16, 2022
Python-Youtube-Downloader - An Open Source Python Youtube Downloader

Python-Youtube-Downloader Hello There This Is An Open Source Python Youtube Down

Flex Tools 3 Jun 14, 2022
Youtube Downloader is a simple but highly efficient Youtube Video Downloader, made completly using Python

Youtube Downloader is a simple but highly efficient Youtube Video Downloader, made completly using Python

Arsh 2 Nov 26, 2022
Youtube-downloader-using-Python - Youtube downloader using Python

Youtube-downloader-using-Python Hii guys !! Fancy to see here Welcome! built by

Lakshmi Deepak 2 Jun 9, 2022
A simple GUI video downloader built off of the python module 'yt-dlp'

Simple-Youtube-DL-Gui Supported Operating Systems Windows 7 (x64), Windows 8 (x64), and Windows 10 (x64) How to use Main Gui Extract program from arch

null 12 Dec 30, 2022
Tkinter based YouTube video downloader works on pytube 11.0.2. Can download YouTube videos in 720p(HD), 144p and even only audio.

YouTube-Downloader Tkinter based YouTube video downloader works on pytube 11.0.2. Can download YouTube videos in 720p(HD), 144p and even only audio. G

Manav Grover 2 Dec 27, 2021
Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Youtube video downloader using Django Using Django as a backend along with pytube module to create Youtbue Video Downloader. https://yt-videos-downloa

Suman Raj Khanal 10 Jun 18, 2022
Youtube Downloader is a Graphic User Interface(GUI) that lets users download a Youtube Video or Audio through a URL

Youtube Downloader This Python and Tkinter based GUI allows users to directly download the Best Resolution Videos and Audios from Youtube. Pa-fy Insta

Samarth Kumar 2 Jun 25, 2022
Youtube Downloader by PyTube é uma ferramenta simples com interface gráfica e escrito em python para baixar vídeos e playlists do youtube...

YouTube Downloader by PyTube O que é o YouTube Downloader by PyTube? YouTube Downloader by PyTube é um software simples para baixar vídeos no YouTube

Elizeu Barbosa Abreu 5 Jul 30, 2022
YouTube-Video-Downloader - Download Youtube Videos for free.

YouTube-Video-Downloader Download Youtube Videos for free. Installing Dependencies:- Windows pip install pytube Mac/Linux pip3 install pytube Clonin

Xception Inc. 1 Jan 1, 2022