A downloader for the ISIS service of TU Berlin

Related tags

Downloader isis_dl
Overview

isis_dl

Tests

A downloading utility for the ISIS tool of TU-Berlin.

Version 0.4

Features

  • Downloads all Material from all courses of your ISIS page.
  • Efficient and dynamic checksum computing for a very good file recognition.
  • You can whitelist / blacklist courses with a given course ID.
  • Multithreaded: A fixed number of threads can be selected at start time.
  • Compatibility: This library will run with any python interpreter that is >= 3.8.

Binary functionality:

  • Building of checksums from existing files.
  • Automatic unpacking of archives.

TL;DR

  1. Use this library instead of isia-tub. It provides a superset of features while having improved performance.
  2. Install using pip install isis_dl. For a manual installation clone this repository and pip install .
  3. For a detailed explanation of Command-Line flags please run isisdl -h.
  4. The first time you run the program you will be prompted if you want to save your password. Look at section encryption for more details.

Installation

You will need a working python3.8 interpreter or above. The script will fail for python3.7 as some new python3.8 features are used.

The recommended installation is via pip - a package manager for python. If pip is not yet installed with the python interpreter run

python3 -m ensurepip

to bootstrap pip.

pip (PyPi)

I have uploaded the repository to the Python Package index (PyPi) where one can download it with the command

pip install isis_dl

Note that you are executing arbitrary code as user. Do this at your own risk!

To run the downloader simply type

isisdl

into your favorite shell.

Please note that, if the virtual environment feature is not used, the ~/.local/bin directory must be in the PATH, otherwise the executable isisdl will not be found.

Manual

Please note that you have to be in a virtual environment in order for this to work as the installation fails otherwise.

Steps:

  • Clone this repository
  • cd isis_dl
  • pip install -e .

Developing

If you want to actively contribute to this repository you will want to install the package in editable mode along with the development requirements:

pip install -r requirements_dev.txt

This creates a symlink to the source code in the pip package location. It will be treated as if it was installed there directly.

There is no method of installation without pip - as the source code expects the module isis_dl to be installed as a package.

Benchmarks

For a comparison between isia-tub and isis_dl please see the Benchmark.md file.

Documentation on features

I am planning on moving this part of the documentation to a dedicated doc site.

File recognition

The file recognition is handled in src/isis_dl/backend/checksums.py.

The main idea is to download a small portion of the file and calculate a hash based on that.

As the requests library provides a file stream, one can only download the first n Bytes and calculate the hash. The problem with this idea is that some files have a header, which is permanently changing.

Unfortunately I don't have an idea why this is the case. In order to circumvent this problem the first portion of the file is skipped based on the file type. The lookup table is located in src/isis_dl/share/settings.py - with the variable being checksum_num_bytes.

The format is <extension>: (<#bytes to ignore>, <#bytes to read>).

This means that one can also set the number of bytes to be read for each file type. For files which store a big header ( I'm looking at you .pdf) the number of bytes to be read is quite high. For others e.g. .mp4 it is not.

Note: If the file extension is not found the default entry None is consulted.

Advantages

  • Only download 512 Bytes of every file.
  • Can verify independently of directory structure / filenames.
  • Lookup is O(1) as a HashSet is used as a datastructure.
  • Up to 255 ** 512 unique files can be saved per course using this method.

Disadvantages

  • For every file in every course x Bytes have to be downloaded.
  • Files are bound to a course.

Note that a default value of 64 suffices to

Can store your password securely

The entire encryption is handled by the src/isis_dl/backend/crypt.py.

The encryption is handled via Fernet

Fernet guarantees that a message encrypted using it cannot be manipulated or read without the key. Fernet is an implementation of symmetric (also known as “secret key”) authenticated cryptography.

The key is generated based on a password you enter and then stored securely.

TODO: This is currently untested. Please enter your password manually for the moment.

Hash Settings

Beware: If you change these settings you will not be able to recover an encrypted file without restoring the settings. I would not recommend changing them.

You may select any hashing algorithm which is supported. This is any hashes.HashAlgorithm. You may also change the number of iterations, which will increase / decrease the time it takes to encrypt / decrypt respectively.

A customizable settings file

The file is located at src/isis_dl/share/settings.py. For the most part you will want to keep the default settings, but if they don't fit your needs, you may easily change them.

Download Directory

The default download directory is ~/isis_dl_downloads. As the intended installation is via pip, there is no good "current working directory", so one cannot use that.

What can be done, however, is migrating this directory to e.g. the Desktop/ or Documents/.

Acknowledgements

isia-tub

Consider checking out the gitlab

This was the original inspiration for this library. At the time isia did not offer the functionality of uri-encoding the password which lead me to create this library. I have recently implemented this functionality into isia in order to benchmark and test both solutions.

Comparison

Downloading my entire isis directory took 22m8s with isia. This is in contrast to the 11m16s it took with isis_dl

mCoding

The structure of this project is heavily inspired by the GitHub of mCoding. Consider giving their video about automated testing a shot.

You might also like...
MMDL (Mega Music Downloader) - A tool to easily download music.
MMDL (Mega Music Downloader) - A tool to easily download music.

mmdl - Mega Music Downloader What is mmdl ❓ TLDR: MMDL is a cli app which allows you to quickly and efficiently download one or multiple songs from Yo

apkizer is a mass downloader for android applications for all available versions.

apkizer apkizer collects all available versions of an Android application from apkpure.com Purpose Sometimes mobile applications can be useful to dig

Pantheon - The fastest YouTube downloader.
Pantheon - The fastest YouTube downloader.

A Youtube downloader written in Python3, using HTTP requests and an API.

Terminal based YouTube player and downloader
Terminal based YouTube player and downloader

termitube NOTE: THIS REPOSITORY IS A FORK OF mps-youtube as mps-youtube has been unmaintained for almost a year now. Features Search and play audio/vi

Youtube playlist downloader with full metadata support
Youtube playlist downloader with full metadata support

ytrake GUI tool to embed metadata for albums on Youtube with youtube-dl. Requires youtube-dl v2021.06.06. Post-processing Album metadata: Usage ytrake

Using Youtube downloader is the fast and easy way to download and save any YouTube video.
Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Youtube video downloader using Django Using Django as a backend along with pytube module to create Youtbue Video Downloader. https://yt-videos-downloa

Advance Image Downloader/Extractor (Job) is a Python-Flask web-based app, which will help the user download the any kind of Images at any date and time over the internet. These images will get downloaded as a job and then let user know that the images have been downloaded by sending them a link over an email. A prometheus exporter for torrent downloader like qbittorrent/transmission/deluge
A prometheus exporter for torrent downloader like qbittorrent/transmission/deluge

downloader-exporter A prometheus exporter for qBitorrent/Transmission/Deluge. Get metrics from multiple servers and offers them in a prometheus format

bing image downloader app used to download bulk images for a specific search term created using streamlit and bing_image_downloader python packages
bing image downloader app used to download bulk images for a specific search term created using streamlit and bing_image_downloader python packages

bing image downloader app bing image downloader app is used to download bulk images for a specific search term. bing image downloader app gets the sea

Releases(untagged-44c24ae2f0ed9de798c2)
  • untagged-44c24ae2f0ed9de798c2(Oct 23, 2021)

    Version0.2

    Version 0.2 is out! yay

    Changelog:

    • Changed download mechanism from 1 Executor per course which downloads with args.num_threads to 1 Executor which downloads everything.
    • Faster instantiation of Files
    • All files get randomly shuffled for a better utilization
    • When interrupted, will finish all remaining downloads and then exit. If prompted again will instantly exit.
    • A better status indicator
    • Moved unzipping from auto → command line argument
    • More tests!
    Source code(tar.gz)
    Source code(zip)
Owner
null
Python-Youtube-Downloader - An Open Source Python Youtube Downloader

Python-Youtube-Downloader Hello There This Is An Open Source Python Youtube Down

Flex Tools 3 Jun 14, 2022
Youtube Downloader is a simple but highly efficient Youtube Video Downloader, made completly using Python

Youtube Downloader is a simple but highly efficient Youtube Video Downloader, made completly using Python

Arsh 2 Nov 26, 2022
Youtube-downloader-using-Python - Youtube downloader using Python

Youtube-downloader-using-Python Hii guys !! Fancy to see here Welcome! built by

Lakshmi Deepak 2 Jun 9, 2022
Youtube Downloader Telegram Bot 😉

Youtube Dl bot ?? Prerequisite ffmpeg install dependencies pip3 install -r requirements.txt Setup Bot - Change configuration config.py File - insta

Aryan Vikash 285 Dec 6, 2022
A scriptable music downloader for Qobuz, Tidal, and Deezer

streamrip A scriptable stream downloader for Qobuz, Tidal, and Deezer. Features Downloads tracks, albums, playlists, discographies, and labels from Qo

null 967 Jan 3, 2023
Bulk Downloader for Reddit

saveddit is a bulk media downloader for reddit pip3 install saveddit Setting up authorization Register an application with Reddit Write down your clie

Pranav 136 Jan 3, 2023
A Udemy downloader that can download DRM protected videos and non-DRM protected videos.

Udemy Downloader with DRM support NOTE This program is WIP, the code is provided as-is and i am not held resposible for any legal repercussions result

Puyodead1 468 Dec 29, 2022
Music and video downloader, Made with love by Bryan Herrera

Python-Mp3Mp4-Downloader Music and video downloader, Made with love by Bryan Herrera Requirements CHOCOLATELY windows command If your system does not

ርᚱ1ናተᛰ ᚻህᚥተპᚱ 104 Dec 27, 2022
📺 YouTube Song Downloader Bot For Telegram 🔮

?? YouTube Song Downloader Bot For Telegram ?? Powerd By TamilBots.

Tamil Bots 146 Dec 31, 2022
music downloader written in python. (Uses jiosaavn API)

music downloader written in python. (Uses jiosaavn API)

Rohn Chatterjee 35 Jul 20, 2022