Fully automated download and parsing for Texas A&M University's Registrar's grade distribution PDFs for years 2014+.

Related tags

Downloader TAMU-GradeDistribution-ParserV2
Overview

TAMU-GradeDistribution-ParserV2

This project is dedicated to helping analyze the massive amounts of data released every semester by Texas A&M University's Registrar's office.

Features:

  • Parse grade report PDFs published by Texas A&M University's Registrar:
    • Automatically adds parsed course data to a mySQL database.

Version 2.0 Roadmap:

  • Grade report parsing
  • Automatically add data to a mySQL database backend
  • Full rewrite with multithreading
  • Fully automated grade report updates (auto add new reports)

How to use:

  1. Set up mySQL table: 
    # create mySQL database table
$ sudo mysql
mysql> use database_name_here;
mysql> CREATE TABLE tamugrades (
           year SMALLINT(4),
           semester VARCHAR(6),
           college VARCHAR(7),
           departmentName VARCHAR(5),
           course VARCHAR(4),
           section VARCHAR(3),
           honors TINYINT(1),
           avgGPA FLOAT(4,3),
           professorName VARCHAR(30),
           numA SMALLINT(3),
           numB SMALLINT(3),
           numC SMALLINT(3),
           numD SMALLINT(3),
           numF SMALLINT(3),
           numI SMALLINT(3),
           numS SMALLINT(3),
           numU SMALLINT(3),
           numQ SMALLINT(3),
           numX SMALLINT(3)
       );
  2. Install dependencies: 
    # automatically install python dependencies
$ python3 -m pip install -r requirements.txt
  3. Generate prefs.json and update file: 
    # run in TAMU-GradeDistribution-ParserV2
$ python3 src/gd_main.py
$ nano prefs.json
  4. Run main python script: 
    # NOTE: building the database will take a while
$ python3 src/gd_main.py
  5. Monitor created logfile (optional): 
    # automatically get and display newest logfile
$ cd logs
$ tail -f $(ls -t | head -1)
You might also like...
Command-line program to download videos from YouTube.com and other video sites

youtube-dl - download videos from youtube.com or other video platforms

youtube-dl 116.4k Jan 7, 2023
The free and open-source Download Manager written in pure Python
The free and open-source Download Manager written in pure Python

The free and open-source Download Manager written in pure Python

pyLoad 2.7k Dec 31, 2022
A Udemy downloader that can download DRM protected videos and non-DRM protected videos.
A Udemy downloader that can download DRM protected videos and non-DRM protected videos.

Udemy Downloader with DRM support NOTE This program is WIP, the code is provided as-is and i am not held resposible for any legal repercussions result

Puyodead1 468 Dec 29, 2022
I sure love the mix of newsboat+mpv+youtube-dl to watch videos from my favourite creators directly from my command line. But sometimes I want to download them beforehand and have them sorted into different folders. Here is the script to do exactly that.

newsboat_video_downloader I sure love the mix of newsboat+mpv+youtube-dl to watch videos from my favourite creators directly from my command line. But

Jocomol 16 Dec 12, 2022
Scripts to download files and folders programmatically from Google Drive

Google Drive Downloader Scripts Every time I need to download a lot of files from Google Drive (e.g. a dataset), it's always incredibly frustrating an

Ivan Evtimov 6 Jul 22, 2021
Python script designed to search and fetch direct download links from nxbrew.com
Python script designed to search and fetch direct download links from nxbrew.com

SwitchGamesDownloader Only for windows nxbrew.com is a website, accessible only using a proxy, where the majority of games for the Nintendo Switch are

Backend 91 Dec 28, 2022
Download history data from binance and save to dataframe or csv file

Binance history data downloader Download history data from binance and save to dataframe or csv file

null 10 Dec 2, 2022
Download Photo and Video from Wall of specific user or community

vkontakte-downloader Download Photo and Video from Wall of specific User or Community on https://vk.com Setup Clone the project git clone https://gith

null 4 Jul 20, 2022
Download all your URI Online Judge source codes and upload to GitHub with simple steps.
Download all your URI Online Judge source codes and upload to GitHub with simple steps.

URI-Code-Downloader Download all your URI Online Judge source codes and upload to GitHub with simple steps. Prerequisites Python 3.x Installing Downlo

Luan Simões 9 Mar 23, 2022
Comments
  • Bump cryptography from 37.0.0 to 38.0.3

    Bump cryptography from 37.0.0 to 38.0.3

    Bumps cryptography from 37.0.0 to 38.0.3.

    Changelog

    Sourced from cryptography's changelog.

    38.0.3 - 2022-11-01

    
* Updated Windows, macOS, and Linux wheels to be compiled with OpenSSL 3.0.7,
  which resolves *CVE-2022-3602* and *CVE-2022-3786*.

    .. _v38-0-2:
    

    38.0.2 - 2022-10-11 (YANKED)

    .. attention::

    This release was subsequently yanked from PyPI due to a regression in OpenSSL.
    • Updated Windows, macOS, and Linux wheels to be compiled with OpenSSL 3.0.6.

    .. _v38-0-1:

    38.0.1 - 2022-09-07

    
* Fixed parsing TLVs in ASN.1 with length greater than 65535 bytes (typically
  seen in large CRLs).

    .. _v38-0-0:
    

    38.0.0 - 2022-09-06

    • Final deprecation of OpenSSL 1.1.0. The next release of cryptography will drop support.
    • We no longer ship manylinux2010 wheels. Users should upgrade to the latest pip to ensure this doesn't cause issues downloading wheels on their platform. We now ship manylinux_2_28 wheels for users on new enough platforms.
    • Updated the minimum supported Rust version (MSRV) to 1.48.0, from 1.41.0. Users with the latest pip will typically get a wheel and not need Rust installed, but check :doc:/installation for documentation on installing a newer rustc if required.
    • :meth:~cryptography.fernet.Fernet.decrypt and related methods now accept both str and bytes tokens.
    • Parsing CertificateSigningRequest restores the behavior of enforcing that the Extension critical field must be correctly encoded DER. See the issue <https://github.com/pyca/cryptography/issues/6368>_ for complete details.
    • Added two new OpenSSL functions to the bindings to support an upcoming pyOpenSSL release.
    • When parsing :class:~cryptography.x509.CertificateRevocationList and

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump pypdf2 from 1.26.0 to 1.27.5

    Bump pypdf2 from 1.26.0 to 1.27.5

    Bumps pypdf2 from 1.26.0 to 1.27.5.

    Release notes

    Sourced from pypdf2's releases.

    Version 1.27.5, 2022-04-15

    Security (SEC)

    • ContentStream_readInlineImage had potential infinite loop (#740)

    Bug fixes (BUG)

    • Fix merging encrypted files (#757)
    • CCITTFaxDecode decodeParms can be an ArrayObject (#756)

    Robustness improvements (ROBUST)

    • title sometimes None (#744)

    Documentation (DOC)

    • Adjust short description of the package

    Tests and Test setup (TST)

    • Rewrite JS tests from unittest to pytest (#746)
    • Increase Test coverage, mainly with filters (#756)
    • Add test for inline images (#758)

    Developer Experience Improvements (DEV)

    • Remove unused Travis-CI configuration (#747)
    • Show code coverage (#754, #755)
    • Add mutmut (#760)

    Miscellaneous

    • STY: Closing file handles, explicit exports, ... (#743)

    All changes: https://github.com/py-pdf/PyPDF2/compare/1.27.4...1.27.5

    Version 1.27.0

    Features

    • Add alpha channel support for png files in Script (#614)

    Bug fixes (BUG)

    • Fix formatWarning for filename without slash (#612)
    • Add whitespace between words for extractText() (#569, #334)
    • "invalid escape sequence" SyntaxError (#522)
    • Avoid error when printing warning in pythonw (#486)
    • Stream operations can be List or Dict (#665)

    Documentation (DOC)

    ... (truncated)

    Changelog

    Sourced from pypdf2's changelog.

    Version 1.27.5, 2022-04-15

    Security (SEC):

    • ContentStream_readInlineImage had potential infinite loop (#740)

    Bug fixes (BUG):

    • Fix merging encrypted files (#757)
    • CCITTFaxDecode decodeParms can be an ArrayObject (#756)

    Robustness improvements (ROBUST):

    • title sometimes None (#744)

    Documentation (DOC):

    • Adjust short description of the package

    Tests and Test setup (TST):

    • Rewrite JS tests from unittest to pytest (#746)
    • Increase Test coverage, mainly with filters (#756)
    • Add test for inline images (#758)

    Developer Experience Improvements (DEV):

    • Remove unused Travis-CI configuration (#747)
    • Show code coverage (#754, #755)
    • Add mutmut (#760)

    Miscellaneous:

    • STY: Closing file handles, explicit exports, ... (#743)

    All changes: https://github.com/py-pdf/PyPDF2/compare/1.27.4...1.27.5

    Version 1.27.4, 2022-04-12

    Bug fixes (BUG):

    • Guard formatting of init.doc string (#738)

    Packaging (PKG):

    • Add more precise license field to setup (#733)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
Owner
TAMU Grade Distribution
This project is dedicated to helping analyze the massive amounts of data released every semester by Texas A&M University's Registrar's office.
TAMU Grade Distribution
GitHub
Fully Automated YouTube Channel ▶️with Added Extra Features.

Fully Automated Youtube Channel ▒█▀▀█ █▀▀█ ▀▀█▀▀ ▀▀█▀▀ █░░█ █▀▀▄ █▀▀ █▀▀█ ▒█▀▀▄ █░░█ ░░█░░ ░▒█░░ █░░█ █▀▀▄ █▀▀ █▄▄▀ ▒█▄▄█ ▀▀▀▀ ░░▀░░ ░▒█░░ ░▀▀▀ ▀▀▀░

sam-sepiol 249 Jan 2, 2023
SABnzbd - The automated Usenet download tool

SABnzbd is an Open Source Binary Newsreader written in Python.

SABnzbd 1.8k Dec 30, 2022
Python Program that downloads gaming required packages based on your Linux Distribution.

LibreGaming Python Program that downloads gaming required packages based on your Linux Distribution. Table of contents Distributions Prerequisites Dep

Ahmed Al Balochi 195 Jan 1, 2023
This script fully automates of downloading tiktok videos, editing them,compiling them and finally uploading them to youtube.

This script fully automates of downloading tiktok videos, editing them,compiling them and finally uploading them to youtube. If you wanted to create a tiktok video compiilation youtubbe channel this script is here to help you :D

Supriyo Sarkar 32 Dec 16, 2022
Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

DeltaLaboratory 2 Nov 16, 2022
FireDM is a python open source (Internet Download Manager) with multi-connections, high speed engine, it downloads general files and videos from youtube and tons of other streaming websites .

python open source (Internet Download Manager) with multi-connections, high speed engine, based on python, LibCurl, and youtube_dl https://github.com/firedm/FireDM

null 1.6k Apr 12, 2022
Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Youtube video downloader using Django Using Django as a backend along with pytube module to create Youtbue Video Downloader. https://yt-videos-downloa

Suman Raj Khanal 10 Jun 18, 2022
Advance Image Downloader/Extractor (Job) is a Python-Flask web-based app, which will help the user download the any kind of Images at any date and time over the internet. These images will get downloaded as a job and then let user know that the images have been downloaded by sending them a link over an email.

Advance Image Downloader/Extractor(Job) Advance Image Downloader/Extractor (Job) is a Python-Flask web-based app, which will help the user download th

Shreyas 13 Aug 27, 2022
Download and save Bing wallpapers and set as background for GNOME desktop

Save Bing wallpapers and set as background for GNOME desktop This script downloads the Bing wallpaper and sets it in the background of your gnome desk

manikamran 2 Nov 6, 2021
Easy automated ebook downloader using openbooks as the backend

Easy automated ebook downloader using openbooks as the backend

null 27 Nov 6, 2022