A Simple modular tool to fetch and parse data related to the stock market.

Overview

๐Ÿ’ stonks-o-fetcher

A Simple modular tool to fetch and parse data related to the stock market.

Build Status codecov

Getting started

For the moment the only source is this repository, so to get the program you have to clone it locally.

Requirements

Python >3.6

The program is tested only on a linux environment (WSL 1 and debian) but should technically work on windows too I think.

Installation

After cloning and entering the root of the project

pip install .

If you are not on python 3

python3 -m pip install .

This will make the program available on your system with the command

stonks-cli

First steps

On first startup you'll have to setup your settings, especially the output path.

There is some validation for fields, so if something is missing you'll see it.

You can use the default ~ to point the path to your home folder, so you can set the path to something like ~/stonks/ or whatever you like.

If you define a filename in your path, meaning that it ends with either .csv or .txt that's the filename it will use to output the data, otherwise the filename will be generated automatically from the settings.

โš ๏ธ File checking

There is not check on existing files yet, and that's on purpose, so if you specify a custom file name it will be overwritten at every execution.

It is recommended to not specify a filename and let the program do its thing. It is also strongly suggested to actually change te path to something familiar.

Controls

The important things are esplained in the program itself, and are mostly out of my control due dependecies, but:

  • Menu navigation: arrow keys and vim bindings
  • Confirm a value: Enter
  • Multiselect when available Space - also enter will add the currently highlighted entry
  • Exit from an input field with no defaults: type an empty space then enter
  • With default values you can press enter to confirm it.

โš ๏ธ Saving your settings

Exiting the application with ESC will NOT save your settings. you have to use the main menu option to do so.

CLI and automation

As of version 0.6.0 the only way to use this program is through the interactive cli menus, but i'm planning on adding the option to launch it with arguments to automate the execution of the process, specify all the required paramenters through arguments and handle different settings files to easily automate the execution through multiple settings.

Contributing

A proper documentation will come later, but here's the gist if you want to contribute on new features.

Components

The project is meant to be easily expandable and flexible. There are two main type of components to consider:

  • Source components
    • Fetchers
    • Parsers
  • Writer components

The name are pretty self explanatory I think.

The whole system is already setup to be almost completely automated Each source handler is included in its own module (folder). The module, through the __init__.py has to export some values:

  • Fetcher - your fetcher class, inheriting from FetcherBase
  • Parser - your parser class, inheriting from ParserBase
  • source - string. Unique value identifying the source handled, can be everyhing
  • friendly_name - string. The text that appears on the CLI
  • description - string a brief description of the source. appears in the cli.

Writers are similar, but instead of Fetcher and Parser and source they must have:

  • Writer - your writer class, inheriting from WriterBase
  • output_type - string unique identifier for the class.

The rest of the attributes remain the same.

โ„น๏ธ You can look at the existing modules inside stonks/components to better understand

There's a manager component that is already set up to import all the valid modules from the components/handlers and components/writers folders, so when your module is ready it should work. Loading is done in the cli module, so that the app is actually empty by itself.

For a module to be valid it has to have the required classes and at least the source/output_type

Custom formatting

If you take a look at the existing components description you'll notice some strange formatting.

The CLI has a custom formatter - because i like colored crayons - to ease highlighing important words. Instead of the standard string.format that replaces the values, here we wrap the words into {} to specify formatting.

#  {word:color}
#  {word:style}
#  {word:color|style}
text = 'This {word:blue} is blue!'
# > this word is blue! - with `word` in blue.

Formatting is done through termcolor, so valid values are the ones in their documentation.

As before, check existing modules to better understand.

โš ๏ธ String content For the moment there are a few issues with the default implementation of string.format that catches various character, specifically the . and : that is used as our delimiter, for now Inserting these character in a block to format will cause problems.

As a rule of thumb, if you write your description and when testing the cli the page doesn't load, it means that there's probably something wrong with the text there.

Testing

Testing is done with pytest and coverage.

You can start a full run with

coverage run -m pytest -v && coverare report -m

Or use whatever integration you like - I'm using vscode and its integrations.

There is an utils file with a bunch of function and a decorator class, used mainly as container for the functions. Most of the tests require at least one decorator if they are not testing for failures.

Pull Requests

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. I'm trying to follow git flow specs to some degree, so eventually the PR toward develop please.

โš ๏ธ Please make sure to update tests as appropriate.

License

MIT

Comments
  • add nasdaq historical data for quotes

    add nasdaq historical data for quotes

    Data is available from https://www.nasdaq.com/market-activity/stocks/gme/historical officially, BUT:

    • the filename itself don't look consistent to me with the data available
    • date ranges cannot be predetermined (as far as I can see)

    See this comment for a working update

    enhancement help wanted 
    opened by dinghino 5
  • Reorganize components and remove validation on constants

    Reorganize components and remove validation on constants

    In order to actually allow custom components to be added we need to remove the validation performed on the scraper.settings.constants.

    Those can be kept for internal reference, for now at least, but it would be better to rethink on how that works, maybe even reorganizing the code.

    The validation itself is technically not needed anymore, I think, since every component passes through the manager to be handled, so the names can actually be anything now.

    It would also be a good idea to actually define naming and import schemes to follow.

    Also, to improve code navigation and import we could change the packages structure to something like

    โ”œโ”€โ”€ components/
    โ”‚   โ”‚   # should import, like it does now, `fetchers` and `writers`,
    โ”‚   โ”‚   # at least for now and include `constants`
    โ”‚   โ”œโ”€โ”€ __init__.py
    โ”‚   โ”œโ”€โ”€ component_base.py
    โ”‚   โ”œโ”€โ”€ base_fetcher.py
    โ”‚   โ”œโ”€โ”€ base_parser.py
    โ”‚   โ”œโ”€โ”€ base_parser.py
    โ”‚   โ”‚
    โ”‚   โ”‚   # one folder for each source to handle
    โ”‚   โ”œโ”€โ”€ <source_name>
    โ”‚   โ”‚   โ”‚   # imports classes and constants, especially the source_name
    โ”‚   โ”‚   โ”œโ”€โ”€ __init__.py
    โ”‚   โ”‚   โ”œโ”€โ”€ <source_name>_fetcher.py
    โ”‚   โ”‚   โ”œโ”€โ”€ <source_name>_parser.py
    โ”‚   โ”‚   โ”‚   # local constants and a SOURCE_NAME for matching
    โ”‚   โ”‚   โ””โ”€โ”€ constants.py
    โ”‚   โ”‚
    โ”‚   โ”‚   # one folder for each output type/writer
    โ”‚   โ”œโ”€โ”€ <output_type>/
    โ”‚   โ”‚   โ”œโ”€โ”€ __init__.py
    โ”‚   โ”‚   โ”œโ”€โ”€<writer_name>.py
    โ”‚   โ”‚   โ”‚   # local constants and OUTPUT_TYPE for matching
    โ”‚   โ”‚   โ””โ”€โ”€constants.py
    

    This could allow anybody to create a new set of handlers for a source/output type, like for example

    โ”œโ”€โ”€ <my_source>/
    โ”‚   โ”œโ”€โ”€ __init__.py
    โ”‚   โ”œโ”€โ”€ <my_source>_fetcher.py
    โ”‚   โ”œโ”€โ”€ <my_source>_parser.py
    โ”‚   โ””โ”€โ”€ constants.py
    
    # classes creation
    from scraper.components import Fetcher  # base class
    
    class MyFetcher(Fetcher):
      pass
    
    from scraper.components import manager
    from my_source import MyParser, MyFetcher, SOURCE_NAME
    
    # Add them to the manager so that both the app and the cli can use them
    manager.register_handler(SOURCE_NAME, MyFetcher, MyParser)
    

    since the interface will be the same regardless and matching is now done through the manager, working this way it's simpler to set up new set of components and organize them.

    enhancement 
    opened by dinghino 4
  • No validation on whole settings

    No validation on whole settings

    we are currently missing a method to actually validate that the settings are there.

    on the cli this is currently done through cli.utils.validate_settings but this should be one directly by the Settings object (ideally in App before starting the actual loop) and the errors should be available somewhere to give feedback.

    This is also needed to complete #67 properly, providing the option to change the dates

    bug high priority internals 
    opened by dinghino 2
  • Unit testing & Major Refactor

    Unit testing & Major Refactor

    As per title. Needed because yes.

    I'm planning on using pytest but we'll see.

    to simulate responses for requests we could use responses, that should do the trick.

    enhancement 
    opened by dinghino 2
  • Update readme for develop branch

    Update readme for develop branch

    A bunch of things changed, for contributors, developers wanting to implement this as a library and even for end users. Update the readme before develop->master

    documentation high priority 
    opened by dinghino 1
  • List of available sources

    List of available sources

    Here is an updated (as much as possible) list of available data that can be scraped through the app.

    A note about NASDAQ DATA

    I've already started working on it, before finding out about the pay wall. They have A BUNCH of data, like all the things but due to the paywall for now ALL THINGS RELATED TO NASDAQ are on halt

    Already available

    List of already available sources implemented. feel free to open issues on bugs/changes you wanna see implemented

    • [x] Historical short volume
      • [x] FINRA
        • direct file access (csv)
        • no authentication needed
        • reported daily
    • [x] Historical fail to deliver
      • [x] SEC
        • direct file access (zip file with csv in .txt format)
        • no authentication needed
        • reported twice a month, reports contain daily data

    Working on

    List of approved suggestions that are work in progress

    • [ ] Historical quotes
      • [ ] NASDAQ
        • REST API
        • requires authentication
        • behind pay wall if not requested through browser
      • [ ] YAHOO
        • REST API
        • requires authentication with optional paywall
        • has free tier with 500 requests/month

    Suggested

    List of noted suggestions

    Rejected

    List of suggestions rejected with some reason

    help wanted good first issue source suggestion 
    opened by dinghino 1
  • Documentation

    Documentation

    It may be time to start writing up some proper documentation, at least for the major parts of the package and some example on how to implement new stuff.

    documentation enhancement 
    opened by dinghino 1
  • launch with arguments

    launch with arguments

    implement argparse or something similar to launch directly bypassing the cli. Useful for automated systems and to just relaunch quickly with options

    • [x] Implement core command structure with click
    • [x] create commands to launch with arguments
    • [x] add option to launch the CLI app
    • [x] Should allow specifying a custom path for a settings file, to allow for separate configurations for eventual automations
    enhancement 
    opened by dinghino 1
  • dynamic descriptions on options in cli

    dynamic descriptions on options in cli

    modular components are functioning cli correctly parses the registered modules

    the description obviously is not updated. I was thinking about using the preview command of the menu creator which - while it adds the not needed preview box, might just do the trick perfectly.

    Working on implementing the preview function to read the description from the module and add custom formatting to allow highlighting and other things, which may be nice.

    enhancement 
    opened by dinghino 1
  • Feature/remove constants validation

    Feature/remove constants validation

    Remove all the CONSTANT classes and related validation.

    Validation will be done using components modules and manager singleton instead, so it should be completely dynamic

    • [x] add constant values to each component module ('handlers' and 'writers')
    • [x] remove current validation method and replace with manager's generated lists
      • [x] (optional) Add validation functions to manager so it's more compact and friendly to use
    • [x] pass test
    • [x] implement new handling in cli
    • [x] update tests to pass correctly

    Finally closes #44 when merged..?

    enhancement 
    opened by dinghino 1
  • 23 improve manager

    23 improve manager

    Major changes

    • Added base classes for all components and manager Handlers
    • Added validation on manager components registration
    • Added manager handling for writers
    • Refactored some code in app and tests to handle the changes
    • Moved components registration in cli app for cleaner package code
    opened by dinghino 1
  • specify output path/template in stonks run

    specify output path/template in stonks run

    Give the ability the option to specify the path to output the data and/or the template when running the run command

    • -o | --output for the output path
    • -p | --pattern for the filename pattern generator
    enhancement outputs cli 
    opened by dinghino 0
  • custom output filename template

    custom output filename template

    some things are already set for this to work but the gist is to have the user be able to specify a format using python formatting your {variable} so that it can know in advance what the file name would be.

    options for this could/should be:

    • start date - with optional formatting
    • end date - with optional formatting
    • source
    • tickers

    The template should be set per writer, when possibile, but for now a global template would be ok. The template should be validated when added (i.e. for missing parenthesis or unknown variables requested

    enhancement outputs 
    opened by dinghino 0
  • Internal error handling and logging

    Internal error handling and logging

    Currently there's some kind of validation of settings in place but it's not how it should be ( #70 ). Also there is no proper logging, with levels and everything, to debug what's happening.

    The idea as it comes to mind is to have something like the manager (or even inside the manager!) to register some error messages with some additional info to be used either as level and as some kind of traceback.

    Errors could have a basic shape of { level: <logging.level>, message: <error-message>} so that they can be outputted with the logging module or better yet loguru which looks awesome and easy to use and wouldn't require much fuss.

    enhancement 
    opened by dinghino 0
  • Customized settings for modules

    Customized settings for modules

    The idea is to expand the settings object and give each component/module its own set of settings, while keeping some as global. Reason being that for example if we implement a MYSQL writer there is not need to have a path to output the data but there is need to configure the mysql connection.

    Same goes with sources: some require keys ( see #65 ) that can either be stored in a separate file or in the settings.

    This would require some refactor of the setting class, maybe creating some base class for basic functionalities and a way to pass around the settings data to each component

    enhancement internals 
    opened by dinghino 1
  • Ability to save settings to a different path

    Ability to save settings to a different path

    This comes in tandem with #67 since I've been requested to be able to launch with arguments, specifying a settings file and the ability to saveload the settings to/from a custom path.

    • Give the user the ability to save the settings on a different path
    • Give the user the ability to load the settings from a file
      • arguments
      • from cli, typing the path
      • from cli, through a navigator
    • Give the user the ability to set a default settings path.

    The app itself should already be capable of most of the work since the instance can be created providing a path to use for the settings file, already has a default one (albeit static for now) and both the to_file and from_file methods accept a path to use.

    Also we already have proper handling for missing or wrongly formatted settings file.

    • [x] refactor CLI to allow saving the settings to another path
    • [ ] refactor CLI to change a default settings path
    • [x] refactor Settings to be able to have a safe fallback and handle a default path
    • [ ] (optional?) store the settings path on a simple file in the project folder so that the app knows where to look for it.
    enhancement 
    opened by dinghino 0
Releases(v0.6.0)
  • v0.6.0(May 4, 2021)

    While still in pre-release status and sub v1, this is the first actual working version of the project.

    User side

    Not much changed from the previous release on the end-user side but improved feedback and navigation.

    • Sources and output type settings now offer a description about what they do and how they work
    • User can properly select a folder to output, using any path they like including ~ to point to their $HOME directory
    • Ability to start anew after completing a run without having to relaunch the program

    ##Dev side This is where most of the work has been done. (see README for an introduction)

    • Major restructuring of the project with a meaningful tree
    • Added custom formatting for the cli to allow easy highlight of important text
    • Automatic and modular sources and writers implementation, allowing to easily write the handlers for a source or a new output class (writer)
    • started working on some kind of documentation/commenting on what the functions do, but will improve later on
    Source code(tar.gz)
    Source code(zip)
  • v0.5.2-alpha(Apr 30, 2021)

  • v0.5.1-alpha(Apr 30, 2021)

Owner
Daniele
Daniele
A stock information collector and parser for Taiwan and US market. Automatically send LINE message if the pre-defined rules are triggered.

agastock ้–‹็™ผๅ‹•ๆฉŸ ๅฐฑๅœจๆตท้‹้ฃ†ๆผฒ็š„2021ๅนด7ๆœˆ๏ผŒๅทฎ้ปž่ทชๅœจๅœฐไธŠๅ–œ่ฟŽ่ฒกๅฏŒ่‡ช็”ฑ็š„็•ถไธ‹๏ผŒEPS่ถ…้ซ˜ๅฅฝๆถˆๆฏไธๆ–ท็š„้•ทๆฆฎ็ซŸ็„ถๅฅ—ๅœจ202ๅ…ƒไธ€ๅŽปไธๅ›ž๏ผŒๆœ‰ๅœ–ๆœ‰็œŸ็›ธ(ๅ“ญ) ๅฟฝ็„ถ้ซ”ๆœƒๅˆฐ่ฟฝ้ซ˜ๆฎบไฝŽไธๆ˜ฏ่พฆๆณ•๏ผŒ้ญฏ่›‡ๆˆ‘ๅพ—้ ้‚่ผฏๅˆ†ๆžไนŸ่ƒฝๅ‡บ้ ญๅคฉ๏ผŒ็ถ“้Žไธ‰ๅ€‹ๆœˆ็„กๆ•ธๅ€‹ไธๅ‡บ้–€็š„ๅ‘จๆœซ๏ผŒ็”ขๅ‡บ็ฐกๅ–ฎ็š„็ˆฌ่Ÿฒๅ’Œๅˆ†ๆžๅทฅๅ…ทใ€‚ ไธŠ้Ž้‡‘่ž็ ”่จ“้™ข็š„้‡ๅŒ–ไบคๆ˜“

Gavin Lee 12 Nov 16, 2022
Stock market bot that will be used to learn about API calls and database connections.

Stock market bot that will be used to learn about API calls and database connections.

null 1 Dec 24, 2021
Fetch Flipkart product details including name, price, MRP and Stock details in general as well as specific to a pincode

Fetch Flipkart product details including name, price, MRP and Stock details in general as well as specific to a pincode

Vishal Das 6 Jul 11, 2022
See trending stock tickers on Reddit and check Stock perfomance

See trending stock tickers on Reddit and check Stock perfomance

Abbas 1.5k Jan 6, 2023
:snake: A simple library to fetch data from the iTunes Store API made for Python >= 3.5

itunespy itunespy is a simple library to fetch data from the iTunes Store API made for Python 3.5 and beyond. Important: Since version 1.6 itunespy no

Fran Gonzรกlez 56 Dec 22, 2022
A simple Discord bot that can fetch definitions and post them in chat.

A simple Discord bot that can fetch definitions and post them in chat. If you are connected to a voice channel, the bot will also read out the definition to you.

Tycho Bellers 4 Sep 29, 2022
A python to scratch API connector. Can fetch data from the API and send it back in cloud variables.

Scratch2py Scratch2py or S2py is a easy to use, versatile tool to communicate with the Scratch API Based of scratchclient by Raihan142857 Installation

null 20 Jun 18, 2022
A combination between python-flask, that fetch and send data from league client during champion select thanks to LCU

A combination between python-flask, that fetch data and send from league client during champion select thanks to LCU and compare picked champs to the gamesDataBase that we need to collect using my other python script and then send the games result to localhost:5000/members that will be read by electron-reactJS script to present the results as a GUI on browser (localhost:5000)

Anas Hamrouni 1 Jan 19, 2022
Fetch fund data from avanza.se using Python and some web scraping with bs4

Py(A)vanza Fetch fund data from avanza.se using Python and some web scraping with bs4. The default way is to display the data in the terminal, apply -

dunderrrrrr 1 Jan 27, 2022
Maubot azuracast - A maubot to fetch data from your radio station

Maubot Azuracast A maubot to fetch data from your radio station Setup Configure

null 3 Mar 14, 2022
Market calendar RESTful API with holiday, late open, and early close. Over 50+ unique exchange calendars for global equity and futures markets.

Trading Calendar Market calendar RESTful API with holiday, late open, and early close. Over 50+ unique exchange calendars for global equity and future

Apptastic Software 1 Feb 3, 2022
Picot - A discord bot made to fetch images from Pexels and unsplash API and provide raw images directly in channels

Picot A discord bot made to fetch images from Pexels and unsplash API and provid

Ayush Chandwani 5 Jan 12, 2022
Tickergram is a Telegram bot to look up quotes, charts, general market sentiment and more.

Tickergram is a Telegram bot to look up quotes, charts, general market sentiment and more.

Alberto Ortega 25 Nov 26, 2022
Parse 11.000 free proxies!

Proxy Machine Description I did this project in order to boost views with the teleboost โœˆ๏ธ in my Telegram channel. You can use it not only for boostin

VLDSLV 77 Jan 8, 2023
Parse discord tokens from any file, even if there is other shit in the file with them.

Discord-Token-Parser Parse discord tokens from any file, even if there is other shit in the file with them. Any. File. I glued together all html from

null 4 May 7, 2022
The Bot provide Hadith API and fetch content via api.hadith.sutanlab.id

Bot Hadith-API on Telegram The Bot provide Hadith API and fetch content via api.hadith.sutanlab.id Built With Python Asynchronous HTTP protocol client

xMan 12 Feb 19, 2022
A python library built on the API of the coderHub.sa, which helps you to fetch the challenges and more

coderHub A python library built on the API of the coderHub.sa, which helps you to fetch the challenges and more Installation โ€ข Features โ€ข Usage โ€ข Lice

TheAwiteb 5 Nov 4, 2022
Simple stock price analytics

mune ยท Mune is an open source python web application built to analyze stocks, named after Homma Munehisa. Currently, the forecasting component is powe

Richard Hong 14 Aug 30, 2021