TickerRain is an open-source web app that stores and analysis Reddit posts in a transparent and semi-interactive manner.

Overview

Tickerrain

TickerRain is an open-source web app that stores and analysis Reddit posts in a transparent and semi-interactive manner.

Overview

A simple webpage will display the sentimental analysis and entities of the last post processed, then it will display DB info and finally three graphs of the most mentioned tickers in Reddit.

Web server

The graphs are updated every 120 seconds and refreshing the page will display the analysis of a new post.

Requirements

Python3 and the following packages:

  • pandas
  • flask
  • redis
  • cairosvg
  • nltk
  • spacy
  • matplotlib
  • asyncpraw
  • cachetools

Other than that you need Cairo, for example, for Ubuntu run apt-get install libpangocairo-1.0-0.

Running

First, make sure you have a Redis DB running.

In the file substoscrap.txt specify what subreddits to analyze.

There are 3 parts, a process to get the submissions and store them in Redis DB, one to process them, and then finally one to run the webserver.

Getting Submissions

Run python news.py with arguments the crendentials for your account reddit API, see more here.

This will start getting posts, comments, and Redditors from Reddit and store them in Redis DB.

Processing Posts

Run python -m spacy download en_core_web_lgto get spacy-model required for processing posts.

Run python process.py, this will connect to the DB and start calculating metrics every 120 seconds, the results will be stored in 3 files, tickers_df_.p.

The metrics computed right now are:

  • Mentions -> Detectes what ticker is being talked about and counts the total mentions of it.
  • Score -> Calculates the log score which takes into account the upvotes and downvotes.
  • Sentiment -> Using Spacy Vader sentimental analysis it aggregates the general sentiment about the ticker.

Flask Web Server

Run python flask_example.py to start the webserver that displays the results, DB infos and the last post being processed. Access it by opening a browser and going to 127.0.0.1:5000

Issues and TODO

Currently, the processing code, using Pandas, needs to be optimized, it needs to use Pandas in a better way. The ticker detection needs to be improved, it emits warnings and misses some.

  • Improve ticker detecting, combining Spacy entities.
  • Optimize Pandas processing.
  • Add more metrics.
  • Improve the design of the Web page.
  • Auto download of tickers.csv from NASDAQ.
Issues
  • Having issues viewing the plots possible bug

    Having issues viewing the plots possible bug

    I'm trying to view the plots but I keep getting an error that the file doesn't exist. I looked at the source it looks like there is a hard coded value

     pf = pd.read_pickle(f'tickers_df_{7}.p')#process.last_processed
    

    just as a guess I think that should be:

    pf = pd.read_pickle(f'tickers_df_{day}.p')
    
    opened by granthoff1107 1
  • Dockerize and minor refactor changes

    Dockerize and minor refactor changes

    Model changes to use the smaller version. Dockerizing Ticker updates etc.

    opened by GrantHoffman 0
  • Tickers initial selection

    Tickers initial selection

    Bug explanation

    There is a small bug when you first filter the tickers by Market Cap. As it stands, you do not remove the nan values from the tickers dataframe.

    Now

    for index, row in tickers_pd.iterrows():
        if(row['Market Cap'] < 300_000_000):
            tickers_pd = tickers_pd.drop(index)
    

    Fix

    tickers_pd = tickers_pd.loc[tickers_pd['Market Cap'] > 300_000_000]
    

    The fix is also faster than the current for-loop.

    opened by PaoloLeonard 0
  • Error running process.py

    Error running process.py

    Process.py throws an error when running calculate_df

    def calculate_df(df):
        data_df = df.filter(['tickers', 'score', 'sentiment'])
    
        tickers_processed = pd.DataFrame(df.tickers.explode().value_counts())
    
        tickers_processed = tickers_processed.rename(columns = {'tickers':'counts'})
    
        tickers_processed['score'] = 0.0
        tickers_processed['sentiment'] = 0.0
    
         for idx, row_tick in enumerate(tickers_processed.iloc):
    

    at line:

    for idx, row_tick in enumerate(tickers_processed.iloc):

    Exception has occurred: NotImplementedError ix is not iterable File "C:\Users\MyUser\Desktop\NLP\trading-bot-base\tickerrain\process.py", line 113, in calculate_df for idx, row_tick in enumerate(tickers_processed.iloc): File "C:\Users\MyUser\Desktop\NLP\trading-bot-base\tickerrain\process.py", line 152, in processed_df return calculate_df(df), calculate_df(df_3), calculate_df(df_1) File "C:\Users\MyUser\Desktop\NLP\trading-bot-base\tickerrain\process.py", line 157, in processing_last last_processed, last_processed_3, last_processed_1 = processed_df() File "C:\Users\MyUser\Desktop\NLP\trading-bot-base\tickerrain\process.py", line 169, in asyncio.run(processing_last())

    opened by granthoff1107 2
  • Add manual instructions for windows

    Add manual instructions for windows

    Create a new App: https://www.reddit.com/prefs/apps

    No need for a redirectUrl: just point to localhost:8080 or something

    Issue Missing redis: Source: https://stackoverflow.com/questions/6476945/how-do-i-run-redis-on-windows

    Download: https://github.com/microsoftarchive/redis/releases/download/win-3.2.100/Redis-x64-3.2.100.zip

    Issue: No Cairo For windows Source: https://weasyprint.readthedocs.io/en/stable/install.html#windows

    install GTK+ libraries:

    Find the proper architecture from the download link Download: https://github.com/tschoonj/GTK-for-Windows-Runtime-Environment-Installer

    opened by granthoff1107 0
  • Dockerizing environment

    Dockerizing environment

    Hello GonVas,

    Nice project. I wrote a Dockerized environment if you are interested in merging it or creating a branch out of it.

    I have attached the patch file.

    patchfile.zip

    opened by entrpn 11
Owner
GonVas
https://www.linkedin.com/in/goncalomoreno/
GonVas
A chain of stores wants a 3-month demand forecast for its 10 different stores and 50 different products.

Demand Forecasting Objective A chain store wants a machine learning project for a 3-month demand forecast for 10 different stores and 50 different pro

null 2 Jan 6, 2022
Pattern Matching for Python 3.7+ in a simple, yet powerful, extensible manner.

Awesome Pattern Matching (apm) for Python pip install awesome-pattern-matching Simple Powerful Extensible Composable Functional Python 3.7+, PyPy3.7+

Julian Fleischer 90 Jan 11, 2022
Headless chatbot that detects spam and posts links to it to chatrooms for quick deletion.

SmokeDetector Headless chatbot that detects spam and posts it to chatrooms. Uses ChatExchange, takes questions from the Stack Exchange realtime tab, a

Charcoal 407 Jan 10, 2022
This program goes thru reddit, finds the most mentioned tickers and uses Vader SentimentIntensityAnalyzer to calculate the ticker compound value.

This program goes thru reddit, finds the most mentioned tickers and uses Vader SentimentIntensityAnalyzer to calculate the ticker compound value.

null 167 Dec 31, 2021
a really simple bot that send you memes from reddit to whatsapp

a really simple bot that send you memes from reddit to whatsapp want to use use it? install the dependencies with pip3 install -r requirements.txt the

pai 10 Nov 28, 2021
A free and open-source chess improvement app that combines the power of Lichess and Anki.

A free and open-source chess improvement app that combines the power of Lichess and Anki. Chessli Project Activity & Issue Tracking PyPI Build & Healt

null 82 Jan 16, 2022
Bionic is Python Framework for crafting beautiful, fast user experiences for web and is free and open source.

Bionic is Python Framework for crafting beautiful, fast user experiences for web and is free and open source. Getting Started This is an example of ho

null 6 Jan 15, 2022
chiarose(XCR) based on chia(XCH) source code fork, open source public chain

chia-rosechain 一个无耻的小活动 | A shameless little event 如果您喜欢这个项目,请点击star 将赠送您520朵玫瑰,可以去 facebook 留下您的(xcr)地址,和github用户名。 If you like this project, please

ddou123 472 Jan 6, 2022
Source-o-grapher is a tool built with the aim to investigate software resilience aspects of Open Source Software (OSS) projects.

Source-o-grapher is a tool built with the aim to investigate software resilience aspects of Open Source Software (OSS) projects.

Aristotle University 2 Oct 10, 2021
TurtleBot Control App - TurtleBot Control App With Python

TURTLEBOT CONTROL APP INDEX: 1. Introduction 2. Environments 2.1. Simulated Envi

Rafanton 3 Jan 17, 2022
🤖🤖 Jarvis is an virtual assistant which can some tasks easy for you like surfing on web opening an app and much more... 🤖🤖

Jarvis ?? ?? Jarvis is an virtual assistant which can some tasks easy for you like surfing on web opening an app and much more... ?? ?? Developer : su

null 1 Nov 8, 2021
Reactjs web app written entirely in python, using transcrypt compiler.

Reactjs web app written entirely in python, using transcrypt compiler.

Dan Shai 22 Dec 11, 2021
A web app for presenting my research in BEM(building energy model) simulation

BEM(building energy model)-SIM-APP The is a web app presenting my research in BEM(building energy model) calibration. You can play around with some pa

null 8 Sep 3, 2021
Web app to find your chance of winning at Texas Hold 'Em

poker_mc Web app to find your chance of winning at Texas Hold 'Em A working version of this project is deployed at poker-mc.ue.r.appspot.com. It's run

Aadith Vittala 7 Sep 15, 2021
Web App for University Project

University Project About I made this web app to finish a project assigned by my teacher. It is written entirely in Python, thanks to streamlit to make

null 10 Dec 30, 2021
Web app for keeping track of buildings in danger of collapsing in the event of an earthquake

Bulina Roșie ???? Un cutremur în București nu este o situație ipotetică. Este o certitudine că acest lucru se va întâmpla. În acest context, la mai bi

Code for Romania 20 Jan 11, 2022
A web app that is written entirely in Python

University Project About I made this web app to finish a project assigned by my teacher. It is written entirely in Python, thanks to streamlit to make

null 10 Dec 30, 2021
Flames Calculater App used to calculate flames status between two names created using python's Flask web framework.

Flames Finder Web App Flames Calculater App used to calculate flames status between two names created using python's Flask web framework. First, App g

Siva Prakash 4 Jan 2, 2022
A docker container (Docker Desktop) for a simple python Web app few unit tested

Short web app using Flask, tested with unittest on making massive requests, responses of the website, containerized

Omar 1 Dec 13, 2021