An attempt to map the areas with active conflict in Ukraine using open source twitter data.

Overview

Contributors Forks Stargazers Issues LinkedIn


Logo

Live Action Map (LAM)

An attempt to use open source data on Twitter to map areas with active conflict. Right now it is used for the Ukraine-Russia conflict, but in the future I hope it can be used for all sorts of dangerous situations.
Report Bug · Add Feature · Website Live! · Join Discord!

Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Roadmap
  5. Contributing
  6. License

About The Project

There are many twitter accounts posting live tweets about locations with conflicts. However, it is difficult to keep track of the locations especially with multiple different sources pointing out different location every few minutes. To make sure people can stay safe and take care of themselves, I have aggregated all the tweets into a single map that is easily accessible.

This project is a work in progress. I am working on adding more features and improving the map.

Website Link Image

How it works:

  • Tweets are sourced using keywords, hashtags and prepositions, such as the phrase "shooting... near ... location".
  • Tweets can also be sourced from known twitter accounts by passing their usernames.
  • Tweets are parsed with NLP and the location is extracted from the tweet, this however is not perfect so we need to filter locations later on.
  • Some tweets might talk about other countries reactions like "The US.." or "Russia.." or "Moscow..", in that case we remove all the locations that are not in Ukraine.
  • Some tweets might talk about multiple locations like "Shooting near the location and the location". In that case both locations are added to the map. Multiple markers can be added to the same location.
  • Finally we add markers for each tweet.
  • Markers will cluster together when you zoom out.
  • A single marker looks like a red pin on a map.
  • A cluster appears as a circle with a number inside it, the color shifts from green to orange to red depending on the number of markers in the cluster.
  • We are not taking data directly because that may be vulnerable to trolling and spamming.
  • We are using the Twitter v2 API to get the tweets, however it does not support parsing location directly from tweets.

(back to top)

Getting Started

To get a local copy up and running follow these simple example steps.

Prerequisites

  • Python
  • tweepy
  • spaCy
  • folium
  • geopy
  • tqdm
  • geography3 (optional, needed for experimental feature)

Installation

Python

  1. Get a free twitter Bearer Token from developer.twitter.com. Remember to create a new app and get the bearer token.
  2. Clone the repo
    git clone https://github.com/kinshukdua/LiveActionMap.git
  3. Install all prerequisites
    pip install -r requirements.txt
  4. Download en_core_web, for more info see --> explosion/spaCy#4577
     python3 -m spacy download en_core_web_sm
  5. Create a .env file based on the .env.example
    cp .env.example .env
  6. Set the Twitter bearer token to your own in the .env file created in the previous step.

Docker

  1. Get a Twitter Bearer Token
  2. Download the docker-compose.yaml-file
    wget https://raw.githubusercontent.com/kinshukdua/LiveActionMap/main/docker/docker-compose.yaml
  3. Create a .env file based on the .env.example
    wget https://raw.githubusercontent.com/kinshukdua/LiveActionMap/main/.env.example -O .env 
  4. Start the stack
    docker-compose up -d
    

(back to top)

Usage

Simply edit hashtags, prepositions and keywords and run scrape.py.

python scrape.py

(back to top)

Roadmap

  • Add tweet scraping
  • Add map
  • Add map clustering
  • Create a server to host the generated map
  • Add better filtering
  • Add tweet link on map
  • Use NLP to indicate danger level
  • Add misinformation prevention algorithm
  • Multi-language Support
    • Ukranian
    • Russian

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project
  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Comments
  • Add CI/CD and Tests

    Add CI/CD and Tests

    As the project grows, we need a way to make sure it is easy to test incoming PRs instead of me manually pulling and testing everything. The best way would be to setup a CI/CD pipeline using Github Actions. At the present moment we don't have any tests except the one introduced in #21. Ideally I would like to setup tests and then a CI/CD pipeline. However, the server running the website right now, is manually using cron jobs. I would instead like to wait for #16 to finish and then create a pipeline with the docker image.

    Meanwhile, feel free to add tests and help me write a CI pipeline.

    enhancement 
    opened by kinshukdua 14
  • UI Changes and wider search

    UI Changes and wider search

    Ok, so I wish to contribute to this project. I made some minor changes, here they are:

    1. Cap tweet preview for the folium marker preview to 100 letters.
    2. Add one more hashtag for tweepy to search (#OSINT).
    3. Adjust default coordinates and zoom.
    4. Change tileset to something more fitting (in my opinion).
    5. Change User Agent which geopy uses for Nominatim.

    Please inform me of anything wrong I did (this is my first pr).

    opened by sahal-mulki 9
  • fix ci, use curl to trigger gitlab deploy

    fix ci, use curl to trigger gitlab deploy

    instead of using a third-party GitHub action.

    #52 I've used just the token secret, the gitlab ref name is hardcoded to master.

    I haven't tested it, as I'd need the secret value and gitlab access.

    opened by edubacco 4
  • Implement a ui

    Implement a ui

    I think it would be a pretty good idea to implement a ui for this instead of just generating a PNG.

    If needed i could spare some time in Vue. By all means i have a free weekend right now.

    opened by DomiiBunn 4
  • Add blacklist twitter userIDs

    Add blacklist twitter userIDs

    Fix #5

    Excluding accounts that spreads misinformation

    This pullrequest added:

    • [x] Changes in README (adding point 4)

    image

    • [x] Adds new env variable to handle blacklisted userIDs
    • [x] Add test folder and some test coverage of the implemented feature
    opened by GabrieleCalarota 3
  • Catch nomatim api error/timeout

    Catch nomatim api error/timeout

    Don't exit the whole application if fetching geolocation data fails. It will look like this:

    Adding markers... (This may take a while)
     10%|█████████████████▌                                       | 50/512 [01:12<14:03,  1.83s/it] 
    Getting geolocation data for the South China Sea failed
     11%|███████████████████                                      | 55/512 [01:26<19:59,  2.63s/it] 
    
    opened by laurin 2
  • Domain name

    Domain name

    I was trying to get a domain on freenom, but it doesn't let me register, for the time being I'll deploy it on my custom domain but if anyone has a spare domain lying around or if they can register a free "liveactionmap" domain on freenom that would be great.

    opened by kinshukdua 2
  • Remove temporary tweets file after every x minutes.

    Remove temporary tweets file after every x minutes.

    Since we're no longer dependent on the cronjob, we need to remove the tweets in the python code itself. However deleting the file after each run as done in an earlier PR is not useful because there's too few data points, we can try to increase the number of results we get from Twitter but that's a separate issue.

    Also let's make sure the schedule time for the scraping and removing tweets don't overlap (they are not multiples) otherwise the tweets file might be deleted while the bot is running outputting an empty map.

    opened by kinshukdua 1
  • Backend API

    Backend API

    Moving to a dynamic backend should be it's own issue

    Originally posted by @Krishna-Sivakumar in https://github.com/kinshukdua/LiveActionMap/issues/24#issuecomment-1053660218

    enhancement 
    opened by DomiiBunn 1
  • Self-host a Nominatim Service

    Self-host a Nominatim Service

    Self-hosting a Nominatim service will help us cut down on time wasted doing API calls. Map building takes the most time. And right now, most of the time while building the map is spent on doing 2 API calls to a public Nominatim service per tweet, which adds a lot of latency.

    Since we already have a docker build, it doesn't hurt to add one or two more containers to our deployment.

    Links to refer: https://github.com/mediagis/nominatim-docker https://nominatim.org/release-docs/latest/admin/Installation/

    This will help us shift to a dynamic backend in the long run.

    enhancement 
    opened by Krishna-Sivakumar 1
  • API: Zones

    API: Zones

    Create a GET API serving /api/zonesproviding a response containing all currently available markers.

    Expected response

    [
        {
            "positions": [
                [
                    50.36429316995319,
                    30.228621662109
                ],
                [
                    50.51303377189189,
                    30.28741424805162
                ],
                [
                    50.60122461757218,
                    30.787151228563896
                ],
                [
                    50.44548234821721,
                    30.77081995469095
                ]
            ],
            "color": "green",
            "title": "Safe Zone",
            "content": "Some safe zone over here"
        }
    ]
    
    enhancement 
    opened by DomiiBunn 0
  • Add a .onion link

    Add a .onion link

    This is so people can access the site using tor, might make some people feel safer.

    This would need a separate frontend that doesn't contain any tracking and no links to anything but out service

    enhancement 
    opened by DomiiBunn 5
  • API: Markers

    API: Markers

    Create a GET API serving /api/markers providing a response containing all currently available markers.

    Expected response

    {
        "markers": [
            {
                "position": [
                    49.038230248475905,
                    31.450182690663947
                ],
                "title": "Twit Title with image",
                "content": "",
                "user": "Twitter User",
                "uri": "https://twitter/postUrl",
                "image": "imageURL",
                "timestamp": "unixtimestamp of tweet"
            },
            {
                "position": [
                    48.038230248475905,
                    31.450182690663947
                ],
                "title": "Twit Title without image",
                "content": "Twit Text",
                "user": "Twitter User",
                "uri": "https://twitter/postUrl",
                "timestamp": "unixtimestamp of tweet"
            }
        ],
        "timestamp": "unixTimeStamp"
    }
    
    enhancement 
    opened by DomiiBunn 2
  • Analitics

    Analitics

    It would be neat to see how many hits the site gets.

    I'd recommend either google or unami(this is more gdpr complaint and can be self hosted on Heroku for free)

    enhancement 
    opened by DomiiBunn 2
  • Dynamic Fronted

    Dynamic Fronted

    How about I make a quick mockup with vuejs that would just read an api. That way once the project is evolved enough it would be enough to just hook one and the other together.

    Originally posted by @Krishna-Sivakumar in https://github.com/kinshukdua/LiveActionMap/issues/24#issuecomment-1053660218

    opened by DomiiBunn 3
Twitter-NLP-Analysis - Twitter Natural Language Processing Analysis

Twitter-NLP-Analysis Business Problem I got last @turk_politika 3000 tweets with

Çağrı Karadeniz 7 Mar 12, 2022
Twitter bot that uses NLP models to summarize news articles referenced in a user's twitter timeline

Twitter-News-Summarizer Twitter bot that uses NLP models to summarize news articles referenced in a user's twitter timeline 1.) Extracts all tweets fr

Rohit Govindan 1 Jan 27, 2022
मराठी भाषा वाचविण्याचा एक प्रयास. इंग्रजी ते मराठीचा शब्दकोश. An attempt to preserve the Marathi language. A lightweight and ad free English to Marathi thesaurus.

For English, scroll down मराठी शब्द मराठी भाषा वाचवण्यासाठी मी हा ओपन सोर्स प्रोजेक्ट सुरू केला आहे. माझ्या मते, आपली भाषा हळूहळू आणि कोणाचाही लक्षात

मुक्त स्त्रोत 19 Dec 26, 2021
This repository serves as a place to document a toy attempt on how to create a generative text model in Catalan, based on GPT-2

GPT-2 Catalan playground and scripts to train a GPT-2 model either from scrath or from another pretrained model.

Laura 1 Jan 28, 2022
A number of methods in order to perform Natural Language Processing on live data derived from Twitter

A number of methods in order to perform Natural Language Processing on live data derived from Twitter

null 1 Nov 24, 2021
TalkNet: Audio-visual active speaker detection Model

Is someone talking? TalkNet: Audio-visual active speaker detection Model This repository contains the code for our ACM MM 2021 paper, TalkNet, an acti

null 121 Sep 23, 2022
Active learning for text classification in Python

Active Learning allows you to efficiently label training data in a small-data scenario.

Webis 327 Sep 19, 2022
Twitter Sentiment Analysis using #tag, words and username

Twitter Sentment Analysis Web App using #tag, words and username to fetch data finds Insides of data and Tells Sentiment of the perticular #tag, words or username.

Kumar Saksham 24 Jul 18, 2022
Download videos from YouTube/Twitch/Twitter right in the Windows Explorer, without installing any shady shareware apps

youtube-dl and ffmpeg Windows Explorer Integration Download videos from YouTube/Twitch/Twitter and more (any platform that is supported by youtube-dl)

Wolfgang 203 Sep 27, 2022
TLA - Twitter Linguistic Analysis

TLA - Twitter Linguistic Analysis Tool for linguistic analysis of communities TLA is built using PyTorch, Transformers and several other State-of-the-

Tushar Sarkar 47 Aug 14, 2022
IndoBERTweet is the first large-scale pretrained model for Indonesian Twitter. Published at EMNLP 2021 (main conference)

IndoBERTweet ?? ???? 1. Paper Fajri Koto, Jey Han Lau, and Timothy Baldwin. IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effe

IndoLEM 36 Aug 25, 2022
This project uses unsupervised machine learning to identify correlations between daily inoculation rates in the USA and twitter sentiment in regards to COVID-19.

Twitter COVID-19 Sentiment Analysis Members: Christopher Bach | Khalid Hamid Fallous | Jay Hirpara | Jing Tang | Graham Thomas | David Wetherhold Pro

null 4 Nov 8, 2021
Code for papers "Generation-Augmented Retrieval for Open-Domain Question Answering" and "Reader-Guided Passage Reranking for Open-Domain Question Answering", ACL 2021

This repo provides the code of the following papers: (GAR) "Generation-Augmented Retrieval for Open-domain Question Answering", ACL 2021 (RIDER) "Read

morning 42 Sep 4, 2022
open-information-extraction-system, build open-knowledge-graph(SPO, subject-predicate-object) by pyltp(version==3.4.0)

中文开放信息抽取系统, open-information-extraction-system, build open-knowledge-graph(SPO, subject-predicate-object) by pyltp(version==3.4.0)

null 6 Sep 23, 2021
💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants

Rasa Open Source Rasa is an open source machine learning framework to automate text-and voice-based conversations. With Rasa, you can build contextual

Rasa 14.9k Sep 24, 2022
An open source library for deep learning end-to-end dialog systems and chatbots.

DeepPavlov is an open-source conversational AI library built on TensorFlow, Keras and PyTorch. DeepPavlov is designed for development of production re

Neural Networks and Deep Learning lab, MIPT 5.9k Sep 24, 2022
💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants

Rasa Open Source Rasa is an open source machine learning framework to automate text-and voice-based conversations. With Rasa, you can build contextual

Rasa 14.9k Sep 28, 2022