NyaaV2 
Setting up for development
This project uses Python 3.7. There are features used that do not exist in 3.6, so make sure to use Python 3.7. This guide also assumes you 1) are using Linux and 2) are somewhat capable with the commandline.
It's not impossible to run Nyaa on Windows, but this guide doesn't focus on that.
Code Quality:
- Before we get any deeper, remember to follow PEP8 style guidelines and run
./dev.py lintbefore committing to see a list of warnings/problems.- You may also use
./dev.py fix && ./dev.py isortto automatically fix some of the issues reported by the previous command.
- You may also use
- Other than PEP8, try to keep your code clean and easy to understand, as well. It's only polite!
Running Tests
The tests folder contains tests for the the nyaa module and the webserver. To run the tests:
- Make sure that you are in the python virtual environment.
- Run
./dev.py testwhile in the repository directory.
Setting up Pyenv
pyenv eases the use of different Python versions, and as not all Linux distros offer 3.7 packages, it's right up our alley.
- Install dependencies https://github.com/pyenv/pyenv/wiki/Common-build-problems
- Install
pyenvhttps://github.com/pyenv/pyenv/blob/master/README.md#installation - Install
pyenv-virtualenvhttps://github.com/pyenv/pyenv-virtualenv/blob/master/README.md - Install Python 3.7.2 with
pyenvand create a virtualenv for the project:pyenv install 3.7.2pyenv virtualenv 3.7.2 nyaapyenv activate nyaa
- Install dependencies with
pip install -r requirements.txt - Copy
config.example.pyintoconfig.py- Change
SITE_FLAVORin yourconfig.pydepending on which instance you want to host
- Change
Setting up MySQL/MariaDB database
You may use SQLite but the current support for it in this project is outdated and rather unsupported.
- Enable
USE_MYSQLflag in config.py - Install latest mariadb by following instructions here https://downloads.mariadb.org/mariadb/repositories/
- Tested versions:
mysql Ver 15.1 Distrib 10.0.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
- Tested versions:
- Run the following commands logged in as your root db user (substitute for your own
config.pyvalues if desired):CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';FLUSH PRIVILEGES;CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;
Finishing up
- Run
python db_create.pyto create the database and import categories- Follow the advice of
db_create.pyand run./db_migrate.py stamp headto mark the database version for Alembic
- Follow the advice of
- Start the dev server with
python run.py - When you are finished developing, deactivate your virtualenv with
pyenv deactivateorsource deactivate(or just close your shell session)
You're now ready for simple testing and development!
Continue below to learn about database migrations and enabling the advanced search engine, Elasticsearch.
Database migrations
- Database migrations are done with flask-Migrate, a wrapper around Alembic.
- If someone has made changes in the database schema and included a new migration script:
- If your database has never been marked by Alembic (you're on a database from before the migrations), run
./db_migrate.py stamp headbefore pulling the new migration script(s).- If you already have the new scripts, check the output of
./db_migrate.py historyinstead and choose a hash that matches your current database state, then run./db_migrate.py stamp.
- If you already have the new scripts, check the output of
- Update your branch (eg.
git fetch && git rebase origin/master) - Run
./db_migrate.py upgrade headto run the migration. Done!
- If your database has never been marked by Alembic (you're on a database from before the migrations), run
- If you have made a change in the database schema:
- Save your changes in
models.pyand ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database) - Run
./db_migrate.py migrate -m "Short description of changes"to automatically generate a migration script for the changes- Check the script (
migrations/versions/...) and make sure it works! Alembic may not able to notice all changes.
- Check the script (
- Run
./db_migrate.py upgradeto run the migration and verify the upgrade works.- (Run
./db_migrate.py downgradeto verify the downgrade works as well, then upgrade again)
- (Run
- Save your changes in
Setting up and enabling Elasticsearch
Installing Elasticsearch
- Install JDK with
sudo apt-get install openjdk-8-jdk - Install Elasticsearch
- From packages...
- Enable the service:
sudo systemctl enable elasticsearch.servicesudo systemctl start elasticsearch.service
- Enable the service:
- or simply extracting the archives and running the files, if you don't feel like permantently installing ES
- From packages...
- Run
curl -XGET 'localhost:9200'and make sure ES is running- Optional: install Kibana as a search debug frontend for ES
Enabling MySQL Binlogging
- Edit your MariaDB/MySQL server configuration and add the following under
[mariadb]:log-bin server_id=1 log-basename=master1 binlog-format=row - Restart MariaDB/MySQL (
sudo service mysql restart) - Copy the example configuration (
es_sync_config.example.json) ases_sync_config.jsonand adjust options in it to your liking (verify the connection options!) - Connect to mysql as root
- Verify that the result of
SHOW VARIABLES LIKE 'binlog_format';isROW - Execute
GRANT REPLICATION SLAVE ON *.* TO 'username'@'localhost';to allow your configured user access to the binlog
- Verify that the result of
Setting up ES
- Run
./create_es.shto create the indices for the torrents:nyaaandsukebei- The output should show
acknowledged: truetwice
- The output should show
- Stop the Nyaa app if you haven't already
- Run
python import_to_es.pyto import all the torrents (on nyaa and sukebei) into the ES indices.- This may take some time to run if you have plenty of torrents in your database.
Enable the USE_ELASTIC_SEARCH flag in config.py and (re)start the application.
Elasticsearch should now be functional! The ES indices won't be updated "live" with the current setup, continue below for instructions on how to hook Elasticsearch up to MySQL binlog.
However, take note that binglog is not necessary for simple ES testing and development; you can simply run import_to_es.py from time to time to reindex all the torrents.
Setting up sync_es.py
sync_es.py keeps the Elasticsearch indices updated by reading the binlog and pushing the changes to the ES indices.
- Make sure
es_sync_config.jsonis configured with the user you grated theREPLICATIONpermissions - Run
import_to_es.pyand copy the outputted JSON into the file specified bysave_locin youres_sync_config.json - Run
sync_es.pyas-is or, for actual deployment, set it up as a service and run it, preferably as the system/root- Make sure
sync_es.pyruns within the venv with the right dependencies!
- Make sure
You're done! The script should now be feeding updates from the database to Elasticsearch.
Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or poke Elasticsearch yourself!
![[Restructure routes 1] Import sorting + Static pages](https://avatars.githubusercontent.com/u/10238474?v=4)
4.2k Dec 31, 2022
1.3k Jan 7, 2023
17.8k Jan 3, 2023
51 Jun 14, 2022
74 Dec 10, 2021
4 Aug 20, 2021
70 Sep 22, 2022
6 Dec 1, 2021
2 Dec 21, 2021
12.9k Jan 4, 2023
5 Jun 28, 2022
87 Dec 27, 2022
3.8k Jan 8, 2023
1.5k Jan 4, 2023
4.7k Jan 5, 2023
591 Dec 13, 2022
29.7k Jan 2, 2023
222.5k Jan 2, 2023
