FGO game data API
HTTP API for FGO game data. Transform the raw game data into something a bit more manageable.
View the API documentation here: https://api.atlasacademy.io.
- Environment variables
- Development environment set up
- Run the API server
- Architecture
- Linting
- Formatting
- Dependencies
- Testing
- Helper scripts
Environment variables
List of environment variables for the main app.
Required variables
DATA: A JSON object string with keys being region strings and values being gamedata's folder location and Postgresql DSN. Not all regions are required in the object. Any combination of regions is accepted.REDISDSN: Redis DSN to a Redis server for caching.
Optional variables
Click to view
REDIS_PREFIX: default tofgoapi. Prefix for redis keys.CLEAR_REDIS_CACHE: default toTrue. If set, will clear the redis cache on start and when the webhook above is used.RATE_LIMIT_PER_5_SEC: default to100. The rate limit per 5 seconds for nice and raw endpoints.RAYSHIFT_API_KEY: default to"". Rayshift.io API key to pull quest data.RAYSHIFT_API_URL: default to https://rayshift.io/api/v1/. Rayshift.io API URL.QUEST_CACHE_LENGTH: default to3600. How long to cache the quest and war endpoints in seconds. Because the rayshift data is updated continously, web and quest endpoints have lower cache time.WRITE_POSTGRES_DATA: default toTrue. Overwrite the data in PostgreSQL when importing.WRITE_REDIS_DATA: default toTrue. Overwrite the data in Redis when importing.ASSET_URL: defaults to https://assets.atlasacademy.io/GameData/. Base URL for the game assets.OPENAPI_URL: default toNone. Set the server URL in the openapi schema export.EXPORT_ALL_NICE: default toFalse. If set toTrue, at start the app will generate nice data of all servant and CE and serve them at the/exportendpoint. It's recommended to serve the files in the/exportfolder using nginx or equivalent webserver to lighten the load on the API server.DOCUMENTATION_ALL_NICE: default toFalse. If set toTrue, there will be links to the exported all nice files in the documentation.GITHUB_WEBHOOK_SECRET: default to"". If set, will add a webhook location at/GITHUB_WEBHOOK_SECRET/updatethat will pull and update the game data. If it's not set, the endpoint is not created.GITHUB_WEBHOOK_GIT_PULL: default toFalse. If set, the app will dogit pullon the gamedata repos when the webhook above is used.GITHUB_WEBHOOK_SLEEP: default to0. If set, will delay the action above byGITHUB_WEBHOOK_SLEEPseconds.
Secrets
Secret variables can be put in the secrets folder instead of being supplied as environment variable:
> cat .\secrets\rayshift_api_key
eca334a9-3289-4ad7-9b92-1ec2bbc3fc19
> cat .\secrets\redisdsn
redis://localhost
config.json
You can make a config.json file with the settings instead of setting the environment variables. Check the config.sample.json file for an example.
Settings at a higher level will override the settings at a lower level.
- Secrets variable
- Enviornment variable
.envfileconfig.json
Development environment set up
Make sure poetry is installed: https://python-poetry.org/docs/#installation.
Docker is recommended to set up the Postgres and redis servers but those can be set up manually as well. Postgres needs the PGroonga extension installed.
> git clone --depth 0 --branch JP https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-jp
> git clone --depth 0 --branch NA https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-na
> git clone --depth 0 --branch CN https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-cn
> git clone --depth 0 --branch KR https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-kr
> git clone --depth 0 --branch TW https://github.com/atlasacademy/fgo-game-data.git fgo-game-data-tw
> git clone https://github.com/atlasacademy/fgo-game-data-api.git
> cd fgo-game-data-api
> cp config.sample.json config.json
> docker-compose up -d
> poetry install
> poetry shell
Run the API server
Run at the project root to start the API server:
> uvicorn app.main:app --reload --log-level debug --reload-dir app
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Started reloader process [16680] using watchgod
INFO fgoapi: Loading game data …
INFO fgoapi: Loaded game data in 15.14s.
INFO: Started server process [33312]
INFO: Waiting for application startup.
INFO: Application startup complete.
DEBUG fgoapi: Processed in 0.21ms.
INFO: 127.0.0.1:56759 - "GET / HTTP/1.1" 307 Temporary Redirect
DEBUG fgoapi: Processed in 0.24ms.
INFO: 127.0.0.1:56759 - "GET /rapidoc HTTP/1.1" 200 OK
Go to http://127.0.0.1:8000 for the API documentation.
Tips:
- Change
write_postgres_datatofalseafter the first run to speed up reloading if it's not needed (schema doesn't change or data hasn't changed).
Architecture
main.py: Main entrypoint of the application.routers/: Routers to deal with incoming requests. The routers call functions fromcoreto get the response data.core/: Build response data. Get raw data from eitherdb/helpers/orredis/helpers.data/: Import translation data into memory. Preprocessing conde for data to be imported into db and redis.db/: DB stuffs.db/helpers/: Functions to be used bycoreto get data from the DB.
redis/: Redis stuffs.redis/helpers/: Functions to be used bycoreto get data from Redis.
schemas/: Response Pydantic models.models/: SQLAlchemy Core Tables.
Linting
pylint and mypy are used to lint the code. pylint's configuration and mypy's configuration are in pyproject.toml.
Formatting
isort and black are used to format the code. isort's configuration is in pyproject.toml and black uses default settings.
isort app tests export scripts; black app tests export scripts
prettier is used to format the json files.
prettier --write tests/*/*.json
prettier --write export/*/Nice*.json
prettier --write export/*/*UserLevel.json --print-width 50
Dependencies
Use poetry to manage the dependencies. Run poetry export after adding a production dependency.
poetry export -f requirements.txt -o requirements.txt
Testing
Run pytest at project root to run the tests or use coverage to get coverage statistics.
coverage run --source=app/ -m pytest; coverage html
Helper scripts
extract_enums.py
Take the dump.cs generated by Il2CppDumper and write the gameenums.py file.
python scripts/extract_enums.py dump.cs_path app/schemas/gameenums.py
update_ce_translation.py
Update equip_names.json with new NA CEs translations. --jp-master and --na-master arguments are not needed if environment variables JP_GAMEDATA and NA_GAMEDATA are set, added to the .env file or set in config.json.
python scripts/update_ce_translation.py --jp-master jp_master_path --na-master na_master_path
load_rayshift_quest_list.py
Update the rayshiftQuest tables with the list of available quests from Rayshift. This script should be run periodically to update the rayshiftQuest list.
python -m scripts.load_rayshift_quest_list
get_test_data.py
Run this script when the master data changed to update the tests or when new tests are added.
python -m tests.get_test_data --raw --nice --basic
3 Nov 11, 2022
1 Feb 10, 2022
3 Oct 25, 2022
4 Nov 7, 2021
199 Jan 2, 2023
11 Sep 18, 2022
2 Mar 1, 2022
5 May 26, 2022
1 Oct 26, 2021
1 Jan 8, 2022
1 Dec 21, 2021
1 Jan 2, 2022
1 Feb 15, 2022
0 Jan 27, 2022
4 Jan 19, 2022
4 Oct 6, 2022
1 Mar 19, 2022
3 Feb 20, 2022
3 Jan 9, 2022