NeurIPS workshop paper 'Counter-Strike Deathmatch with Large-Scale Behavioural Cloning'

Overview

Counter-Strike Deathmatch with Large-Scale Behavioural Cloning

Tim Pearce, Jun Zhu
Offline RL workshop, NeurIPS 2021
Paper: https://arxiv.org/abs/2104.04258
Four minute introduction video: https://youtu.be/rnz3lmfSHv0
Gameplay examples: https://youtu.be/KTY7UhjIMm4

Contents

  1. Code Overview
  2. Trained Models
  3. Datasets
  4. Requirements
  5. License
  6. Disclaimer
  7. Maintenance

Code Overview

We briefly describe the workflow: dataset capture -> data processing -> training -> testing.

  1. Use dm_record_data.py to scrape data as a spectator, or dm_record_data_me_wasd.py to record when actively playing. This creates .npy files with screenshots and metadata.
  2. Run dm_infer_actions.py on .npy files from step 1 to infer actions from the metadata ('inverse dynamics model'). These are appended to the same .npy files. The visulations and stats produced by this script can be used to clean the data -- e.g. if the metadata from GSI and RAM disagrees for some variables, this is a sign that the data might be unreliable, if there are long periods of immobility (vel_static), this can mean you were tracking a motionless player. We suggest deleting such files.
  3. Run dm_pretrain_process.py on .npy files from step 2, this creates new .hdf5 files containing screenshots and onehot targets.
  4. (Optional.) Run tools_extract_metaadata.py to pull out metadata from the .npy files, saves as new .npy file.
  5. Run dm_train_model.py to train a model.
  6. Use tools_create_stateful.py to create a 'stateful' version of the model.
  7. Script dm_run_agent.py runs the model in the CSGO environment.

Brief overview of each script's purpose.

  • config.py : Contains key global hyperparameters and settings. Also several functions used across many scripts.
  • screen_input.py : Gets screenshot of the CSGO window, does some cropping and downsampling. Uses win32gui, win32ui, win32con, OpenCV.
  • key_output.py : Contains functions to send mouse clicks, set mouse position and key presses. Uses ctypes.
  • key_output.py : Contains functions to check for mouse location, mouse clicks and key presses. Uses win32api and ctypes.
  • meta_utils.py : Contains functions to set up RAM reading. Also contains a server class to connect to CSGO's GSI tool -- you should update MYTOKENHERE variable if setting up GSI.
  • dm_hazedumper_offsets.py : Contains offsets to read variables from RAM directly. They updated by the function in meta_utils.py.
  • dm_record_data.py : Records screenshots and metadata when in spectator mode. (Note these record scripts can be temperamental, and depend on hazedumper offsets being up to date.)
  • dm_record_data_me_wasd.py : Records screenshots and metadata when actively playing on local machine.
  • dm_infer_actions.py : For recorded dataset, infer actions from metadata and append to same file. Also stats indicative of data quality, and some functionality to rename files sequentially.
  • dm_pretrain_process.py : Takes .npy files from dm_infer_actions.py, and creates twin .hdf5 files ready for fast data loading with inputs and labels as expected by model. Also a function to strip image from .npy files.
  • dm_train_model.py : Trains the neural network model. Either from scratch or from a previous checkpoint.
  • dm_run_agent.py : Runs a trained model in CSGO. Option to collect metadata (eg score) using IS_GSI=True.
  • tools_dataset_inspect.py : Minimal code to open different datasets and accompanying metadata files.
  • tools_extract_metadata.py : Cuts out metadata (currvars, infer_a and helper_arr) from the original .npy files, and saves as new .npy without image. Aggregates 100 files together.
  • tools_dataset_stats.py : Collates metadata from the large online dataset, and produces summary stats and figures as in appendix A.
  • tools_create_stateful.py : This takes a trained 'non-stateful' model and saves a cloned 'stateful' version that should be used at test time (otherwise LSTM states reset between each forward pass). Required due to keras weirdness.
  • tools_map_coverage_analysis.py : Visualises map coverage of different agent and datasets, computes Earth mover's distance.
  • tools_view_save_egs.py : Opens and displays sequences of images .hdf5 and metadata .npy. Option of overlaying inferred actions. Option of saving as .png.
  • console_csgo_setup.txt : Console commands for setting up CSGO levels.

Trained Models

Four trained models are provided. There are 'non-stateful' (use during training) and 'stateful' (use at test time) versions of each.
Download link: https://drive.google.com/drive/folders/11n5Nxj6liSXKP6_vl-FjwfRHtWiSO6Ry?usp=sharing

  • ak47_sub_55k_drop_d4 : Pretrained on AK47 sequences only.
  • ak47_sub_55k_drop_d4_dmexpert_28 : Finetuned on expert deathmatch data.
  • ak47_sub_55k_drop_d4_aimexpertv2_60 : Finetuned on aim mode expert data.
  • July_remoterun7_g9_4k_n32_recipe_ton96__e14 : Pretrained on full dataset.

Datasets

For access to all datasets, please email the first author (tp424 at cam dot ac dot uk) with the title 'csgo dataset access' and a one sentence summary of how you intend to use the data. The email address you send from will be granted access to the cloud account. A brief description of dataset and directory structure is given below.

  • dataset_dm_scraped_dust2/hdf5_dm_july2021_*.hdf5

    • total files: 5500
    • approx size: 700 GB
    • map: dust2
    • gamemode: deathmatch
    • source: scraped from online servers
  • dataset_dm_expert_dust2/hdf5_dm_july2021_expert_*.hdf5

    • total files: 190
    • approx size: 24 GB
    • map: dust2
    • gamemode: deathmatch
    • source: manually created, clean actions
  • dataset_aim_expert/hdf5_aim_july2021_expert_*.hdf5

    • total files: 45
    • approx size: 6 GB
    • map: aim map
    • gamemode: aim mode
    • source: manually created, clean actions
  • dataset_dm_expert_othermaps/hdf5_dm_nuke_expert_*.hdf5

    • total files: 10
    • approx size: 1 GB
    • map: nuke
    • gamemode: deathmatch
    • source: manually created, clean actions
  • dataset_dm_expert_othermaps/hdf5_dm_nuke_expert_*.hdf5

    • total files: 10
    • approx size: 1 GB
    • map: nuke
    • gamemode: deathmatch
    • source: manually created, clean actions
  • dataset_dm_expert_othermaps/hdf5_dm_inferno_expert_*.hdf5

    • total files: 10
    • approx size: 1 GB
    • map: mirage
    • gamemode: deathmatch
    • source: manually created, clean actions
  • dataset_metadata/currvarsv2_dm_july2021_*_to_*.npy, currvarsv2_dm_july2021_expert_*_to_*.npy, currvarsv2_dm_mirage_expert_1_to_100.npy, currvarsv2_dm_inferno_expert_1_to_100.npy, currvarsv2_dm_nuke_expert_1_to_100.npy, currvarsv2_aim_july2021_expert_1_to_100.npy

    • total files: 55 + 2 + 1 + 1 + 1 + 1 = 61
    • approx size: 6 GB
    • map: as per filename
    • gamemode: as per filename
    • source: as per filename
  • location_trackings_backup/

    • total files: 305
    • approx size: 0.5 GB
    • map: dust2
    • gamemode: deathmatch
    • source: contains metadata used to compute map coverage analysis
      • currvarsv2_agentj22 is the agent trained over the full online dataset
      • currvarsv2_agentj22_dmexpert20 is previous model finetuned on the clean expert dust2 dataset
      • currvarsv2_bot_capture is medium difficulty built-in bot

Structure of .hdf5 files (image and action labels):

Each file contains an ordered sequence of 1000 frames (~1 minute) of play. This contains screenshots, as well as processed action labels. We chose .hdf5 format for fast dataloading, since a subset of frames can be accessed without opening the full file. The lookup keys are as follows (where i is frame number 0-999)

  • frame_i_x: is the image
  • frame_i_xaux: contains actions applied in previous timesteps, as well as health, ammo, and team. see dm_pretrain_preprocess.py for details, note this was not used in our final version of the agent
  • frame_i_y: contains target actions in flattened vector form; [keys_pressed_onehot, Lclicks_onehot, Rclicks_onehot, mouse_x_onehot, mouse_y_onehot]
  • frame_i_helperarr: in format [kill_flag, death_flag], each a binary variable, e.g. [1,0] means the player scored a kill and did not die in that timestep

Structure of .npy files (scraped metadata):

Each .npy file contains metadata corresponding to 100 .hdf5 files (as indicated by file name) They are dictionaries with keys of format: file_numi_frame_j for file number i, and frame number j in 0-999 The values are of format [curr_vars, infer_a, frame_i_helperarr] where,

  • curr_vars: contains a dictionary of the metadata originally scraped -- see dm_record_data.py for details
  • infer_a: are inferred actions, [keys_pressed,mouse_x,mouse_y,press_mouse_l,press_mouse_r], with mouse_x and y being continuous values and keys_pressed is in string format
  • frame_i_helperarr: is a repeat of the .hdf5 file

Requirements

Python and OS requirements

Below are the Python package versions used in development. The OS used for interacting with the game (testing and recording) was Windows, and for model training was Linux.

Python: '3.6.9' 
tensorflow: '2.3.0'
tensorflow.keras: '2.4.0'
h5py: '2.10.0'
numpy: '1.18.5'
OpenCV (cv2): '4.4.0'
scipy: '1.4.1'
ctypes: '1.1.0'
json: '2.0.9'

Hardware

Running the agent requires making forward passes through a large convolutional neural networks at 16 fps. If you do not have a CUDA enabled GPU, the agent's performance may degrade.

CSGO requirements

We collected the datasets and conducted testing over game versions 1.37.7.0 to 1.38.0.1. CSGO is continually updated and this may affect performance. The map dust2 received minor updates in 1.38.0.2. To test on the map from version 1.38.0.1, download here: https://steamcommunity.com/sharedfiles/filedetails/?id=2606435621. Future updates to gameplay may also degrade performance, consider rolling back the CSGO game version in this case.

Game State Integration (GSI) is used to pull out some metadata about the game. The dm_run_agent.py script is written so that it may be run without installing GSI (option IS_GSI). If you'd like to record data or extract metadata while running the agent, you'll need to set up GSI: https://www.reddit.com/r/GlobalOffensive/comments/cjhcpy/game_state_integration_a_very_large_and_indepth/ and update MYTOKENHERE in meta_utils.py.

Game settings (resolution, cross hair, mouse sensitivity etc) are documented in the paper, appendix E.

License

This repo can be used for personal projects and open-sourced research. We do not grant a license for its commercial use in any form. If in doubt, please contact us for permission.

Disclaimer

Whilst our code is not intended for cheating/hacking purposes, it's possible that Valve may detect the usage of some of these scripts in game (for example simulated mouse movements and RAM parsing) which in turn might lead to suspicion of cheating. We accept no liability for these sort of issues. Use it at your own risk!

Maintenance

This repo shares code used for academic research. It's not production ready. It's unlikely to be robust across operating systems, python versions, python packages, future CSGO updates etc. There's no plan to actively maintain this repo for these purposes, nor to fix minor bugs. If you'd like to help out with this, please reach out.

You might also like...
Open-AI's DALL-E for large scale training in mesh-tensorflow.

DALL-E in Mesh-Tensorflow [WIP] Open-AI's DALL-E in Mesh-Tensorflow. If this is similarly efficient to GPT-Neo, this repo should be able to train mode

Apache Spark - A unified analytics engine for large-scale data processing

Apache Spark Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an op

[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
[ICLR 2021, Spotlight] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks, ICLR 2021 (Spotlight) Demo | Paper [NEW!] Time to play with our interac

SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

The SLIDE package contains the source code for reproducing the main experiments in this paper. Dataset The Datasets can be downloaded in Amazon-

This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.
This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.

Skeleton Aware Multi-modal Sign Language Recognition By Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li and Yun Fu. Smile Lab @ Northeastern

Official implementation of
Official implementation of "Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets" (CVPR2021)

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets This is the official implementation of "Towards Good Pract

Official Implementation and Dataset of
Official Implementation and Dataset of "PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency", CVPR 2021

Portrait Photo Retouching with PPR10K Paper | Supplementary Material PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask an

An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicity.
An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicity.

Fast Face Classification (F²C) This is the code of our paper An Efficient Training Approach for Very Large Scale Face Recognition or F²C for simplicit

DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021)
DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021)

DeepLM DeepLM: Large-scale Nonlinear Least Squares on Deep Learning Frameworks using Stochastic Domain Decomposition (CVPR 2021) Run Please install th

Comments
  • Record Crashes

    Record Crashes

    Hi, I have this issue: I open dm_record_data.py, and nothing happens, I press Q and nothing happens too. I set SHOW_IMAGE to True and window opens, that says resized that is crashing.

    Video of error:

    https://user-images.githubusercontent.com/48152410/152793200-c9b00f2f-64c8-449a-aae4-87947a06b05f.mp4 PS: I just requested for datasets, my email is r. h. 1234 @ gmail.com

    opened by DuroDaCoder 6
  • Wonder if accessing game memory is necessary when training?

    Wonder if accessing game memory is necessary when training?

    Hi TeaPearce,

    Thanks for providing such a well written paper and presenting this project for the behaviral cloning track, really appreciate it. Me and my team are inspired from this and we are planing to do a research on other MMORPG games.

    Before we start our project, I would like to consult you some questions regarding to this project codes which would defenetly help us to understand the concept better.

    1. In the file dm_record_data.py and dm_record_data_me_wasd.py, the application is actually open up the game process and access game's memorry to get internal game data like player position, x,y,z and height etc. So the question is, is it necessary to access game memory while recording game play data(especially for MMORPG/FPS game) for behavioural cloning working properly?
    2. If it is true above, after done training the model by using the data above, is it necessary to access game's memory when running agents to use model to play actual game? The reason I'm asking is because usually accessing to game's memory will be considered as hacks.
    3. Last but not the least, for 3D games, is the FOV, positions of player, yaw, pitch and roll required to gather as the meta data in order to do the behaviral cloning and make agents play like human?

    Sorry if I'm asking too many questions! Really hope you can answer them since our team is now very excited about what the AI can achieve in the MMORPG games!

    Best regards, Whaley

    opened by bbsxjy 1
  • Adding Conda Environment

    Adding Conda Environment

    According to conda, the requirements are unsatisfiable, as tensorflow 2.3.0 requires python 3.7 or 3.8. This configuration enabled me to run dm_run_agent.py without crashing.

    opened by David-Durst 0
  • Other games

    Other games

    Hey there TeaPearce,

    I saw a video by bycloud last year that mentioned your project. I loved to see it and inspired me that time to learn how to code and get into ai. I started to train an ai for image regognision, for the game I love the most, Siege. I failed that time and tried to look up your code and couldn't find it (probably wasn't open source then). Now I got back intrested in it and was wondering if I can try to rewrite some parts of your code to make it work on Siege. Like I said, I love Siege and does not want to harm it in anyway, the bot will probably only play against other bots and maybe some human players but then on like privat servers, if it works and I am allowed to change the code.

    I am trying to read your code and your comments are very helpful to under stand it.

    Beside that, thanks for inspiring me Best regards, Bengeljo

    opened by Bengeljo 2
Owner
Tim Pearce
AI researcher \\ Curr: Tsinghua Uni \\ Prev: Cambridge Uni
Tim Pearce
Guiding evolutionary strategies by (inaccurate) differentiable robot simulators @ NeurIPS, 4th Robot Learning Workshop

Guiding Evolutionary Strategies by Differentiable Robot Simulators In recent years, Evolutionary Strategies were actively explored in robotic tasks fo

Vladislav Kurenkov 4 Dec 14, 2021
Cobalt Strike teamserver detection.

Cobalt-Strike-det Cobalt Strike teamserver detection. usage: cobaltstrike_verify.py [-l TARGETS] [-t THREADS] optional arguments: -h, --help show this

TimWhite 17 Sep 27, 2022
This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

This is a Pytorch implementation of the paper: Self-Supervised Graph Transformer on Large-Scale Molecular Data.

null 212 Dec 25, 2022
The implementation of the CVPR2021 paper "Structure-Aware Face Clustering on a Large-Scale Graph with 10^7 Nodes"

STAR-FC This code is the implementation for the CVPR 2021 paper "Structure-Aware Face Clustering on a Large-Scale Graph with 10^7 Nodes" ?? ?? . ?? Re

Shuai Shen 87 Dec 28, 2022
Official Implement of CVPR 2021 paper “Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting”

RGBT Crowd Counting Lingbo Liu, Jiaqi Chen, Hefeng Wu, Guanbin Li, Chenglong Li, Liang Lin. "Cross-Modal Collaborative Representation Learning and a L

null 37 Dec 8, 2022
null 190 Jan 3, 2023
Official repository for the paper, MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Understanding.

MidiBERT-Piano Authors: Yi-Hui (Sophia) Chou, I-Chun (Bronwin) Chen Introduction This is the official repository for the paper, MidiBERT-Piano: Large-

null 137 Dec 15, 2022
A pytorch implementation of the CVPR2021 paper "VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild"

VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild A pytorch implementation of the CVPR2021 paper "VSPW: A Large-scale Dataset for Video

null 45 Nov 29, 2022
PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].

Smooth ReLU in PyTorch Unofficial PyTorch reimplementation of the Smooth ReLU (SmeLU) activation function proposed in the paper Real World Large Scale

Christoph Reich 10 Jan 2, 2023
Code to reproduce the experiments from our NeurIPS 2021 paper " The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective"

Code To run: python runner.py new --save <SAVE_NAME> --data <PATH_TO_DATA_DIR> --dataset <DATASET> --model <model_name> [options] --n 1000 - train - t

Geoff Pleiss 5 Dec 12, 2022