Hello, thank you for creating and providing this great project! I plan to use this project for my bachelor thesis. Therefore, I am mainly interested in the inference functionality provided with infer.py on branch server (branch infer seems to be outdated). I am aware of the VS Code extension and the public JSON API. I, however, prefer to use this project locally.

Since infer.py takes a pre-trained model as a program argument, I followed all the steps in the README to train such a model. Unfortunately, the script crashes with the following message (excerpt):

onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: tok for the following indices
 index: 0 Got: 7 Expected: 1
 Please fix either the inputs or the model.

Below you can find a link to a Google Colab notebook with all the steps from start (downloading the ManyTypes4Py dataset, pip-installing type4py, preprocessing) to finish (training a model, trying to infer the types of a single file) and the corresponding output from when I ran it the last time (including the full error backtrace on the bottom):

https://colab.research.google.com/drive/1kRIffMlgGCeW55wXelksGrXfSd0WjhKQ?usp=sharing

It should be relatively self-explanatory. Evidently, I use a fork of this project and not the project itself. The differences are minor though: In learn.py, I just re-uncommented the .to(DEVICE)-calls (c42144d) as otherwise it would lead to a crash in the notebook (vectors are on different devices). The remaining changes don't affect Python files and are not relevant to this issues. Further, I am using venv, although I doubt this has any negative influence on the execution of this project.

My question is, how can I successfully use infer.py? How can I obtain a proper compatible model for it? Are any of those steps in the linked notebook incorrect?

When using the preprocess command with only the -o argument, the code crashes with the following

UnboundLocalError: local variable 'train_files_vars' referenced before assignment

This is because in the following extract

https://github.com/saltudelft/type4py/blob/93828c3d1a3460dc29cace398ca0dcb10ea14daf/type4py/preprocess.py#L302-L319

the train_files_vars variable is only initialised in the if branch

In #8, I published Docker images for using Type4Py locally on users' machines. This PR also creates Docker images to deploy Type4Py in both production and development environments. Specifically, it makes the following changes:

• The Type4Py server detects whether the Docker image is running in local mode or production mode based on given ENV vars.
• Creates a separate Docker file to build images that allows performing the model inference on GPUs.
• Adds GH workflows to build Docker images for production and dev. environments and also supporting CPU/GPU.
• Add a unit test to test the local model before publishing its Docker image in GH Action.
• A bash script to test all the Docker images for production, dev, local environments.
• Use one config file for both production and dev. envs.

This PR makes the following changes:

• A new --rvth CLI argument that applies VTHs with a probability (default 0.5) [ONLY FOR PRODUCTION]
• Improves memory consumption when loading datasets
• Add reduce command, which uses PCA to reduce the dimension of type clusters and the size of the whole model.
• Disable telemetry data in the server module when running the Docker version.
• For inference, drops dependency on PyTorch to reduce the size of the Docker image
• Adding Docker file and config file for building Docker images of Type4Py

• Type checking the prediction of Type4Py (experimental)
• Converting the Type4Py's PyTorch model to the ONNX format
• Adding a web server to query the model and support the VSCode extension.
• Computing MRR for evaluation

• Extend preprocess and vectorize modules for variables prediction
• Extend data_loaders module for loading variables' tensors
• Separating ubiquitous and common types when evaluating
• Considering Visible Type Hints in preprocess module
• Evaluating different tasks in the eval module i.e., parameters, return, and variable
• Adding the Type4Py model with different configurations
• Improving the performance of data loaders
• In eval, consider MRR instead of weighted metrics
• predict now creates a JSON file for predictions
• Improve the memory consumption of predict
• Improvements to predict: better type aliasing & reducing the depth of parametric types
• Determining no. of batches when predicting different from the training phase

• Minor fix to the preprocess module for training the ManyTypes4Py Dataset
• Merging the JSON file of processed projects by the LibSA4Py pipeline in the preprocess module.
• Using the LibSA4Py pipeline when running the extract CLI command.
• Removed legacy pipeline and unused code

• Add Python logging instead of print in the modules
• Write logs of each module to its own log file. E.g. predict.py, learn.py
• Improve log messages in the modules

I experimented with the type prediction (http://localhost:5001/api/predict?tc=0) using the provided docker image. I noticed that depending on the analysed source code, I get different amounts of type predictions per parameter/return/variable type. Is it possible to retrieve a fixed number of predicted types? For example, I would like to retrieve the Top-10 type predictions for each parameter and return type.

Best regards Florian

Before this PR, we had to compute triplets on the fly to obtain a training sample, slowing down training and making the GPU wait for data. This PR makes an improvement to TripletDataset by pre-computing triplets before training the model and giving up to 6 times faster training speed. This way, obtaining a training sample to create batches is almost instant. However, with this improvement, each anchor has only one corresponding positive and negative example in every epoch.

The speed of preprocess is quite significantly improved by using parallel_apply() for processing functions' arguments and reducing the depth of parametric types. parallel_apply() is used from pandarallel.

The JSON output file is not JSON conformant in two aspects:

1. Single quotes (') are used instead of double quotes(")
2. Some words such as None, True or False are not wrapped in any quotes at all

This may affect some simpler JSON parsers, better JSON parsers can handle these minor errors just fine.

'error': None
#should be
"error": "None"

It would be interesting to see how well the TypeWriter algorithm (https://software-lab.org/publications/TypeWriter_arXiv_1912.03768.pdf) for searching type annotation suggestions works against type4py. We might get dramatically better results for two reasons:

• type4py's ML model seems to perform quite a bit better
• today's pyre incremental is orders of magnitude faster than pyre was when the TypeWriter paper was written, so we may be able to try many more combinations and get correspondingly better results

At one point we'd considered hacking this very quickly as an internal project in my company, but we ran out of time. I think it would be better done open-source anyway because then

• it would be easier to try out against external projects
• we could publish our results with code if they are interesting enough to be worth a paper
• the entire OSS community could benefit

I'm unsure if I can find time to prioritize this in the next 6 months at work but it's a little more likely if I treat it as a side project, which would also open the door to an informal weekend hackathon as a way to kick it off :)

I could do this in a separate repository or inside of type4py. What do you think @mir-am ? And does this sound interesting to you?