This is a graphql api build using ariadne python that serves a graphql-endpoint at port 3002 to perform language translation and identification using deep learning in python pytorch.

Language Translation and Identification

this machine/deep learning api that will be served as a graphql-api using ariadne, to perform the following tasks.

1. Language Identification

Identifying the language which the text belongs to using a simple text classification model. This model will be able to identify 7 different languages:

1. english (en)
2. french (fr)
3. german (de)
4. spanish (es)
5. italian (it)
6. portuguese (pt)
7. swedish (sw)

2. Language Translation

1. eng-de (english to german)
2. de-eng (german to english)
3. eng-af (english to afrikaans)
4. af-eng (afrikaans to german)
5. fr-eng (french to german)
6. eng-fr (english to french)
7. es-eng (spanish to german)
8. eng-es (english to spanish)
9. it-eng (italian to german)
10. eng-it (english to italian)
11. pt-eng (portuguese to german)
12. eng-pt (english to portuguese)
13. sw-eng (swedish to german)
14. eng-sw (english to swedish)

Starting the server

To start the server first you need to install all the packages that we used and make sure you have the .pt files for both the translation and identification models. To install the packages you need to run the following command:

Note that to save the .pt files for model you have to train the models first. The notebooks for doing so can be found on the repositories links that are given at the end of this README file.

pip install -r requirements.txt

Models Metrics Summary

1. Language Translation models

2. Language Identification models

For language identification i used the model based on fasttext paper for quick training on google colab GPU

Language Translation Model (graphql api)

The graphql server is running on http://127.0.0.1:3002/graphql if you send the following graphql mutation:

mutation Translator($input: TranslationInputType!) {
  translate(input: $input) {
    from_
    meta {
      name
      language
      author
      package
      description
      project
    }
    translation
    sent
  }
}

With the following query variables:

{
  "input": {
    "to": "eng",
    "from_": "it",
    "text": "ciao , come stai ?"
  }
}

You will get the following response:

{
  "data": {
    "translate": {
      "from_": "it",
      "meta": {
        "author": "@crispengari",
        "description": "language identification and translation graphql api.",
        "language": "python",
        "name": "ml backend",
        "package": "pytorch",
        "project": "noteme"
      },
      "sent": "ciao , come stai ?",
      "translation": "hello , how are you ? ."
    }
  }
}

Language Identification Model (graphql api)

To identify the language that the text is written in, we run the following mutation on http://127.0.0.1:3002/graphql

mutation Identify($input: IdentificationInputType!) {
  identify(input: $input) {
    probability
    label
    lang
    prediction {
      code
      id
      name
    }
    predictions {
      prediction {
        code
        id
        name
      }
      probability
    }
  }
}

With the following query variables:

{
  "input": {
    "text": "how are you?"
  }
}

To get the following response:

{
  "data": {
    "identify": {
      "label": 0,
      "lang": "eng",
      "prediction": {
        "code": "eng",
        "id": 0,
        "name": "english"
      },
      "predictions": [
        {
          "prediction": {
            "code": "eng",
            "id": 0,
            "name": "english"
          },
          "probability": 1
        },
        {
          "prediction": {
            "code": "swe",
            "id": 1,
            "name": "swedish"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "fra",
            "id": 2,
            "name": "french"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "deu",
            "id": 3,
            "name": "germany"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "ita",
            "id": 4,
            "name": "italian"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "por",
            "id": 5,
            "name": "portuguese"
          },
          "probability": 0
        },
        {
          "prediction": {
            "code": "afr",
            "id": 6,
            "name": "afrikaans"
          },
          "probability": 0
        }
      ],
      "probability": 1
    }
  }
}

Why graphql?

With graphql we allow the client to select fields he/she is interested in. And this give us an advantage of using a single endpoint for example http://127.0.0.1:3002/graphql for all the identification and translation models.

Why language translation?

This project was build to translate simple and complex sentences for 7 different languages. The idea was brought forward with the project likeme where we perform some processing on user's caption using pytorch deep learning models. The following steps were considered to preprocess the caption:

1. identify the language the caption in
2. translate the given caption to a certain language.

Notebooks

1. Translation models

• All the notebooks for the translation models are found here

2. Identification model

• The notebook for language identification model is found here