Dataflow Flex Template in Python

This repository contains a template for a Dataflow Flex Template written in Python that can easily be used to build Dataflow jobs to run in STOIX using Dataflow runner.

The code is based on the same example data as Google Cloud Python Quickstart, "King Lear" which is a tragedy written by William Shakespeare.

The Dataflow job reads the file content, count occurencies of each word and inserts it to a BigQuery table. The schedule date is also added to the table name producing a sharded table for the output.

Source data:

https://storage.cloud.google.com/dataflow-samples/shakespeare/kinglear.txt
gs://dataflow-samples/shakespeare/kinglear.txt

Template maintained by STOIX.

Configuration

The job is configured with the following pipeline options:

stoix_scheduled - Scheduled datetime as RFC3339
input_file - Text to read
output_dataset - BigQuery dataset for output table
output_table_prefix - BigQuery output table name prefix
project - Google Cloud project id

When using Dataflow runner, stoix_scheduled is automatically set and other pipeline options can be added as described in the Dataflow runner README.

Test the code

Tox is used to format, test and lint the code. Make sure to install it with pip install tox and then just run tox within the project folder.

Run pipeline

In order to work with the code locally, you can use Python virtual environments. Make sure to use Python version 3.7.10 as it is the version supported by Google Dataflow.

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -e .

Run on local machine

See quickstart python for further description of arguments.

python -m main \
    --region europe-north1 \
    --runner DirectRunner \
    --stoix_scheduled 2021-01-01T00:00:00Z \
    --input_file gs://dataflow-samples/shakespeare/kinglear.txt \
    --output_table_prefix kinglear \
    --output_dataset 
   
     \
    --project 
    
      \
    --temp_location gs://
     
      /tmp/

Build Docker image for STOIX

In order to run the pipeline the Flex Template needs to be packaged in a Docker image and pushed to a Docker image repository. In this example Docker Hub is used.

Set the tag to the name and version of your pipeline, e.g: stoix/count-words:1.0.0.

$ docker build --tag stoix/count-words:1.0.0 .

Then upload the image to the Docker image repository.

$ docker push stoix/count-words:1.0.0

Run Dataflow on STOIX

Now the Dataflow Flex Template job can be ran using Dataflow runner. Add a new job with the image stoix/dataflow-runner and the following environment variables:

GCP_PROJECT_ID:
GCP_REGION: europe-north1
GCP_SERVICE_ACCOUNT: BASE64 encoded service account JSON
JOB_IMAGE: stoix/count-words:1.0.0
JOB_NAME_PREFIX: count-words
JOB_PARAM_INPUT_FILE: gs://dataflow-samples/shakespeare/kinglear.txt
JOB_PARAM_OUTPUT_DATASET: dataflow
JOB_PARAM_OUTPUT_TABLE_PREFIX: kinglear
JOB_SDK_LANGUAGE: python

Note: When running this in production, set GCP_SERVICE_ACCOUNT as a secret instead of environment variable.