Hi, in the last step: Terraform apply I have this error:

Error: Error creating Dataproc cluster: googleapi: Error 400: User not authorized to act as service account '[email protected]'. To act as a service account, user must have one of [Owner, Editor, Service Account Actor] roles. See https://cloud.google.com/iam/docs/understanding-service-accounts for additional details., badRequest

I tried many things to solve but I don't know what is the problem, please help.

After creating docker container for the kafka instance and running the script to send message to it with the eventsim, the number of topics in the control center (9021) would not increase. I remained at 40.

Notice the warning at the top. What could that mean please?

I used e2-standard-2 for the kafka and airflow instances I then used e2-medium for both master and worker in the spark cluster Apparently, the kafka and airflow instances were successful.

google_dataproc_cluster.mulitnode_spark_cluster: Creating...
╷
│ Error: Error creating Dataproc cluster: googleapi: Error 400: Multiple validation errors:
│  - Insufficient 'CPUS' quota. Requested 6.0, available 4.0.
│  - Insufficient 'IN_USE_ADDRESSES' quota. Requested 3.0, available 2.0.
│  - This request exceeds CPU quota. Some things to try: request fewer workers (a minimum of 2 is required), 
use smaller master and/or worker machine types (such as n1-standard-2)., badRequest
│
│   with google_dataproc_cluster.mulitnode_spark_cluster,
│   on main.tf line 93, in resource "google_dataproc_cluster" "mulitnode_spark_cluster":
│   93: resource "google_dataproc_cluster" "mulitnode_spark_cluster" {
│

│ Error: Invalid value for network: project: required field is not set │ │ with google_compute_firewall.rules, │ on main.tf line 19, in resource "google_compute_firewall" "port_rules": │ 19: resource "google_compute_firewall" "port_rules" { │ ╵ ╷ │ Error: project: required field is not set │ │ with google_compute_instance.kafka_vm_instance, │ on main.tf line 35, in resource "google_compute_instance" "kafka_vm_instance": │ 35: resource "google_compute_instance" "kafka_vm_instance" { │ ╵ ╷ │ Error: project: required field is not set │ │ with google_compute_instance.airflow_vm_instance, │ on main.tf line 56, in resource "google_compute_instance" "airflow_vm_instance": │ 56: resource "google_compute_instance" "airflow_vm_instance" { │ ╵ ╷ │ Error: project: required field is not set │ │ with google_storage_bucket.bucket, │ on main.tf line 75, in resource "google_storage_bucket" "bucket": │ 75: resource "google_storage_bucket" "bucket" { │ ╵ ╷ │ Error: project: required field is not set │ │ with google_dataproc_cluster.mulitnode_spark_cluster, │ on main.tf line 93, in resource "google_dataproc_cluster" "mulitnode_spark_cluster": │ 93: resource "google_dataproc_cluster" "mulitnode_spark_cluster" { │ ╵ ╷ │ Error: project: required field is not set │ │ with google_bigquery_dataset.stg_dataset, │ on main.tf line 139, in resource "google_bigquery_dataset" "stg_dataset": │ 139: resource "google_bigquery_dataset" "stg_dataset" { │ ╵ ╷ │ Error: project: required field is not set │ │ with google_bigquery_dataset.prod_dataset, │ on main.tf line 146, in resource "google_bigquery_dataset" "prod_dataset": │ 146: resource "google_bigquery_dataset" "prod_dataset" { │

I have a question regarding the command for spark-submit:

spark-submit \
    --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 \
    stream_all_events.py

What is the meaning of the packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2?