← DataAI ETL marketsAll usage guidesFunction catalog

Use DataAI ETL with Google Cloud Dataproc

Execution model: DataAI runs in a customer Dataproc cluster or Serverless batch. Google Cloud supplies the Spark runtime, service account, VPC, Cloud Storage, catalogs, BigQuery/connectors, Cloud Logging, and scheduling.

1. Prepare customer-controlled resources

  1. Create a private Cloud Storage prefix for one immutable DataAI release.
  2. Upload matching API, quality, core, and functions JARs. Upload the shaded CLI only for JSON-driven quality processing.
  3. Grant the Dataproc service account read access to the artifact prefix and minimum approved input/output access.
  4. Create isolated source and output datasets, tables, or storage prefixes.
  5. Use a Dataproc cluster or Serverless batch compatible with Java 17 APIs and Scala 2.12 artifacts.

2. Submit the job

Embedded application

Provide all DataAI JAR URIs through Dataproc's supported --jars or dependency configuration. Submit the customer application JAR that reads a DataFrame, calls DataAiPipeline or a function family, and explicitly writes approved outputs.

Configuration-driven quality job

  1. Store a customer JSON configuration in Cloud Storage and localize it for the driver.
  2. Use main class com.dataai.etl.spark.cli.DataAiJob.
  3. Pass --config <localized-file>.
  4. Use dedicated staging tables because named CLI outputs are overwritten.

3. Choose outputs

Customer code may persist clean/rejected rows, findings, profiles, analytics, time series, business/market outputs, map readiness, insight outputs, or matrix results to approved BigQuery, catalog, or Cloud Storage destinations. The embedded API performs no automatic write.

4. Schedule and observe

  1. Run first with approved non-production data.
  2. Inspect Cloud Logging and retain the Dataproc batch/job identifier.
  3. Reconcile input and output row counts and validate quality gates.
  4. Check matrix convergence metadata when balancing is selected.
  5. Test service-account denial, invalid configuration, timeout, retry, and partial-write behavior.
  6. Create the production workflow only after a clean rerun on the exact runtime image.

If a Marketplace image acts as the licensed artifact carrier, follow its documented extraction/copy process and do not add JARs from another release ZIP. Connectors, BigQuery writes, formats, and save modes remain customer-controlled.

Usage is successful when: one DataAI version loads, the service account remains least-privilege, Cloud Logging records the run, selected outputs reconcile, and retries do not duplicate or overwrite unintended data.