DataAI ETL Google Cloud (Dataproc) — Production Installation Guide

Product: Google Cloud (Dataproc)

Distribution: Production

Source document: Google/docs/INSTALLATION_AND_USAGE.md

Source SHA-256: acc68380f1c94acdbd188a631c51d0af52c20a0928341b06fb5e82e024ca183b

Download the applicable package | Read the applicable license

Downloads and verification: Google package 1.0.0 | Production checksums

Installation and Usage for Google Cloud Marketplace

Delivery model

The container stores DataAI JARs in /opt/dataai/lib. Customers copy licensed artifacts into a governed Cloud Storage bucket and reference them with Dataproc --jars or cluster initialization. The image has no DataAI service, telemetry, or customer-data callback.

JAR installation boundary

The production container candidate includes commercial DataAI Spark JARs in the image. Private Cloud Storage delivery does not install them by itself; attaching the authorized objects with Dataproc --jars or cluster initialization explicitly installs them for that workload. Both routes require the customer's applicable Yanbor LLC authorization. Third-party platforms, runtimes, and drivers remain customer-provided and subject to their respective vendor license terms.

Production-candidate artifacts

Install these matching-version modules together:

dataai-spark-api-1.0.0.jar
dataai-spark-quality-1.0.0.jar
dataai-spark-core-1.0.0.jar
dataai-spark-functions-1.0.0.jar

The CLI JAR is optional. Spark and Hadoop are provided by the customer runtime and are not bundled by DataAI.

Generic Spark usage

Add all required JARs to the Spark driver and executor classpaths using the target platform's supported library or --jars mechanism. Then call DataAiPipeline and the functions in com.dataai.etl.spark.functions. The complete inventory is in FUNCTION_CATALOG.md.

Customer code supplies Dataset<Row> inputs and decides whether returned DataFrames are written. DataAI performs no automatic persistence.

Production

Deploy immutable version 1.0.0 only after Yanbor release authorization, commercial entitlement, and target-platform validation. Obtain it from the approved marketplace artifact, authenticated Maven repository, or customer artifact repository. Verify checksums and pin the exact version.

Troubleshooting

UnsupportedClassVersionError: select a Java 17-compatible runtime.
Class not found: install API, quality, core, and functions at the same version.
Scala linkage error: use a runtime compatible with Scala 2.12.
Spark linkage error: validate against the documented Spark 3.5.x baseline.
Permission failure: use customer-approved identity and storage permissions; DataAI does not bypass platform controls.