← DataAI ETL marketsAll usage guidesFunction catalog

Use DataAI ETL with AWS Glue or Amazon EMR

Execution model: DataAI runs in customer-managed Spark compute on AWS Glue Spark, Amazon EMR, or EMR Serverless. AWS supplies identity, networking, S3, Glue Data Catalog, logs, runtime, and scheduling. AWS Lambda is not the Spark runtime for this package.

1. Prepare the job

  1. Place one exact DataAI release in a private S3 artifact prefix or use the approved Marketplace fulfillment.
  2. Attach the API, quality, core, and functions JARs with the platform's supported dependent-JAR mechanism. Attach the shaded CLI only for a JSON-driven quality job.
  3. Grant the job role read access to DataAI artifacts and minimum approved access to input/output locations.
  4. Use isolated Glue databases, S3 prefixes, or Delta/Iceberg/Hive tables for initial outputs.
  5. Confirm the chosen AWS Spark runtime is compatible with Java 17 APIs and Scala 2.12 artifacts.

2. Choose an execution path

Embedded Spark application

Read the customer input as a Spark DataFrame, call DataAiPipeline or functions from com.dataai.etl.spark.functions, and explicitly write selected result DataFrames using the customer's approved AWS format and catalog.

Configuration-driven quality job

  1. Store the JSON configuration in a customer S3 location accessible to the driver.
  2. Use main class com.dataai.etl.spark.cli.DataAiJob.
  3. Localize/pass the configuration as --config <file>.
  4. Use dedicated output tables because the quality CLI overwrites named outputs.

3. Run from the selected AWS product

RuntimeUsage
AWS Glue SparkAdd DataAI JAR S3 URIs through the job's dependent-JAR configuration; keep connectors and secrets in Glue/customer facilities.
Amazon EMRAttach the JARs through the approved bootstrap, step, or --jars mechanism and submit the customer application.
EMR ServerlessUse the approved application image/dependencies, job role, entry point, and S3 configuration under the documented fulfillment model.

4. Observe and validate

  1. Inspect CloudWatch and EMR/Glue job logs and retain the job/run identifier.
  2. Reconcile source, clean, rejected, finding, and profile counts.
  3. Verify analytical outputs and matrix convergence when used.
  4. Test one invalid rule, insufficient access, job timeout, retry, and partial-write scenario.
  5. Confirm no unselected S3 prefix, table, or catalog object changed.
  6. Pin the exact JAR version in the scheduled job.

Do not combine a Marketplace image's DataAI libraries with a different release ZIP. DataAI performs no automatic persistence in the embedded API; customer code chooses writes. The quality CLI overwrites configured tables.

Usage is successful when: one DataAI version loads on driver and executors, IAM remains least-privilege, CloudWatch captures the run, outputs reconcile, and retries are safe.