Use DataAI ETL with Oracle AIDP Spark

1. Prepare the Oracle job

1. Use the exact DataAI JAR version installed in the approved Oracle artifact location.
2. Create separate Object Storage locations for licensed DataAI artifacts, the customer application/configuration, and customer data.
3. Grant the Oracle job identity read access to the artifact objects and minimum approved input/output permissions.
4. Create isolated source and output schemas and verify the Oracle runtime uses Java 17, Spark 3.5-compatible APIs, and Scala 2.12.

2. Choose an execution method

Configuration-driven quality job

1. Create a JSON configuration selecting the source table, optional output tables, normalization, record keys, rules, and minimum quality score.
2. Set main class com.dataai.etl.spark.cli.DataAiJob.
3. Distribute/localize the JSON file to the driver.
4. Pass --config <localized-file> as application arguments.

Embedded Java/Spark application

DataAiResult result = DataAiPipeline
    .fromTable(spark, "staging.customer_orders")
    .normalize()
    .recordKey("order_id")
    .profile()
    .validate(RuleSpec.required("customer-required", "customer_id"))
    .execute();

result.requireMinimumQualityScore(90.0);

Customer code explicitly writes selected DataFrames using Oracle-supported formats, catalogs, schemas, and save modes.

3. Use DataAI functions

Call quality, analytics, time-series, business, market, map-readiness, matrix-balancing, and insight functions from the installed Java libraries. Keep correlation lists bounded and define matrix iteration/tolerance controls. DataAI returns DataFrames and metadata; it does not automatically persist them.

4. Operate and validate

1. Run first with fictional or approved non-production data.
2. Inspect Oracle job logs and capture the application/run identifier.
3. Query every selected output and reconcile source, clean, rejected, and finding counts.
4. Verify matrix convergence metadata when balancing is used.
5. Test insufficient permissions, invalid rules, failed quality gates, retry behavior, and duplicate-write protection.
6. Schedule only after the exact Oracle runtime image and dependencies pass validation.

Do not mix JARs from different DataAI releases. The quality CLI overwrites named output tables; use dedicated staging tables. Embedded applications control their own save mode.

Usage is successful when: the Oracle job loads one DataAI version, executes under least privilege, returns expected results, writes only selected outputs, and exposes complete customer-controlled logs.