Use DataAI ETL with InterSystems IRIS

1. Prepare the IRIS connection

1. Use an isolated namespace and least-privilege identity.
2. Install an InterSystems JDBC driver approved for the IRIS server and Java 17.
3. Keep the driver and DataAI adapter JAR on both Spark driver and executor classpaths.
4. Load the IRIS user and password from the customer's secret manager.
5. Approve source tables, output tables, fetch size, batch size, and maximum concurrent JDBC connections.

2. Read, process, and explicitly write

IrisJdbcOptions iris = IrisJdbcOptions
    .forServer("iris.internal", 1972, "DATAAI")
    .credentials(System.getenv("IRIS_USER"), System.getenv("IRIS_PASSWORD"))
    .fetchSize(5000)
    .batchSize(1000)
    .build();

Dataset<Row> input = IrisDataFrames.readTable(
    spark, iris, "Source.CustomerOrders");

DataAiResult result = DataAiPipeline.fromDataset(input)
    .normalize()
    .recordKey("order_id")
    .profile()
    .validate(RuleSpec.required("customer-required", "customer_id"))
    .execute();

IrisPipelineOutputBundle outputs = IrisPipelineOutputs.from(result);
IrisDataFrames.writer(outputs.qualityFindings(), iris)
    .option("dbtable", IrisOutputNames.QUALITY_FINDINGS)
    .mode(SaveMode.Append)
    .save();

The adapter never chooses an IRIS target table or save mode and never calls save() automatically. Customer code authorizes every write.

3. Use large IRIS tables carefully

1. Start with readTable(...) on a bounded validation table.
2. Use readTablePartitioned(...) only after an IRIS administrator approves a numeric partition field, lower/upper bounds, partition count, and JDBC concurrency.
3. Prefer staging tables and an IRIS-controlled merge procedure for production.
4. Validate partial failures and retries before scheduling.

4. Publish advanced-function outputs

Run any function from dataai-spark-functions. Add run metadata with IrisFunctionOutputs.withRunMetadata(...). For matrix balancing, use IrisFunctionOutputs.matrixBalance(...) so convergence, iteration count, and maximum error are retained before the customer-selected JDBC write.

5. Validate in IRIS

• Query source and output counts in the same namespace.
• Verify clean, rejected, finding, profile, run, analytical, and matrix tables selected by the application.
• Check SQL privileges, indexes, batch behavior, retries, connection limits, and rollback/merge handling.
• Confirm that no unselected table changed.

Usage is successful when: approved IRIS rows are read, DataAI results reconcile, explicit writes reach only approved staging tables, and retries do not duplicate or lose output.