Use DataAI ETL with Talend or MuleSoft

1. Talend Spark job usage

1. Configure Talend's approved Maven/module repository with the exact installed DataAI version.
2. Keep spark-sql_2.12 supplied by the target Spark runtime; do not package a conflicting Spark copy.
3. Read the customer source into a Spark Dataset<Row>.
4. Call the DataAI pipeline or a selected function from a customer Java routine/job.
5. Explicitly write only approved outputs with the customer's Talend/Spark connectors and save-mode policy.

DataAiResult result = DataAiPipeline.fromDataset(input)
    .normalize()
    .recordKey("business_key")
    .profile()
    .validate(RuleSpec.required("required-id", "customer_id"))
    .execute();

Dataset<Row> clean = result.cleanRows();
Dataset<Row> rejected = result.rejectedRows();
Dataset<Row> findings = result.findings();

Connect these DataFrames to approved Talend output components, or persist them in customer code. Reconcile row counts before continuing the Talend job.

2. Talend orchestration alternative

For the packaged quality CLI, use a Talend command/job component to invoke spark-submit with main class com.dataai.etl.spark.cli.DataAiJob and --config <customer-json>. Require exit code 0 before starting downstream Talend work.

3. MuleSoft usage

1. Choose a customer-approved Spark job interface: scheduler API, message queue, workflow endpoint, or another existing integration.
2. Have MuleSoft send only a job identifier and approved parameters—not raw credentials in payloads.
3. Poll or receive completion status and require successful Spark termination.
4. Use an existing database, Databricks, or other approved connector to read DataAI output tables.
5. Return bounded summaries or identifiers through Mule APIs; do not unintentionally expose rejected rows or sensitive findings.

4. Advanced functions and validation

Analytics, time series, business, market, map-readiness, insight, and matrix-balancing functions run in the Talend/customer Spark job. MuleSoft can trigger the wrapper job and consume its outputs but does not execute those Spark algorithms itself.

• Pin one DataAI version in the dependency tree.
• Test quality failures, Spark failures, orchestration timeouts, retries, and idempotency.
• Verify only selected tables and APIs expose outputs.
• Keep repository and platform credentials in approved secret stores.

Usage is successful when: Talend or the customer Spark job produces governed outputs, MuleSoft orchestrates without embedding Spark libraries, and retries cannot duplicate or publish unauthorized data.