Use DataAI ETL from a .NET Pipeline

1. Verify the installed runtime layout

1. Confirm the application restores Yanbor.DataAI.Etl.Spark and its runtime package at one approved version.
2. Confirm the application output contains the DataAI library JARs under dataai/jars and the shaded quality CLI under dataai/cli.
3. Run spark-submit --version under the same identity that will run the .NET application.
4. Confirm that the Spark identity can read the source table and write only the approved output tables.

2. Build a quality job configuration

using Yanbor.DataAI.Etl.Spark;

var configuration = new DataAiQualityJobConfiguration
{
    SourceTable = "staging.customer_orders",
    CleanTable = "dataai.customer_orders_clean",
    RejectedTable = "dataai.customer_orders_rejected",
    ProfileTable = "dataai.customer_orders_profiles",
    FindingsTable = "dataai.customer_orders_findings",
    Normalize = true,
    RecordKeyColumns = ["order_id"],
    Rules =
    [
        DataAiRuleSpec.Required("customer-required", "customer_id"),
        DataAiRuleSpec.Between("amount-range", "amount", 0, 100000)
    ],
    MinimumQualityScore = 90
};

configuration.WriteJson("dataai-job.json");

Leave an optional output table blank only when that output must not be written. Do not put passwords or tokens in this JSON.

3. Build and launch the Spark command

var command = DataAiSparkSubmitBuilder.BuildQualityJob(
    "dataai-job.json",
    options: new DataAiSparkSubmitOptions
    {
        SparkSubmitExecutable = "spark-submit",
        Master = "local[*]",
        SparkArguments = ["--conf", "spark.sql.session.timeZone=UTC"]
    });

var start = new System.Diagnostics.ProcessStartInfo(command.FileName)
{
    UseShellExecute = false
};
foreach (string argument in command.Arguments)
    start.ArgumentList.Add(argument);

using var process = System.Diagnostics.Process.Start(start)
    ?? throw new InvalidOperationException("Spark could not be started.");
process.WaitForExit();
if (process.ExitCode != 0)
    throw new InvalidOperationException($"DataAI Spark job failed: {process.ExitCode}");

The command builder does not start a process, transmit data, or choose credentials. Customer code starts the process under an approved identity. The quality CLI overwrites each named output table, so use dedicated staging tables.

4. Validate and continue the pipeline

1. Require process exit code 0.
2. Query clean, rejected, findings, and profile tables.
3. Reconcile clean plus rejected counts with source counts and inspect _dataai_record_key.
4. Use a success precedence path to publish clean rows or trigger the next orchestrator step.
5. Send a failure to the customer's logging and incident system without copying sensitive rows into application logs.

5. Use advanced DataAI functions

The .NET helper exposes the function catalog but does not implement Spark DataFrame extensions. For analytics, market models, map readiness, insights, or matrix balancing, package a customer Java/Spark application that calls the DataAI Java API and use DataAiSparkSubmitBuilder.BuildLibraryApplication(...) to submit that application with the installed DataAI JARs.

Usage is successful when: the command exits successfully, only selected tables change, quality counts reconcile, the minimum-score gate behaves as designed, and retries do not create unintended output.