Use DataAI ETL with Databricks

Execution model: DataAI runs in customer Databricks compute with JARs installed from an approved Unity Catalog volume or private repository. The supplied notebooks demonstrate quality, analytics/market functions, matrix balancing, and governed BI outputs.

1. Prepare a governed workspace

Use a customer-controlled Unity Catalog catalog, schema, volume, and workspace folder.
Attach the same-version DataAI API, quality, core, and functions JARs to approved compute or a Databricks Job.
Add the volume path to the standard-access-mode allowlist when required.
Restart compute after library installation.
Grant the job identity USE CATALOG, USE SCHEMA, READ VOLUME, source reads, and only approved output privileges.

2. Run the notebooks in order

00_INSTALL_AND_VERIFY.py — verify runtime versions, paths, and DataAI class loading.
01_QUALITY_PIPELINE.py — run normalization, profiling, rules, clean/rejected routing, and quality gates.
02_ANALYTICS_AND_MARKET.py — run selected analytics and market examples.
03_MATRIX_BALANCING.py — balance matrix cells against row/column controls and inspect convergence.
04_PUBLISH_FOR_BI.py — prepare optional governed BI views/tables.

Set dataai_base_path to the customer volume. Replace sample catalog/schema values with isolated customer-approved locations.

3. Control persistence

Leave persist_results and publish_views false during the first computational validation.
Review output catalog, schema, table names, retention, ownership, and grants.
Enable persistence only after approval.
Record run_id, library version, completion time, source version, and job run identifier with published outputs.

4. Create an operational Job

Create a Databricks Workflow/Job under a least-privilege service principal.
Pin the exact DataAI libraries and approved compute policy.
Sequence source preparation, DataAI processing, output validation, and BI refresh tasks.
Configure retries only after write behavior is idempotent or protected by run-specific tables/merge logic.
Send failures to customer monitoring without copying sensitive rows into notifications.

5. Validate

Verify class loading on driver and executors.
Reconcile source, clean, rejected, finding, and profile counts.
Test the minimum-quality gate and one intentionally failing rule.
Confirm matrix balancing reports convergence, iterations, and maximum error.
Validate Unity Catalog lineage, grants, row/column controls, and BI views.
Test a fresh compute restart so the result does not depend on notebook state.

A notebook result displayed successfully does not authorize publication. Customer code/settings choose every persisted table or view. Do not mix different DataAI JAR versions on the same compute.

Usage is successful when: notebook 00 verifies installation, all selected functions return expected results, optional persistence reaches only approved Unity Catalog objects, and a clean Databricks Job rerun is repeatable.