# DataAI ETL Databricks Installation and Usage

## 1. Confirm prerequisites

Use a Unity Catalog-enabled workspace and compute compatible with Java 17,
Spark 3.5.x, and Scala 2.12. Databricks Runtime 16.4 LTS with the Scala 2.12
image is the initial validation target. Do not claim support until the selected
runtime and access mode pass the provider checklist.

This package contains immutable `1.0.0` production-candidate artifacts.
Commercial deployment requires Yanbor release authorization, the customer's
applicable entitlement, and completion of the documented Databricks runtime
validation gates.

## 2. Create customer-controlled storage

Adapt `../sql/CREATE_CONSUMER_OUTPUTS.sql` and create a catalog, schema, and
volume under the customer's governance. Example names are intentionally
generic and may be replaced.

The customer controls grants. The identity running the notebooks needs:

- `USE CATALOG` on the selected catalog.
- `USE SCHEMA` on the selected schema.
- `READ VOLUME` for the delivered DataAI files.
- `CREATE TABLE` only if optional output persistence is enabled.

## 3. Upload the authorized `1.0.0` payload

Upload the package contents to a Unity Catalog volume. With the Databricks CLI:

```powershell
databricks fs mkdir dbfs:/Volumes/<catalog>/<schema>/<volume>/dataai-etl
databricks fs cp .\lib dbfs:/Volumes/<catalog>/<schema>/<volume>/dataai-etl/lib -r --overwrite
databricks fs cp .\data dbfs:/Volumes/<catalog>/<schema>/<volume>/dataai-etl/data -r --overwrite
databricks fs cp .\docs dbfs:/Volumes/<catalog>/<schema>/<volume>/dataai-etl/docs -r --overwrite
```

Do not upload Maven credentials, Databricks tokens, license certificates,
customer agreements, or customer data with the delivered payload.

## 4. Install the DataAI JARs

Install these four libraries on the cluster or Job compute:

```text
/Volumes/<catalog>/<schema>/<volume>/dataai-etl/lib/dataai-spark-api-1.0.0.jar
/Volumes/<catalog>/<schema>/<volume>/dataai-etl/lib/dataai-spark-quality-1.0.0.jar
/Volumes/<catalog>/<schema>/<volume>/dataai-etl/lib/dataai-spark-core-1.0.0.jar
/Volumes/<catalog>/<schema>/<volume>/dataai-etl/lib/dataai-spark-functions-1.0.0.jar
```

In the Databricks UI, edit the compute resource, open **Libraries**, select
**Install new**, choose the Unity Catalog volume source, and add each JAR.
Restart compute after installation.

For standard access mode, an administrator may need to add the volume path to
the library allowlist. The DataAI classes use named Java packages.

The CLI JAR is optional and is intended for configuration-driven Spark jobs,
not for the notebook examples.

## 5. Import the notebooks

Import the `notebooks/` directory into a customer-controlled workspace folder.
Run them in this order:

1. `00_INSTALL_AND_VERIFY.py`
2. `01_QUALITY_PIPELINE.py`
3. `02_ANALYTICS_AND_MARKET.py`
4. `03_MATRIX_BALANCING.py`
5. `04_PUBLISH_FOR_BI.py`

Set the `dataai_base_path` widget to:

```text
/Volumes/<catalog>/<schema>/<volume>/dataai-etl
```

## 6. Verify the runtime

The first notebook checks that the four required DataAI classes are visible,
prints the DataAI development version, and reports the Spark, Scala, and Java
runtime versions. A missing class normally means the JAR was not installed on
the attached compute or the compute was not restarted.

## 7. Run the quality example

The quality notebook reads fictional orders, normalizes values, creates a
record key, profiles fields, applies validation rules, and returns clean rows,
rejected rows, findings, profiles, and a run summary.

The notebook does not persist output by default. Set `persist_results` to
`true` only after specifying a customer-controlled catalog and schema.

## 8. Run analytics and matrix examples

The analytics notebook calls the same Java APIs available to Java/Scala Spark
jobs and displays DataFrames through PySpark. It demonstrates grouped totals,
market segments, scenarios, and automatic quality diagnostics.

The matrix notebook balances fictional region/channel cells against row and
column controls. It exposes iteration count, maximum error, convergence status,
balanced values, coefficients, and differences.

## 9. Publish optional governed outputs

The BI notebook creates views only when `publish_views` is explicitly set to
`true`. It never chooses a catalog, schema, table name, or grant automatically.
The resulting views can be queried from Databricks SQL or connected to Power BI
and Tableau using their native Databricks connectors.

## 10. Production fulfillment

For licensed production use, obtain the authorized, signed `1.0.0` artifacts
from Yanbor LLC's authenticated Maven repository or an approved customer
artifact repository. Pin the exact version in Jobs and cluster policies. Do
not overwrite an existing production version.

## Troubleshooting

### `JavaPackage` is not callable or class is missing

Install all four required JARs on the attached compute and restart it. Confirm
the volume path and allowlist settings.

### `UnsupportedClassVersionError`

The compute JVM is older than Java 17. Select a Java 17-compatible runtime.

### Scala linkage errors

Use the Scala 2.12 runtime image. Do not use a Scala 2.13-only or Spark 4-only
runtime with the current artifacts.

### Writes fail

Keep `persist_results=false`, or grant the notebook identity the customer-
approved Unity Catalog privileges. DataAI does not bypass Unity Catalog.

### Matrix does not converge

Check that row and column targets have equal totals, controls cover the input
keys, and the iteration and tolerance settings are suitable.
