Check and Install DataAI ETL Prerequisites on Windows

Step 1.1 — Open PowerShell

1. Open the Windows Start menu.
2. Type PowerShell.
3. Open PowerShell directly from Start.
4. Copy and run the complete check below. The Spark check may take several seconds.

Step 1.2 — Run the prerequisite and folder inventory

$dotnetCommand = Get-Command dotnet -ErrorAction SilentlyContinue
$dotnetFolder = if ($dotnetCommand) { Split-Path -Parent $dotnetCommand.Source } else { $null }
$dotnet8 = $false
if ($dotnetCommand) {
    $dotnet8 = [bool](dotnet --list-sdks 2>$null | Where-Object { $_ -match '^8\.' })
}

$javaHome = $env:JAVA_HOME
if ([string]::IsNullOrWhiteSpace($javaHome)) {
    $javaHome = [Environment]::GetEnvironmentVariable("JAVA_HOME", "User")
}
$javaCommand = Get-Command java -ErrorAction SilentlyContinue
$javaExe = if ($javaCommand) {
    $javaCommand.Source
} elseif ($javaHome) {
    Join-Path $javaHome "bin\java.exe"
} else {
    $null
}
$javaFolderExists = [bool]($javaHome -and (Test-Path -LiteralPath (Join-Path $javaHome "bin\java.exe")))
$java17 = $false
if ($javaExe -and (Test-Path -LiteralPath $javaExe)) {
    $javaVersionText = (& $javaExe -version 2>&1 | Out-String)
    $java17 = $javaVersionText -match 'version "17\.'
}

$sparkHome = $env:SPARK_HOME
if ([string]::IsNullOrWhiteSpace($sparkHome)) {
    $sparkHome = [Environment]::GetEnvironmentVariable("SPARK_HOME", "User")
}
$sparkCommand = Get-Command spark-submit.cmd -ErrorAction SilentlyContinue
if (-not $sparkCommand) { $sparkCommand = Get-Command spark-submit -ErrorAction SilentlyContinue }
$sparkExe = if ($sparkCommand) {
    $sparkCommand.Source
} elseif ($sparkHome) {
    Join-Path $sparkHome "bin\spark-submit.cmd"
} else {
    $null
}
$sparkFolderExists = [bool]($sparkHome -and (Test-Path -LiteralPath (Join-Path $sparkHome "bin\spark-submit.cmd")))
$spark350 = $false
$scala212 = $false
$processJavaHomeBeforeCheck = $env:JAVA_HOME
$processSparkHomeBeforeCheck = $env:SPARK_HOME
if ([string]::IsNullOrWhiteSpace($env:JAVA_HOME) -and $javaHome) { $env:JAVA_HOME = $javaHome }
if ([string]::IsNullOrWhiteSpace($env:SPARK_HOME) -and $sparkHome) { $env:SPARK_HOME = $sparkHome }
if ($sparkExe -and (Test-Path -LiteralPath $sparkExe)) {
    $sparkVersionText = (& $sparkExe --version 2>&1 | Out-String)
    $spark350 = $sparkVersionText -match 'version 3\.5\.0'
    $scala212 = $sparkVersionText -match 'Scala version 2\.12\.'
}
if ($null -eq $processJavaHomeBeforeCheck) { Remove-Item Env:JAVA_HOME -ErrorAction SilentlyContinue } else { $env:JAVA_HOME = $processJavaHomeBeforeCheck }
if ($null -eq $processSparkHomeBeforeCheck) { Remove-Item Env:SPARK_HOME -ErrorAction SilentlyContinue } else { $env:SPARK_HOME = $processSparkHomeBeforeCheck }

$dotnetStatus = if ($dotnet8) { "READY" } else { "DOWNLOAD / INSTALL" }
$javaStatus = if ($java17 -and $javaFolderExists -and $javaCommand) {
    "READY"
} elseif ($java17 -and $javaFolderExists) {
    "CONFIGURE JAVA_HOME / PATH"
} else {
    "DOWNLOAD / INSTALL"
}
$sparkStatus = if ($spark350 -and $scala212 -and $sparkFolderExists -and $sparkCommand) {
    "READY"
} elseif ($spark350 -and $scala212 -and $sparkFolderExists) {
    "CONFIGURE SPARK_HOME / PATH"
} else {
    "DOWNLOAD / INSTALL"
}

@(
    [pscustomobject]@{
        Prerequisite = ".NET 8 SDK"; Status = $dotnetStatus
        Folder = if ($dotnetFolder) { $dotnetFolder } else { "<not found>" }
        FolderExists = [bool]($dotnetFolder -and (Test-Path -LiteralPath $dotnetFolder))
    }
    [pscustomobject]@{
        Prerequisite = "Java 17 JDK"; Status = $javaStatus
        Folder = if ($javaHome) { $javaHome } else { "<JAVA_HOME not set>" }
        FolderExists = $javaFolderExists
    }
    [pscustomobject]@{
        Prerequisite = "Spark 3.5.0 / Scala 2.12"; Status = $sparkStatus
        Folder = if ($sparkHome) { $sparkHome } else { "<SPARK_HOME not set>" }
        FolderExists = $sparkFolderExists
    }
) | Format-Table -AutoSize

Step 1.3 — Decide what to do

• READY: do not download or reinstall this prerequisite.
• CONFIGURE: the correct software folder exists, but the environment variables or Path need configuration.
• DOWNLOAD / INSTALL: the required version or installation folder was not found. Follow that product's download section.

Version matters: Java 8, 11, 21, or 25 does not satisfy this exact Java 17 prerequisite. Spark 4 or Spark built with Scala 2.13 does not satisfy the Spark 3.5.0/Scala 2.12 prerequisite. A .NET 8 runtime without an 8.0 SDK does not satisfy the SDK prerequisite.

Step 2.1 — Check the folders

$downloadFolder = Join-Path $env:USERPROFILE "Downloads"
$sparkRoot = "C:\Tools\Spark"
$sparkVersionFolder = Join-Path $sparkRoot "spark-3.5.0-bin-hadoop3"

@(
    [pscustomobject]@{ Purpose = "Downloads"; Folder = $downloadFolder; Exists = Test-Path -LiteralPath $downloadFolder }
    [pscustomobject]@{ Purpose = "Spark installation root"; Folder = $sparkRoot; Exists = Test-Path -LiteralPath $sparkRoot }
    [pscustomobject]@{ Purpose = "Extracted Spark 3.5.0"; Folder = $sparkVersionFolder; Exists = Test-Path -LiteralPath $sparkVersionFolder }
) | Format-Table -AutoSize

Step 2.2 — Create only missing working folders

The Microsoft .NET and Temurin Java installers create their own folders under C:\Program Files. Do not create those product folders manually. If the Downloads or Spark root folder is missing, open PowerShell as Administrator and run:

$downloadFolder = Join-Path $env:USERPROFILE "Downloads"
New-Item -ItemType Directory -Force -Path $downloadFolder
New-Item -ItemType Directory -Force -Path "C:\Tools\Spark"

The folder spark-3.5.0-bin-hadoop3 will be created when the Spark archive is extracted.

Step 3.1 — Download or install

Open PowerShell as Administrator and run:

winget install --id Microsoft.DotNet.SDK.8 --source winget

If WinGet is unavailable, use the official Microsoft .NET 8 download page. Under Build apps - SDK, download the Windows installer matching the customer's architecture. Do not select a runtime-only download.

Step 3.2 — Close, reopen, and verify

1. Wait for the installer to finish.
2. Close the elevated PowerShell window.
3. Exit the entire terminal-hosting application.
4. Open PowerShell directly from Start.
5. Run:

dotnet --list-sdks
Test-Path -LiteralPath "$env:ProgramFiles\dotnet\dotnet.exe"
Get-ChildItem -LiteralPath "$env:ProgramFiles\dotnet\sdk" -Directory -Filter "8.0.*"

At least one 8.0.x SDK directory must be listed. dotnet --version may show a newer SDK.

Step 4.1 — If Java 17 exists but needs configuration

If the result was CONFIGURE JAVA_HOME / PATH, locate the JDK directory containing bin\java.exe. JAVA_HOME points to the JDK directory, not its bin folder.

1. Press Windows key + R and enter sysdm.cpl.
2. Select Advanced and Environment Variables.
3. Create or edit the user variable JAVA_HOME with the full JDK folder.
4. Edit the user Path and add %JAVA_HOME%\bin.
5. Click OK in every window.
6. Exit every terminal host and reopen PowerShell from Start.

Step 4.2 — If Java 17 is missing, download the JDK

1. Open the official Eclipse Temurin download page.
2. Select JDK 17 - LTS.
3. Select Windows and the customer's architecture, normally x64.
4. Select JDK, not a JRE-only package.
5. Download and run the Windows MSI installer.
6. In Custom Setup, enable both Add to PATH and Set or update JAVA_HOME.
7. Complete the installer. It creates its directory, normally under C:\Program Files\Eclipse Adoptium.

See the official Temurin Windows MSI instructions for details.

Step 4.3 — Close, reopen, and verify

1. Close all terminals.
2. Exit Windows Terminal, Visual Studio, VS Code, or another terminal host completely.
3. Open PowerShell from Start.
4. Run:

$env:JAVA_HOME
where.exe java
Test-Path -LiteralPath (Join-Path $env:JAVA_HOME "bin\java.exe")
java -version

The folder test must return True, and Java must report 17.0.x.

Step 5.1 — If Spark exists but needs configuration

If the result was CONFIGURE SPARK_HOME / PATH, confirm that the existing Spark folder contains bin\spark-submit.cmd. Open Environment Variables, set SPARK_HOME to that folder, and add %SPARK_HOME%\bin to the user Path. Save all windows, exit the terminal host, and reopen PowerShell from Start.

Step 5.2 — If Spark is missing, download the correct files

• spark-3.5.0-bin-hadoop3.tgz
• spark-3.5.0-bin-hadoop3.tgz.sha512

Save both files in Downloads. Do not download the file ending in scala2.13.tgz. The archive without a Scala suffix is the required Scala 2.12 build.

Archived release: Spark 3.5.0 is historical. Use this exact version for DataAI ETL compatibility evaluation and apply the customer's security policies before production use.

Step 5.3 — Verify the SHA-512 checksum

$download = Join-Path $env:USERPROFILE "Downloads"
$archive = Join-Path $download "spark-3.5.0-bin-hadoop3.tgz"
$checksumFile = "$archive.sha512"

if (-not (Test-Path -LiteralPath $archive)) { throw "Spark archive was not found in Downloads." }
if (-not (Test-Path -LiteralPath $checksumFile)) { throw "Spark checksum file was not found in Downloads." }

$expected = ((Get-Content -LiteralPath $checksumFile -Raw) -split '\s+')[0].ToLowerInvariant()
$actual = (Get-FileHash -LiteralPath $archive -Algorithm SHA512).Hash.ToLowerInvariant()
if ($actual -ne $expected) { throw "Spark checksum mismatch. Download the archive again from Apache." }
Write-Host "Spark download checksum verified." -ForegroundColor Green

Step 5.4 — Create the folder and extract Spark

Open PowerShell as Administrator and run:

New-Item -ItemType Directory -Force -Path "C:\Tools\Spark"
tar -xzf "$env:USERPROFILE\Downloads\spark-3.5.0-bin-hadoop3.tgz" -C "C:\Tools\Spark"
Test-Path -LiteralPath "C:\Tools\Spark\spark-3.5.0-bin-hadoop3\bin\spark-submit.cmd"

The final command must return True.

Step 5.5 — Configure Spark and verify

1. Open Environment Variables with sysdm.cpl.
2. Create the user variable SPARK_HOME with value C:\Tools\Spark\spark-3.5.0-bin-hadoop3.
3. Add %SPARK_HOME%\bin to the user Path.
4. Click OK in every window.
5. Exit every terminal host and reopen PowerShell from Start.
6. Run:

$env:SPARK_HOME
where.exe spark-submit
Test-Path -LiteralPath (Join-Path $env:SPARK_HOME "bin\spark-submit.cmd")
spark-submit --version

The folder test must return True. Spark must report 3.5.0, Scala 2.12.x, and Java 17.0.x.

Write-Host ".NET 8 SDK" -ForegroundColor Cyan
dotnet --list-sdks
Test-Path -LiteralPath "$env:ProgramFiles\dotnet\dotnet.exe"

Write-Host "Java 17 JDK" -ForegroundColor Cyan
$env:JAVA_HOME
Test-Path -LiteralPath (Join-Path $env:JAVA_HOME "bin\java.exe")
java -version

Write-Host "Spark 3.5.0 and Scala 2.12" -ForegroundColor Cyan
$env:SPARK_HOME
Test-Path -LiteralPath (Join-Path $env:SPARK_HOME "bin\spark-submit.cmd")
spark-submit --version

Write-Host "Persistent user settings" -ForegroundColor Cyan
[Environment]::GetEnvironmentVariable("JAVA_HOME", "User")
[Environment]::GetEnvironmentVariable("SPARK_HOME", "User")

Write-Host "Persistent machine settings" -ForegroundColor Cyan
[Environment]::GetEnvironmentVariable("JAVA_HOME", "Machine")
[Environment]::GetEnvironmentVariable("SPARK_HOME", "Machine")

Write-Host "Current PowerShell settings" -ForegroundColor Cyan
$env:JAVA_HOME
$env:SPARK_HOME

$env:JAVA_HOME = if ([Environment]::GetEnvironmentVariable("JAVA_HOME", "User")) { [Environment]::GetEnvironmentVariable("JAVA_HOME", "User") } else { [Environment]::GetEnvironmentVariable("JAVA_HOME", "Machine") }
$env:SPARK_HOME = if ([Environment]::GetEnvironmentVariable("SPARK_HOME", "User")) { [Environment]::GetEnvironmentVariable("SPARK_HOME", "User") } else { [Environment]::GetEnvironmentVariable("SPARK_HOME", "Machine") }
$env:Path = "$env:JAVA_HOME\bin;$env:SPARK_HOME\bin;$env:Path"
java -version
spark-submit --version

& "C:\Tools\Spark\spark-3.5.0-bin-hadoop3\bin\spark-submit.cmd" --version