Package com.dataai.etl.spark.functions

Class MatrixFunctions

java.lang.Object

com.dataai.etl.spark.functions.MatrixFunctions

public final class MatrixFunctions
extends Object

Method Summary

Modifier and Type	Method	Description
static MatrixBalanceResult	balance(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input, String rowField, String columnField, String valueField, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> rowTargets, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> columnTargets, String targetColumn, int maximumIterations, double tolerance)
static MatrixBalanceResult	balanceWithMinimumDeparture(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input, String rowField, String columnField, String valueField, String rowOrdinalField, String columnOrdinalField, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> rowTargets, org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> columnTargets, String targetColumn, int maximumIterations, double tolerance, int rowSplit, int columnSplit)	Balances a matrix and then reproduces DataAI's minimum-departure adjustment for the top-right and bottom-left matrix regions.
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	crossTab(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input, String rowField, String columnField, String valueField, Aggregation aggregation)
static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	minimumDepartureAdjustment(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input, String rowOrdinalField, String columnOrdinalField, String startingValueField, String balancedValueField, int rowSplit, int columnSplit)	Adjusts the two off-diagonal regions of an already balanced matrix while retaining the starting matrix's difference between their totals.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

crossTab

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> crossTab(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input,
 String rowField,
 String columnField,
 String valueField,
 Aggregation aggregation)

balance

public static MatrixBalanceResult balance(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input,
 String rowField,
 String columnField,
 String valueField,
 org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> rowTargets,
 org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> columnTargets,
 String targetColumn,
 int maximumIterations,
 double tolerance)

balanceWithMinimumDeparture

public static MatrixBalanceResult balanceWithMinimumDeparture(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input,
 String rowField,
 String columnField,
 String valueField,
 String rowOrdinalField,
 String columnOrdinalField,
 org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> rowTargets,
 org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> columnTargets,
 String targetColumn,
 int maximumIterations,
 double tolerance,
 int rowSplit,
 int columnSplit)

Balances a matrix and then reproduces DataAI's minimum-departure adjustment for the top-right and bottom-left matrix regions.

Row and column ordinals must be non-negative integral fields. Spark DataFrames have no intrinsic row order, so callers must provide these fields explicitly. The split values are zero-based boundaries: the top-right region has rowOrdinal < rowSplit and columnOrdinal >= columnSplit; the bottom-left region uses the opposite comparisons.

minimumDepartureAdjustment

public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> minimumDepartureAdjustment(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> input,
 String rowOrdinalField,
 String columnOrdinalField,
 String startingValueField,
 String balancedValueField,
 int rowSplit,
 int columnSplit)

Adjusts the two off-diagonal regions of an already balanced matrix while retaining the starting matrix's difference between their totals.

The returned DataFrame has the same columns as the input, with only balancedValueField replaced. This method performs a bounded aggregate action to calculate the four region totals.