Latu.ML.Stat (latu_ml v0.4.2)

pyspark.ml.stat: three tests that run where the data is, and Summarizer.

The tests are chi-square, correlation and Kolmogorov-Smirnov. None of them fits anything and none caches anything — each is one method on the server's own helper object, and each hands back a lazy Latu.DataFrame with a row or two in it. Nothing reaches the server until you collect.

summary/3 and its ten shortcuts are Summarizer: aggregate statistics over a Vector column, as column expressions for Latu.agg/2.

Most of this reads a features column, which is a Vector — and a Vector column cannot be collected on 4.2.0 (docs/deviations.md). The tests' results are ordinary columns, so that only bites where you collect the frame you passed in rather than the one you got back. A summary's vector-valued metrics are Vectors too; Latu.ML.Functions.vector_to_array/2 reads them.

Latu.ML.helper/3 is the layer underneath the tests, and Latu.ML.helpers/0 the table they are generated from — except that these three are not generated: a flat module cannot have two functions called test, so they are named here.

Summary

Types

column()

A column, as anywhere in Latu.

metric()

One of Summarizer's ten metrics, in Elixir spelling.

Functions

chi_square_test(data, features_col, label_col, flatten \\ false)

Pearson's independence test for every feature against a label.

correlation(data, column, method \\ "pearson")

The correlation matrix of a column of Vectors.

count(features, opts \\ [])

Summarizer's count of a Vector column, as one expression.

kolmogorov_smirnov_test(data, sample_col, dist_name, params)

A one-sample, two-sided Kolmogorov-Smirnov test against a theoretical distribution.

max(features, opts \\ [])

Summarizer's max of a Vector column, as one expression.

mean(features, opts \\ [])

Summarizer's mean of a Vector column, as one expression.

min(features, opts \\ [])

Summarizer's min of a Vector column, as one expression.

norm_l1(features, opts \\ [])

Summarizer's normL1 of a Vector column, as one expression.

norm_l2(features, opts \\ [])

Summarizer's normL2 of a Vector column, as one expression.

num_nonzeros(features, opts \\ [])

Summarizer's numNonzeros of a Vector column, as one expression.

std(features, opts \\ [])

Summarizer's std of a Vector column, as one expression.

sum(features, opts \\ [])

Summarizer's sum of a Vector column, as one expression.

summary(features, metrics, opts \\ [])

Summary statistics over a Vector column, as one struct. Spark's Summarizer.

variance(features, opts \\ [])

Summarizer's variance of a Vector column, as one expression.

Types

column()

@type column() :: atom() | String.t()

A column, as anywhere in Latu.

metric()

@type metric() ::
  :mean
  | :sum
  | :variance
  | :std
  | :count
  | :num_nonzeros
  | :max
  | :min
  | :norm_l2
  | :norm_l1

One of Summarizer's ten metrics, in Elixir spelling.

Functions

chi_square_test(data, features_col, label_col, flatten \\ false)

@spec chi_square_test(Latu.DataFrame.t(), column(), column(), boolean()) ::
  Latu.DataFrame.t()

Pearson's independence test for every feature against a label.

A lazy builder. features_col is a Vector column and label_col any numeric one; both are treated as categorical, one contingency table per feature.

With flatten: false — the default — the answer is a single row of pValues, degreesOfFreedom and statistics, one entry per feature. With flatten: true it is one row per feature, carrying featureIndex, pValue, degreesOfFreedom and statistic.

Latu.ML.Stat.chi_square_test(features, :features, :label, true) |> Latu.collect()

correlation(data, column, method \\ "pearson")

@spec correlation(Latu.DataFrame.t(), column(), String.t()) :: Latu.DataFrame.t()

The correlation matrix of a column of Vectors.

A lazy builder. method is "pearson" or "spearman"; the answer is one row and one column, named for the method and the column, holding a Matrix.

Spearman is a rank correlation and needs a sort per column, so cache the input before asking for one if the lineage is expensive to recompute.

Latu.ML.Stat.correlation(features, :features)

count(features, opts \\ [])

@spec count(column(), keyword()) :: Latu.Plan.expression()

Summarizer's count of a Vector column, as one expression.

summary/3 for the one metric, with its count field read out. Same :weight option.

kolmogorov_smirnov_test(data, sample_col, dist_name, params)

@spec kolmogorov_smirnov_test(Latu.DataFrame.t(), column(), String.t(), [number()]) ::
  Latu.DataFrame.t()

A one-sample, two-sided Kolmogorov-Smirnov test against a theoretical distribution.

A lazy builder. dist_name is "norm" — the only one Spark supports — and params are that distribution's, mean then standard deviation. The answer is one row of pValue and statistic.

Latu.ML.Stat.kolmogorov_smirnov_test(sample, :value, "norm", [0.0, 1.0])

PySpark takes those parameters as varargs (test(df, col, "norm", 0.0, 1.0)); here they are a list, because a trailing list reads better in Elixir than a trailing splat.

max(features, opts \\ [])

@spec max(column(), keyword()) :: Latu.Plan.expression()

Summarizer's max of a Vector column, as one expression.

summary/3 for the one metric, with its max field read out. Same :weight option.

mean(features, opts \\ [])

@spec mean(column(), keyword()) :: Latu.Plan.expression()

Summarizer's mean of a Vector column, as one expression.

summary/3 for the one metric, with its mean field read out. Same :weight option.

min(features, opts \\ [])

@spec min(column(), keyword()) :: Latu.Plan.expression()

Summarizer's min of a Vector column, as one expression.

summary/3 for the one metric, with its min field read out. Same :weight option.

norm_l1(features, opts \\ [])

@spec norm_l1(column(), keyword()) :: Latu.Plan.expression()

Summarizer's normL1 of a Vector column, as one expression.

summary/3 for the one metric, with its normL1 field read out. Same :weight option.

norm_l2(features, opts \\ [])

@spec norm_l2(column(), keyword()) :: Latu.Plan.expression()

Summarizer's normL2 of a Vector column, as one expression.

summary/3 for the one metric, with its normL2 field read out. Same :weight option.

num_nonzeros(features, opts \\ [])

@spec num_nonzeros(column(), keyword()) :: Latu.Plan.expression()

Summarizer's numNonzeros of a Vector column, as one expression.

summary/3 for the one metric, with its numNonzeros field read out. Same :weight option.

std(features, opts \\ [])

@spec std(column(), keyword()) :: Latu.Plan.expression()

Summarizer's std of a Vector column, as one expression.

summary/3 for the one metric, with its std field read out. Same :weight option.

sum(features, opts \\ [])

@spec sum(column(), keyword()) :: Latu.Plan.expression()

Summarizer's sum of a Vector column, as one expression.

summary/3 for the one metric, with its sum field read out. Same :weight option.

summary(features, metrics, opts \\ [])

@spec summary(column(), [metric()], keyword()) :: Latu.Plan.expression()

Summary statistics over a Vector column, as one struct. Spark's Summarizer.

alias Latu.ML.Stat

Latu.agg(scored, s: Stat.summary(:features, [:mean, :count]))

An aggregate expression, so it belongs in Latu.agg/2. The struct carries one field per metric under Spark's own spelling (numNonzeros, normL2), and Latu.Column.get_field/2 reads one out; mean/2 and its nine siblings do that for a single metric in one call.

Every metric but :count is a Vector, which Latu.collect/2 refuses. Read it through Latu.ML.Functions.vector_to_array/2 or Latu.to_nx/2:

Latu.agg(scored, mean: Functions.vector_to_array(Stat.mean(:features)))

The wire is aggregate_metrics, a function in Spark's internal registry with the metric names as an array of literals; an unknown name is refused here, naming the ten.

Options

:weight — a column, or a number, weighting each row. Defaults to 1.0, as PySpark's does.

variance(features, opts \\ [])

@spec variance(column(), keyword()) :: Latu.Plan.expression()

Summarizer's variance of a Vector column, as one expression.

summary/3 for the one metric, with its variance field read out. Same :weight option.