# `Latu.ML.Stat`
[🔗](https://github.com/zero-one-group/latu_ml/blob/v0.4.2/lib/latu/ml/stat.ex#L1)

`pyspark.ml.stat`: three tests that run where the data is, and `Summarizer`.

The tests are chi-square, correlation and Kolmogorov-Smirnov. None of them fits anything and
none caches anything — each is one method on the server's own helper object, and each hands
back a lazy `Latu.DataFrame` with a row or two in it. Nothing reaches the server until you
collect.

`summary/3` and its ten shortcuts are `Summarizer`: aggregate statistics over a `Vector`
column, as column expressions for `Latu.agg/2`.

Most of this reads a `features` column, which is a `Vector` — and a Vector column cannot
be collected on 4.2.0 (`docs/deviations.md`). The tests' *results* are ordinary columns, so
that only bites where you collect the frame you passed in rather than the one you got back.
A summary's vector-valued metrics are Vectors too; `Latu.ML.Functions.vector_to_array/2`
reads them.

`Latu.ML.helper/3` is the layer underneath the tests, and `Latu.ML.helpers/0` the table they
are generated from — except that these three are not generated: a flat module cannot have two
functions called `test`, so they are named here.

# `column`

```elixir
@type column() :: atom() | String.t()
```

A column, as anywhere in Latu.

# `metric`

```elixir
@type metric() ::
  :mean
  | :sum
  | :variance
  | :std
  | :count
  | :num_nonzeros
  | :max
  | :min
  | :norm_l2
  | :norm_l1
```

One of `Summarizer`'s ten metrics, in Elixir spelling.

# `chi_square_test`

```elixir
@spec chi_square_test(Latu.DataFrame.t(), column(), column(), boolean()) ::
  Latu.DataFrame.t()
```

Pearson's independence test for every feature against a label.

A lazy builder. `features_col` is a `Vector` column and `label_col` any numeric one; both are
treated as categorical, one contingency table per feature.

With `flatten: false` — the default — the answer is a single row of `pValues`,
`degreesOfFreedom` and `statistics`, one entry per feature. With `flatten: true` it is one
row per feature, carrying `featureIndex`, `pValue`, `degreesOfFreedom` and `statistic`.

    Latu.ML.Stat.chi_square_test(features, :features, :label, true) |> Latu.collect()

# `correlation`

```elixir
@spec correlation(Latu.DataFrame.t(), column(), String.t()) :: Latu.DataFrame.t()
```

The correlation matrix of a column of `Vector`s.

A lazy builder. `method` is `"pearson"` or `"spearman"`; the answer is one row and one
column, named for the method and the column, holding a `Matrix`.

Spearman is a rank correlation and needs a sort per column, so cache the input before asking
for one if the lineage is expensive to recompute.

    Latu.ML.Stat.correlation(features, :features)

# `count`

```elixir
@spec count(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `count` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `count` field read out. Same `:weight` option.

# `kolmogorov_smirnov_test`

```elixir
@spec kolmogorov_smirnov_test(Latu.DataFrame.t(), column(), String.t(), [number()]) ::
  Latu.DataFrame.t()
```

A one-sample, two-sided Kolmogorov-Smirnov test against a theoretical distribution.

A lazy builder. `dist_name` is `"norm"` — the only one Spark supports — and `params` are that
distribution's, mean then standard deviation. The answer is one row of `pValue` and
`statistic`.

    Latu.ML.Stat.kolmogorov_smirnov_test(sample, :value, "norm", [0.0, 1.0])

PySpark takes those parameters as varargs (`test(df, col, "norm", 0.0, 1.0)`); here they are
a list, because a trailing list reads better in Elixir than a trailing splat.

# `max`

```elixir
@spec max(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `max` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `max` field read out. Same `:weight` option.

# `mean`

```elixir
@spec mean(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `mean` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `mean` field read out. Same `:weight` option.

# `min`

```elixir
@spec min(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `min` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `min` field read out. Same `:weight` option.

# `norm_l1`

```elixir
@spec norm_l1(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `normL1` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `normL1` field read out. Same `:weight` option.

# `norm_l2`

```elixir
@spec norm_l2(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `normL2` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `normL2` field read out. Same `:weight` option.

# `num_nonzeros`

```elixir
@spec num_nonzeros(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `numNonzeros` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `numNonzeros` field read out. Same `:weight` option.

# `std`

```elixir
@spec std(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `std` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `std` field read out. Same `:weight` option.

# `sum`

```elixir
@spec sum(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `sum` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `sum` field read out. Same `:weight` option.

# `summary`

```elixir
@spec summary(column(), [metric()], keyword()) :: Latu.Plan.expression()
```

Summary statistics over a `Vector` column, as one struct. Spark's `Summarizer`.

    alias Latu.ML.Stat

    Latu.agg(scored, s: Stat.summary(:features, [:mean, :count]))

An aggregate expression, so it belongs in `Latu.agg/2`. The struct carries one field per
metric under Spark's own spelling (`numNonzeros`, `normL2`), and `Latu.Column.get_field/2`
reads one out; `mean/2` and its nine siblings do that for a single metric in one call.

Every metric but `:count` is a `Vector`, which `Latu.collect/2` refuses. Read it through
`Latu.ML.Functions.vector_to_array/2` or `Latu.to_nx/2`:

    Latu.agg(scored, mean: Functions.vector_to_array(Stat.mean(:features)))

The wire is `aggregate_metrics`, a function in Spark's internal registry with the metric
names as an array of literals; an unknown name is refused here, naming the ten.

## Options

  * `:weight` — a column, or a number, weighting each row. Defaults to `1.0`, as PySpark's
    does.

# `variance`

```elixir
@spec variance(column(), keyword()) :: Latu.Plan.expression()
```

`Summarizer`'s `variance` of a `Vector` column, as one expression.

`summary/3` for the one metric, with its `variance` field read out. Same `:weight` option.

---

*Consult [api-reference.md](api-reference.md) for complete listing*