# `Latu.ML`
[🔗](https://github.com/zero-one-group/latu_ml/blob/v0.4.2/lib/latu/ml.ex#L1)

Spark MLlib from Elixir, over Spark Connect.

The algorithms run on the cluster, where the data is; the Elixir side holds plans and
handles. That is the whole difference from `Scholar`, which runs on one node in memory and
hands you a model you can inspect and ship. Reach for Scholar when the training set fits a
node; reach for this when it does not, or when the features already live in a lakehouse.

## The shapes

Two, and Spark chose them, not this package:

  * **Actions** reach the server and return `{:ok, _} | {:error, %Latu.Error{}}`, with `!`
    twins that raise — `fit/2`, `evaluate/2`, `attribute/2`, `save/3`, `load/3`, `delete/1`.
  * **Lazy builders** take a struct and hand one back, touching nothing — `transform/2`
    returns a `Latu.DataFrame` and no bytes move until you collect it.

## A model is not a value

`fit/2` hands back a `Latu.ML.Model`: a reference into a per-session cache on the server,
which offloads to disk under memory pressure and is gone when the session ends. Nothing on
the BEAM can notice a model going away, and there is no finalizer to lean on, so the caller
says when it ends — `with_model/3` where the model's life fits an expression, `fit/2` plus
`delete/1` where it does not.

## Example

    alias Latu.ML
    alias Latu.ML.{Classification, Feature}

    df = Latu.sql!(session, "SELECT * FROM training")

    assembler = Feature.vector_assembler(input_cols: [:x1, :x2], output_col: :features)
    features = ML.transform(assembler, df)

    lr = Classification.logistic_regression(max_iter: 10, reg_param: 0.01)

    {:ok, coefficients} =
      ML.with_model(lr, features, fn model ->
        ML.attribute(model, :coefficients)
      end)

# `cache_entry`

```elixir
@type cache_entry() :: %{id: String.t(), class: String.t(), size: non_neg_integer()}
```

One object the session's ML cache is holding, as `cache_info/1` reports it.

`size` is the bytes the cache charged the object: `0` for a summary always, and `0` for a
model too where the session has memory control off.

# `deletable`

```elixir
@type deletable() :: Latu.ML.Model.t() | Latu.ML.PipelineModel.t() | searched()
```

Anything `delete/1` can release: a model, a composite that owns models, or a list.

# `loadable`

```elixir
@type loadable() :: atom() | {String.t(), Latu.ML.Plan.kind()}
```

How `load/3` names what is at a path — an operator name from the registry, a generated model
module, or `{class, kind}` for a class this package does not know.

# `param_map`

```elixir
@type param_map() :: [{String.t(), atom(), term()}]
```

One combination from a grid: which operator's param, by uid, and what to set it to.

A uid rather than a struct because the operator being tuned is often a stage of a pipeline,
and because it is how Spark's saved format keys them (`parent` and `name`). Build these with
`param_grid/2` rather than by hand.

# `persistable`

```elixir
@type persistable() ::
  Latu.ML.Model.t()
  | Latu.ML.Estimator.t()
  | Latu.ML.Transformer.t()
  | Latu.ML.Evaluator.t()
  | Latu.ML.Pipeline.t()
  | Latu.ML.PipelineModel.t()
```

Anything `save/3` writes and `load/3` answers with: a fitted model, or an operator that has
not been fitted.

# `searched`

```elixir
@type searched() ::
  Latu.ML.CrossValidatorModel.t() | Latu.ML.TrainValidationSplitModel.t()
```

What a grid search hands back: the winner, and the score of everything it tried.

`transform/2` scores with the winner, and `delete/1` releases it along with any fold models
the search was asked to keep.

# `assign_clusters`

```elixir
@spec assign_clusters(Latu.ML.Helper.t(), Latu.DataFrame.t()) :: Latu.DataFrame.t()
```

Assign a cluster to every vertex of an affinity matrix — Power Iteration Clustering.

A lazy builder, not an action: the frame it hands back carries a `Fetch` on the server's own
helper object and nothing has run. What goes in is an edge list — `src`, `dst` and `weight`
by default — and what comes back is `id` and `cluster`.

`PowerIterationClustering` is neither fitted nor applied; this is the one thing it does, and
that is why it is a verb here rather than a model you keep. Its params ride the call as
positional arguments, so every one is sent whether you set it or not.

    pic = Latu.ML.Clustering.power_iteration_clustering(k: 2, max_iter: 10)
    clusters = Latu.ML.assign_clusters(pic, edges)

# `attribute`

```elixir
@spec attribute(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t()) ::
  {:ok, term()} | {:error, Latu.Error.t()}
```

Read a model attribute whose answer is a value — a coefficient, an intercept, a count.

An action. A `Vector` or `Matrix` comes back as an `Nx.Tensor`, or a
`Latu.ML.SparseVector` where densifying would be this package's decision rather than yours.

**Either spelling.** `Latu.ML.attributes/1` lists snake_case names and the server's allowlist
is camelCase, so `:area_under_roc` and `"areaUnderROC"` both work and the table is a list of
things you can actually ask for. A name on the allowlist is taken as written, so a wire name
never means anything else.

A name neither spelling accounts for goes to the server as written and comes back as
`CONNECT_ML.ATTRIBUTE_NOT_ALLOWED`: a typo is its error and not this package's, and that is
also what keeps a class the registry does not know reachable at all. The generated accessor
modules — `Latu.ML.Classification.LogisticRegressionModel` and its 41 siblings — are that
allowlist as named functions and are what to reach for first; this is the layer underneath.

# `attribute`

```elixir
@spec attribute(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t(), [term()]) ::
  {:ok, term()} | {:error, Latu.Error.t()}
```

Read an attribute that takes arguments — `predict`, `fMeasureByLabel`, `recommendForAllUsers`.

An action, and the same `Fetch` as `attribute/2` with the arguments attached to its last
method. **The types are not yours to state**: `priv/ml_attributes.exs` records them from
PySpark's own annotations, so `predict` knows it wants a `Vector` and `recommendForAllUsers`
an `int`, and a wrong count or a wrong kind is refused here rather than on the server.

The generated accessors are these too, and are what to reach for first —
`Latu.ML.Classification.LogisticRegressionModel.predict/2` is this call with the class
checked and the arguments named.

    {:ok, label} = Latu.ML.attribute(model, "predict", [Nx.tensor([1.0, 0.0])])
    {:ok, f1} = Latu.ML.attribute(summary, "fMeasureByLabel", [1.0])

A DataFrame argument is passed as one and rides the relation arm — `Latu.ML.attribute(model,
"evaluateEachIteration", [frame, "logLoss"])`. Attributes that answer with a *frame* go to
`attribute_frame/3` instead. The `evaluate` **attribute** — a model's, not the evaluator verb
`evaluate/2` — answers with another cached object rather than a value, and has no verb of its
own: asking for it here reaches the server and comes back as a protocol error naming the arm
it answered on.

Needs the model's class, since that is what the types are recorded against. A model built by
`Latu.ML.Estimator.new/2` without one has nothing to look them up in.

# `attribute!`

```elixir
@spec attribute!(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t()) ::
  term()
```

See `attribute/2`. Raises instead of returning an error.

# `attribute!`

```elixir
@spec attribute!(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t(), [
  term()
]) :: term()
```

See `attribute/3`. Raises instead of returning an error.

# `attribute_frame`

```elixir
@spec attribute_frame(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t()) ::
  Latu.DataFrame.t()
```

Read a model attribute whose answer is a DataFrame — `gaussiansDF`, `itemFactors` and their
kind.

A lazy builder, not an action: the same `Fetch` `attribute/2` sends, wrapped as a relation
rather than a command, so this hands back a `Latu.DataFrame` and nothing has run.

Which attributes answer this way is not a judgement call — the registry records it from
PySpark's own `try_remote_attribute_relation` decorator, and the generated accessors are
split by it. Asking for a literal-valued attribute here builds a relation the server will
refuse.

# `attribute_frame`

```elixir
@spec attribute_frame(Latu.ML.Model.t() | Latu.ML.Summary.t(), atom() | String.t(), [
  term()
]) ::
  Latu.DataFrame.t()
```

A frame-valued attribute that takes arguments — `describeTopics`, `recommendForAllUsers`,
`approxNearestNeighbors`.

`attribute_frame/2` and `attribute/3` in one: still a lazy builder, still typed from
`priv/ml_attributes.exs` rather than from the caller.

    users = Latu.ML.attribute_frame(model, "recommendForAllUsers", [5])
    near = Latu.ML.attribute_frame(model, "approxNearestNeighbors", [df, key, 3, "distance"])

# `attributes`

```elixir
@spec attributes(
  module()
  | Latu.ML.Model.t()
  | Latu.ML.Summary.t()
  | String.t()
  | nil
) :: [map()] | nil
```

What the server will let you `Fetch` from a class, and how each one answers.

Takes a generated accessor module, a fitted `Latu.ML.Model`, a `Latu.ML.Summary`, or a JVM
class name. The server's allowlist is inherited, so this is the union up the class's
hierarchy rather than what its own entry says.

Where the class is not known — an `LDA` fit, or a summary whose class the fitted model picks
— this answers with what **every** class it could be allows, since only those are certain to
be fetchable. Name the class directly to see its own, wider list.

    iex> Latu.ML.attributes(Latu.ML.Clustering.KMeansModel) |> Enum.map(& &1.name)
    [:cluster_center_matrix, :has_summary, :num_features, :predict, :summary, :to_string]

# `best_index`

```elixir
@spec best_index(
  Latu.ML.CrossValidatorModel.t()
  | Latu.ML.TrainValidationSplitModel.t()
) ::
  non_neg_integer()
```

Which param map won, as an index into the validator's `param_maps`.

Pure. The metric is maximised or minimised according to `larger_better?/1` on the evaluator
that scored it, and ties go to the **first** — so the order `param_grid/2` built the grid in
is what separates two param maps that scored the same.

# `cache_info`

```elixir
@spec cache_info(Latu.Session.t()) ::
  {:ok, [cache_entry()]} | {:error, Latu.Error.t()}
```

Everything the session's ML cache is holding: one entry per model and summary.

An action. Each entry is `%{id: ..., class: ..., size: ...}` — the reference a
`%Latu.ML.Model{}` carries, the JVM class, and the size in bytes the cache charged it. The
server renders these as JSON strings and this decodes them, so a shape the server changes
arrives as `{:error, %Latu.Error{kind: :protocol}}` rather than as a surprise.

`size` is `0` for a summary, and `0` for a model too when the session has memory control
turned off: `MLCache` only measures what it is going to budget.

    {:ok, held} = Latu.ML.cache_info(session)
    Enum.map(held, & &1.class)

Takes a session rather than a model — the cache belongs to the session, and the point of
asking is usually that you have lost track of what is in it.

# `cache_info!`

```elixir
@spec cache_info!(Latu.Session.t()) :: [cache_entry()]
```

See `cache_info/1`. Raises instead of returning an error.

# `clean_cache`

```elixir
@spec clean_cache(Latu.Session.t()) ::
  {:ok, non_neg_integer()} | {:error, Latu.Error.t()}
```

Drop everything the session's ML cache holds, and say how many objects that was.

An action, and a blunt one: `MLCache.clear` empties the map and the offload directory
together, so **every** model and summary in the session goes at once and every
`%Latu.ML.Model{}` still in hand becomes a reference to nothing — the next fetch on one
answers `CONNECT_ML.CACHE_INVALID`.

`delete/1` and `with_model/3` are the ordinary way to give a model back. This is for a
session you are reusing and want to start clean in, and for the end of a long-lived one.

# `clean_cache!`

```elixir
@spec clean_cache!(Latu.Session.t()) :: non_neg_integer()
```

See `clean_cache/1`. Raises instead of returning an error.

# `cross_validator`

```elixir
@spec cross_validator(keyword()) :: Latu.ML.CrossValidator.t()
```

A k-fold cross-validation, as data. See `Latu.ML.CrossValidator`.

A builder, not an action: nothing here reaches a server and no session is needed, because a
validator has no wire form at all. `fit/2` is what searches.

    cv =
      ML.cross_validator(
        estimator: lr,
        param_maps: ML.param_grid(lr, reg_param: [0.1, 0.01]),
        evaluator: Evaluation.binary_classification_evaluator(),
        num_folds: 3,
        seed: 42
      )

`estimator:`, `param_maps:` and `evaluator:` are required. `num_folds:` defaults to 3,
`seed:` to none — and with no seed the fold cut differs between builds, so pass one for a
search you can repeat. `fold_col:` names a column that already holds a fold number, in which
case no draw is added at all. `collect_sub_models: true` keeps every fold's models instead of
releasing them as they are scored.

# `delete`

```elixir
@spec delete(deletable() | [deletable()]) :: :ok | {:error, Latu.Error.t()}
```

Release a model, or a list of them, from the server's ML cache.

An action, and the explicit half of the resource rule. The server answers with the references
it dropped rather than staying silent, so a failure here is a real one.

**Everything goes in one `Delete`**, whatever shape it arrives in. A list is what
`attribute(model, "trees")` needs — the server registers every sub-model it hands back, so a
hundred-tree forest is a hundred cache entries and giving them back one at a time is a hundred
round trips. A composite is flattened: a `PipelineModel` owns the models inside it, nested
pipelines included, and a search result owns its winner and the fold models where it was asked
to keep them. And a **mixed list** of any of those is fine, because a script that fitted a
pipeline and a search writes `delete([fitted, best])`.

Inside a composite, a stage that holds nothing on the server is skipped rather than refused,
so deleting a pipeline of transformers is `:ok` and sends nothing. **At the top level the
contract is strict**: anything that is not one of the four is refused by name, because a
silent `:ok` for an argument this cannot release is the worse answer.

# `delete!`

```elixir
@spec delete!(deletable() | [deletable()]) :: :ok
```

See `delete/1`. Raises instead of returning an error.

# `error_kind`

```elixir
@spec error_kind(Latu.Error.t()) :: atom() | nil
```

Which ML failure this is, as an atom, or `nil` for anything that is not one.

Match on this rather than on `error_class`, and never on the message. The class strings carry
a `CONNECT_ML.` prefix that Spark's own documentation leaves off, so a matcher written from
the docs compares against a string the server never sends — and the messages are worse: see
`hint/1`.

    case Latu.ML.attribute(model, :coefficients) do
      {:ok, value} -> value
      {:error, error} ->
        case Latu.ML.error_kind(error) do
          :cache_invalid -> refit()
          _other -> raise error
        end
    end

The five: `:cache_invalid` (the object is gone), `:attribute_not_allowed` (the name is not on
this class's allowlist), `:summary_lost` (the cache dropped a summary), `:model_too_large`
and `:cache_full` (a fit refused against a budget). A reference that never existed at all
arrives as a bare `INTERNAL_ERROR` with no class, and answers `nil` here.

# `evaluate`

```elixir
@spec evaluate(Latu.ML.Evaluator.t(), Latu.DataFrame.t()) ::
  {:ok, float()} | {:error, Latu.Error.t()}
```

Score a frame with an evaluator: one number comes back.

An action, and the third of Spark's three verbs. The frame is where the session comes from,
as it is for `fit/2`, and it is a frame a model has already scored — an evaluator reads
`prediction_col` and `label_col`, not features. Which number it is is the evaluator's
`metric_name`, whose default is its own: `areaUnderROC` for a binary classifier, `rmse` for
a regressor.

    scored = ML.transform(model, test)
    {:ok, auc} = ML.evaluate(Evaluation.binary_classification_evaluator(), scored)

Nothing is cached: unlike a fit, an evaluate answers with the metric itself.

# `evaluate!`

```elixir
@spec evaluate!(Latu.ML.Evaluator.t(), Latu.DataFrame.t()) :: float()
```

See `evaluate/2`. Raises instead of returning an error.

# `find_frequent_sequential_patterns`

```elixir
@spec find_frequent_sequential_patterns(Latu.ML.Helper.t(), Latu.DataFrame.t()) ::
  Latu.DataFrame.t()
```

Find the frequent sequential patterns in a frame of sequences — PrefixSpan.

`assign_clusters/2`'s twin in shape: a lazy builder over the helper object, for the other
operator that is neither fitted nor applied. The input column is `sequence`, an
`array<array<T>>`; what comes back is `sequence` and `freq`.

    ps = Latu.ML.FPM.prefix_span(min_support: 0.5, max_pattern_length: 5)
    patterns = Latu.ML.find_frequent_sequential_patterns(ps, sequences)

# `fit`

```elixir
@spec fit(Latu.ML.Estimator.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.Model.t()} | {:error, Latu.Error.t()}
@spec fit(Latu.ML.Pipeline.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.PipelineModel.t()} | {:error, Latu.Error.t()}
@spec fit(Latu.ML.CrossValidator.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.CrossValidatorModel.t()} | {:error, Latu.Error.t()}
@spec fit(Latu.ML.TrainValidationSplit.t(), Latu.DataFrame.t()) ::
  {:ok, Latu.ML.TrainValidationSplitModel.t()} | {:error, Latu.Error.t()}
```

Fit an estimator, or a whole pipeline, on a frame.

An action: the fit runs on the server and the result is a handle. The frame is where the
session comes from, so there is no session argument.

The model carries the estimator's uid and params, because a `Fit` answers with an object
reference and nothing else — no uid, no params, no warning.

## A pipeline

A `Latu.ML.Pipeline` is not a server operator: there is no `Fit` for one on the wire, so this
folds over its stages and does one action per estimator, exactly as PySpark does. What comes
back is a `Latu.ML.PipelineModel` that **owns** every model it fitted — `delete/1` on it
releases them all, and `with_model/3` brackets the whole thing.

The fold has one surprise worth stating. The frame is transformed by each stage on the way
through, but only **up to the last estimator**: stages after it are carried into the pipeline
model untouched, because there is nothing left to learn from them. And if a later stage fails,
the models fitted before it are released before the error is returned, so a half-built
pipeline leaks nothing.

    {:ok, model} = ML.fit(ML.pipeline([assembler, scaler, lr]), training)

## A grid search

A `Latu.ML.CrossValidator` or `Latu.ML.TrainValidationSplit` has no `Fit` on the wire either.
This runs the search: `num_folds * length(param_maps)` fits, each scored by the evaluator on
its held-out slice, and then **one more** — the winning param map refit on the whole frame,
because the fold models each saw only a fraction of the rows.

Serial, one fit at a time. PySpark runs the grid on a thread pool; whether several processes
can drive one session's channel at once is not something this package has measured, so it
does not claim it. `docs/decisions.md`.

A sub-model is released as soon as its metric is read, so a 5-by-6 search holds one cache
entry rather than thirty. `collect_sub_models: true` keeps them and hands them back, and they
are then yours — `delete/1` on the returned model releases them along with the winner.

    cv = ML.cross_validator(estimator: lr, param_maps: grid, evaluator: evaluator)
    {:ok, searched} = ML.fit(cv, training)
    searched.avg_metrics

# `fit!`

```elixir
@spec fit!(
  Latu.ML.Estimator.t()
  | Latu.ML.Pipeline.t()
  | Latu.ML.CrossValidator.t()
  | Latu.ML.TrainValidationSplit.t(),
  Latu.DataFrame.t()
) :: Latu.ML.Model.t() | Latu.ML.PipelineModel.t() | searched()
```

See `fit/2`. Raises instead of returning an error.

# `helper`

```elixir
@spec helper(Latu.Session.t(), atom(), [term()]) ::
  {:ok, term()} | {:error, Latu.Error.t()}
```

Call a method on the server's own helper object.

The escape hatch under `Latu.ML.Stat`, `Latu.ML.Feature.load_default_stop_words/2` and the
generated `Latu.ML.Feature.StringIndexerModel.from_labels/3` and its siblings — the same
relationship `attribute/3` has to the generated accessors.

`ConnectHelper` is a singleton the server resolves by name: there is no model behind it,
nothing is cached and nothing can be deleted. Its arguments are **positional and complete** —
every declared one is sent, including the `uid` the three model-building methods take, which
the named constructors generate and this does not.

    {:ok, locale} = ML.helper(session, :stop_words_remover_get_default_or_us)
    {:ok, words} = ML.helper(session, :stop_words_remover_load_default_stop_words, ["german"])

A method whose answer is a DataFrame goes to `helper_frame/2` instead; asking for one here
raises rather than building a relation the server would refuse. `Latu.ML.helpers/0` is the
table of what there is.

# `helper!`

```elixir
@spec helper!(Latu.Session.t(), atom(), [term()]) :: term()
```

See `helper/3`. Raises instead of returning an error.

# `helper_frame`

```elixir
@spec helper_frame(atom(), [term()]) :: Latu.DataFrame.t()
```

Call a helper method whose answer is a DataFrame.

A lazy builder, where `helper/3` is an action — the same split `attribute_frame/2` and
`attribute/2` have, and the same `Fetch` underneath. Five of the eleven answer this way, and
every one of them takes the frame it works on as an argument, so the session comes from
there and there is none to pass.

    pairs = ML.helper_frame(:correlation, [frame, "features", "pearson"])

# `helpers`

```elixir
@spec helpers() :: [map()]
```

Every method on the server's own helper object, as the registry holds them.

The `ConnectHelper` route: eleven methods in the same allowlist a model's attributes are in,
and attributes of nothing.

Each is a map with `:name`, the `:method` that goes on the wire, how it `:returns`
(`:value`, `:frame` or `:model`), its `:args` named and typed, and the `:python` call site
all of that was read from — a helper has no class and so no signature, so the call site is
the only thing that says what it takes.

Ten of the eleven were read that way. `handleOverwrite` has no such call site — PySpark
reaches it through `_call_java` — so its arguments are pinned in the extractor with the call
site they were read off, and its row carries `read_by_hand: true` rather than passing for
something derived. `save/3` is what uses it: an operator's own `overwrite:` rides
`MlCommand.Write`, but a pipeline writes a **directory**, and clearing one is this method.

    iex> Latu.ML.helpers() |> Enum.frequencies_by(& &1.returns) |> Enum.sort()
    [frame: 5, model: 3, value: 3]

# `hint`

```elixir
@spec hint(Latu.Error.t()) :: String.t() | nil
```

What to do about an ML failure, in this package's words, or `nil` if it has nothing to add.

Spark's own message reaches the caller untouched — that is Latu's rule, and its text is
usually the better one. These five are the exception, because their text is **wrong** in ways
that send a reader the wrong way:

  * `CACHE_INVALID` describes the missing object as a *Summary object* even for a model,
    blames a 15-minute idle eviction even for a model deleted a moment earlier, and
    recommends `model.evaluate(dataset)`. All three measured false.
  * every one of them opens "Generic Spark Connect ML error".

So this is a second opinion beside the server's, never a replacement for it.

    {:error, error} = Latu.ML.attribute(deleted, :coefficients)
    Latu.ML.hint(error)

# `larger_better?`

```elixir
@spec larger_better?(Latu.ML.Evaluator.t()) :: boolean()
```

Whether a bigger number from `evaluate/2` is a better one, for this evaluator.

Pure — it reaches no server, because there is nothing there to ask. `isLargerBetter` is not
on the server's allowlist, so PySpark's six evaluators each override it client-side to make
it work over Connect, and those overrides are what the registry extracts. This resolves one
against the evaluator's own `metric_name`, set or default.

    iex> Latu.ML.larger_better?(Latu.ML.Evaluation.regression_evaluator())
    false
    iex> Latu.ML.larger_better?(Latu.ML.Evaluation.regression_evaluator(metric_name: "r2"))
    true

Cross-validation and train/validation split pick their best model by this, which is why it
is public: a caller comparing metrics by hand should not have to remember that `rmse` is
minimised and `r2` maximised.

# `load`

```elixir
@spec load(Latu.Session.t(), loadable(), String.t(), keyword()) ::
  {:ok, persistable()} | {:error, Latu.Error.t()}
```

Read a model or an operator back from a path.

An action, and the other half of `save/3`. What is at the path is **named rather than
discovered**: Spark's metadata carries the class, but a `Read` has to say which class to
load before the server will look, so the caller states what they expect and a mismatch is
the server's refusal.

Three spellings, one per way a caller knows what they saved:

  * an operator name from the registry — `:logistic_regression` for the estimator,
    `:logistic_regression_model` for the model it fits;
  * a generated model module — `Latu.ML.Classification.LogisticRegressionModel`;
  * `{class, kind}` for a JVM class this package does not know, where `kind` is
    `:estimator`, `:transformer`, `:evaluator` or `:model`.

A **model** comes back holding a fresh cache reference: reading one costs a cache entry
exactly as fitting one does, and it is yours to `delete/1`. It carries no training frame, so
`summary/1` on it has nothing to rebuild from — and nothing to rebuild either, because a
saved model does not carry its summary. Everything else comes back as inert data with
nothing cached, carrying the uid and the params the metadata held.

## Examples

    {:ok, model} = ML.load(session, :logistic_regression_model, "/tmp/lr")
    scored = ML.transform(model, features)

    {:ok, lr} = ML.load(session, :logistic_regression, "/tmp/lr-unfitted")

# `load!`

```elixir
@spec load!(Latu.Session.t(), loadable(), String.t(), keyword()) :: persistable()
```

See `load/4`. Raises instead of returning an error.

# `model_size`

```elixir
@spec model_size(Latu.ML.Model.t()) ::
  {:ok, non_neg_integer()} | {:error, Latu.Error.t()}
```

What the server thinks a model weighs, in bytes.

An action, and a *server* estimate — `Model.estimatedSize`, which nothing client-side can
compute. It is the number the cache's budgets are spent against, so it is what explains a
`CONNECT_ML.MODEL_SIZE_OVERFLOW_EXCEPTION` on a fit that was refused, and what to measure
before fitting a hundred of something.

    {:ok, bytes} = Latu.ML.model_size(model)

# `model_size!`

```elixir
@spec model_size!(Latu.ML.Model.t()) :: non_neg_integer()
```

See `model_size/1`. Raises instead of returning an error.

# `operator`

```elixir
@spec operator(atom()) :: Latu.ML.Operator.t() | nil
```

One operator by name, or `nil`.

    iex> Latu.ML.operator(:standard_scaler).class
    "org.apache.spark.ml.feature.StandardScaler"

# `operator!`

```elixir
@spec operator!(atom()) :: Latu.ML.Operator.t()
```

See `operator/1`. Raises on a miss, naming the operators whose spelling is close.

# `operators`

```elixir
@spec operators(keyword()) :: [Latu.ML.Operator.t()]
```

Every operator this package knows, as `Latu.ML.Operator` structs.

The table the constructors are generated from, handed back so you can look at it. Filters are
`and`-ed, and an unknown filter key raises rather than quietly matching nothing.

## Options

  * `:kind` — `:estimator`, `:transformer`, `:evaluator` or `:model`.
  * `:group` — PySpark's own module grouping: `:classification`, `:clustering`,
    `:evaluation`, `:feature`, `:fpm`, `:recommendation`, `:regression`.
  * `:status` — `:probed`, `:built` or `:missing`. See `t:Latu.ML.Operator.status/0`.

## Example

    iex> Latu.ML.operators(kind: :evaluator) |> Enum.map(& &1.name)
    [:binary_classification_evaluator, :clustering_evaluator,
     :multiclass_classification_evaluator, :multilabel_classification_evaluator,
     :ranking_evaluator, :regression_evaluator]

# `param_grid`

```elixir
@spec param_grid([
  {Latu.ML.Estimator.t() | Latu.ML.Transformer.t() | Latu.ML.Evaluator.t(),
   keyword([term()])}
]) :: [param_map()]
```

See `param_grid/2`. Takes the pairs together, so the product spans several operators.

# `param_grid`

```elixir
@spec param_grid(
  Latu.ML.Estimator.t() | Latu.ML.Transformer.t() | Latu.ML.Evaluator.t(),
  keyword([term()])
) :: [param_map()]
```

A grid of params to try, as data.

Every combination of the values given, which is what cross-validation and train/validation
split search over. A builder in the sense that nothing runs: no session, no server, and the
params are checked against the operator's own table now rather than after ten fits.

    grid = ML.param_grid(lr, reg_param: [0.1, 0.01], max_iter: [10, 100])
    length(grid)
    #=> 4

Each entry says **which** operator's param it sets, by uid. That matters because the thing
being tuned is often a pipeline and the param belongs to one stage of it:

    lr = Classification.logistic_regression()
    pipeline = ML.pipeline([assembler, lr])
    grid = ML.param_grid(lr, reg_param: [0.1, 0.01])

    ML.cross_validator(estimator: pipeline, param_maps: grid, evaluator: evaluator)

To vary params on two operators at once, pass the pairs together — the product is taken
across all of them, not per operator:

    ML.param_grid([{assembler, [handle_invalid: ["skip", "keep"]]}, {lr, [max_iter: [10]]}])

## Order

The **last** param given varies fastest, as `itertools.product` does it in PySpark. This is
worth knowing because two param maps that score identically are separated by nothing but
their position: the best one is the first maximum.

# `params`

```elixir
@spec params(atom()) :: [Latu.ML.Param.t()]
```

An operator's param table, for a look in `iex`.

    iex> Latu.ML.params(:vector_assembler) |> Enum.map(& &1.name)
    [:handle_invalid, :input_cols, :output_col]

# `pipeline`

```elixir
@spec pipeline([Latu.ML.Pipeline.stage()]) :: Latu.ML.Pipeline.t()
```

A pipeline: stages fitted as one, in order.

A builder, not an action — there is no `Fit` for a pipeline on the wire, so this reaches no
server and needs no session. `Latu.ML.fit/2` is what folds over the stages, and what it hands
back is a `Latu.ML.PipelineModel`.

A stage is an estimator, a transformer, a model already fitted, or another pipeline. Anything
else is refused here rather than partway through a fit that has already cached two models.

    ML.pipeline([
      Feature.vector_assembler(input_cols: [:x1, :x2], output_col: :features),
      Feature.standard_scaler(input_col: :features, output_col: :scaled),
      Classification.logistic_regression(features_col: :scaled, max_iter: 10)
    ])

# `save`

```elixir
@spec save(persistable(), String.t(), keyword()) :: :ok | {:error, Latu.Error.t()}
```

Write a model or an unfitted operator to a path, in Spark's own on-disk format.

An action. The path is the **server's**, not this machine's: the write happens where the
session is, so a bare path is a path on the driver, and anything a second machine has to
read wants a URL the cluster's filesystem understands (`s3://…`, `hdfs://…`).

What comes out is Spark's own format, so PySpark and Scala read it as readily as `load/3`
does — and the reverse, which is what the interop check in `dev/README.md` measures rather
than assumes.

## Options

  * `:overwrite` — replace what is already at the path. Defaults to `false`, where the
    server refuses and names the path.
  * `:options` — the writer's own options, a map of strings, handed to Spark untouched.
  * `:session` — **required for an unfitted operator, and refused for a model.** An
    estimator, a transformer and an evaluator are inert data with no session behind them; a
    model carries its own, and a second one beside it would be a way to write to the wrong
    server.
  * `:sub_models` — **a search only.** Whether to write the fold models beside the winner.
    Defaults to whether the search collected any, which is PySpark's default too, though it
    reaches it through a lowercased key in the writer's option map rather than an option of
    its own. Asking for them where the search kept none is refused rather than silently
    writing nothing.

## A search

A `Latu.ML.CrossValidator` and its model lay out a directory rather than a single operator:
`metadata` with the validator's params and its grid, then `estimator/` and `evaluator/`, and
for a fitted one `bestModel/` and the metrics. `subModels/foldN/M/` where they were kept — a
split has no `foldN` level, because it has one.

    :ok = ML.save(searched, "/tmp/cv", overwrite: true, sub_models: true)

## Examples

    :ok = ML.save(model, "/tmp/lr", overwrite: true)
    :ok = ML.save(lr, "/tmp/lr-unfitted", session: session)

# `save!`

```elixir
@spec save!(persistable(), String.t(), keyword()) :: :ok
```

See `save/3`. Raises instead of returning an error.

# `summary`

```elixir
@spec summary(Latu.ML.Model.t()) :: Latu.ML.Summary.t()
```

The training summary a fitted model carries.

A lazy builder: Spark has no separate cache key for a summary, so this composes the reference
from the model's own and nothing is sent. Its attributes are on the generated summary module
for the class — `Latu.ML.Classification.BinaryLogisticRegressionSummary` and its kind.

Raises for a model class that has no summary, naming the ten that do. Ten of the 43 is not an
oversight: a summary is what an estimator recorded *while fitting*, and most do not record
one.

    summary = Latu.ML.summary(model)
    {:ok, iterations} = Latu.ML.attribute(summary, :totalIterations)

A summary is the one object the server's cache **drops** rather than offloading, and a fetch
that finds it gone answers `CONNECT_ML.MODEL_SUMMARY_LOST`. `attribute/2` recovers from that
on its own, with the frame this struct carries; nothing is asked of the caller.

# `train_validation_split`

```elixir
@spec train_validation_split(keyword()) :: Latu.ML.TrainValidationSplit.t()
```

A single train/validation split, as data. See `Latu.ML.TrainValidationSplit`.

The same builder as `cross_validator/1` with one split instead of k folds, so each param map
is scored once rather than averaged. `train_ratio:` defaults to 0.75.

    tvs =
      ML.train_validation_split(
        estimator: lr,
        param_maps: ML.param_grid(lr, reg_param: [0.1, 0.01]),
        evaluator: Evaluation.binary_classification_evaluator()
      )

# `transform`

```elixir
@spec transform(
  Latu.ML.Transformer.t()
  | Latu.ML.Model.t()
  | Latu.ML.PipelineModel.t()
  | searched(),
  Latu.DataFrame.t()
) :: Latu.DataFrame.t()
```

Apply a transformer or a fitted model to a frame.

A lazy builder, not an action: the frame it hands back carries the ML relation and nothing
has run. `Latu.schema/1` will answer from it without executing, and `Latu.collect/2` is
what finally reaches the server.

# `with_model`

```elixir
@spec with_model(
  Latu.ML.Estimator.t() | Latu.ML.Pipeline.t(),
  Latu.DataFrame.t(),
  (Latu.ML.Model.t()
   | Latu.ML.PipelineModel.t() -&gt;
     result)
) ::
  result | {:error, Latu.Error.t()}
when result: var
```

Fit, use, release — the bracket, in `File.open/3`'s shape.

The model is deleted in an `after`, so it goes even if the function raises. A delete that
itself fails is logged rather than raised: the useful result is the one in hand, and the
session's end bounds the leak anyway.

Returns whatever the function returned, or the fit's error if there was never a model.

    {:ok, coefficients} =
      Latu.ML.with_model(lr, features, fn model ->
        Latu.ML.attribute(model, :coefficients)
      end)

# `with_model!`

```elixir
@spec with_model!(
  Latu.ML.Estimator.t() | Latu.ML.Pipeline.t(),
  Latu.DataFrame.t(),
  (Latu.ML.Model.t()
   | Latu.ML.PipelineModel.t() -&gt;
     result)
) ::
  result
when result: var
```

See `with_model/3`. Raises where the fit would have returned an error.

---

*Consult [api-reference.md](api-reference.md) for complete listing*