# `Latu.ML.Functions`
[🔗](https://github.com/zero-one-group/latu_ml/blob/v0.4.2/lib/latu/ml/functions.ex#L1)

Converting between a `Vector` column and an ordinary array of numbers.

`pyspark.ml.functions`' two, and the way to read a `Vector` column: `Latu.collect/2` and
`Latu.to_explorer/2` both refuse one, because Spark describes it as a UDT that does not say
its SQL type (`docs/deviations.md`).

    alias Latu.ML.Functions

    scored = Latu.ML.transform(model, features)

    scored
    |> Latu.select([:prediction, features: Functions.vector_to_array(:features)])
    |> Latu.collect()

Both are lazy: each builds a column expression and reaches no server.

Spark keeps these in an **internal** function registry rather than the builtin one, so they
are unreachable from SQL — `Latu.sql/3` on `SELECT vector_to_array(…)` answers
`UNRESOLVED_ROUTINE` on any 4.2.0 server, and that is by design rather than a gap. Over
Connect they resolve, which is what these two build.

Both are UDFs on the server rather than compiled expressions, so they cost a pass over the
rows, and `vector_to_array` densifies a sparse `Vector` rather than refusing it.

# `column`

```elixir
@type column() :: atom() | String.t() | Latu.Plan.expression()
```

A column.

An atom or a string names one; anything else is used as the expression it is, so a built
column such as `Latu.Column.fun/3`'s result goes straight through.

# `array_to_vector`

```elixir
@spec array_to_vector(column()) :: Latu.Plan.expression()
```

An array of numbers as a dense `Vector` column.

The input is an `array<double>`; the answer is a `Vector` every ML operator accepts, and one
`Latu.collect/2` refuses like any other.

    Functions.array_to_vector(:embedding)

`VectorAssembler` is the other way to build one, and the better one where the features are
already separate columns — it takes them without a round trip through an array.

# `dtypes`

```elixir
@spec dtypes() :: [String.t()]
```

The dtypes `vector_to_array/2` takes, which are the two Spark registers.

# `vector_to_array`

```elixir
@spec vector_to_array(column(), String.t()) :: Latu.Plan.expression()
```

A `Vector` column as an array of numbers.

`dtype` is `"float64"` — the default — for an `array<double>`, or `"float32"` for an
`array<float>`. Dense and sparse vectors alike; a sparse one comes back dense.

    Functions.vector_to_array(:features)
    Functions.vector_to_array(:features, "float32")

The dtype is checked here rather than on the server, which would answer
`INVALID_PARAMETER_VALUE.DTYPE` a round trip later.

---

*Consult [api-reference.md](api-reference.md) for complete listing*