# embeddings_visualizer

Python API: `luxonis_train.attached_modules.visualizers.embeddings_visualizer`

Projects the embeddings of a batch to two dimensions with PCA and draws them colored by identity label.

## Classes

### EmbeddingsVisualizer

Visualize embedding spaces as two-dimensional plots.

The KDE plot on the left and the scatter plot on the right show the 2-D PCA of the embeddings. Each color is one identity label.

 * `Inputs:`: * `prediction_canvas`, `target_canvas` (`Tensor`): [B, 3, H, W]
    * `predictions` (`Tensor`): [B, D] embeddings
    * `target` (`Tensor`): [B] identity labels
 * `Outputs:`: * `tuple[Tensor, Tensor]`: [1, 3, 512, 512] KDE and scatter plots of the 2-D PCA

> **References**
> * Source: This project.
 * License: Apache-2.0 (this project)

> **Notes**
> Projects embeddings with PCA, filters z-score outliers, and renders Seaborn plots as tensors.

> **Example**
> Attached to a `GhostFaceNetHead` in `model.nodes`:

```yaml
- name: GhostFaceNetHead
  inputs: [GhostFaceNet]
  visualizers:
    - name: EmbeddingsVisualizer
```

 * `Compatible with:`: * Used by:
   [EmbeddingsModel](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/config/predefined_models/embeddings/v1/model.md)
    * Nodes:
      [GhostFaceNetHead](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/nodes/heads/ghostfacenet_head.md)

#### Methods

##### init

```python
def __init__(z_score_threshold: float = 3, **kwargs):
```

Initialize the visualizer and store the outlier threshold.

Parameters

 * `z_score_threshold` (`float`): Limit for the absolute z-score of a projected point.
   [forward](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/embeddings_visualizer.md)
   keeps a point only when the absolute z-score of each of its two coordinates is below the limit. The z-scores use the mean and
   the standard deviation of the batch on each axis. Defaults to `3`.
 * `**kwargs`: Keyword arguments forwarded to
   [BaseVisualizer](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/base_visualizer.md),
   such as `scale` and `node`.

##### forward

```python
def forward(prediction_canvas: Tensor, target_canvas: Tensor, predictions: Tensor, target: Tensor) -> tuple[Tensor, Tensor]:
```

Project the embeddings to two dimensions and plot them.

The method replaces `NaN` and infinite values with `0`. When `predictions` is a CPU tensor, this replacement writes into it. Then
it fits a two-component PCA with `random_state=42`. When the explained variance of the second component is below `1e-12`, the
embeddings are effectively one-dimensional. The method then fixes the sign of the first projected coordinates. It computes their
dot product with the row sums of the embeddings and flips the coordinates when the product is negative. Next,
[_filter_outliers](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/embeddings_visualizer.md)
drops the points whose absolute z-score reaches `z_score_threshold` on either axis. Finally it renders a KDE plot and a scatter
plot with
[kde_plot](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/embeddings_visualizer.md)
and
[scatter_plot](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/embeddings_visualizer.md).
The method ignores both canvases.

Logs the number of dropped outliers at the `INFO` level.

Parameters

 * `prediction_canvas` (`Tensor`): Ignored.
 * `target_canvas` (`Tensor`): Ignored.
 * `predictions` (`Tensor`): Embeddings of shape `[B, D]`.
 * `target` (`Tensor`): The `id` metadata label of each embedding, of shape `[B]`. The method casts it to `int`.

Returns

 * `tuple[Tensor, Tensor]`: The KDE plot and the scatter plot, each a `uint8` image of shape `[1, 3, 512, 512]`.

##### kde_plot

```python
def kde_plot(ax: plt.Axes, emb: np.ndarray, labels: np.ndarray):
```

Draw one filled KDE per label on the axes.

Each label gets its color from
[BaseVisualizer.colormap](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/base_visualizer.md).
The KDE uses `bw_adjust=1.5` and `alpha=0.9`. `warn_singular=False` silences the Seaborn warning for a label whose points cannot
form a density.

Parameters

 * `ax` (`plt.Axes`): Axes to draw on.
 * `emb` (`np.ndarray`): Projected points of shape `[N, 2]`.
 * `labels` (`np.ndarray`): Label of each point, of shape `[N]`.

##### plot_to_tensor

```python
def plot_to_tensor(embeddings_2d: np.ndarray, ids_np: np.ndarray, plot_func: Callable[[plt.Axes, np.ndarray, np.ndarray], None]) -> Tensor:
```

Render a plot of the projected embeddings as an image tensor.

Creates a `10 x 10` inch figure and limits both axes to the range of the points. Then it calls `plot_func` on the axes, hides the
axes, and converts the figure with
[figure_to_torch](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/utils.md).

> **Example**
> ```pycon
>>> import numpy as np
>>> def no_op(ax, points, labels):
...     pass
>>> EmbeddingsVisualizer.plot_to_tensor(
...     np.zeros((0, 2)), np.zeros(0, dtype=int), no_op
... ).shape
torch.Size([1, 3, 512, 512])
```

Parameters

 * `embeddings_2d` (`np.ndarray`): Projected points of shape `[N, 2]`. May be empty; then the axis limits stay at their defaults.
 * `ids_np` (`np.ndarray`): Labels of shape `[N]`.
 * `plot_func` (`Callable[[plt.Axes, np.ndarray, np.ndarray], None]`): Draws on the axes it receives, for example [kde_plot](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/embeddings_visualizer.md) or [scatter_plot](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/embeddings_visualizer.md).

Returns

 * `Tensor`: A `uint8` image of shape `[1, 3, 512, 512]`.

##### scatter_plot

```python
def scatter_plot(ax: plt.Axes, emb: np.ndarray, labels: np.ndarray):
```

Draw all points on the axes, colored by label.

Uses `s=300`, `alpha=0.9`, and no legend. The palette maps each label to its color in [BaseVisualizer.colormap](https://docs.luxonis.com/software-v3/ai-inference/model-source/training/luxonis-train/luxonis-train-api-reference/attached_modules/visualizers/base_visualizer.md).

Parameters

 * `ax` (`plt.Axes`): Axes to draw on.
 * `emb` (`np.ndarray`): Projected points of shape `[N, 2]`.
 * `labels` (`np.ndarray`): Label of each point, of shape `[N]`.

#### Attributes

##### supported_tasks