Project still active?

[KIC]

I am wondering if this project is still active and the special cases (pca) soms are still being implemented?

I am also wondering whether I can use a traditional torch data loader for a dataset which clearly doesn't fit into my memory?

[LouisTier]

Hi @KIC ,

Yes, the project is still active. There is ongoing development focused on bug fixes, documentation, performance improvements, and new SOM mechanisms.

Current / planned work includes:

• migrating documentation from pydoc to furo;
• refining documentation to better match the current package;
• integrating FAISS for optimized search (BMU identification, sample collection, etc.);
• extending SOM mechanisms:
  • PBC,
  • growing SOMs,
  • hierarchical SOMs;
• improving visualization (e.g., tracking map evolution across epochs, U-matrix, BMU/neighborhood highlighting);
• handling edge cases (e.g., batches with a single sample).

Regarding “special cases”: if you are referring to PCA-based initialization, it is already implemented. You can use it via the initialization_mode parameter:

som = SOM(
    x=10,
    y=10,
    num_features=n_features,
    initialization_mode="pca",
)

Concerning the use of a PyTorch DataLoader: internally, the current implementation already builds a TensorDataset and a DataLoader from the input tensor, based on the specified batch_size (see fit in torchsom/core/som.py). If memory pressure occurs during training, reducing batch_size is the first lever.

However, if the dataset itself does not fit into memory, the current API still requires passing a full tensor to fit(data). Supporting out-of-core training via a user-provided Dataset/DataLoader is not directly implemented yet and would require adapting the training loop to consume an iterable rather than a preloaded tensor.

This is a valid use case and could be considered for future extensions.

[KIC]

Cool! I have like 500M datapoints, so I had to add the possibility for a DataLoader basically everywhere. I have submitted a PR #4 in case you are interested. Otherwise, I just keep merging from your repo.