Skip to content

v0.1.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Feb 06:55
· 23 commits to main since this release
v0.1.0
07975f7

v0.1.0 (2025-02-12)

Feature

  • feat: pypi packaging and auto-release with semantic release (0ff8888)

Unknown

  • Merge pull request #37 from chanind/pypi-package

feat: pypi packaging and auto-release with semantic release (a711efe)

  • simplify matryoshka loss (43421f5)

  • Use torch.split() instead of direct indexing for 25% speedup (505a445)

  • Fix matryoshka spelling (aa45bf6)

  • Fix incorrect auxk logging name (784a62a)

  • Add citation (77f2690)

  • Make sure to detach reconstruction before calculating aux loss (db2b564)

  • Merge pull request #36 from saprmarks/aux_loss_fixes

Aux loss fixes, standardize decoder normalization (34eefda)

  • Standardize and fix topk auxk loss implementation (0af1971)

  • Normalize decoder after optimzer step (200ed3b)

  • Remove experimental matroyshka temperature (6c2fcfc)

  • Make sure x is on the correct dtype for jumprelu when logging (c697d0f)

  • Import trainers from correct relative location for submodule use (8363ff7)

  • By default, don't normalize Gated activations during inference (52b0c54)

  • Also update context manager for matroyshka threshold (65e7af8)

  • Disable autocast for threshold tracking (17aa5d5)

  • Add torch autocast to training loop (832f4a3)

  • Save state dicts to cpu (3c5a5cd)

  • Add an option to pass LR to TopK trainers (8316a44)

  • Add April Update Standard Trainer (cfb36ff)

  • Merge pull request #35 from saprmarks/code_cleanup

Consolidate LR Schedulers, Sparsity Schedulers, and constrained optimizers (f19db98)

  • Consolidate LR Schedulers, Sparsity Schedulers, and constrained optimizers (9751c57)

  • Merge pull request #34 from adamkarvonen/matroyshka

Add Matroyshka, Fix Jump ReLU training, modify initialization (92648d4)

  • Add a verbose option during training (0ff687b)

  • Prevent wandb cuda multiprocessing errors (370272a)

  • Log dead features for batch top k SAEs (936a69c)

  • Log number of dead features to wandb (77da794)

  • Add trainer number to wandb name (3b03b92)

  • Add notes (810dbb8)

  • Add option to ignore bos tokens (c2fe5b8)

  • Fix jumprelu training (ec961ac)

  • Use kaiming initialization if specified in paper, fix batch_top_k aux_k_alpha (8eaa8b2)

  • Format with ruff (3e31571)

  • Add temperature scaling to matroyshka (ceabbc5)

  • norm the correct decoder dimension (5383603)

  • Fix loading matroyshkas from_pretrained() (764d4ac)

  • Initial matroyshka implementation (8ade55b)

  • Make sure we step the learning rate scheduler (1df47d8)

  • Merge pull request #33 from saprmarks/lr_scheduling

Lr scheduling (316dbbe)

  • Properly set new parameters in end to end test (e00fd64)

  • Standardize learning rate and sparsity schedules (a2d6c43)

  • Merge pull request #32 from saprmarks/add_sparsity_warmup

Add sparsity warmup (a11670f)

  • Add sparsity warmup for trainers with a sparsity penalty (911b958)

  • Clean up lr decay (e0db40b)

  • Track lr decay implementation (f0bb66d)

  • Remove leftover variable, update expected results with standard SAE improvements (9687bb9)

  • Merge pull request #31 from saprmarks/add_demo

Add option to normalize dataset, track thresholds for TopK SAEs, Fix Standard SAE (67a7857)

  • Also scale topk thresholds when scaling biases (efd76b1)

  • Use the correct standard SAE reconstruction loss, initialize W_dec to W_enc.T (8b95ec9)

  • Add bias scaling to topk saes (484ca01)

  • Fix topk bfloat16 dtype error (488a154)

  • Add option to normalize dataset activations (81968f2)

  • Remove demo script and graphing notebook (57f451b)

  • Track thresholds for topk and batchtopk during training (b5821fd)

  • Track threshold for batchtopk, rename for consistency (32d198f)

  • Modularize demo script (dcc02f0)

  • Begin creation of demo script (712eb98)

  • Fix JumpReLU training and loading (552a8c2)

  • Ensure activation buffer has the correct dtype (d416eab)

  • Merge pull request #30 from adamkarvonen/add_tests

Add end to end test, upgrade nnsight to support 0.3.0, fix bugs (c4eed3c)

  • Merge pull request #26 from mntss/batchtokp_aux_fix

Fix BatchTopKSAE training (2ec1890)

  • Check for is_tuple to support mlp / attn submodules (d350415)

  • Change save_steps to a list of ints (f1b9b80)

  • Add early stopping in forward pass (05fe179)

  • Obtain better test results using multiple batches (067bf7b)

  • Fix frac_alive calculation, perform evaluation over multiple batches (dc30720)

  • Complete nnsight 0.2 to 0.3 changes (807f6ef)

  • Rename input to inputs per nnsight 0.3.0 (9ed4af2)

  • Add a simple end to end test (fe54b00)

  • Create LICENSE (32fec9c)

  • Fix BatchTopKSAE training (4aea538)

  • dtype for loading SAEs (932e10a)

  • Merge pull request #22 from pleask/jumprelu

Implement jumprelu training (713f638)

  • Merge branch 'main' into jumprelu (099dbbf)

  • Merge pull request #21 from pleask/separate-wandb-runs

Use separate wandb runs for each SAE being trained (df60f52)

  • Merge branch 'main' into jumprelu (3dfc069)

  • implement jumprelu training (16bdfd9)

  • handle no wandb (8164d32)

  • Merge pull request #20 from pleask/batchtopk

Implement BatchTopK (b001fb0)

  • separate runs for each sae being trained (7d3b127)

  • add batchtopk (f08e00b)

  • Move f_gate to encoder's dtype (43bdb3b)

  • Ensure that x_hat is in correct dtype (3376f1b)

  • Preallocate buffer memory to lower peak VRAM usage when replenishing buffer (90aff63)

  • Perform logging outside of training loop to lower peak memory usage (57f8812)

  • Remove triton usage (475fece)

  • Revert to triton TopK implementation (d94697d)

  • Add relative reconstruction bias from GDM Gated SAE paper to evaluate() (8984b01)

  • git push origin main:Merge branch 'ElanaPearl-small_bug_fixes' into main (2d586e4)

  • simplifying readme (9c46e06)

  • simplify readme (5c96003)

  • add missing imports (7f689d9)

  • fix arg name in trainer_config (9577d26)

  • update sae training example code (9374546)

  • Merge branch 'main' of https://github.com/saprmarks/dictionary_learning into main (7d405f7)

  • GatedSAE: moved feature re-normalization into encode (f628c0e)

  • documenting JumpReLU SAE support (322b6c0)

  • support for JumpReluAutoEncoders (57df4e7)

  • Add submodule_name to PAnnealTrainer (ecdac03)

  • host SAEs on huggingface (0ae37fe)

  • fixed batch loading in examine_dimension (82485d7)

  • Merge pull request #17 from saprmarks/collab

Merge Collab Branch (cdf8222)

added annealing and trainer_param_callback (61927bc)

add ListaTrainer (d4d2fd9)

  • additional evluation metrics (fa2ec08)

  • add GroupSAETrainer (60e6068)

  • added annealing and trainer_param_callback (18e3fca)

  • Merge remote-tracking branch 'upstream/collab' into collab (4650c2a)

  • fixing neuron resampling (a346be9)

  • improvements to saving and logging (4a1d7ae)

  • can export buffer config (d19d8d9)

  • fixing evaluation.py (c91a581)

  • fixing bug in neuron resampling (67a03c7)

  • add ListaTrainer (880f570)

  • fixing neuron resampling in standard trainer (3406262)

  • improvements to training and evaluating (b111d40)

  • Factoring out SAETrainer class (fabd001)

  • updating syntax for buffer (035a0f9)

  • updating readme for from_pretrained (70e8c2a)

  • from_pretrained (db96abc)

  • Change syntax for specifying activation dimensions and batch sizes (bdf1f19)

  • Merge branch 'main' of https://github.com/saprmarks/dictionary_learning into main (86c7475)

  • activation_dim for IdentityDict is optional (be1b68c)

  • update umap requirement (776b53e)

  • Merge pull request #10 from adamkarvonen/shell_script_change

Add sae_set_name to local_path for dictionary downloader (33b5a6b)

  • Add sae_set_name to local_path for dictionary downloader (d6163be)

  • dispatch no longer needed when loading models (69c32ca)

  • removed in_and_out option for activation buffer (cf6ad1d)

  • updating readme with 10_32768 dictionaries (614883f)

  • upgrade to nnsight 0.2 (cbc5f79)

  • downloader script (7a305c5)

  • fixing device issue in buffer (b1b44f1)

  • added pretrained_dictionary_downloader.sh (0028ebe)

  • added pretrained_dictionary_downloader.sh (8b63d8d)

  • added pretrained_dictionary_downloader.sh (6771aff)

  • efficiency improvements (94844d4)

  • adding identity dict (76bd32f)

  • debugging interp (2f75db3)

  • Merge branch 'main' of https://github.com/saprmarks/dictionary_learning into main (86812f5)

  • warns user when evaluating without enough data (246c472)

  • cleaning up interp (95d7310)

  • examine_dimension returns mbottom_tokens and logit stats (40137ff)

  • continuing merge (db693a6)

  • progress on merge (949b3a7)

  • changes to buffer.py (792546b)

  • fixing some things in buffer.py (f58688e)

  • updating requirements (a54b496)

  • updating requirements (a1db591)

  • identity dictionary (5e1f35e)

  • bug fix for neuron resampling (b281b53)

  • UMAP visualizations (81f8e1f)

  • better normalization for ghost_loss (fc74af7)

  • neuron resampling without replacement (4565e9a)

  • simplifications to interp functions (2318666)

  • Second nnsight 0.2 pass through (3bcebed)

  • Conversion to nnsight 0.2 first pass (cac410a)

  • detaching another thing in ghost grads (2f212d6)

  • Neuron resampling no longer errors when resampling zero neurons (376dd3b)

  • NNsight v0.2 Updates (90bbc76)

  • cosmetic improvements to buffer.py (b2bd5f0)

  • fix to ghost grads (9531fe5)

  • fixing table formatting (0e69c8c)

  • Fixing some table formatting (75f927f)

  • gpt2-small support (f82146c)

  • fixing bug relevant to UnifiedTransformer support (9ec9ce4)

  • Getting rid of histograms (31d09d7)

  • Fixing tables in readme (5934011)

  • Updates to the readme (a5ca51e)

  • Fixing ghost grad bugs (633d583)

  • Handling ghost grad case with no dead neurons (4f19425)

  • adding support for buffer on other devices (f3cf296)

  • support for ghost grads (25d2a62)

  • add an implementation of ghost gradients (2e09210)

  • fixing a bug with warmup, adding utils (47bbde1)

  • remove HF arg from buffer. rename search_utils to interp (7276f17)

  • typo fix (3f6b922)

  • Merge branch 'main' of https://github.com/saprmarks/dictionary_learning into main (278084b)

  • added utils for converting hf dataset to generator (82fff19)

  • add ablated token effects to ; restore support for HF datasets (799e2ca)

  • merge in function for examining features (986bf96)

  • easier submodule/dictionary feature examination (2c8b985)

  • Adding lr warmup after every time neurons are resampled (429c582)

  • fixing issues with EmptyStream exception (39ff6e1)

  • Minor changes due to updates in nnsight (49bbbac)

  • Revert "restore support for streaming HF datasets"

This reverts commit b43527b. (23ada98)

Update README.md (fd79bb3)

Assets 6
Loading