This is the official repository for the CIKM'26 paper "Pitfalls with Downsampled Data in AutoML for Clustering" by Philipp Jahn, Gabriel Marques Tavares, Anna Beer, Udo Schlegel, and Thomas Seidl.

Requirements

The framework is designed for Python 3.12 on Linux. Timeout checks for optimization are performed using SIGALRM, which is available only on Linux.

The pip freeze output for our virtual environment is in requirements.txt. It may be necessary to install swig and cmake before the other packages, as pyrfr, which is required by SMAC3, can cause trouble otherwise.

DENSIRED requires numpy==2.0 or higher, which is incompatible with some other packages that require earlier versions of numpy, and is thus left out of requirements.txt. It is only needed if make_densired_data.py or intuition_figure_maker.py is run, otherwise the stored datasets are used.

Datasets

To start with, datasets are required for running any experiments:

• /data/ contains datasets (the datasets are either included, are set up to download automatically upon request or have download instructions provided below)
• make_densired_data.py was used to generate datasets using the densired package. The generated datasets are also included in the repository.
• data_handler.py manages the datasets.

All datasets are referenced through their keys:

Name	Key
Aggregation	aggregation
Complex-9	complex9
DENSIRED	densired
DENSIRED_N	densired_noise
Wine-Quality	wine_quality
Isolet	isolet
HAR	har
Pendigits	pendigits
HTRU2	htru2
Magic-Gamma	magic_gamma
Letter	letter
GTSRB	gtsrb
Statlog (Shuttle)	shuttle
Sensorless Drive Diagnosis	sensorless
MNIST	mnist
Fashion-MNIST	fmnist
Kuzushiji-MNIST	kmnist
ScalingTestDown	scaling1
ScalingTestUp-1	scaling2
ScalingTestUp-2	scaling4

• HAR can be downloaded here. (Place in /data/uci_download)
• Sensorless Drive Diagnosis can be downloaded here (Place in /data/uci_download)
• Letter can be downloaded in Parmar Milaan's Clustering Dataset repository here (The filename is letter.arff. Place in /data/uci_milaan9/)
• Complex-9 and Aggregation can be downloaded in Parmar Milaan's Clustering Dataset repository here (The filenames are complex9.arff and aggregation.arff, respectively. Place in /data/synthetic_milaan9/)
• All other datasets are either already in the data folder or are downloaded automatically when requested (through the packages ucimlrepo or clustpy).

Downsampling

Downsampling is performed using subset_handler.py

Parameter	Function
--ds	Dataset to downsample
--size	Ratio/size to downsample to
--sampling	Downsampling method to use
--data_seed	Seed for random instantiation of downsampling strategy

Here, the following keys correspond to the implemented downsampling strategies:

Name	Key
Uniform Random Sampling	random
LWC	lwc_square
k-Centroid	kcentroid
DenDis	dendis
DisDen	disden
ProTraS	protras

• /data/ contains downsampled datasets with the folder name corresponding to the used downsampling strategy, for licensing reasons only the subset of datasets created by us are included; other subsets can be automatically recreated by requesting the respective subsamples (the affected datasets are only uniform random subsamples)

Hyperparameter Optimization

The optimization using SMAC3 is performed using findParameters.py. For unsupervised scores, use findParameterUnsup.py.

Parameter	Function
--ds	Dataset
--size	Downsampling size
--sampling	Downsampling method
--method	Clustering method to optimize
--budget	Optimization time budget in seconds
--data_seed	Seed for downsampling (-1 iterates over 0,1,2 for non-deterministic methods, each requiring time corresponding --budget)
--smac_seed	Seed for SMAC3 (-1 iterates over 0,1,2, each requiring time corresponding --budget (this stacks with --data_seed )
--supervised	Whether to use the supervised or unsupervised score (Boolean as Integer, 1=Supervised, 0=Unsupervised), in findParameterUnsup.py this takes disco for DISCO and sil for Sil
--overwrite	Whether to overwrite existing optimization runs (Boolean as Integer, 1=Yes, 0=No)

Clustering methods follow this naming scheme. K-Means, DBSCAN and Agglomerative Clustering use their Scikit-Learn implementations. SHADE uses the code from the original authors' repository.

Name	Key
k-Means	kmeans
DBSCAN	dbscan
Agglomerative	agglomerative_nn
SHADE	shade

• After execution, /opt_logs/ will contain the logged information on all tested configurations during optimization. The log files are sorted into folders /opt_logs/dataset_clustering-algorithm/ and all files are named like this: log_dataset_clustering-algorithm_metric_downsampling-method_downsampling-size_optimization-budget_data-seed_smac-seed.txt.
• After execution, /param_logs/ will contain the information on the final configurations obtained from optimization. The files are sorted into folders /param_logs/dataset_clustering-algorithm/ and all files are named like this: log_dataset_clustering-algorithm_metric_downsampling-method_downsampling-size_optimization-budget.csv.

The configuration spaces for this are handled through config_handler.py. If a new dataset is added, it needs to receive an entry in the class_num-dictionary to function (as downsampled datasets may drop class labels, but the default hyperparameters should stay the same).

The clustering algorithms are handled through clustering_handler.py.

Scaling Behavior Analysis

The hyperparameter scaling analysis is a two-step process. First, a grid search across has to be performed for every dataset that is supposed to be compared (our used keys were: scaling1, scaling2, scaling4). The grid search is performed using evaluateParamterGrid.py. It supports downsampled/upsampled datasets, but ultimately was doen using fixed datasets.

Parameter	Function
--ds	Dataset
--size	Downsampling size
--sampling	Downsampling method
--method	Clustering method
--param_configs	Number of configurations to test
--primary	First hyperparameter to iterate over
--secondary	Secondary hyperparameter to iterate over
--tertiary	Third hyperparameter to iterate over

• After execution, /grid_evals/ will contain the results for this. They follow the naming scheme of dataset_downsampling-method_downsampling-size_clustering-algorithm_primary_secondary_tertiary_config-number.txt. Each line corresponds to an evaluated run.

We used the following configurations:

• --method kmeans --primary n_clusters --secondary init --tertiary none
• --method dbscan --primary eps --secondary min_samples --tertiary none
• --method agglomerative --primary n_clusters --secondary n_neighbors --tertiary linkage
• --method shade --primary plr --secondary clr --tertiary min_points

These are the processed into dictionary form with scaling_dict_maker.py, which requires the specification of the dataset with --ds.

• After execution, /grid_dicts/ contains the extraction dictionaries. The naming scheme is dataset_method_metric.npy.

To extract information on the scaling behavior/make the basis for tables in the paper, we use scaling_table_maker.py, which automatically outputs the results for the examined clustering algorithms. (Note: this will produce two tables per method, one for each ScalingTest variant)

Configuration Evaluation

To evaluate the obtained hyperparameter configurations, they are rerun to get their performance on (potentially) different dataset sizes. This is done with evaluateParameter.py for supervised scores and with evaluateParameterUnsup.py for DISCO and Sil

Parameter	Function
--ds	Dataset
--size	Downsampling size of original (downsampled) dataset
--sampling	Downsampling method of original (downsampled) dataset
--evalsize	Downsampling size of evaluation (typical full) datatset
--sampling	Downsampling method of evaluation (typical full) datatset (not used for size greater or equal to 1)
--method	Clustering method to optimize
--budget	Optimization time budget in seconds
--supervised	Whether to use the supervised or unsupervised score (Boolean as Integer, 1=Supervised, 0=Unsupervised (unused)), this is replaced with --optmetric for evaluateParameterUnsup.py which takes disco for DISCO and sil for Sil
--scaling	Which scaling behavior to apply

The scaling behaviors are bundled as follows. Hyperparameters not listed here are unaffected by scaling (i.e. if no hyperparameters of an algorithm are included, the scaling option is equivalent to no scaling):

• none applies no scaling to any hyperparameter

• sample_mult applies linear scaling to min_samples (DBSCAN), n_neighbors (Agglomerative Clustering) and min_points/plr/clr (SHADE)

• sample_mult_no_lr applies linear scaling to min_samples (DBSCAN), n_neighbors (Agglomerative Clustering) and min_points (SHADE)

• root_clunum applies root scaling to n_clusters (k-Means, Agglomerative Clustering)

• /eval_logs contains the evaluations of the obtained hyperparameter configurations on relevant datasets. The files are sorted into folders /eval_logs/dataset_clustering-algorithm and all files are named according to log_dataset_clustering-algorithm_supervision_from_downsampling-method-1_downsampling-size-1_to_downsampling-method-2_downsampling-size-2_optimization-budget_scaling-method.txt

Paper Tools

To facilitate the information extraction from the gathered data in order to put it into the paper, we added several tools to automate the process.

• evaluationTableMaker.py/evaluationTableMakerICVI.py/evaluationTableMakerOtehrClustering.py/evaluationTableMakerTime.py produce tables for the final results (soem editing required)
• differenceTableMaker.py/differenceTableMakerOTherClsutering.py produces tables comparing two scaling functions
• gridFigureMaker2.py produces parameter grid figures in 2x2 shape (legend_maker.py produces its legend)
• data_figure_maker.py makes figures for data sets
• sampling_figure_maker.py makes figures for the subsampling metrics (specifically BoP for Isolet)
• motivation_figure_maker.py makes figures using different configurations of a clustering algorithm
• granularity_figure_maker.ipynb makes figures for the behavior of scaling hyperparmeters (the corresponding configuration needs to be evaluated with evaluateParameterGranularity.py first; setup is the same as other evaluateParameter-methods, but produces multiple evaluations per configuration)
• incumbentFigureMaker.py makes a figure that compares the currently-best-configuration for each step in the HPO process to the best achieved result (these need to be extracted with incumbent_extractor.py, which requires the corresponding optimization logs in /opt_logs/. The incumbent files for DBSCAN are already included in the repository.)
• intuition_figure_maker.ipynb makes the intuition figures
• subsetRanker.py computes the ranks of performance scores for downsampling methods
• score_extractor.py computes the iCVI values for datasets given the ground truth labels

External Code

• BoP code is in /similarity/BoP.py and originally comes from the official implementation repository. (MIT License Copyright (c) 2023 Klaus-Tu)
• The DISCO code was initially provided by the original authors (it corresponds to the version prior to Commit f94306f, which sped up the DCTree calculation). The code can be found in the official repository. Handling to treat all labels < 0 as noise was added by us. The disco.py code needs to be placed in /metrics/
• The SHADE code is in /clustering/SHADE and originally comes from the official implementation repository (BSD 3-Clause License Copyright (c) 2026 pasiweber)

Citation

Please cite this repository as follows:

@inproceedings{jahn2026automlpitfalls,
  author       = {Philipp Jahn and
                  Gabriel Marques Tavares and
                  Anna Beer and
                  Udo Schlegel and
                  Thomas Seidl},
  title        = {Pitfalls with Downsampled Data in AutoML for Clustering},
  booktitle    = {Proceedings of the 35th {ACM} International Conference on Information
                  and Knowledge Management, {CIKM} 2026, Rome, Italy, November
                  07-11, 2026},
  pages        = {in press},
  publisher    = {{ACM}},
  year         = {2026},
  url          = {https://doi.org/10.1145/3799682.3840859},
  doi          = {10.1145/3799682.3840859},
}