CLUE: Crystal Learning for Understandable Explanations

This repository contains my project for the LLM Hackathon for Applications in Materials Science & Chemistry 2025 (https://llmhackathon.github.io/)

Motivation:

In traditional physics and chemistry, explanations and definitions are the scaffolding for progress: they guide model improvement, enable fair comparison between approaches, and turn results into reusable knowledge. Black-box ML models intrinsically do not provide this, so special methods to interpret and explain decisions of ML models need to be developed to improve trust, build better evaluation methods (not just rooted in statistics of outputs), refine models more purposefully.

Hackathon target:

We aim to develop CLUE. CLUE generates physically valid crystal counterfactuals, and based on them generates natural language explanations for the structure-property relationships.

Implementation

Logic

Here we do the following:

1. Make property prediction for the compound provided by the user
2. Compose the list of possible counterfactuals. This is done by queiring all entries from Materials project which contain at least one of the elements from the initial forlmula. Then for each elements other 5 elements are determined which are closest on Pettifor scale to them (so, there is higher probability of substitution). All these elements are compiled into a list of possible elements. The list of compounds is filtered to contain only compounds with possible elements.
3. For all possible counterfactuals property is predicted and only compounds with the opposite value of property are retained.
4. We rank possible counterfactuals with respect to the similarity to the sample compund. To assess similarity we use a combination of composition similarity (using ElMD package), similarity of volumes per atom, and similarity of structural fingerprints (based on SOAP). After ranking we retain 10 most similar to the sample.
5. The gpt-4.1-mini is prompted with the sample information and information about counterfactuals and asked to explain the output of the model predicting the property. In the prompt structures of all compounds are included in the form of Robocrytallographer descriptions.
6. All explainations and lists of counterfactuals are stored in 'outputs/' folder

Distance metric

• Distance between two compositions is calculated using Earth-Mover distance with modified Pettifor scale as implementd in ElMD package. To constrain the distances to [0,1] interval we scale them by the largest distance calculated for the dataframe. Similarity is calculated as exp(-scaled_distance).
• Each structure is represented by SOAP descriptors that capture local atomic environments. Atoms equivalent by symmetry are averaged into orbit vectors, forming a compact structural signature. To compare two structures, orbit vectors are matched across species using the Hungarian algorithm, with similarities weighted by the Pettifor scale (so F is closer to Cl than to Pb). The final similarity score is the average across matched orbits, penalizing unmatched ones, and ranges from 0 (dissimilar) to 1 (identical).
• As structure similarity is implemented in a way imsensitive to uniform cell expansion, similarity with respect to volume per atom is added.

Other ways to generate counterfactuals which were explored

We also exxplored other ways to generate counterfactuals such as generating substitutions in the initial structure, and using CrystalLLM generator on mutated compositions. However due to lack of time this work is not included.

Model

As a predictive model whose predictions we explain we choose RF from scikit-learn (due to the lack of time and usually good performance of it on materials data). We predict metallicity. To train the model we use Jarvis 3d_dft dataset (metal:nonmetal ration is 0.75:0.25). We use 80:20 train:test split. As features we use combination of matminer composition and structural features with SOAP (calculated for lattice, with erased atom identitites).

Performance of trained model on the test dataset:

• accuracy = 0.92
• matthews_corrcoef = 0.81
• f1-score = 0.94

Run it yourself

To run it youself:

• Install Python (>=3.12,<3.14)
• Install Poetry
• Create a clean environment (here it is done with venv, but conda, etc. can also be used)

python -m venv .venv
source .venv/bin/activate

• Clone the repository

git clone https://github.com/epatyukova/llm2025-hackathon-CLUE.git
cd llm2025-hackathon-CLUE

• Install dependencies

poetry install

• Put your cif-file in 'data/cif_files'. Run prediction and explainer on cif file.

python clue/explaination_generation.py --cif_file your_structure.cif

The outputs can be found in 'outputs/' folder.

References:

1. A Perspective on Explanations of Molecular Prediction Models, Geemi P. Wellawatte, Heta A. Gandhi, Aditi Seshadri, and Andrew D. White, Journal of Chemical Theory and Computation 2023 19 (8), 2149-2160, DOI: 10.1021/acs.jctc.2c01235
2. Human interpretable structure-property relationships in chemistry using explainable machine learning and large language models, Wellawatte, G.P., Schwaller, P., Commun Chem 8, 11 (2025). DOI: 10.1038/s42004-024-01393-y
3. Robocrystallographer: Automated crystal structure text descriptions and analysis, Ganose, A., & Jain, A., MRS Communications, 2019, 9(3), 874-881. DOI: 10.1557/mrc.2019.94
4. The Earth Mover’s Distance as a Metric for the Space of Inorganic Compositions, Cameron J. Hargreaves, Matthew S. Dyer, Michael W. Gaultois, Vitaliy A. Kurlin, and Matthew J Rosseinsky, Chemistry of Materials 2020 32 (24), 10610-10620, DOI: 10.1021/acs.chemmater.0c03381
5. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design, npj Computational Materials 6, 173 (2020). DOI: 10.1038/s41524-020-00440-1
6. Accelerated data-driven materials science with the Materials Project, Horton, M.K., Huck, P., Yang, R.X. et al., Nat. Mater. (2025). DOI: 10.1038/s41563-025-02272-0
7. MC3D: The Materials Cloud computational database of experimentally known stoichiometric inorganics, Sebastiaan P. Huber et al., https://arxiv.org/abs/2508.19223
8. Python Materials Genomics (pymatgen) : A Robust, Open-Source Python Library for Materials Analysis, Shyue Ping Ong et al, Computational Materials Science, 2013, 68, 314–319. DOI: 10.1016/j.commatsci.2012.10.028
9. Crystal structure generation with autoregressive large language modeling, Antunes, L.M., Butler, K.T. & Grau-Crespo, R, Nat Commun 15, 10570 (2024). DOI: 10.1038/s41467-024-54639-7
10. DScribe: Library of descriptors for machine learning in materials science, Lauri Himanen et al, Computer Physics Communications, 2020, 247, 106949. DOI: 10.1016/j.cpc.2019.106949
11. ChatGPT5 was used for writing parts of the code and project description.