Merge pull request #14 from SCANL/master

sync master and dev

Author: cnewman

.github/workflows/tests.yml

@@ -2,9 +2,9 @@ name: SCALAR Tagger CI
 
 on:
   push:
-    branches: [ main, develop ]
+    branches: [ master, develop, distilbert ]
   pull_request:
-    branches: [ main, develop ]
+    branches: [ master, develop ]
 
 jobs:
   test-docker:
@@ -78,12 +78,12 @@ jobs:
       
       - name: Start tagger server
         run: |
-          ./main -r &
+          python main --mode run --model_type lm_based &
           
           # Wait for up to 5 minutes for the service to start and load models
           timeout=300
           while [ $timeout -gt 0 ]; do
-            if curl -s "http://localhost:8080/cache/numberArray/DECLARATION" > /dev/null; then
+            if curl -s "http://localhost:8080/numberArray/DECLARATION" > /dev/null; then
               echo "Service is ready"
               break
             fi
@@ -101,7 +101,7 @@ jobs:
       
       - name: Test tagger endpoint
         run: |
-          response=$(curl -s "http://localhost:8080/cache/numberArray/DECLARATION")
+          response=$(curl -s "http://localhost:8080/numberArray/DECLARATION")
           if [ -z "$response" ]; then
             echo "No response from tagger"
             exit 1
@@ -112,4 +112,4 @@ jobs:
         uses: actions/cache@v3
         with:
           path: ~/.cache/gensim-data/fasttext-wiki-news-subwords-300*
-          key: ${{ runner.os }}-fasttext-model
+          key: ${{ runner.os }}-fasttext-model

.gitignore

@@ -2,4 +2,4 @@ output/
 __pycache__/
 code2vec/
 cache/
-input.txt
+input.txt

Dockerfile

@@ -2,10 +2,11 @@ FROM python:3.12-slim
 
 # Install (and build) requirements
 COPY requirements.txt /requirements.txt
-RUN apt-get update && \
-    apt-get install -y git curl && \
+RUN apt-get clean && rm -rf /var/lib/apt/lists/* && \
+    apt-get update --fix-missing && \
+    apt-get install --allow-unauthenticated -y git curl && \
     pip install -r requirements.txt && \
-    rm -rf /var/lib/apt/lists/*
+    apt-get clean && rm -rf /var/lib/apt/lists/*
 
 COPY . .
 RUN pip install -e .
@@ -69,6 +70,6 @@ CMD date; \
     fi; \
     date; \
     echo "Running..."; \
-    /main -r --words words/abbreviationList.csv
+    /main --mode train --model_type lm_based --words words/abbreviationList.csv
 
-ENV TZ=US/Michigan
+ENV TZ=US/Michigan

LICENSE

@@ -1,152 +1,180 @@
-# SCALAR Part-of-speech tagger
-This the official release of the SCALAR Part-of-speech tagger
+# SCALAR Part-of-Speech Tagger for Identifiers
 
-There are two ways to run the tagger. This document describes both ways.
+**SCALAR** is a part-of-speech tagger for source code identifiers. It supports two model types:
 
-1. Using Docker compose (which runs the tagger's built-in server for you)
-2. Running the tagger's built-in server without Docker
+- **DistilBERT-based model with CRF layer** (Recommended: faster, more accurate)
+- Legacy Gradient Boosting model (for compatibility)
 
-## Getting Started with Docker
+---
 
-To run SCNL tagger in a Docker container you can clone the repository and pull the latest docker impage from `srcml/scanl_tagger:latest`
+## Installation
 
-Make sure you have Docker and Docker Compose installed:
-https://docs.docker.com/engine/install/
-https://docs.docker.com/compose/install/
+Make sure you have `python3.12` installed. Then:
 
-```
-git clone git@github.com:SCANL/scanl_tagger.git
+```bash
+git clone https://github.com/SCANL/scanl_tagger.git
 cd scanl_tagger
-docker compose pull
-docker compose up
+python -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
 ```
 
-## Getting Started without Docker
-You will need `python3.12` installed. 
+---
 
-You'll need to install `pip` -- https://pip.pypa.io/en/stable/installation/
-
-Set up a virtual environtment: `python -m venv /tmp/tagger` -- feel free to put it somewhere else (change /tmp/tagger) if you prefer
+## Usage
 
-Activate the virtual environment: `source /tmp/tagger/bin/activate` (you can find how to activate it here if `source` does not work for you -- https://docs.python.org/3/library/venv.html#how-venvs-work)
+You can run SCALAR in multiple ways:
 
-After it's installed and your virtual environment is activated, in the root of the repo, run `pip install -r requirements.txt`
+### CLI (with DistilBERT or GradientBoosting model)
 
-Finally, we require the `token` and `target` vectors from [code2vec](https://github.com/tech-srl/code2vec). The tagger will attempt to automatically download them if it doesn't find them, but you could download them yourself if you like. It will place them in your local directory under `./code2vec/*`
+```bash
+python main --mode run --model_type lm_based         # DistilBERT (recommended)
+python main --mode run --model_type tree_based       # Legacy model
+```
 
-## Usage
+Then query like:
 
 ```
-usage: main [-h] [-v] [-r] [-t] [-a ADDRESS] [--port PORT] [--protocol PROTOCOL]
-            [--words WORDS]
-
-options:
-  -h, --help            show this help message and exit
-  -v, --version         print tagger application version
-  -r, --run             run server for part of speech tagging requests
-  -t, --train           run training set to retrain the model
-  -a ADDRESS, --address ADDRESS
-                        configure server address
-  --port PORT           configure server port
-  --protocol PROTOCOL   configure whether the server uses http or https
-  --words WORDS         provide path to a list of acceptable abbreviations
+http://127.0.0.1:8080/GetValue/FUNCTION
 ```
 
-`./main -r` will start the server, which will listen for identifier names sent via HTTP over the route:
-
-http://127.0.0.1:5000/{cache_selection}/{identifier_name}/{code_context}
-
-**NOTE: ** On docker, the port is 8080 instead of 5000.
-
-"cache selection" will save results to a separate cache if it is set to "student"
-
-"code context" is one of:
+Supports context types:
 - FUNCTION
-- ATTRIBUTE
 - CLASS
+- ATTRIBUTE
 - DECLARATION
 - PARAMETER
 
-For example:
+---
+
+## Training
+
+You can retrain either model (default parameters are currently hardcoded):
+
+```bash
+python main --mode train --model_type lm_based
+python main --mode train --model_type tree_based
+```
+
+---
 
-Tag a declaration: ``http://127.0.0.1:5000/cache/numberArray/DECLARATION``
+## Evaluation Results
 
-Tag a function: ``http://127.0.0.1:5000/cache/GetNumberArray/FUNCTION``
+### DistilBERT (LM-Based Model) — Recommended
 
-Tag an class: ``http://127.0.0.1:5000/cache/PersonRecord/CLASS``
+| Metric                   | Score   |
+|--------------------------|---------|
+| **Macro F1**             | 0.9032  |
+| **Token Accuracy**       | 0.9223  |
+| **Identifier Accuracy**  | 0.8291  |
 
-#### Note
-Kebab case is not currently supported due to the limitations of Spiral. Attempting to send the tagger identifiers which are in kebab case will result in the entry of a single noun. 
+| Label | Precision | Recall | F1    | Support |
+|-------|-----------|--------|-------|---------|
+| CJ    | 0.88      | 0.88   | 0.88  | 8       |
+| D     | 0.98      | 0.96   | 0.97  | 52      |
+| DT    | 0.95      | 0.93   | 0.94  | 45      |
+| N     | 0.94      | 0.94   | 0.94  | 418     |
+| NM    | 0.91      | 0.93   | 0.92  | 440     |
+| NPL   | 0.97      | 0.97   | 0.97  | 79      |
+| P     | 0.94      | 0.92   | 0.93  | 79      |
+| PRE   | 0.79      | 0.79   | 0.79  | 68      |
+| V     | 0.89      | 0.84   | 0.86  | 110     |
+| VM    | 0.79      | 0.85   | 0.81  | 13      |
 
-You will need to have a way to parse code and filter out identifier names if you want to do some on-the-fly analysis of source code. We recommend [srcML](https://www.srcml.org/). Since the actual tagger is a web server, you don't have to use srcML. You could always use other AST-based code representations, or any other method of obtaining identifier information. 
+**Inference Performance:**
+- Identifiers/sec: 225.8
 
+---
 
-## Tagset
+### Gradient Boost Model (Legacy)
 
-**Supported Tagset**
-| Abbreviation |                 Expanded Form                |                   Examples                   |
-|:------------:|:--------------------------------------------:|:--------------------------------------------:|
-|       N      |                     noun                     | Disneyland, shoe, faucet, mother             |
-|      DT      |                  determiner                  | the, this, that, these, those, which         |
-|      CJ      |                  conjunction                 | and, for, nor, but, or, yet, so              |
-|       P      |                  preposition                 | behind, in front of, at, under, above        |
-|      NPL     |                  noun plural                 | Streets, cities, cars, people, lists         |
-|      NM      | noun modifier  (**noun-adjunct**, adjective) | red, cold, hot, **bit**Set, **employee**Name |
-|       V      |                     verb                     | Run, jump, spin,                             |
-|      VM      |            verb modifier  (adverb)           | Very, loudly, seriously, impatiently         |
-|       D      |                     digit                    | 1, 2, 10, 4.12, 0xAF                         |
-|      PRE     |                   preamble                   | Gimp, GLEW, GL, G, p, m, b                   |
+| Metric               | Score     |
+|----------------------|-----------|
+| Accuracy             | 0.8216    |
+| Balanced Accuracy    | 0.9160    |
+| Weighted Recall      | 0.8216    |
+| Weighted Precision   | 0.8245    |
+| Weighted F1          | 0.8220    |
+| Inference Time       | 249.05s   |
 
-**Penn Treebank to SCALAR tagset**
+**Inference Performance:**
+- Identifiers/sec: 8.6
 
-|   Penn Treebank Annotation  | SCALAR Tagset            |
-|:---------------------------:|:------------------------:|
-|       Conjunction (CC)      |     Conjunction (CJ)     |
-|          Digit (CD)         |         Digit (D)        |
-|       Determiner (DT)       |      Determiner (DT)     |
-|      Foreign Word (FW)      |         Noun (N)         |
-|       Preposition (IN)      |      Preposition (P)     |
-|        Adjective (JJ)       |    Noun Modifier (NM)    |
-| Comparative Adjective (JJR) |    Noun Modifier (NM)    |
-| Superlative Adjective (JJS) |    Noun Modifier (NM)    |
-|        List Item (LS)       |         Noun (N)         |
-|          Modal (MD)         |         Verb (V)         |
-|      Noun Singular (NN)     |         Noun (N)         |
-|      Proper Noun (NNP)      |         Noun (N)         |
-|  Proper Noun Plural (NNPS)  |     Noun Plural (NPL)    |
-|      Noun Plural (NNS)      |     Noun Plural (NPL)    |
-|         Adverb (RB)         |    Verb Modifier (VM)    |
-|   Comparative Adverb (RBR)  |    Verb Modifier (VM)    |
-|        Particle (RP)        |    Verb Modifier (VM)    |
-|         Symbol (SYM)        |         Noun (N)         |
-|     To Preposition (TO)     |      Preposition (P)     |
-|          Verb (VB)          |         Verb (V)         |
-|          Verb (VBD)         |         Verb (V)         |
-|          Verb (VBG)         |         Verb (V)         |
-|          Verb (VBN)         |         Verb (V)         |
-|          Verb (VBP)         |         Verb (V)         |
-|          Verb (VBZ)         |         Verb (V)         |
+---
 
-## Training the tagger
-You can train this tagger using the `-t` option (which will re-run the training routine). For the moment, most of this is hard-coded in, so if you want to use a different data set/different seeds, you'll need to modify the code. This will potentially change in the future.
+## Supported Tagset
+
+| Tag   | Meaning                            | Examples                       |
+|-------|------------------------------------|--------------------------------|
+| N     | Noun                               | `user`, `Data`, `Array`        |
+| DT    | Determiner                         | `this`, `that`, `those`        |
+| CJ    | Conjunction                        | `and`, `or`, `but`             |
+| P     | Preposition                        | `with`, `for`, `in`            |
+| NPL   | Plural Noun                        | `elements`, `indices`          |
+| NM    | Noun Modifier (adjective-like)     | `max`, `total`, `employee`     |
+| V     | Verb                               | `get`, `set`, `delete`         |
+| VM    | Verb Modifier (adverb-like)        | `quickly`, `deeply`            |
+| D     | Digit                              | `1`, `2`, `10`, `0xAF`         |
+| PRE   | Preamble / Prefix                  | `m`, `b`, `GL`, `p`            |
+
+---
+
+## Docker Support (Legacy only)
+
+For the legacy server, you can also use Docker:
+
+```bash
+docker compose pull
+docker compose up
+```
+
+---
+
+## Notes
+
+- **Kebab case** is not supported (e.g., `do-something-cool`).
+- Feature and position tokens (e.g., `@pos_0`) are inserted automatically.
+- Internally uses [WordNet](https://wordnet.princeton.edu/) for lexical features.
+- Input must be parsed into identifier tokens. We recommend [srcML](https://www.srcml.org/) but any AST-based parser works.
+
+---
+
+## Citations
+
+Please cite:
+
+```
+@inproceedings{newman2025scalar,
+  author    = {Christian Newman and Brandon Scholten and Sophia Testa and others},
+  title     = {SCALAR: A Part-of-speech Tagger for Identifiers},
+  booktitle = {ICPC Tool Demonstrations Track},
+  year      = {2025}
+}
+
+@article{newman2021ensemble,
+  title={An Ensemble Approach for Annotating Source Code Identifiers with Part-of-speech Tags},
+  author={Newman, Christian and Decker, Michael and AlSuhaibani, Reem and others},
+  journal={IEEE Transactions on Software Engineering},
+  year={2021},
+  doi={10.1109/TSE.2021.3098242}
+}
+```
 
-## Errors?
-Please make an issue if you run into errors
+---
 
-# Please Cite the Paper(s)!
+## Training Data
 
-Newman, Christian, Scholten , Brandon, Testa, Sophia, Behler, Joshua, Banabilah, Syreen, Collard, Michael L., Decker, Michael, Mkaouer, Mohamed Wiem, Zampieri, Marcos, Alomar, Eman Abdullah, Alsuhaibani, Reem, Peruma, Anthony, Maletic, Jonathan I., (2025), “SCALAR: A Part-of-speech Tagger for Identifiers”, in the Proceedings of the 33rd IEEE/ACM International Conference on Program Comprehension - Tool Demonstrations Track (ICPC), Ottawa, ON, Canada, April 27 -28, 5 pages TO APPEAR.
+You can find the most recent SCALAR training dataset [here](https://github.com/SCANL/scanl_tagger/blob/master/input/tagger_data.tsv)
 
-Christian  D.  Newman,  Michael  J.  Decker,  Reem  S.  AlSuhaibani,  Anthony  Peruma,  Satyajit  Mohapatra,  Tejal  Vishnoi, Marcos Zampieri, Mohamed W. Mkaouer, Timothy J. Sheldon, and Emily Hill, "An Ensemble Approach for Annotating Source Code Identifiers with Part-of-speech Tags," in IEEE Transactions on Software Engineering, doi: 10.1109/TSE.2021.3098242.
+---
 
-# Training set
-The data used to train this tagger can be found in the most recent database update in the repo -- https://github.com/SCANL/scanl_tagger/blob/master/input/scanl_tagger_training_db_11_29_2024.db
+## More from SCANL
 
-# Interested in our other work?
-Find our other research [at our webpage](https://www.scanl.org/) and check out the [Identifier Name Structure Catalogue](https://github.com/SCANL/identifier_name_structure_catalogue)
+- [SCANL Website](https://www.scanl.org/)
+- [Identifier Name Structure Catalogue](https://github.com/SCANL/identifier_name_structure_catalogue)
 
-# WordNet
-This project uses WordNet to perform a dictionary lookup on the individual words in each identifier:
+---
 
-Princeton University "About WordNet." [WordNet](https://wordnet.princeton.edu/). Princeton University. 2010
+## Trouble?
 
+Please [open an issue](https://github.com/SCANL/scanl_tagger/issues) if you encounter problems!

README.md

@@ -20,3 +20,4 @@ services:
       - words:/words
     ports:
       - "${PORT-8080}:5000"
+    restart: always