compute accuracy of word embedding model in fasttext

[lili2018]

Hi, I trained two word embedding model with following command
./fasttext skipgram -input traindata.txt -output sg-model -dim 300 -lr 0.05
./fasttext cbow -input traindata.txt -output cbow-model -dim 300 -lr 0.05
Now, I want to compute accuracy of these models.
How can do it in fasttext?

[zaghaghi]

Hi @lili2018
word embedding models are trained in an unsupervised way so you can use a benchmark dataset to evaluate these models.

[a11apurva]

@lili2018, You can pick a few word from your data-set and query for the nearest words to it. It wont give you an absolute value of accuracy but you can get a feel/sense how accurate your models are.

[Witiko]

I assume @lili2018 is asking how to easily reproduce the analogy results, such as those in Table 2 of Bojanowski et al. (2018). I had the same question, and the answer is that you can use the fastText Python module for that:

$ git clone https://github.com/facebookresearch/fastText.git
$ cd fastText
$ git show --oneline --no-patch HEAD
6dd2e11 Fix Aligned word vectors URL

$ ./get-wikimedia.sh
Saving data in data/wikimedia/20190607
Choose a language (e.g. en, bh, fr, etc.): en
Chosen language: en
Continue to download (WARNING: This might be big and can take a long time!)(y/n)? y
Starting download...
Processing data/wikimedia/20190607/enwiki-latest-pages-articles.xml.bz2

$ make
$ ./fasttext skipgram -input data/wikimedia/20190607/wiki.en.txt \
>                     -output model \
>                     -bucket 2000000 \
>                     -dim 300 \
>                     -epoch 5 \
>                     -loss ns \
>                     -lr 0.05 \
>                     -minn 3 \
>                     -maxn 6 \
>                     -minCount 5 \
>                     -neg 5 \
>                     -t 0.0001 \
>                     -ws 5 \
>                     -thread 32
Read 4650M words
Number of words:  2876780
Number of labels: 0
Progress: 100.0% words/sec/thread:    7768 lr:  0.000000 loss:  0.089633 ETA:   0h 0m

$ wget http://download.tensorflow.org/data/questions-words.txt
$ pip install .
$ python python/doc/examples/compute_accuracy.py model.bin questions-words.txt 200000
      capital-common-countries: ACCURACY TOP1: 95.26 %  (482 / 506)       Total accuracy: 95.26 %   Semantic accuracy: 95.26 %   Syntactic accuracy: 0.00 %
                 capital-world: ACCURACY TOP1: 91.67 %  (4147 / 4524)     Total accuracy: 92.03 %   Semantic accuracy: 92.03 %   Syntactic accuracy: 0.00 %
                      currency: ACCURACY TOP1: 13.12 %  (106 / 808)       Total accuracy: 81.11 %   Semantic accuracy: 81.11 %   Syntactic accuracy: 0.00 %
                 city-in-state: ACCURACY TOP1: 71.46 %  (1763 / 2467)     Total accuracy: 78.24 %   Semantic accuracy: 78.24 %   Syntactic accuracy: 0.00 %
                        family: ACCURACY TOP1: 81.82 %  (414 / 506)       Total accuracy: 78.45 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 0.00 %
     gram1-adjective-to-adverb: ACCURACY TOP1: 49.50 %  (491 / 992)       Total accuracy: 75.52 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 49.50 %
                gram2-opposite: ACCURACY TOP1: 52.22 %  (424 / 812)       Total accuracy: 73.74 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 50.72 %
             gram3-comparative: ACCURACY TOP1: 87.24 %  (1162 / 1332)     Total accuracy: 75.24 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 66.23 %
             gram4-superlative: ACCURACY TOP1: 72.73 %  (768 / 1056)      Total accuracy: 75.04 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 67.87 %
      gram5-present-participle: ACCURACY TOP1: 64.49 %  (681 / 1056)      Total accuracy: 74.24 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 67.19 %
   gram6-nationality-adjective: ACCURACY TOP1: 90.87 %  (1453 / 1599)     Total accuracy: 75.94 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 72.72 %
              gram7-past-tense: ACCURACY TOP1: 63.65 %  (993 / 1560)      Total accuracy: 74.83 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 71.04 %
                  gram8-plural: ACCURACY TOP1: 89.41 %  (1191 / 1332)     Total accuracy: 75.88 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 73.55 %
            gram9-plural-verbs: ACCURACY TOP1: 73.10 %  (636 / 870)       Total accuracy: 75.75 %   Semantic accuracy: 78.45 %   Syntactic accuracy: 73.51 %
Questions seen / total: 19420 19544   99.37 %