[
{
"path": ".gitignore",
"content": "__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndist/\ndevelop-eggs/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don’t work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n"
},
{
"path": "README.md",
"content": "ETNLP: A Toolkit for Extraction, Evaluation and Visualization of Pre-trained Word Embeddings\n=====\n\n# Table of contents\n1. [Introduction](#introduction)\n2. [More about ETNLP](#moreaboutETNLP)\n3. [Installation and How to Use](#installation_and_howtouse)\n4. [Download Resources](#Download_Resources)\n\n\n# I. Overview \n## A glimpse of ETNLP:\n- Github: https://github.com/vietnlp/etnlp\n- Video: https://vimeo.com/317599106\n- Paper: https://arxiv.org/abs/1903.04433\n\n\n# II. How do I cite ETNLP?\nPlease CITE paper the Arxiv paper whenever ETNLP (or the pre-trained embeddings) is used to produce published results or incorporated into other software:\n\n```\n@inproceedings{vu:2019n,\n title={ETNLP: A Visual-Aided Systematic Approach to Select Pre-Trained Embeddings for a Downstream Task},\n author={Vu, Xuan-Son and Vu, Thanh and Tran, Son N and Jiang, Lili},\n booktitle={Proceedings of the International Conference Recent Advances in Natural Language Processing (RANLP)},\n year={2019}\n }\n```\n\n\n# III. More about ETNLP :\n## 1. Embedding Evaluator: \nTo compare quality of embedding models on the word analogy task.\n- Input: a pre-trained embedding vector file (word2vec format), and word analogy file.\n- Output: (1) evaluate quality of the embedding model based on the MAP/P@10 score, (2) Paired t-tests to show significant level between different word embeddings.\n\n### 1.1. Note: The word analogy list is created by:\n- Adopt from the English list by selecting suitable categories and translating to the target language (i.e., Vietnamese). \n- Removing inappropriate categories (i.e., category 6, 10, 11, 14) in the target language (i.e., Vietnamese).\n- Adding custom category that is suitable for the target language (e.g., cities and their zones in Vietnam for Vietnamese).\nSince most of this process is automatically done, it can be applied in other languages as well.\n\n### 1.2. Selected categories for Vietnamese: \n> 1. capital-common-countries\n> 2. capital-world\n> 3. currency: E.g., Algeria | dinar | Angola | kwanza\n> 4. city-in-zone (Vietnam's cities and its zone)\n> 5. family (boy|girl | brother | sister)\n> 6. gram1-adjective-to-adverb (NOT USED)\n> 7. gram2-opposite (e.g., acceptable | unacceptable | aware | unaware)\n> 8. gram3-comparative (e.g., bad | worse | big | bigger)\n> 9. gram4-superlative (e.g., bad | worst | big | biggest)\n> 10. gram5-present-participle (NOT USED)\n> 11. gram6-nationality-adjective-nguoi-tieng (e.g., Albania | Albanian | Argentina | Argentinean)\n> 12. gram7-past-tense (NOT USED)\n> 13. gram8-plural-cac-nhung (e.g., banana | bananas | bird | birds) (NOT USED)\n> 14. gram9-plural-verbs (NOT USED)\n\n### 1.3 Evaluation results (in details)\n\n* Analogy: Word Analogy Task\n\n* NER (w): NER task with hyper-parameters selected from the best F1 on validation set.\n\n* NER (w.o): NER task without selecting hyper-parameters from the validation set.\n\n| Model | NER.w | NER.w.o \t| Analogy \t|\n|------------------------------\t|------------- | ------------------\t|------------------\t|\n| BiLC3 + w2v \t| 89.01 | 89.41 \t| 0.4796 |\n| BiLC3 + Bert_Base \t| 88.26 | 89.91 | 0.4609 |\n| BiLC3 + w2v_c2v \t| 89.46 | 89.46 \t| 0.4796 |\n| BiLC3 + fastText \t| 89.65 | 89.84 \t| 0.4970 |\n| BiLC3 + Elmo \t| 89.67 | 90.84 \t| **0.4999** |\n| BiLC3 + MULTI_WC_F_E_B | **91.09** | **91.75** \t| 0.4906|\n\n\n## 2. Embedding Extractor: To extract embedding vectors for other tasks.\n- Input: (1) list of input embeddings, (2) a vocabulary file.\n- Output: embedding vectors of the given vocab file in `.txt`, i.e., each line conains the embedding for a word. The file then be compressed in .gz format. This format is widely used in existing NLP Toolkits (e.g., Reimers et al. [1]).\n\n### Extra options:\n- `-input-c2v`: character embedding file\n- `solveoov:1`: to solve OOV words of the 1st embedding. Similarly for more than one embedding: e.g., `solveoov:1:2`.\n\n[1] Nils Reimers and Iryna Gurevych, Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging, 2017, http://arxiv.org/abs/1707.09861, arXiv.\n\n## 3. Visualizer: to explore the embedding space and compare between different embeddings.\n\n### Screenshot of viewing multiple-embeddings side-by-side (Vietnamese):\n![Alt text](images/etnlp_view_multi_embeddings.png \"Screenshot multiple-embeddings side-by-side\")\n\n### Screenshot of viewing each embedding interactively (Vietnamese):\n![Alt text](images/etnlp_view_embs.png \"Screenshot example of viewing each embedding interactively\")\n\n### Screenshot of viewing each embedding side-by-side (English):\n![Alt text](images/english_sbs.png \"Screenshot example of viewing each embedding interactively\")\n\n\n# IV. Installation and How to use ETNLP \n## 1. Installation:\n\nFrom source codes (Python 3.6.x):\n> 1. cd src/codes/\n> 2. pip install -r requirements.txt\n> 3. python setup.py install\n\nFrom pip (python 3.6.x)\n> 1. sudo apt-get install python3-dev\n> 2. pip install cython\n> 3. pip install git+git://github.com/vietnlp/etnlp.git\n\nOR:\n> 1. pip install etnlp \n\n## 2. Examples\n> 1. cd src/examples\n> 2. python test1_etnlp_preprocessing.py\n> 3. python test2_etnlp_extractor.py\n> 4. python test3_etnlp_evaluator.py\n> 5. python test4_etnlp_visualizer.py\n### Example of using Fasttext-Sent2Vec:\n- 01. Install: https://github.com/epfml/sent2vec\n```\n01. git clone https://github.com/epfml/sent2vec\n02. cd sent2vec; pip install .\n```\n\n- 02. Extract embeddings for sentences (no requirement for tokenization before extracting embedding of sentences).\n```\nimport sent2vec\nmodel = sent2vec.Sent2vecModel()\nmodel.load_model('opendata_wiki_lowercase_words.bin')\nemb = model.embed_sentence(\"tôi là sinh viên đh công nghệ, đại học quôc gia hà nội\") \nembs = model.embed_sentences([\"tôi là sinh viên\", \"tôi là nhà thơ\", \"tôi là bác sĩ\"])\n\n```\n\n\n### 3. Visualization\nSide-by-side visualization:\n> 1. sh src/codes/04.run_etnlp_visualizer_sbs.sh\n\nInteractive visualization:\n> 1. sh src/codes/04.run_etnlp_visualizer_inter.sh\n\n\n# V. Available Lexical Resources \n## 1. Word Analogy List for Vietnamese\n\n| Word Analogy List | Download Link (NER Task)| Download Link (General)| \n|------------------------------|---------------|---------------|\n| Vietnamese (This work) | [Link1](https://drive.google.com/file/d/1eA5yvla4BhAIfWsmZherT1GEW6gzDC-1/view?usp=sharing)| [Link1](https://drive.google.com/file/d/1YJ9d5rVKMMKF1xWWZi26_sNpgULTvxwg/view?usp=sharing)|\n| English (Mirkolov et al. [2]) | [Link2]| [Link2](https://drive.google.com/file/d/10rWxGu8-nbQmYC8wrIussSZjY0lDh6RP/view?usp=sharing)|\n| Portuguese (Hartmann et al. [3]) | [Link3]| [Link3](https://github.com/nathanshartmann/portuguese_word_embeddings/blob/master/analogies/testset/LX-4WAnalogies.txt)|\n\n\n\n## 2. Multiple pre-trained embedding models for Vietnamese\n\n- Training data: Wiki in Vietnamese:\n\n| # of sentences | # of tokenized words| \n|------------------------------|---------------|\n| 6,685,621 | 114,997,587 |\n\n\n- Download Pre-trained Embeddings: \n(Note: The MULTI_WC_F_E_B is the concatenation of four embeddings: W2V_C2V, fastText, ELMO, and Bert_Base.)\n\n| Embedding Model | Download Link (NER Task) | Download Link (AIVIVN SentiTask) | Download Link (General) | \n|------------------------------|---------------|---------------|---------------|\n| w2v | [Link1](https://drive.google.com/file/d/1LHaZ8LXxteHzod42naqJZYCwwq5mI9aL/view?usp=sharing) (dim=300)| [Link1] | [Link1] |\n| w2v_c2v | [Link2](https://drive.google.com/file/d/1-M9Tb9l8mNmP3RKxZiZNK1Vpbng2yw4l/view?usp=sharing) (dim=300)| [Link2] | [Link2] |\n| fastText | [Link3](https://drive.google.com/file/d/1dHCPhKFjtDjbrUeeymheDnlhjtaljPGE/view?usp=sharing) (dim=300)| [Link3] | [Link3] |\n| fastText-[Sent2Vec](https://github.com/epfml/sent2vec) | [Link3]| [Link3] | [Link3](https://drive.google.com/file/d/1BzL1mpdfqCCJioCdAlTVshbrz0lGfP2D/view?usp=sharing) (dim=300, 6GB, trained on 20GB of [news data](https://github.com/binhvq/news-corpus) and Wiki-data of ETNLP. |\n| Elmo | [Link4](https://drive.google.com/file/d/1zDaSD8NsZNXGyd9iVOxTcb7CP61Ixo-r/view?usp=sharing) (dim=1024)| [Link4](https://drive.google.com/file/d/1jVJtF0f6SbtUd-t3bnywP6mFnz0QXPIx/view?usp=sharing) (dim=1024)| [Link4](https://drive.google.com/file/d/1XPsTzg1Gex-Hh2nl9344YlZc1orOVBDp/view?usp=sharing) (dim=1024, 731MB and 1.9GB after extraction.)|\n| Bert_base | [Link5](https://drive.google.com/file/d/16fRkmIHiB16OlM8WdFmoApGtLMf6YJJ8/view?usp=sharing) (dim=768)| [Link5] | [Link5] |\n| MULTI_WC_F_E_B | [Link6](https://drive.google.com/file/d/1gq7b8hs31VzoeO3n3C__ftlDnE_iBZW2/view?usp=sharing) (dim=2392)| [Link6] | [Link6] |\n\n\n# VI. Versioning\nFor transparency and insight into our release cycle, and for striving to maintain backward compatibility, ETNLP will be maintained under the Semantic Versioning guidelines as much as possible.\n\nReleases will be numbered with the following format:\n\n`..`\n\nAnd constructed with the following guidelines:\n\n* Breaking backward compatibility bumps the major (and resets the minor and patch)\n* New additions without breaking backward compatibility bumps the minor (and resets the patch)\n* Bug fixes and misc changes bumps the patch\n\nFor more information on SemVer, please visit http://semver.org/.\n"
},
{
"path": "src/codes/00.run_etnlp_preprocessing.sh",
"content": "#!/bin/sh\nexport PYTHONPATH=\"$PYTHONPATH:$PWD\"\nINPUT_FILES=\"../data/glove2vec_dicts/glove1.vec;../data/glove2vec_dicts/glove2.vec\"\nOUTPUT_FILES=\"../data/glove2vec_dicts/glove1_w2v.vec;../data/glove2vec_dicts/glove2_w2v.vec\"\n# do_normalize: use this flag to normalize in case of multiple embeddings.\npython ./etnlp_api.py -input $INPUT_FILES -output $OUTPUT_FILES -args \"glove2w2v\"\n"
},
{
"path": "src/codes/01.run_etnlp_evaluator.sh",
"content": "#!/bin/sh\nexport PYTHONPATH=\"$PYTHONPATH:$PWD\"\nINPUT_FILES=\"../data/embedding_dicts/ELMO_23.vec;../data/embedding_dicts/FastText_23.vec;../data/embedding_dicts/W2V_C2V_23.vec;../data/embedding_dicts/MULTI_23.vec\"\nANALOGY_FILE=\"../data/embedding_analogies/vi/solveable_analogies_vi.txt\"\nOUT_FILE=\"../data/embedding_analogies/vi/Multi_evaluator_results.txt\"\npython ./etnlp_api.py -input $INPUT_FILES -output $OUT_FILE -analoglist $ANALOGY_FILE -args eval"
},
{
"path": "src/codes/02.run_etnlp_extractor.sh",
"content": "#!/bin/sh\nexport PYTHONPATH=\"$PYTHONPATH:$PWD\"\nINPUT_FILES=\"../data/embedding_dicts/ELMO_23.vec;../data/embedding_dicts/FastText_23.vec;../data/embedding_dicts/W2V_C2V_23.vec;../data/embedding_dicts/MULTI_23.vec\"\nC2V=\"../data/embedding_dicts/C2V.vec\"\nOUTPUT=\"../data/embedding_dicts/MULTI_W_F_B_E.vec\"\nVOCAB_FILE=\"../data/vocab.txt\"\npython ./etnlp_api.py -input $INPUT_FILES -vocab $VOCAB_FILE -input_c2v $C2V -args \"extract\" -output $OUTPUT\n"
},
{
"path": "src/codes/03.run_etnlp_visualizer_inter.sh",
"content": "#!/bin/sh\nexport PYTHONPATH=\"$PYTHONPATH:$PWD\"\nINPUT_FILES=\"../data/embedding_dicts/ELMO_23.vec;../data/embedding_dicts/FastText_23.vec;../data/embedding_dicts/W2V_C2V_23.vec;../data/embedding_dicts/MULTI_23.vec\"\npython3 ./etnlp_api.py -input $INPUT_FILES -args visualizer -port 8889\n"
},
{
"path": "src/codes/04.run_etnlp_visualizer_sbs.sh",
"content": "#!/bin/sh\nexport PYTHONPATH=\"$PYTHONPATH:$PWD\"\nINPUT_FILES=\"../data/embedding_dicts/ELMO_23.vec;../data/embedding_dicts/FastText_23.vec;../data/embedding_dicts/W2V_C2V_23.vec;../data/embedding_dicts/MULTI_23.vec\"\n# python ./visualizer/visualizer_sbs.py -input $INPUT_FILES -args visualizer\npython3 ./visualizer/visualizer_sbs.py $INPUT_FILES\n"
},
{
"path": "src/codes/api/__init__.py",
"content": ""
},
{
"path": "src/codes/api/embedding_evaluator.py",
"content": "import logging\nimport gensim\nimport argparse\nfrom gensim.models.keyedvectors import WordEmbeddingsKeyedVectors, Word2VecKeyedVectors\nfrom gensim import utils, matutils\nfrom six import string_types\nfrom numpy import dot, float32 as REAL, array, ndarray, argmax\nfrom utils import embedding_io, emb_utils\nfrom embeddings.embedding_configs import EmbeddingConfigs\n\nlogger = logging.getLogger(__name__)\n\n\nclass new_Word2VecKeyedVectors(Word2VecKeyedVectors):\n def __init__(self, vector_size):\n super(Word2VecKeyedVectors, self).__init__(vector_size=vector_size)\n\n def most_similar(self, positive=None, negative=None, topn=10, restrict_vocab=None, indexer=None):\n \"\"\"\n Find the top-N most similar words. Positive words contribute positively towards the\n similarity, negative words negatively.\n\n This method computes cosine similarity between a simple mean of the projection\n weight vectors of the given words and the vectors for each word in the model.\n The method corresponds to the `word-analogy` and `distance` scripts in the original\n word2vec implementation.\n\n If topn is False, most_similar returns the vector of similarity scores.\n\n `restrict_vocab` is an optional integer which limits the range of vectors which\n are searched for most-similar values. For example, restrict_vocab=10000 would\n only check the first 10000 word vectors in the vocabulary order. (This may be\n meaningful if you've sorted the vocabulary by descending frequency.)\n\n Example::\n\n >>> trained_model.most_similar(positive=['woman', 'king'], negative=['man'])\n [('queen', 0.50882536), ...]\n\n \"\"\"\n if positive is None:\n positive = []\n if negative is None:\n negative = []\n\n self.init_sims()\n\n if isinstance(positive, string_types) and not negative:\n # allow calls like most_similar('dog'), as a shorthand for most_similar(['dog'])\n positive = [positive]\n\n # add weights for each word, if not already present; default to 1.0 for positive and -1.0 for negative words\n positive = [\n (word, 1.0) if isinstance(word, string_types + (ndarray,)) else word\n for word in positive\n ]\n negative = [\n (word, -1.0) if isinstance(word, string_types + (ndarray,)) else word\n for word in negative\n ]\n\n # compute the weighted average of all words\n all_words, mean = set(), []\n for word, weight in positive + negative:\n if isinstance(word, ndarray):\n mean.append(weight * word)\n else:\n mean.append(weight * self.word_vec(word, use_norm=True))\n if word in self.vocab:\n all_words.add(self.vocab[word].index)\n if not mean:\n raise ValueError(\"cannot compute similarity with no input\")\n mean = matutils.unitvec(array(mean).mean(axis=0)).astype(REAL)\n\n if indexer is not None:\n return indexer.most_similar(mean, topn)\n\n limited = self.syn0norm if restrict_vocab is None else self.syn0norm[:restrict_vocab]\n dists = dot(limited, mean)\n if not topn:\n return dists\n best = matutils.argsort(dists, topn=topn + len(all_words), reverse=True)\n # ignore (don't return) words from the input\n result = [(self.index2word[sim], float(dists[sim])) for sim in best if sim not in all_words]\n return result[:topn]\n\n def new_accuracy(self, questions, restrict_vocab=30000, most_similar=most_similar, case_insensitive=True):\n \"\"\"\n Compute accuracy of the model. `questions` is a filename where lines are\n 4-tuples of words, split into sections by \": SECTION NAME\" lines.\n See questions-words.txt in\n https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/word2vec/source-archive.zip\n for an example.\n\n The accuracy is reported (=printed to log and returned as a list) for each\n section separately, plus there's one aggregate summary at the end.\n\n Use `restrict_vocab` to ignore all questions containing a word not in the first `restrict_vocab`\n words (default 30,000). This may be meaningful if you've sorted the vocabulary by descending frequency.\n In case `case_insensitive` is True, the first `restrict_vocab` words are taken first, and then\n case normalization is performed.\n\n Use `case_insensitive` to convert all words in questions and vocab to their uppercase form before\n evaluating the accuracy (default True). Useful in case of case-mismatch between training tokens\n and question words. In case of multiple case variants of a single word, the vector for the first\n occurrence (also the most frequent if vocabulary is sorted) is taken.\n\n This method corresponds to the `compute-accuracy` script of the original C word2vec.\n\n \"\"\"\n print(\"INFO: Using new accuracy\")\n ok_vocab = [(w, self.vocab[w]) for w in self.index2word[:restrict_vocab]]\n ok_vocab = {w.upper(): v for w, v in reversed(ok_vocab)} if case_insensitive else dict(ok_vocab)\n\n oov_counter, idx_cnt, is_vn_counter = 0, 0, 0\n sections, section = [], None\n for line_no, line in enumerate(utils.smart_open(questions)):\n # TODO: use level3 BLAS (=evaluate multiple questions at once), for speed\n line = utils.to_unicode(line)\n\n if line.startswith(': '):\n # a new section starts => store the old section\n if section:\n sections.append(section)\n self.log_accuracy(section)\n section = {'section': line.lstrip(': ').strip(), 'correct': [], 'incorrect': []}\n else:\n # Count number of analogy to check\n idx_cnt += 1\n if not section:\n raise ValueError(\"missing section header before line #%i in %s\" % (line_no, questions))\n try:\n if case_insensitive:\n a, b, c, expected = [word.upper() for word in line.split(\" | \")]\n else:\n a, b, c, expected = [word for word in line.split(\" | \")]\n # print(\"Line : \", line)\n # print(\"a, b, c, expected: %s, %s, %s, %s\"%(a, b, c, expected))\n # input(\">>> Wait ...\")\n except ValueError:\n logger.info(\"SVX: ERROR skipping invalid line #%i in %s\", line_no, questions)\n print(\"Line : \", line)\n print(\"a, b, c, expected: %s, %s, %s, %s\" % (a, b, c, expected))\n input(\">>> Wait ...\")\n continue\n\n # In case of Vietnamese, word analogy can be a phrase\n if \" \" in a or \" \" in b or \" \" in c or \" \" in expected:\n is_vn_counter += 1\n pass\n else:\n if a not in ok_vocab or b not in ok_vocab or c not in ok_vocab or expected not in ok_vocab:\n logger.debug(\"SVX: skipping line #%i with OOV words: %s\", line_no, line.strip())\n oov_counter += 1\n continue\n\n original_vocab = self.vocab\n self.vocab = ok_vocab\n ignore = {a, b, c} # input words to be ignored\n predicted = None\n # find the most likely prediction, ignoring OOV words and input words\n sims = most_similar(self, positive=[b, c], negative=[a], topn=False, restrict_vocab=restrict_vocab)\n self.vocab = original_vocab\n for index in matutils.argsort(sims, reverse=True):\n predicted = self.index2word[index].upper() if case_insensitive else self.index2word[index]\n if predicted in ok_vocab and predicted not in ignore:\n if predicted != expected:\n logger.debug(\"%s: expected %s, predicted %s\", line.strip(), expected, predicted)\n break\n if predicted == expected:\n section['correct'].append((a, b, c, expected))\n else:\n section['incorrect'].append((a, b, c, expected))\n\n if section:\n # store the last section, too\n sections.append(section)\n self.log_accuracy(section)\n\n total = {\n 'OOV/Total/VNCompound_Words': [oov_counter, (idx_cnt), is_vn_counter],\n 'section': 'total',\n 'correct': sum((s['correct'] for s in sections), []),\n 'incorrect': sum((s['incorrect'] for s in sections), []),\n }\n self.log_accuracy(total)\n sections.append(total)\n return sections\n\n\ndef convert_conll_format_to_normal(connl_file, out_file):\n \"\"\"\n read file conll format\n return format : One sentence per line\n sentences_arr: [EU rejects German call .., ...]\n tags_arr: [B-ORG O B-MIST O ..., ...]\n \"\"\"\n f = open(connl_file)\n sentences = []\n sentence = \"\"\n for line in f:\n # print(\"line: \", line)\n if len(line) == 0 or line.startswith('-DOCSTART') or line[0] == \"\\n\":\n sentences.append(sentence.rstrip())\n sentence = \"\"\n continue\n else:\n splits = line.split('\\t')\n sentence += splits[1].rstrip() + \" \"\n\n # To handle the last sentence.\n if len(sentence) > 0:\n sentences.append(sentence)\n del sentence\n\n # Write to output\n if out_file is None:\n out_file = connl_file + \".std.txt\"\n writer = open(out_file, \"w\")\n for sen in sentences:\n writer.write(sen + \"\\n\")\n writer.flush()\n writer.close()\n\n return sentences\n\n\ndef verify_word_analogies(file):\n \"\"\"\n Verify the word analogy file.\n :param file:\n :return:\n \"\"\"\n f_reader = open(file, \"r\")\n\n valid_cnt, invalid_cnt = 0, 0\n\n for line in f_reader:\n # print(\"line: \", line)\n if len(line) == 0 or line.startswith('-DOCSTART') or line[0] == \"\\n\":\n continue\n else:\n splits = line.split('\\t')\n if len(splits) != 4:\n invalid_cnt += 1\n else:\n valid_cnt += 1\n\n print(\"Valid analogy: %s, invalid analogy: %s\" % (valid_cnt, invalid_cnt))\n\n\ndef check_oov_of_word_analogies(w2v_format_emb_file, analogy_file, is_vn=True, case_sensitive=True):\n emb_model = gensim.models.KeyedVectors.load_word2vec_format(w2v_format_emb_file,\n binary=False,\n unicode_errors='ignore')\n\n f_reader = open(analogy_file, \"r\")\n vocab_arr = []\n for line in f_reader:\n if not case_sensitive:\n line = line.lower()\n\n if line.startswith(': '):\n continue\n else:\n for word in line.split(\" | \"):\n # In Vietnamese, we have compound and single word.\n # if is_vn:\n # if \" \" in word:\n # print(\"I should not going here\")\n # single_words = word.split(\" \")\n # for single_word in single_words:\n # vocab_arr.append(single_word)\n # For other languages.\n # else:\n vocab_arr.append(word)\n\n print(\"Before unique set: len = \", len(vocab_arr))\n unique_vocab_arr = set(vocab_arr)\n print(\"After unique set: len = \", len(unique_vocab_arr))\n valid_word_cnt = 0\n for word in unique_vocab_arr:\n if word in emb_model:\n valid_word_cnt += 1\n\n print(\"With Is_VN = %s, case_sensitive = %s, Valid word = %s/%s\" % (is_vn,\n case_sensitive,\n valid_word_cnt,\n len(unique_vocab_arr)))\n\n\ndef evaluator_api(input_files, analoglist, output, embed_config=None):\n \"\"\"\n\n :param input_files:\n :param analoglist:\n :param output:\n :param embed_config:\n :return:\n \"\"\"\n if embed_config is None:\n embed_config = EmbeddingConfigs() # Initialize default config for embedding.\n local_embedding_names, local_word_embeddings = embedding_io.load_word_embeddings(input_files, embed_config)\n # emb_utils.print_analogy('man', 'him', 'woman', emb_words)\n local_output_str = emb_utils.eval_word_analogy_4_all_embeddings(analoglist,\n local_embedding_names,\n local_word_embeddings,\n output_file=output)\n print(\"OUTPUT: \", local_output_str)\n\n\nif __name__ == \"__main__\":\n \"\"\"\n Evaluates a given word embedding model.\n To use:\n evaluate.py path_to_model [-restrict]\n optional restrict argument performs an evaluation using the original\n Mikolov restriction of vocabulary\n \"\"\"\n\n desc = \"Evaluates a word embedding model\"\n parser = argparse.ArgumentParser(description=desc)\n parser.add_argument(\"-input\",\n required=True,\n default=\"../data/embedding_dicts/ELMO_23.vec\",\n help=\"Input multiple word embeddings, each model separated by a `;`.\")\n parser.add_argument(\"-analoglist\",\n nargs=\"?\",\n # default=\"../data/embedding_analogies/vi/analogy_vn_seg.txt.std.txt\",\n default=\"../data/embedding_analogies/vi/solveable_analogies_vi.txt\",\n help=\"Input analogy file to run the word analogy evaluation.\")\n\n parser.add_argument(\"-r\",\n nargs=\"?\",\n default=False,\n help=\"Vocabulary restriction\")\n\n parser.add_argument(\"-checkoov\",\n nargs=\"?\",\n default=False,\n help=\"Check OOV percentage\")\n\n parser.add_argument(\"-lang\",\n nargs=\"?\",\n default=\"VI\",\n help=\"Specify language, by default, it's Vietnamese.\")\n\n parser.add_argument(\"-lowercase\",\n nargs=\"?\",\n default=True,\n help=\"Lowercase all word analogies? (depends on how the emb was trained).\")\n\n parser.add_argument(\"-output\",\n nargs=\"?\",\n default=\"../data/embedding_analogies/vi/results_out.txt\",\n help=\"Output file of word analogy task\")\n\n parser.add_argument(\"-remove_redundancy\",\n nargs=\"?\",\n default=True,\n help=\"Remove redundancy in predicted words\")\n\n print(\"Params: \", parser)\n\n args = parser.parse_args()\n\n embedding_config = EmbeddingConfigs()\n\n paths_of_models = args.input\n testset = args.analoglist\n is_vietnamese = args.lang\n output_file = args.output\n\n # use restriction?\n restriction = None\n if args.r:\n restriction = 30000\n\n # set logging definitions\n logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',\n level=logging.INFO)\n\n if args.checkoov:\n print(\"Checking OOV ...\")\n check_oov_of_word_analogies(paths_of_models, testset, is_vn=is_vietnamese)\n\n if not args.checkoov:\n print(\"Evaluating embeddings on the word analogy task ...\")\n if is_vietnamese:\n print(\" ... for ETNLP's evaluation approach.\")\n embedding_names, word_embeddings = embedding_io.load_word_embeddings(paths_of_models, embedding_config)\n # emb_utils.print_analogy('man', 'him', 'woman', emb_words)\n output_str = emb_utils.eval_word_analogy_4_all_embeddings(testset, embedding_names, word_embeddings,\n output_file=args.output_file)\n print(\"#\"*20)\n print(output_str)\n print(\"#\" * 20)\n\n else:\n print(\" ... for Mirkolov et al.'s evaluation approach.\")\n word_analogy_obj = new_Word2VecKeyedVectors(1024)\n\n # load and evaluate\n model = word_analogy_obj.load_word2vec_format(\n paths_of_models,\n binary=False,\n unicode_errors='ignore')\n\n model.accuracy = word_analogy_obj.new_accuracy\n\n acc = model.accuracy(testset, restrict_vocab=restriction, case_insensitive=False)\n print(\"Acc = \", acc)\n\n print(\"DONE\")\n"
},
{
"path": "src/codes/api/embedding_extractor.py",
"content": "from embeddings import embedding_utils\nfrom pathlib import Path\nimport numpy as np\nimport os\nimport logging\nimport gzip\nfrom embeddings.embedding_configs import EmbeddingConfigs\n\n\ndef get_multi_embedding_models(config: EmbeddingConfigs):\n \"\"\"\n\n :param config:\n :return:\n \"\"\"\n model_paths_list = config.model_paths_list\n model_names_list = config.model_names_list\n model_dims_list = config.model_dims_list\n char_model_path = config.char_model_path\n char_model_dims = config.char_model_dims\n\n if char_model_path:\n char_model = embedding_utils.reload_char2vec_model(char_model_path, char_model_dims)\n else:\n char_model = None\n\n embedding_models = embedding_utils.reload_embedding_models(model_paths_list,\n model_names_list,\n model_dims_list,\n char_model)\n # doc_vector = embedding_models.get_vector_of_document(tokenized_text)\n return embedding_models\n\n\ndef get_emb_dim(emb_file):\n idx = 0\n dim = 0\n with open(emb_file, \"r\") as reader:\n if idx == 0:\n line = reader.readline().rstrip()\n dim = int(line.split(\" \")[1])\n return dim\n\n\ndef extract_embedding_for_vocab_file(paths_of_emb_models, vocab_words_file, c2v_emb_file, output_file, output_format):\n \"\"\"\n\n :param paths_of_emb_models:\n :param vocab_words_file:\n :param c2v_emb_file:\n :param output_file:\n :param output_format:\n :return:\n \"\"\"\n config = EmbeddingConfigs()\n config.output_format = output_format\n config.model_paths_list = paths_of_emb_models.split(\";\")\n embedding_file_names = []\n embedding_dims = []\n\n if c2v_emb_file:\n config.char_model_path = c2v_emb_file\n config.char_model_dims = get_emb_dim(c2v_emb_file)\n\n print(\"02. Extracting word embeddings ...\")\n if paths_of_emb_models and paths_of_emb_models.__contains__(\";\"):\n files = paths_of_emb_models.split(\";\")\n for emb_file in files:\n embedding_name = os.path.basename(os.path.normpath(emb_file))\n embedding_file_names.append(embedding_name)\n embedding_dim = get_emb_dim(emb_file)\n embedding_dims.append(embedding_dim)\n elif paths_of_emb_models: # In case there is only one embedding\n embedding_name = os.path.basename(os.path.normpath(paths_of_emb_models))\n embedding_file_names.append(embedding_name)\n embedding_dim = get_emb_dim(paths_of_emb_models)\n embedding_dims.append(embedding_dim)\n else:\n raise Exception(\"List of embeddings cannot be None.\")\n\n # Data type:\n embedding_names = [\"word2vec\"]*len(embedding_dims) # embedding type, only support w2v and c2v type now\n config.model_names_list = embedding_names\n config.model_dims_list = embedding_dims\n\n # Do extracting embeddings\n extract_embedding_vectors(vocab_words_file, output_file, config)\n print(\"Done\")\n\n\ndef extract_embedding_vectors(vocab_words_file, output_file, config: EmbeddingConfigs):\n \"\"\"\n\n :param vocab_words_file:\n :param output_file:\n :param config:\n :return:\n \"\"\"\n # Load vocab\n with Path(vocab_words_file).open() as f:\n word_to_idx = {line.strip(): idx for idx, line in enumerate(f)}\n size_vocab = len(word_to_idx)\n\n # Output writer\n fwriter = open(output_file, \"w\")\n\n # Array of zeros\n dim_size = sum(config.model_dims_list)\n found = 0\n print('Reading embedding file (may take a while)')\n\n embedding_models = get_multi_embedding_models(config)\n\n embeddings = np.zeros((size_vocab, dim_size))\n\n line_idx = 0\n for word in word_to_idx.keys():\n word_idx = word_to_idx[word]\n\n word = word.rstrip()\n try:\n if line_idx % 100000 == 0:\n print('- At line {}'.format(line_idx))\n\n w2v_vector = embedding_models.get_word_vector_of_multi_embeddings(word)\n\n if w2v_vector is not None and len(w2v_vector) > 0:\n embeddings[word_idx] = w2v_vector\n line = \"%s %s\" % (word, \" \".join(str(scalar) for scalar in w2v_vector))\n fwriter.write(line + \"\\n\")\n fwriter.flush()\n found += 1\n\n logging.debug(\"Embedding: \", w2v_vector)\n except Exception as e:\n logging.debug(\"Unexpected error: word = %s, error = %s\" % (word, e))\n pass\n line_idx += 1\n\n print('- done. Found {} vectors for {} words'.format(found, size_vocab))\n fwriter.close()\n\n # Open file again to add meta data:\n src = open(output_file, \"r\")\n meta_line = \"%s %s\\n\"%(found, dim_size)\n oline = src.readlines()\n # Here, we prepend the string we want to on first line\n oline.insert(0, meta_line)\n src.close()\n\n # We again open the file in WRITE mode\n src = open(output_file, \"w\")\n src.writelines(oline)\n src.close()\n # Done with writing.\n\n if config.output_format.__contains__(\".gz\"):\n content = open(output_file, \"rb\").read()\n gzip_out_file = output_file + '.gz'\n with gzip.open(gzip_out_file, 'wb') as f:\n f.write(content)\n print(\"Saved embedding to %s\" % (gzip_out_file))\n\n if config.output_format.__contains__(\".npz\"):\n npz_out_file = output_file + '.npz'\n np.savez_compressed(npz_out_file, embeddings=embeddings)\n print(\"Saved embedding to %s\"%(npz_out_file))\n return\n\n"
},
{
"path": "src/codes/api/embedding_preprocessing.py",
"content": "# Convert to a standard word2vec format\n\nimport gensim\nfrom utils import embedding_io\nimport sys\nfrom threading import Thread\nfrom embeddings.embedding_configs import EmbeddingConfigs\n\n\ndef convert_to_w2v(vocab_file, embedding_file, out_file):\n \"\"\"\n Export from a word2vec file by filtering out vocabs based on the input vocab file.\n :param vocab_file:\n :param embedding_file:\n :param out_file:\n :return: word2vec file\n \"\"\"\n std_vocab = []\n with open(vocab_file) as f:\n for word in f:\n std_vocab.append(word)\n\n print (\"Loaded NER vocab_size = %s\" % (len(std_vocab)))\n is_binary = False\n if embedding_file.endswith(\".bin\"):\n is_binary = True\n\n print(\"Loading w2v model ...\")\n\n emb_model = gensim.models.KeyedVectors.load_word2vec_format(embedding_file,\n binary=is_binary,\n unicode_errors='ignore')\n\n print(\"LOADED model: vocab_size = %s\" % (len(emb_model.wv.vocab)))\n f_writer = open(out_file, \"w\")\n for word in std_vocab:\n word = word.rstrip()\n line = None\n if word in emb_model:\n vector = \" \".join(str(item) for item in emb_model[word])\n # word = word.lower()\n line = \"%s %s\" % (word, vector)\n else:\n word = word.lower()\n if word in emb_model:\n vector = \" \".join(str(item) for item in emb_model[word])\n line = \"%s %s\" % (word, vector)\n # print(\"LINE: \", line)\n if line:\n f_writer.write(line + \"\\n\")\n f_writer.close()\n\n\ndef test():\n vocab_file = \"../data/vnner_BiLSTM_CRF/vocab.words.txt\"\n embedding_file = \"../data/embedding_dicts/elmo_embeddings_large.txt\"\n out_file = \"../data/embedding_dicts/elmo_1024dims_wiki_normalcase2lowercase_NER.vec\"\n convert_to_w2v(vocab_file, embedding_file, out_file)\n print(\"Out file: \", out_file)\n print(\"DONE\")\n\n\ndef load_and_save_2_word2vec_model(input_model_path, output_model_path, embedding_config):\n \"\"\"\n Process one embedding model\n :param input_model_path:\n :param output_model_path:\n :return:\n \"\"\"\n model_in = embedding_io.load_word_embedding(input_model_path, embedding_config)\n embedding_io.save_model_to_file(model_in, output_model_path)\n print(\"Write model back to \", output_model_path)\n\n\ndef load_and_save_2_word2vec_models(input_embedding_files_str, output_embedding_files_str, embedding_config):\n \"\"\"\n Multi-threaded processing to export to word2vec format\n :param input_embedding_files_str:\n :param output_embedding_files_str:\n :return:\n \"\"\"\n if input_embedding_files_str.__contains__(\";\"):\n input_model_files = input_embedding_files_str.split(\";\")\n else:\n input_model_files = [input_embedding_files_str]\n\n if output_embedding_files_str.__contains__(\";\"):\n output_model_files = output_embedding_files_str.split(\";\")\n else:\n output_model_files = [output_embedding_files_str]\n\n # Double check input files and output files.\n assert (len(output_model_files) == len(input_model_files)), \\\n \"Number of input files and output files must be equal. Exiting ...\"\n\n # create a list of threads\n threads = []\n\n for model_in, model_out in zip(input_model_files, output_model_files):\n # We start one thread per file.\n process = Thread(target=load_and_save_2_word2vec_model, args=[model_in, model_out, embedding_config])\n process.start()\n threads.append(process)\n # load_and_save_2_word2vec_model(model_in, model_out)\n\n # This to ensure each thread has finished processing the input file.\n for process in threads:\n process.join()\n\n\nif __name__ == \"__main__\":\n\n if len(sys.argv) != 2:\n print(\"Missing input arguments. Input format: ./*.py . Exiting ...\")\n exit(0)\n\n embedding_config = EmbeddingConfigs()\n # We don't need to be word2vec format for pre-processing here but it still shows warning\n # if input files aren't in w2v format.\n embedding_config.is_word2vec_format = True\n embedding_config.do_normalize_emb = False # If you don't want to normalize the embedding vectors.\n\n if sys.argv[1].__contains__(\";\"):\n in_model_files = sys.argv[1].split(\";\")\n else:\n in_model_files = [sys.argv[1]]\n\n out_model_files = [input_model_path + \".extracted.vec\" for input_model_path in in_model_files]\n\n load_and_save_2_word2vec_models(in_model_files, out_model_files)\n"
},
{
"path": "src/codes/api/embedding_visualizer.py",
"content": "# 1. Read embedding file\n# 2. Convert to tensorboard\n# 3. Visualize\n\n# encoding: utf-8\nimport sys, os\nimport gensim\nimport tensorflow as tf\nimport numpy as np\nfrom tensorflow.contrib.tensorboard.plugins import projector\nimport logging\nfrom tensorboard import default\nfrom tensorboard import program\n\n\nclass TensorBoardTool:\n\n def __init__(self, dir_path):\n self.dir_path = dir_path\n\n def run(self, emb_name, port):\n # Remove http messages\n # log = logging.getLogger('sonvx').setLevel(logging.INFO)\n logging.basicConfig(level=logging.INFO)\n logging.propagate = False\n # Start tensorboard server\n tb = program.TensorBoard(default.get_plugins(), default.get_assets_zip_provider())\n tb.configure(argv=[None, '--logdir', self.dir_path, '--port', str(port)])\n url = tb.launch()\n sys.stdout.write('TensorBoard of %s at %s \\n' % (emb_name, url))\n\n\ndef convert_multiple_emb_models_2_tf(emb_name_arr, w2v_model_arr, output_path, port):\n \"\"\"\n\n :param emb_name_arr:\n :param w2v_model_arr:\n :param output_path:\n :param port:\n :return:\n \"\"\"\n idx = 0\n # define the model without training\n sess = tf.InteractiveSession()\n config = projector.ProjectorConfig()\n\n for w2v_model in w2v_model_arr:\n emb_name = emb_name_arr[idx]\n\n meta_file = \"%s.tsv\" % emb_name\n placeholder = np.zeros((len(w2v_model.wv.index2word), w2v_model.vector_size))\n\n with open(os.path.join(output_path, meta_file), 'wb') as file_metadata:\n for i, word in enumerate(w2v_model.wv.index2word):\n placeholder[i] = w2v_model[word]\n # temporary solution for https://github.com/tensorflow/tensorflow/issues/9094\n if word == '':\n print(\"Empty Line, should replaced by any thing else, or will cause a bug of tensorboard\")\n file_metadata.write(u\"{0}\".format('').encode('utf-8') + b'\\n')\n else:\n file_metadata.write(u\"{0}\".format(word).encode('utf-8') + b'\\n')\n\n word_embedding_var = tf.Variable(placeholder, trainable=False, name=emb_name)\n tf.global_variables_initializer().run()\n sess.run(word_embedding_var)\n\n # adding into projector\n embed = config.embeddings.add()\n embed.tensor_name = emb_name\n embed.metadata_path = meta_file\n idx += 1\n\n saver = tf.train.Saver()\n writer = tf.summary.FileWriter(output_path, sess.graph)\n\n # Specify the width and height of a single thumbnail.\n projector.visualize_embeddings(writer, config)\n all_emb_name = \"_\".join(emb_name for emb_name in emb_name_arr)\n saver.save(sess, os.path.join(output_path, '%s.ckpt' % all_emb_name))\n # tf.flags.FLAGS.logdir = output_path\n # print('Running `tensorboard --logdir={0}` to run visualize result on tensorboard'.format(output_path))\n # tb.run_main()q\n tb_tool = TensorBoardTool(output_path)\n tb_tool.run(all_emb_name, port)\n return\n\n\ndef convert_one_emb_model_2_tf(emb_name, model, output_path, port):\n \"\"\"\n\n :param model: Word2Vec model\n :param output_path:\n :return:\n \"\"\"\n # emb_name = \"word_embedding\"\n meta_file = \"%s.tsv\"%emb_name\n placeholder = np.zeros((len(model.wv.index2word), model.vector_size))\n\n with open(os.path.join(output_path, meta_file), 'wb') as file_metadata:\n for i, word in enumerate(model.wv.index2word):\n placeholder[i] = model[word]\n # temporary solution for https://github.com/tensorflow/tensorflow/issues/9094\n if word == '':\n print(\"Empty Line, should replaced by any thing else, or will cause a bug of tensorboard\")\n file_metadata.write(u\"{0}\".format('').encode('utf-8') + b'\\n')\n else:\n file_metadata.write(u\"{0}\".format(word).encode('utf-8') + b'\\n')\n\n # define the model without training\n sess = tf.InteractiveSession()\n\n word_embedding_var = tf.Variable(placeholder, trainable=False, name=emb_name)\n sess.run(word_embedding_var)\n # tf.global_variables_initializer().run()\n\n saver = tf.train.Saver()\n writer = tf.summary.FileWriter(output_path, sess.graph)\n\n # adding into projector\n config = projector.ProjectorConfig()\n embed = config.embeddings.add()\n embed.tensor_name = emb_name\n embed.metadata_path = meta_file\n\n # Specify the width and height of a single thumbnail.\n projector.visualize_embeddings(writer, config)\n saver.save(sess, os.path.join(output_path, '%s.ckpt'%emb_name))\n # tf.flags.FLAGS.logdir = output_path\n # print('Running `tensorboard --logdir={0}` to run visualize result on tensorboard'.format(output_path))\n # tb.run_main()q\n tb_tool = TensorBoardTool(output_path)\n tb_tool.run(emb_name, port)\n return\n\n\ndef visualize_multiple_embeddings_individually(paths_of_emb_models):\n output_root_dir = \"../data/embedding_tf_data/\"\n starting_port = 6006\n embedding_names = []\n print(\"Loaded all word embeddings, going to visualize ...\")\n\n if paths_of_emb_models and paths_of_emb_models.__contains__(\";\"):\n files = paths_of_emb_models.split(\";\")\n for emb_file in files:\n\n embedding_name = os.path.basename(os.path.normpath(emb_file))\n\n tf_data_folder = output_root_dir + embedding_name\n\n if not os.path.exists(tf_data_folder):\n os.makedirs(tf_data_folder)\n\n is_binary = False\n\n if emb_file.endswith(\".bin\"):\n is_binary = True\n\n emb_model = gensim.models.KeyedVectors.load_word2vec_format(emb_file, binary=is_binary)\n\n convert_one_emb_model_2_tf(embedding_name, emb_model, tf_data_folder, starting_port)\n\n embedding_names.append(embedding_name)\n\n starting_port += 1\n\n while True:\n print(\"Type exit to quite the visualizer: \")\n user_input = input()\n if user_input == \"exit\":\n break\n return\n\n\ndef visualize_multiple_embeddings_all_in_one(paths_of_emb_models, port):\n output_root_dir = \"../data/embedding_tf_data/\"\n starting_port = port\n embedding_names = []\n print(\"Loaded all word embeddings, going to visualize ...\")\n\n embedding_name_arr = []\n w2v_embedding_model_arr = []\n\n if paths_of_emb_models and paths_of_emb_models.__contains__(\";\"):\n files = paths_of_emb_models.split(\";\")\n for emb_file in files:\n\n embedding_name = os.path.basename(os.path.normpath(emb_file))\n embedding_name_arr.append(embedding_name)\n\n is_binary = False\n\n if emb_file.endswith(\".bin\"):\n is_binary = True\n\n emb_model = gensim.models.KeyedVectors.load_word2vec_format(emb_file, binary=is_binary)\n w2v_embedding_model_arr.append(emb_model)\n embedding_names.append(embedding_name)\n\n # print(\"View side-by-side word similarity of multiple embeddings at: http://Sons-MBP.lan:8089\")\n\n all_emb_name = \"_\".join(emb_name for emb_name in embedding_name_arr)\n tf_data_folder = output_root_dir + all_emb_name\n if not os.path.exists(tf_data_folder):\n os.makedirs(tf_data_folder)\n\n convert_multiple_emb_models_2_tf(embedding_name_arr, w2v_embedding_model_arr, tf_data_folder, starting_port)\n\n while True:\n print(\"Type exit to quite the visualizer: \")\n user_input = input()\n if user_input == \"exit\":\n break\n return\n\n\ndef visualize_multiple_embeddings(paths_of_emb_models, port):\n \"\"\"\n API to other part to call, don't modify this function.\n :param paths_of_emb_models:\n :param port:\n :return:\n \"\"\"\n visualize_multiple_embeddings_all_in_one(paths_of_emb_models, port)\n\n\nif __name__ == \"__main__\":\n \"\"\"\n Just run `python w2v_visualizer.py word2vec.model visualize_result`\n \"\"\"\n try:\n model_path = sys.argv[1]\n output_path = sys.argv[2]\n except Exception as e:\n print(\"Please provide model path and output path %s \" % e)\n\n # model = Word2Vec.load(model_path)\n model = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=True)\n convert_one_emb_model_2_tf(model, output_path)\n"
},
{
"path": "src/codes/embeddings/__init__.py",
"content": ""
},
{
"path": "src/codes/embeddings/embedding_configs.py",
"content": "\n\nclass EmbeddingConfigs(object):\n \"\"\"\n Configuration information\n \"\"\"\n is_word2vec_format = True\n do_normalize_emb = True\n model_paths_list = []\n model_names_list = []\n model_dims_list = []\n char_model_path = None\n char_model_dims = -1\n output_format = \".txt;.npz;.gz\"\n"
},
{
"path": "src/codes/embeddings/embedding_models.py",
"content": "from gensim.models import KeyedVectors as Word2Vec\nimport numpy as np\nfrom embeddings import embedding_utils\nfrom utils import file_utils\nimport os, re\nimport logging\n\n\nDEBUG = False\n\n\nclass Model_Constants(object):\n word2vec = \"word2vec\"\n char2vec = \"char2vec\"\n private_word2vec = \"private_word2vec\"\n elmo = \"elmo\"\n\n\nclass Embedding_Model(object):\n def __init__(self, name, vector_dim):\n self.name = name\n self.model = None\n self.char_model = None\n self.vocabs_list = None\n self.vector_dim = vector_dim\n # TODO: update this changeable param later\n # unk, random, mean, replace_by_character_embedding\n self.unknown_word = \"replace_by_character_embedding\"\n # self.MAX_DIM = 400 # No longer use MAX_DIM, now it depends on input dims\n\n def load_model(self, model_path):\n if self.name == Model_Constants.word2vec or self.name == Model_Constants.elmo:\n if model_path.endswith(\".bin\"):\n self.model = Word2Vec.load_word2vec_format(model_path, binary=True)\n else:\n self.model = Word2Vec.load_word2vec_format(model_path, binary=False)\n elif self.name == Model_Constants.char2vec:\n self.model = dict()\n print(\"Loading model_path = \", model_path)\n file = open(model_path, \"r\")\n for line in file:\n elements = line.split()\n if len(elements) > 100: # because embedding dim is higher than 100.\n # char_model[elements[0]] = np.array(map(float, elements[1:])).tolist()\n self.model[elements[0]] = np.array([float(i) for i in elements[1:]]).tolist()\n return self.model\n elif self.name == Model_Constants.private_word2vec:\n self.model, _, self.vocabs_list = embedding_utils.reload_embeddings(model_path)\n else:\n raise Exception(\"Unknown embedding models!\")\n\n def is_punct(self, word):\n arr_list = [\n '!',\n '\"',\n '%',\n '&',\n \"'\",\n \"''\",\n '(',\n '(.',\n ')',\n '*',\n '+',\n ',',\n '-',\n '---',\n '.',\n '..',\n '...',\n '....',\n '/',\n ]\n if word in arr_list:\n return True\n else:\n return False\n\n def is_number(self, word):\n regex = r\"^[0-9]+\"\n matches = re.finditer(regex, word, re.MULTILINE)\n matchNum = 0\n for matchNum, match in enumerate(matches):\n matchNum = matchNum + 1\n if matchNum > 0:\n return True\n else:\n return False\n\n def set_char_model(self, char_model):\n self.char_model = char_model\n\n def load_vocabs_list(self, vocab_file_path):\n \"\"\"\n Load vocabs list for private w2v model. Has to be pickle file.\n :param vocab_file_path:\n :return:\n \"\"\"\n if vocab_file_path:\n self.vocabs_list = file_utils.load_obj(vocab_file_path)\n\n def get_char_vector(self, char_model, word):\n \"\"\"\n char_model here is an instance of embedding_model\n :param char_model: an instance of embedding_model\n :param word:\n :return:\n \"\"\"\n if char_model is None:\n # Sonvx on March 20, 2019: we now allow the char_model is None,\n # cannot call this get_char_vector in such case.\n raise Exception(\"Char_model is None! Cannot use character-embedding.\")\n\n out_char_2_vec = []\n char_vecs = []\n chars = list(word)\n vecs = []\n for c in chars:\n if c in char_model.model:\n emb_vector = char_model.model[c]\n vecs.append(emb_vector)\n if DEBUG:\n input(\">>>>>>\")\n print(\"Char_emb_vector=\", emb_vector)\n\n # char_vecs.extend(list(vecs))\n\n if len(vecs) > 0:\n out_char_2_vec = np.mean(vecs, axis=0)\n\n if DEBUG:\n print(\">>> Output of char2vec: %s\"%(out_char_2_vec))\n input(\">>>> outc2v ...\")\n\n return out_char_2_vec\n\n def is_unknown_word(self, word):\n \"\"\"Check whether or not a word is unknown\"\"\"\n is_unknown_word = False\n if self.vocabs_list is not None:\n if word not in self.vocabs_list:\n is_unknown_word = True\n else:\n if word not in self.model:\n is_unknown_word = True\n return is_unknown_word\n\n def get_word_vector(self, word):\n \"\"\"\n Handle unknown word: In case of our private word2vec, we have a vocabs_list to check. With regular models,\n we can check inside the model. Note that by default, we use char-model to handle unknown words.\n :param word:\n :param char_model:\n :return:\n \"\"\"\n\n rtn_vector = []\n\n # try first time with normal case\n is_unknown_word = self.is_unknown_word(word)\n\n # try 2nd times with lowercase.\n if is_unknown_word:\n word = word.lower()\n is_unknown_word = self.is_unknown_word(word)\n\n # unknown word\n if is_unknown_word and self.char_model:\n # Sonvx on March 20, 2019: solve unknown only when char_model is SET.\n rtn_vector = self.get_vector_of_unknown(word)\n else:\n # normal case\n if self.name == Model_Constants.word2vec:\n rtn_vector = self.model[word]\n\n # For now we have self.vector_dim, max_dim, and len(rtn_vector)\n # Update: move to use self.vector_dim only\n if len(rtn_vector) > self.vector_dim:\n print(\"Warning: auto trim to %s/%s dimensions\"%(self.vector_dim, len(rtn_vector)))\n rtn_vector = self.model[word][:self.vector_dim]\n\n elif self.name == Model_Constants.elmo:\n rtn_vector = self.model[word]\n\n if self.vector_dim == len(rtn_vector)/2:\n vector1 = rtn_vector[:self.vector_dim]\n vector2 = rtn_vector[self.vector_dim:]\n print(\"Notice: auto average to b[i] = (a[i] + a[i + %s])/2 /%s dimensions\" % (self.vector_dim,\n len(rtn_vector)))\n rtn_vector = np.mean([vector1, vector2], 0)\n elif len(rtn_vector) > self.vector_dim:\n print(\"Warning: auto trim to %s/%s dimensions\" % (self.vector_dim, len(rtn_vector)))\n rtn_vector = self.model[word][:self.vector_dim]\n\n elif self.name == Model_Constants.char2vec:\n rtn_vector = self.get_char_vector(self, word)\n\n elif self.name == Model_Constants.private_word2vec:\n # Handle unknown word - Not need for now since we handle unknown words first\n if word not in self.vocabs_list:\n word = \"UNK\"\n word_idx = self.vocabs_list.index(word)\n emb_vector = self.model[word_idx]\n rtn_vector = emb_vector\n\n # final check before returning vector\n if DEBUG:\n print(\">>> DEBUG: len(rtn_vector) = %s\" % (len(rtn_vector)))\n input(\">>> before returning vector ...\")\n if len(rtn_vector) < 1:\n return np.zeros(self.vector_dim)\n else:\n if len(rtn_vector) == self.vector_dim:\n return rtn_vector\n # TODO: find a better way to represent unknown word by character to have same-size with word-vector-size\n # For now, I add 0 to the [current-len, expected-len]\n else:\n logging.debug(\"Model name = %s, Current word = %s, Current size = %s, expected size = %s\"\n %(self.name, word, len(rtn_vector), self.vector_dim))\n return np.append(rtn_vector, np.zeros(self.vector_dim - len(rtn_vector)))\n\n def get_vector_of_unknown(self, word):\n \"\"\"\n If word is UNK, use char_vector model instead.\n :param word:\n :return:\n \"\"\"\n # Here we handle features based on the w2v model where\n # numbers and punctuations are encoded as , \n if self.name == Model_Constants.word2vec:\n if self.is_number(word):\n rtn_vector = self.model[\"\"]\n elif self.is_punct(word):\n rtn_vector = self.model[\"\"]\n else:\n rtn_vector = self.get_char_vector(self.char_model, word)\n\n if rtn_vector is not None:\n if len(rtn_vector) > self.vector_dim:\n print(\"Warning: auto trim to %s/%s dimensions\"%(self.vector_dim, len(rtn_vector)))\n return rtn_vector[:self.vector_dim]\n else:\n return rtn_vector\n # otherwise, using c2v to build-up the embedding vector\n else:\n return self.get_char_vector(self.char_model, word)\n\n\nclass Embedding_Models(object):\n \"\"\"\n Using all available embedding models to generate vectors\n \"\"\"\n def __init__(self, list_models):\n self.list_models = list_models # list of embedding_model_objs: ['word2vec', 'char2vec', 'private_word2vec']\n\n def add_model(self, emb_model, char_model):\n \"\"\"\n Add new model into the collection of embedding models. Note that, every model has to add char_model to handle\n unknown word.\n :param emb_model:\n :param char_model:\n :return:\n \"\"\"\n if char_model is None:\n print(\"Warning: char_model is None -> cannot solve OOV word. Keep going ...\")\n # Sonvx on March 20, 2019: change to allow None char_model\n # raise Exception(\"char_model cannot be None.\")\n if isinstance(emb_model, Embedding_Model):\n emb_model.set_char_model(char_model)\n self.list_models.append(emb_model)\n else:\n raise Exception(\"Not an instance of embedding_model class.\")\n\n def get_vector_of_document(self, document):\n \"\"\"\n Get all embedding vectors for one document\n :param document:\n :return:\n \"\"\"\n doc_vector = []\n # debug_dict = {}\n # print (\"len_doc = \", len(document))\n for word in document:\n\n all_vectors_of_word = []\n\n # get all embedding vectors of a word\n for emb_model in self.list_models:\n emb_vector = emb_model.get_word_vector(word)\n # print(\"len_emb_vector = \", len(emb_vector))\n all_vectors_of_word.extend(emb_vector)\n\n # if word in debug_dict.keys():\n # debug_dict[word].append(len(emb_vector))\n # else:\n # debug_dict[word] = [len(emb_vector)]\n\n # stack a combined vector of all words\n doc_vector.append(all_vectors_of_word)\n\n # print(\"list of words and emb size = \", debug_dict)\n # get the mean of them to represent a document\n doc_vector = np.mean(doc_vector, axis=0)\n\n return doc_vector\n\n def get_word_vector_of_multi_embeddings(self, word):\n \"\"\"\n Get all embedding vectors for one document\n :param word:\n :return:\n \"\"\"\n word_vector = []\n for emb_model in self.list_models:\n emb_vector = emb_model.get_word_vector(word)\n word_vector.extend(emb_vector)\n\n return word_vector\n\n\n\n\n\n"
},
{
"path": "src/codes/embeddings/embedding_utils.py",
"content": "import os\nfrom utils import file_utils\nfrom embeddings.embedding_models import Embedding_Model, Embedding_Models\n\n\ndef reload_char2vec_model(model_path, model_dim):\n char_model = Embedding_Model(\"char2vec\", model_dim)\n char_model.load_model(model_path)\n return char_model\n\n\ndef reload_embedding_models(model_paths_list, model_names_list, model_dims_list, char_model):\n \"\"\"\n Reload collection of embedding models to serve feature extraction task.\n :param model_paths_list:\n :param model_names_list:\n :param model_dims_list:\n :param char_model:\n :return:\n \"\"\"\n # model path list and name list must be equal.\n print(\"model_paths_list = \", model_paths_list)\n print(\"model_formats_list = \", model_names_list)\n assert (len(model_names_list) == len(model_paths_list)), \"Not equal length\"\n assert (len(model_names_list) == len(model_dims_list)), \"Not equal length\"\n\n all_emb_models = Embedding_Models([])\n\n for model_idx in range(len(model_paths_list)):\n # get model path based on index\n model_path = model_paths_list[model_idx]\n model_name = model_names_list[model_idx]\n model_dim = model_dims_list[model_idx]\n\n if model_path is not None:\n\n emb_model = Embedding_Model(model_name, model_dim)\n emb_model.load_model(model_path)\n\n # add to final list of emb_models\n all_emb_models.add_model(emb_model, char_model)\n\n return all_emb_models\n\n\ndef save_embedding_models_tofolder(dir_path, final_embeddings, reverse_dictionary, vocabulary_size):\n \"\"\"\n Save all trained word-embedding model of the custom word2vec.\n :param final_embeddings:\n :param reverse_dictionary:\n :param vocabulary_size:\n :return:\n \"\"\"\n\n if not os.path.exists(dir_path):\n os.makedirs(dir_path)\n\n def save_to_word2vec_model(vocabs_list):\n # print(\"Saving word2vec format ...\")\n filewriter = open(os.path.join(dir_path, \"word2vec.txt\"), \"w\", encoding=\"utf-8\")\n\n filewriter.write(\"%s %s\\n\" % (len(vocabs_list), len(final_embeddings[0])))\n for word in vocabs_list:\n word_idx = vocabs_list.index(word)\n emb_vector = final_embeddings[word_idx]\n line = ' '.join([\"%s\" % (x) for x in emb_vector])\n filewriter.write(word + \" \" + line + \"\\n\")\n\n filewriter.close()\n # print(\"Done!\")\n\n file_utils.save_obj(final_embeddings, os.path.join(dir_path, \"final_embeddings\"))\n # We don't need to save reversed_dictionary\n # file_utils.save_obj(reverse_dictionary, os.path.join(FLAGS.trained_models, \"reversed_dictionary\"))\n vocab_list = [reverse_dictionary[i] for i in range(vocabulary_size)]\n save_to_word2vec_model(vocab_list)\n file_utils.save_obj(vocab_list, os.path.join(dir_path, \"words_dictionary\"))\n\n\ndef save_embedding_models(FLAGS, final_embeddings, reverse_dictionary, vocabulary_size):\n \"\"\"\n Keep for old implementation.\n :param FLAGS:\n :param final_embeddings:\n :param reverse_dictionary:\n :param vocabulary_size:\n :return:\n \"\"\"\n save_embedding_models_tofolder(FLAGS.trained_models, final_embeddings,\n reverse_dictionary, vocabulary_size)\n\n\ndef reload_embeddings(trained_models_dir):\n \"\"\"\n Reload trained word-embedding model of the custom word2vec.\n :param trained_models_dir:\n :return:\n \"\"\"\n final_embeddings = file_utils.load_obj(os.path.join(trained_models_dir, \"final_embeddings\"))\n # reverse_dictionary = file_utils.load_obj(os.path.join(trained_models_dir, \"reversed_dictionary\"))\n reverse_dictionary = None\n labels = file_utils.load_obj(os.path.join(trained_models_dir, \"words_dictionary\"))\n return final_embeddings, reverse_dictionary, labels\n\n\ndef create_single_utf8_file(input_dir, output_file):\n import glob\n # path = './wiki_data/*.txt'\n # out = './wiki_all.vi.utf8.txt'\n files = glob.glob(input_dir)\n for file in files:\n with open(output_file, \"a\") as myfile:\n with open(file, \"r\") as fp:\n for line in fp:\n line = line.strip().lower()\n line = line.decode('utf-8', 'ignore').encode(\"utf-8\")\n myfile.write(line)\n print(\"done\")\n"
},
{
"path": "src/codes/etnlp_api.py",
"content": "import argparse\nfrom api import embedding_preprocessing, embedding_evaluator, embedding_extractor, embedding_visualizer\nfrom visualizer import visualizer_sbs\nimport logging\nimport os\nfrom embeddings.embedding_configs import EmbeddingConfigs\n__version__ = \"0.1.3\"\n\n\nembedding_config = EmbeddingConfigs()\n\nif __name__ == \"__main__\":\n \"\"\"\n ETNLP: a toolkit for evaluate, extract, and visualize multiple word embeddings\n \"\"\"\n\n _desc = \"Evaluates a word embedding model\"\n _parser = argparse.ArgumentParser(description=_desc)\n _parser.add_argument(\"-input\",\n required=True,\n default=\"../data/embedding_dicts/elmo_embeddings.txt\",\n #\n help=\"model\")\n _parser.add_argument(\"-analoglist\",\n nargs=\"?\",\n # default=\"../data/embedding_analogies/vi/analogy_vn_seg.txt.std.txt\",\n default=\"./data/embedding_analogy/solveable_analogies_vi.txt\",\n help=\"testset\")\n\n _parser.add_argument(\"-args\",\n nargs=\"?\",\n default=\"eval\",\n help=\"Run evaluation\")\n\n _parser.add_argument(\"-lang\",\n nargs=\"?\",\n default=\"VI\",\n help=\"Specify language, by default, it's Vietnamese.\")\n\n _parser.add_argument(\"-vocab\",\n nargs=\"?\",\n default=\"../data/vocab.txt\",\n help=\"Vocab to be extracted\")\n\n _parser.add_argument(\"-port\",\n nargs=\"?\",\n default=8889,\n help=\"Port for visualization\")\n\n _parser.add_argument(\"-input_c2v\",\n nargs=\"?\",\n default=None,\n help=\"C2V embedding\")\n\n _parser.add_argument(\"-output\",\n nargs=\"?\",\n default=\"../data/embedding_analogies/vi/results_out.txt\",\n help=\"Output file of word analogy task\")\n\n _parser.add_argument(\"-output_format\",\n nargs=\"?\",\n default=\".txt\",\n help=\"Format of output file of the extracted embedding.\")\n\n _args = _parser.parse_args()\n\n # Set logging level\n logging.basicConfig(level=logging.INFO)\n logging.disable(logging.INFO)\n os.environ['TF_CPP_MIN_LOG_LEVEL'] = '5'\n\n input_embedding_files_str = _args.input\n analoglist = _args.analoglist\n is_vietnamese = _args.lang\n output_files_str = _args.output\n options_str = _args.args\n vocab_file = _args.vocab\n output_format = _args.output_format\n port = _args.port\n\n # By default, we process all embeddings as word2vec format.\n embedding_preprocessing.is_word2vec_format = True\n\n if options_str == 'eval':\n print(\"Starting evaluator ...\")\n embedding_evaluator.evaluator_api(input_files=input_embedding_files_str, analoglist=analoglist,\n output=output_files_str)\n print(\"Done evaluator !\")\n elif options_str == 'visualizer':\n print(\"Starting visualizer ...\")\n embedding_visualizer.visualize_multiple_embeddings(input_embedding_files_str, port)\n print(\"Done visualizer !\")\n elif options_str.startswith(\"extract\"):\n print(\"Starting extractor ...\")\n embedding_extractor.extract_embedding_for_vocab_file(input_embedding_files_str, vocab_file,\n _args.input_c2v, output_files_str, output_format)\n print(\"Done extractor !\")\n elif options_str.startswith(\"glove2w2v\"):\n print(\"Starting pre-processing: convert to word2vec format ...\")\n embedding_config.is_word2vec_format = False\n if options_str.__contains__(\"do_normalize\"):\n embedding_config.do_normalize_emb = True\n else:\n embedding_config.do_normalize_emb = False\n embedding_preprocessing.load_and_save_2_word2vec_models(input_embedding_files_str,\n output_files_str,\n embedding_config)\n\n else:\n print(\"Invalid options\")\n\n print(\"Done!\")\n\n\n\n\n"
},
{
"path": "src/codes/requirements.txt",
"content": "gensim==3.4.0\nscipy==1.1.0\nsix==1.12.0\nsetuptools==40.6.2\ntensorflow==1.12.0\nFlask==1.0.2\ntensorboard==1.12.0\nnumpy==1.15.4\nscikit_learn==0.20.3\ntyping==3.6.6\n"
},
{
"path": "src/codes/setup.py",
"content": "from setuptools import setup, find_packages\nfrom etnlp_api import __version__\n\n\nwith open(\"../../README.md\", \"r\") as fh:\n long_description = fh.read()\n\nsetup(\n name='ETNLP',\n version=__version__,\n # packages=['api', 'utils', 'embeddings', 'visualizer'],\n packages=find_packages(),\n py_modules=['etnlp_api'],\n long_description=long_description,\n long_description_content_type=\"text/markdown\",\n url='https://github.com/vietnlp/etnlp',\n license='MIT',\n author='vietnlp',\n author_email='sonvx.coltech@gmail.com',\n description='ETNLP: Embedding Toolkit for NLP Tasks'\n)\n# from setuptools import setup, find_packages\n# import sys\n#\n# with open('requirements.txt') as f:\n# reqs = f.read()\n# setup(\n# name='ETNLP',\n# version='0.1.0',\n# description='ETNLP: Embedding Toolkit for NLP Tasks',\n# python_requires='>=3.5',\n# packages=find_packages(exclude=('data')),\n# install_requires=reqs.strip().split('\\n'),\n# )\n"
},
{
"path": "src/codes/utils/__init__.py",
"content": ""
},
{
"path": "src/codes/utils/emb_utils.py",
"content": "from sklearn.metrics.pairwise import cosine_similarity\nfrom typing import Any, Iterable, List, Optional, Set, Tuple\n\nfrom utils.vectors import Vector\nfrom utils import vectors\nfrom utils.word import Word\nfrom utils import eval_utils\nfrom gensim import utils as genutils\nimport logging\nimport numpy as np\nfrom scipy import stats\n\n# Timing info for most_similar (100k words):\n# Original version: 7.3s\n# Normalized vectors: 3.4s\nlogger = logging.getLogger(__name__)\n\n\ndef most_similar(base_vector: Vector, words: List[Word]) -> List[Tuple[float, Word]]:\n \"\"\"Finds n words with smallest cosine similarity to a given word\"\"\"\n words_with_distance = [(vectors.cosine_similarity_normalized(base_vector, w.vector), w) for w in words]\n # We want cosine similarity to be as large as possible (close to 1)\n sorted_by_distance = sorted(words_with_distance, key=lambda t: t[0], reverse=True)\n # Sonvx: remove duplications (not understand why yet, probably because the w2v?)\n # sorted_by_distance = list(set(sorted_by_distance))\n return sorted_by_distance\n\n\ndef print_most_similar(words: List[Word], text: str) -> None:\n base_word = find_word(text, words)\n if not base_word:\n print(\"Unknown word: %s\"%(text))\n return\n print(\"Words related to %s:\" % (base_word.text))\n sorted_by_distance = [\n word.text for (dist, word) in\n most_similar(base_word.vector, words)\n if word.text.lower() != base_word.text.lower()\n ]\n print(', '.join(sorted_by_distance[:10]))\n\n\ndef read_word() -> str:\n return input(\"Type a word: \")\n\n\ndef find_word(text: str, words: List[Word]) -> Optional[Word]:\n try:\n return next(w for w in words if text == w.text)\n except StopIteration:\n return None\n\n\ndef closest_analogies_OLD(\n left2: str, left1: str, right2: str, words: List[Word]\n) -> List[Tuple[float, Word]]:\n word_left1 = find_word(left1, words)\n word_left2 = find_word(left2, words)\n word_right2 = find_word(right2, words)\n if (not word_left1) or (not word_left2) or (not word_right2):\n return []\n vector = vectors.add(\n vectors.sub(word_left1.vector, word_left2.vector),\n word_right2.vector)\n closest = most_similar(vector, words)[:10]\n\n def is_redundant(word: str) -> bool:\n \"\"\"\n Sometimes the two left vectors are so close the answer is e.g.\n \"shirt-clothing is like phone-phones\". Skip 'phones' and get the next\n suggestion, which might be more interesting.\n \"\"\"\n word_lower = word.lower()\n return (\n left1.lower() in word_lower or\n left2.lower() in word_lower or\n right2.lower() in word_lower)\n\n closest_filtered = [(dist, w) for (dist, w) in closest if not is_redundant(w.text)]\n return closest_filtered\n\n\ndef closest_analogies_vectors(\n word_left2: Word, word_left1: Word, word_right2: Word, words: List[Word]) \\\n -> List[Tuple[float, Word]]:\n \"\"\"\n Sonvx:\n :param word_left2:\n :param word_left1:\n :param word_right2:\n :param words:\n :param remove_redundancy: remove suggestions if they contain the given words.\n :return:\n \"\"\"\n # print(\">>>> Remove redundancy = \", remove_redundancy)\n # input(\">>>>\")\n vector = vectors.add(\n vectors.sub(word_left1.vector, word_left2.vector),\n word_right2.vector)\n closest = most_similar(vector, words)[:10]\n\n def is_redundant(word: str) -> bool:\n \"\"\"\n Sometimes the two left vectors are so close the answer is e.g.\n \"shirt-clothing is like phone-phones\". Skip 'phones' and get the next\n suggestion, which might be more interesting.\n \"\"\"\n word_lower = word.lower()\n return (\n word_left1.text.lower() in word_lower or\n word_left2.text.lower() in word_lower or\n word_right2.text.lower() in word_lower)\n # It doesn't work this way for Vietnamese, so we try both of this to test for now\n if False:\n closest_filtered = [(dist, w) for (dist, w) in closest if not is_redundant(w.text)]\n else:\n closest_filtered = closest\n return closest_filtered\n\n\ndef get_avg_vector(word, embedding_words):\n\n if \" \" in word:\n single_words = word.split(\" \")\n list_vector = []\n\n for single_word in single_words:\n word_vec = find_word(single_word, embedding_words)\n if word_vec:\n list_vector.append(word_vec.vector)\n else:\n # Try again with lowercase\n single_word = single_word.lower()\n word_vec = find_word(single_word, embedding_words)\n if word_vec:\n list_vector.append(word_vec.vector)\n\n # print(\"list_vector: \", list_vector)\n # input(\">>>>>>>>\")\n\n returned_Word = Word(word, vectors.mean_list(list_vector), 1)\n else:\n returned_Word = find_word(word, embedding_words)\n\n # print(\"Avg returned vector = \", returned_vector)\n # input(\">>>>\")\n\n return returned_Word\n\n\ndef run_paired_ttests(all_map_arr, embedding_names):\n \"\"\"\n Run Paired t-tests on MAP results\n :param all_map_arr:\n :param embedding_names:\n :return:\n \"\"\"\n str_out = \"\"\n num_embs = len(all_map_arr)\n\n # Verify to make sure they have the same length\n if all_map_arr and embedding_names:\n for i in range(0, num_embs - 1):\n for j in range(i + 1, num_embs):\n if len(all_map_arr[i]) != len(all_map_arr[j]):\n raise Exception(\"Two embedding (%s, %s) have different MAP list, sizes: %s vs. %s\"\n % (embedding_names[i], embedding_names[j], len(all_map_arr[i]), len(all_map_arr[j])))\n else:\n logging.error(\"Inputs are NULL\")\n\n result_str_ttest_arr = []\n for i in range(0, num_embs - 1):\n for j in range(i + 1, num_embs):\n stat_test_ret = stats.ttest_rel(all_map_arr[i], all_map_arr[j])\n # if stat_test_ret.pvalue >= 0.05:\n result = \"%s vs. %s: %s\" % (embedding_names[i], embedding_names[j], stat_test_ret)\n str_out += result + \"\\n\"\n\n return str_out\n\n\ndef eval_word_analogy_4_all_embeddings(word_analogies_file, embedding_names: List[str],\n word_embeddings: List[List[Word]], output_file):\n \"\"\"\n Run word analogy for all embeddings\n :param word_analogies_file:\n :param embedding_names:\n :param word_embeddings:\n :param output_file:\n :return:\n \"\"\"\n fwriter = open(output_file, \"w\")\n idx = 0\n all_map_arr = []\n console_output_str = \"\"\n category = \": | Word Analogy Task results\\n\"\n fwriter.write(category)\n console_output_str += category\n\n for word_embedding in word_embeddings:\n embedding_name = embedding_names[idx]\n map_at_10, map_arr, result_str = eval_word_analogies(word_analogies_file, word_embedding, embedding_name)\n all_map_arr.append(map_arr)\n meta_info = \"\\nEmbedding: %s\"%(embedding_names[idx])\n fwriter.write(meta_info + \"\\n\")\n fwriter.write(result_str)\n fwriter.write(\"MAP_arr = %s\"%(map_arr))\n fwriter.write(\"MAP@10 = %s\" % (map_at_10))\n fwriter.flush()\n console_output_str += meta_info + \"\\n\" + \"MAP@10 = %s\" % (map_at_10) + \"\\n\"\n idx += 1\n\n # Getting significant Paired t-tests\n category = \"\\n: | Paired t-tests results\\n\"\n fwriter.write(category)\n console_output_str += category\n ttests_result = run_paired_ttests(all_map_arr, embedding_names)\n console_output_str += ttests_result\n fwriter.write(ttests_result)\n fwriter.flush()\n fwriter.close()\n\n return console_output_str\n\n\ndef eval_word_analogies(word_analogies_file, words: List[Word], embedding_name):\n \"\"\"\n Sonvx: Evaluate word analogy for one embedding.\n :param word_analogies_file:\n :param words:\n :return:\n \"\"\"\n # input(\"GO checking >>>>\")\n oov_counter, idx_cnt, is_vn_counter, phrase_cnt = 0, -1, 0, 0\n sections, section = [], None\n # map_arr = []\n out_str = \"\"\n map_ret_dict = {}\n\n for line_no, line in enumerate(genutils.smart_open(word_analogies_file)):\n # TODO: use level3 BLAS (=evaluate multiple questions at once), for speed\n line = genutils.to_unicode(line)\n line = line.rstrip()\n if line.startswith(': |'):\n # a new section starts => store the old section\n if section:\n sections.append(section)\n section = {'section': line.lstrip(': ').strip(), 'correct': [], 'incorrect': []}\n else:\n # Count number of analogy to check\n idx_cnt += 1\n\n # Set default map value\n map_ret_dict[idx_cnt] = 0.0\n\n if not section:\n raise ValueError(\"missing section header before line #%i in %s\" % (line_no, word_analogies_file))\n try:\n # a - b + c = expected\n # Input: Baghdad | Irac | Bangkok | Thai_Lan\n # Baghdad - Irac = Bangkok - Thai_Lan\n # -> Baghdad - Irac + Thai_Lan = Bangkok\n # =>\n a, b, expected, c = [word for word in line.split(\" | \")]\n except ValueError:\n logger.debug(\"SVX: ERROR skipping invalid line #%i in %s\", line_no, word_analogies_file)\n print(\"Line : \", line)\n print(\"a, b, c, expected: %s, %s, %s, %s\" % (a, b, c, expected))\n # input(\">>> Wait ...\")\n continue\n\n # In case of Vietnamese, word analogy can be a phrase\n if \" \" in expected:\n print(\"INFO: we don't support to find word analogies for phrase for NOW.\")\n phrase_cnt += 1\n continue\n elif \" \" in a or \" \" in b or \" \" in c:\n is_vn_counter += 1\n word_left1 = get_avg_vector(a, words)\n word_left2 = get_avg_vector(b, words)\n word_right2 = get_avg_vector(c, words)\n else:\n word_left1 = find_word(a, words)\n word_left2 = find_word(b, words)\n word_right2 = find_word(c, words)\n\n if (not word_left1) or (not word_left2) or (not word_right2):\n logger.debug(\"SVX: skipping line #%i with OOV words: %s\", line_no, line.strip())\n oov_counter += 1\n continue\n\n # Write solable analogy to a file\n # fsolveable_writer.write(line + \"\\n\")\n\n logger.debug(\"word_left1 = %s\", word_left1.text)\n logger.debug(\"word_left2 = %s\", word_left2.text)\n logger.debug(\"word_right2 = %s\", word_right2.text)\n\n # Start finding close word:\n # Note: we can only find 1 expected word in Vietnamese for NOW\n top10_candidate = closest_analogies_vectors(word_left2, word_left1,\n word_right2, words)\n list_candidate_arr = []\n for tuple in top10_candidate:\n list_candidate_arr.append(tuple[1].text)\n\n logger.debug(\"Expected Word: %s, candidate = %s\" % (expected, list_candidate_arr))\n # input(\">>>>>\")\n # Calculate MAP@10 score\n this_map_result = eval_utils.mapk(expected, list_candidate_arr, word_level=True)\n if this_map_result >= 0:\n this_map_result = round(this_map_result, 6)\n # map_arr[idx_cnt] = this_map_result\n else:\n this_map_result = 0.0\n # map_arr.append(0.0)\n # map_arr[idx_cnt] = this_map_result\n\n map_ret_dict[idx_cnt] = this_map_result\n\n if expected in list_candidate_arr:\n section['correct'].append((a, b, c, expected))\n out_line = \"%s - %s + %s = ?; Expect: %s, candidate: %s\" % \\\n (word_left1, word_left2, word_right2, expected, list_candidate_arr)\n out_str += out_line + \"\\n\"\n\n # else:\n # section['incorrect'].append((a, b, c, expected))\n\n # fsolveable_writer.close()\n if section:\n # store the last section, too\n sections.append(section)\n\n map_arr = list(map_ret_dict.values())\n logger.debug(\"map_arr = \", map_arr)\n logger.debug(\"MAP_RET_DICT = \", map_ret_dict)\n # input(\"Check result dict: >>>>>\")\n\n total = {\n \"Emb_Name: \" + embedding_name + '/OOV/Total/VN_Solveable_Cases/VN_Phrase_Target':\n [oov_counter, (idx_cnt + 1), is_vn_counter, phrase_cnt],\n 'MAP@10': np.mean(map_arr)\n # ,\n # 'section': 'total'\n # ,\n # 'correct': sum((s['correct'] for s in sections), []),\n # 'incorrect': sum((s['incorrect'] for s in sections), []),\n }\n # print (out_str)\n # print(total)\n # logger.info(total)\n\n sections.append(total)\n sections_str = \"\\n%s\\n\" % sections\n\n return np.mean(map_arr), map_arr, sections_str\n\n\ndef print_analogy(left2: str, left1: str, right2: str, words: List[Word]) -> None:\n analogies = closest_analogies_OLD(left2, left1, right2, words)\n if (len(analogies) == 0):\n # print(f\"{left2}-{left1} is like {right2}-?\")\n print(\"%s-%s is like %s-?\"%(left2, left1, right2))\n # man-king is like woman-king\n # input: man is to king is like woman is to ___?(queen).\n else:\n (dist, w) = analogies[0]\n # alternatives = ', '.join([f\"{w.text} ({dist})\" for (dist, w) in analogies])\n # print(f\"{left2}-{left1} is like {right2}-{w.text}\")\n print(\"%s-%s is like %s-%s\"%(left2, left1, right2, w.text))\n\n"
},
{
"path": "src/codes/utils/embedding_io.py",
"content": "from typing import Iterable, List, Set\n\nfrom itertools import groupby\nimport numpy as np\nimport re\nimport utils.vectors as v\nfrom utils.word import Word\nimport logging\nimport os\nfrom embeddings.embedding_configs import EmbeddingConfigs\n\n\ndef save_model_to_file(embedding_model: List[Word], model_file_out: str):\n \"\"\"\n Save loaded model back to file (to remove duplicated items).\n :param embedding_model:\n :param model_file_out:\n :return:\n \"\"\"\n fwriter = open(model_file_out, \"w\")\n\n meta_data = \"%s %s\\n\"%(len(embedding_model), len(embedding_model[0].vector))\n fwriter.write(meta_data)\n fwriter.flush()\n for w_Word in embedding_model:\n line = w_Word.text + \" \" + \" \".join(str(scalar) for scalar in w_Word.vector.tolist())\n fwriter.write(line + \"\\n\")\n fwriter.flush()\n fwriter.close()\n\n\ndef load_word_embeddings(file_paths: str, emb_config: EmbeddingConfigs) -> List[List[Word]]:\n \"\"\"\n Sonvx: load multiple embeddings: e.g., ;\n :param file_paths:\n :param emb_config:\n :return:\n \"\"\"\n embedding_models = []\n embedding_names = []\n if file_paths and file_paths.__contains__(\";\"):\n files = file_paths.split(\";\")\n for emb_file in files:\n word_embedding = load_word_embedding(emb_file.replace(\"\\\"\", \"\"), emb_config)\n embedding_name = os.path.basename(os.path.normpath(emb_file))\n embedding_models.append(word_embedding)\n embedding_names.append(embedding_name)\n else:\n return [load_word_embedding(file_paths), emb_config]\n\n return embedding_names, embedding_models\n\n\ndef load_word_embedding(file_path: str, emb_config: EmbeddingConfigs) -> List[Word]:\n \"\"\"\n Load and cleanup the data.\n :param file_path:\n :param emb_config:\n :return:\n \"\"\"\n # print(f\"Loading {file_path}...\")\n print(\"Loading %s ...\"%(file_path))\n words = load_words_raw(file_path, emb_config)\n # print(f\"Loaded {len(words)} words.\")\n print(\"Loaded %s words.\" %(len(words)))\n\n # Test\n word1 = words[1]\n print(\"Vec Len(word1) = \", len(word1.vector))\n\n # num_dimensions = most_common_dimension(words)\n # words = [w for w in words if len(w.vector) == dims]\n # print(f\"Using {num_dimensions}-dimensional vectors, {len(words)} remain.\")\n\n # words = remove_stop_words(words)\n # print(f\"Removed stop words, {len(words)} remain.\")\n\n # ords = remove_duplicates(words)\n # print(f\"Removed duplicates, {len(words)} remain.\")\n\n logging.debug(\"Embedding words: \", words[:10])\n print(\"Emb_vocab_size = \", len(words))\n # input(\"Done loading embedding: >>>>\")\n\n return words\n\n\ndef load_words_raw(file_path: str, emb_config: EmbeddingConfigs) -> List[Word]:\n \"\"\"\n Load the file as-is, without doing any validation or cleanup.\n :param file_path:\n :param emb_config:\n :return:\n \"\"\"\n\n def parse_line(line: str, frequency: int) -> Word:\n # print(\"Line=\", line)\n tokens = line.split(\" \")\n word = tokens[0]\n if emb_config.do_normalize_emb:\n vector = v.normalize(np.array([float(x) for x in tokens[1:]]))\n else:\n vector = np.array([float(x) for x in tokens[1:]])\n return Word(word, vector, frequency)\n\n # Sonvx: NOT loading the same word twice.\n\n unique_dict = {}\n\n words = []\n # Words are sorted from the most common to the least common ones\n frequency = 1\n\n duplicated_entry = 0\n\n idx_counter, vocab_size, emb_dim = 0, 0, 0\n with open(file_path) as f:\n for line in f:\n line = line.rstrip()\n\n # print(\"Processing line: \", line)\n\n if idx_counter == 0 and emb_config.is_word2vec_format:\n try:\n meta_info = line.split(\" \")\n vocab_size = int(meta_info[0])\n emb_dim = int(meta_info[1])\n idx_counter += 1\n continue\n except Exception as e:\n print(\"meta_info = \"%(meta_info))\n logging.error(\"Input embedding has format issue: Error = %s\" % (e))\n\n # if len(line) < 20: # Ignore the first line of w2v format.\n # continue\n\n w = parse_line(line, frequency)\n\n # Svx: only load if the word is not existed in the list.\n if w.text not in unique_dict:\n unique_dict[w.text] = frequency\n words.append(w)\n frequency += 1\n else:\n duplicated_entry += 1\n # print(\"Loading the same word again\")\n\n # # Svx: check if the embedding dim is the same with the metadata, random check only\n if idx_counter == 10:\n if len(w.vector) != emb_dim:\n message = \"Metadata and the real vector size do not match: meta:real = %s:%s\" \\\n % (emb_dim, len(w.vector))\n logging.error(message)\n raise ValueError(message)\n idx_counter += 1\n\n if duplicated_entry > 0:\n logging.debug(\"Loading the same word again: %s\"%(duplicated_entry))\n\n # Final check:\n if (frequency - 1) != vocab_size:\n msg = \"Loaded %s/%s unique vocab.\" % ((frequency - 1), vocab_size)\n logging.info(msg)\n\n return words\n\n\ndef iter_len(iter: Iterable[complex]) -> int:\n return sum(1 for _ in iter)\n\n\ndef most_common_dimension(words: List[Word]) -> int:\n \"\"\"\n There is a line in the input file which is missing a word\n (search -0.0739, -0.135, 0.0584).\n \"\"\"\n lengths = sorted([len(word.vector) for word in words])\n dimensions = [(k, iter_len(v)) for k, v in groupby(lengths)]\n print(\"Dimensions:\")\n for (dim, num_vectors) in dimensions:\n # print(f\"{num_vectors} {dim}-dimensional vectors\")\n print(\"%s %s-dimensional vectors\"%(num_vectors, dim))\n most_common = sorted(dimensions, key=lambda t: t[1], reverse=True)[0]\n return most_common[0]\n\n\n# We want to ignore these characters,\n# so that e.g. \"U.S.\", \"U.S\", \"US_\" and \"US\" are the same word.\nignore_char_regex = re.compile(\"[\\W_]\")\n\n# Has to start and end with an alphanumeric character\nis_valid_word = re.compile(\"^[^\\W_].*[^\\W_]$\")\n\n\ndef remove_duplicates(words: List[Word]) -> List[Word]:\n seen_words: Set[str] = set()\n unique_words: List[Word] = []\n for w in words:\n canonical = ignore_char_regex.sub(\"\", w.text)\n if not canonical in seen_words:\n seen_words.add(canonical)\n # Keep the original ordering\n unique_words.append(w)\n return unique_words\n\n\ndef remove_stop_words(words: List[Word]) -> List[Word]:\n return [w for w in words if (\n len(w.text) > 1 and is_valid_word.match(w.text))]\n\n\n# Run \"smoke tests\" on import\nassert [w.text for w in remove_stop_words([\n Word('a', [], 1),\n Word('ab', [], 1),\n Word('-ab', [], 1),\n Word('ab_', [], 1),\n Word('a.', [], 1),\n Word('.a', [], 1),\n Word('ab', [], 1),\n])] == ['ab', 'ab']\nassert [w.text for w in remove_duplicates([\n Word('a.b', [], 1),\n Word('-a-b', [], 1),\n Word('ab_+', [], 1),\n Word('.abc...', [], 1),\n])] == ['a.b', '.abc...']\n"
},
{
"path": "src/codes/utils/eval_utils.py",
"content": "\n\"\"\"\nMAP@K word level and character level are explained in detail in this paper:\n\ndpUGC: Learn Differentially Private Representationfor User Generated Contents\nXuan-Son Vu, Son N. Tran, Lili Jiang\nIn: Proceedings of the 20th International Conference on Computational Linguistics and\nIntelligent Text Processing, April, 2019, (to appear)\n\nPlease cite the above paper if you use codes in this file.\n\n\"\"\"\n\n\ndef apk(actual, predicted, k=10):\n \"\"\"\n Computes the average precision at k.\n This function computes the average prescision at k between two lists of\n items.\n Parameters\n ----------\n actual : list\n A list of elements that are to be predicted (order doesn't matter)\n predicted : list\n A list of predicted elements (order does matter)\n k : int, optional\n The maximum number of predicted elements\n Returns\n -------\n score : double\n The average precision at k over the input lists\n \"\"\"\n if len(predicted) > k:\n predicted = predicted[:k]\n\n score = 0.0\n num_hits = 0.0\n\n for i, p in enumerate(predicted):\n if p in actual and p not in predicted[:i]:\n num_hits += 1.0\n score += num_hits / (i + 1.0)\n\n if not actual:\n return 0.0\n\n return score / min(len(actual), k)\n\n\ndef mapk(actual, predicted, k=10, word_level=True):\n \"\"\"\n Computes the mean average precision at k.\n This function computes the mean average prescision at k between two lists\n of lists of items.\n Parameters\n ----------\n actual : list\n A list of lists of elements that are to be predicted\n (order doesn't matter in the lists)\n predicted : list\n A list of lists of predicted elements\n (order matters in the lists)\n k : int, optional\n The maximum number of predicted elements\n Returns\n -------\n score : double\n The mean average precision at k over the input lists\n \"\"\"\n # print(\"Sending arr = \", arr)\n if word_level:\n return calc_map(actual, predicted, topK=k)\n else:\n # arr = [apk(a, p, k) for a, p in zip(actual, predicted)]\n # return np.mean(arr)\n return calc_map_character_level(actual, predicted, topK=k)\n\n\ndef calc_map(actual, predicted, topK=10):\n \"\"\"\n\n :param actual:\n :param predicted:\n :param topK:\n :return:\n \"\"\"\n # print(\"Input: actual %s, predicted %s\"%(actual, predicted))\n if len(predicted) > topK:\n predicted = predicted[:topK]\n idx = 1\n hit = 0\n map_arr = []\n for answer in predicted:\n if answer in actual[:topK]:\n hit += 1\n val = (hit * 1.0) / (idx * 1.0)\n # print(\"hit = %s, idx = %s\"%(hit, idx))\n map_arr.append(val)\n # print(\"hit: %s, map_arr = %s\"%(answer, map_arr))\n idx += 1\n # print(\"map_arr = %s done\", map_arr)\n if len(map_arr) > 0:\n return np.mean(map_arr)\n else:\n return 0.0\n\n\ndef calc_map_character_level(actual, predicted, topK=10):\n \"\"\"\n\n :param actual:\n :param predicted:\n :param topK:\n :return:\n \"\"\"\n # print(\"Input: actual %s, predicted %s\" % (actual, predicted))\n if len(predicted) > topK:\n predicted = predicted[:topK]\n\n if len(actual) > topK:\n actual = actual[:topK]\n\n rank = 1\n hit = 0\n actual_seq = ''.join([word for word in actual])\n predicted_seq = ''.join([word for word in predicted])\n map_arr = []\n for char in predicted_seq:\n if char in actual_seq[:rank]:\n hit += 1\n val = (hit * 1.0) / (rank * 1.0)\n # print(\"hit = %s, idx = %s\" % (hit, rank))\n map_arr.append(val)\n # print(\"hit: %s, map_arr = %s\" % (char, map_arr))\n rank += 1\n # print(\"map_arr = %s done\", map_arr)\n return np.mean(map_arr)\n\n\nimport unittest\nimport numpy as np\n\n\ndef test_apk(self):\n self.assertAlmostEqual(apk(range(1, 6), [6, 4, 7, 1, 2], 2), 0.25)\n self.assertAlmostEqual(apk(range(1, 6), [1, 1, 1, 1, 1], 5), 0.2)\n predicted = range(1, 21)\n predicted.extend(range(200, 600))\n self.assertAlmostEqual(apk(range(1, 100), predicted, 20), 1.0)\n\n\ndef test_mapk(self):\n self.assertAlmostEqual(mapk([range(1, 5)], [range(1, 5)], 3), 1.0)\n self.assertAlmostEqual(mapk([[1, 3, 4], [1, 2, 4], [1, 3]],\n [range(1, 6), range(1, 6), range(1, 6)], 3), 0.685185185185185)\n self.assertAlmostEqual(mapk([range(1, 6), range(1, 6)],\n [[6, 4, 7, 1, 2], [1, 1, 1, 1, 1]], 5), 0.26)\n self.assertAlmostEqual(mapk([[1, 3], [1, 2, 3], [1, 2, 3]],\n [range(1, 6), [1, 1, 1], [1, 2, 1]], 3), 11.0 / 18)\n\n\nif __name__ == '__main__':\n a1 = [\"1\", '2', '3', '4']\n b1 = ['1', '5', '2', '8']\n print(mapk(a1, b1, 4))\n\n a1 = [\"15\"]\n b1 = [\"1\", \"2\", \"3\", \"4\", \"5\",\"6\",\"7\",\"8\",\"9\",\"10\"]\n\n print(\"MapK:\", mapk(a1, b1, 4))\n\n # unittest.main()\n"
},
{
"path": "src/codes/utils/file_utils.py",
"content": "import pickle\n\n\ndef save_obj(obj, file_path):\n with open(file_path + '.pkl', 'wb') as f:\n pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)\n\n\ndef load_obj(file_path):\n with open(file_path + '.pkl', 'rb') as f:\n return pickle.load(f)\n\n\ndef get_unique_vocab(analogy_file_path, write_out_file):\n \"\"\"\n\n :param analogy_file_path:\n :param write_out_file:\n :return:\n \"\"\"\n vocab_dict = {}\n with open(analogy_file_path, \"r\") as freader:\n for line in freader:\n if line.__contains__(\" | \"):\n word_parts = line.split(\" | \")\n for word in word_parts:\n word = word.rstrip()\n vocab_dict[word] = 0\n\n fwriter = open(write_out_file, \"w\")\n for word in vocab_dict.keys():\n fwriter.write(word + \"\\n\")\n fwriter.close()\n\n print(\"Write dictionary file to %s\"%(write_out_file))\n\n return vocab_dict\n\n\nif __name__ == '__main__':\n get_unique_vocab(\"../data/embedding_analogies/portuguese/LX-4WAnalogies-ETNLP.txt\",\n \"../data/embedding_analogies/portuguese/vocab.txt\")\n"
},
{
"path": "src/codes/utils/string_utils.py",
"content": "import six\n\n\ndef convert_to_unicode(text):\n \"\"\"Converts `text` to Unicode (if it's not already), assuming utf-8 input.\"\"\"\n if six.PY3:\n if isinstance(text, str):\n return text\n elif isinstance(text, bytes):\n return text.decode(\"utf-8\", \"ignore\")\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n elif six.PY2:\n if isinstance(text, str):\n return text.decode(\"utf-8\", \"ignore\")\n elif isinstance(text, unicode):\n return text\n else:\n raise ValueError(\"Unsupported string type: %s\" % (type(text)))\n else:\n raise ValueError(\"Not running on Python2 or Python 3?\")\n\n\n"
},
{
"path": "src/codes/utils/vectors.py",
"content": "from typing import List, Any, Optional\n\nimport math\nimport numpy as np\n\n# Adopt from https://github.com/mkonicek/nlp/vecters.py\n\n# Vector = np.ndarray[float]\nVector = 'np.ndarray[float]'\nvector_type = 'np.ndarray[float]'\n\n# Vector = np.ndarray(dtype=float)\n\n\ndef l2_len(v: vector_type) -> float:\n return math.sqrt(np.dot(v, v))\n\n\ndef dot(v1: vector_type, v2: vector_type) -> float:\n assert v1.shape == v2.shape\n return np.dot(v1, v2)\n\n\ndef mean(v1: vector_type, v2: vector_type) -> Vector:\n \"\"\"\n Added by Sonvx: get mean of 2 vectors.\n :param v1:\n :param v2:\n :return:\n \"\"\"\n assert v1.shape == v2.shape\n return np.mean([v1, v2], axis=0)\n\n\ndef mean_list(v1: List[Vector]) -> Vector:\n \"\"\"\n Added by Sonvx: get mean of 2 vectors.\n :param v1:\n :return:\n \"\"\"\n if len(v1) > 0:\n return np.mean(v1, axis=0)\n else:\n return None\n\n\ndef add(v1: vector_type, v2: vector_type) -> Vector:\n assert v1.shape == v2.shape\n return np.add(v1, v2)\n\n\ndef sub(v1: vector_type, v2: vector_type) -> Vector:\n assert v1.shape == v2.shape\n return np.subtract(v1, v2)\n\n\ndef normalize(v: vector_type) -> Vector:\n return v / l2_len(v)\n\n\ndef cosine_similarity_normalized(v1: vector_type, v2: vector_type) -> float:\n \"\"\"\n Returns the cosine of the angle between the two vectors.\n Each of the vectors must have length (L2-norm) equal to 1.\n Results range from -1 (very different) to 1 (very similar).\n \"\"\"\n return dot(v1, v2)\n"
},
{
"path": "src/codes/utils/word.py",
"content": "from typing import List\nfrom utils.vectors import Vector\n\n# Adopt from https://github.com/mkonicek/nlp/Word.py\n\n\nclass Word:\n \"\"\"A single word (one line of the input vector embedding file)\"\"\"\n\n def __init__(self, text: str, vector: Vector, frequency: int) -> None:\n self.text = text\n self.vector = vector\n self.frequency = frequency\n\n def __repr__(self) -> str:\n vector_preview = ', '.join(map(str, self.vector[:2]))\n # return f\"{self.text} [{vector_preview}, ...]\"\n return \"%s [%s, ...]\"%(self.text, vector_preview)\n"
},
{
"path": "src/codes/visualizer/README.md",
"content": "# Requirements:\n- ```pip install gensim flask```\n- Download any pre-trained embeddings and put it into ../03.run_etnlp_visualizer_inter.sh\n\n# How to run\n> 1. sh ../03.run_etnlp_visualizer_inter.sh\n> 2. Visit http://localhost:8089\n\n# Screenshot\n\n![Alt text](https://github.com/vietnlp/etnlp/blob/master/images/etnlp_view_multi_embeddings.png \"Screenshot example of one given input\")\n\n"
},
{
"path": "src/codes/visualizer/__init__.py",
"content": ""
},
{
"path": "src/codes/visualizer/outof_w2vec.dict",
"content": "\n\n\n\n\n\n\n\n\n\n'news' \n\n\n\n\n\n'news' \n\n\n\n\n\n'news' \n\n\n\n\n\n'news' \n\n\n\n\n\n'news' \n\n\n\n\n\n\n'news' \n\n\n\n\n\n'news' \n\n'back' \n'back' \n'back' \n'back' \n'news' \n'news' \n'back' \n'back' \n'back' \n'back' \n'news' \n'news' \n'back' \n'back' \n'back' \n'back' \n'news' \n'news' \n'lovely' \n'lovely' \n'lovely' \n'lovely' \n'love' \n'love' \n"
},
{
"path": "src/codes/visualizer/static/style.css",
"content": ".container-4{\n overflow: hidden;\n width: 300px;\n vertical-align: middle;\n white-space: nowrap;\n}\n\n.container-4 input#search{\n width: 300px;\n height: 50px;\n background: #2b303b;\n border: none;\n font-size: 10pt;\n float: left;\n color: #fff;\n padding-left: 15px;\n -webkit-border-radius: 5px;\n -moz-border-radius: 5px;\n border-radius: 5px;\n}\n\n.container-4 input#search::-webkit-input-placeholder {\n color: #65737e;\n}\n\n.container-4 input#search:-moz-placeholder { /* Firefox 18- */\n color: #65737e;\n}\n\n.container-4 input#search::-moz-placeholder { /* Firefox 19+ */\n color: #65737e;\n}\n\n.container-4 input#search:-ms-input-placeholder {\n color: #65737e;\n}\n\n.container-4 button.icon{\n -webkit-border-top-right-radius: 5px;\n -webkit-border-bottom-right-radius: 5px;\n -moz-border-radius-topright: 5px;\n -moz-border-radius-bottomright: 5px;\n border-top-right-radius: 5px;\n border-bottom-right-radius: 5px;\n\n border: none;\n background: #232833;\n height: 50px;\n width: 50px;\n color: #4f5b66;\n opacity: 0;\n font-size: 10pt;\n\n -webkit-transition: all .55s ease;\n -moz-transition: all .55s ease;\n -ms-transition: all .55s ease;\n -o-transition: all .55s ease;\n transition: all .55s ease;\n}\n\n.container-4:hover button.icon, .container-4:active button.icon, .container-4:focus button.icon{\n outline: none;\n opacity: 1;\n margin-left: -50px;\n}\n\n.container-4:hover button.icon:hover{\n background: white;\n}\n\ndiv#answers {\n background-color: #f2f2f2;\n padding-top: 2px;\n padding-bottom: 2px;\n padding-left: 100px; \n}\n"
},
{
"path": "src/codes/visualizer/templates/app.html",
"content": "\n\n\n \n \n Title\n\n\n\n