[ { "path": ". gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\ncover/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\n.pybuilder/\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n# For a library or package, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# .python-version\n\n# pipenv\n# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n# However, in case of collaboration, if having platform-specific dependencies or dependencies\n# having no cross-platform support, pipenv may install dependencies that don't work, or not\n# install all needed dependencies.\n#Pipfile.lock\n\n# poetry\n# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.\n# This is especially recommended for binary packages to ensure reproducibility, and is more\n# commonly ignored for libraries.\n# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control\n#poetry.lock\n\n# pdm\n# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.\n#pdm.lock\n# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it\n# in version control.\n# https://pdm.fming.dev/#use-with-ide\n.pdm.toml\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n# pytype static type analyzer\n.pytype/\n\n# Cython debug symbols\ncython_debug/\n\n# PyCharm\n# JetBrains specific template is maintained in a separate JetBrains.gitignore that can\n# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore\n# and can be added to the global gitignore or merged into this file. For a more nuclear\n# option (not recommended) you can uncomment the following to ignore the entire idea folder.\n#.idea/" }, { "path": ".github/FUNDING.yml", "content": "# These are supported funding model platforms\n\ngithub: Barqawiz # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]\npatreon: # Replace with a single Patreon username\nopen_collective: # Replace with a single Open Collective username\nko_fi: # Replace with a single Ko-fi username\ntidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel\ncommunity_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry\nliberapay: # Replace with a single Liberapay username\nissuehunt: # Replace with a single IssueHunt username\notechie: # Replace with a single Otechie username\ncustom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']\n" }, { "path": "LICENSE.md", "content": "The MIT License (MIT)\n\nCopyright (c) 2017 Shakkala Project\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n" }, { "path": "PIP_README.md", "content": "# Shakkala Project V 2.1 مشروع شكّالة\n\n## Introduction\nThe Shakkala project presents a recurrent neural network for Arabic text vocalization that automatically forms Arabic characters (تشكيل الحروف) to enhance text-to-speech systems. The model can also be used in other applications such as improving search results. In the beta version, the model was trained on over a million sentences, including a majority of historical Arabic data from books and some modern data from the internet. The accuracy of the model reached up to 95%, and in some data sets it achieved even higher levels of accuracy depending on complexity and data distribution. This innovative approach has the potential to significantly improve the quality of writing and text-to-speech systems for the Arabic language.\n\n## Requirements\n\n```\npip install shakkala\n```\n\nNote: Shakkala has been tested with Tensorflow 2.9.3.
\n\n## Code Examples (How to)\nCheck full example in (demo.py) file.
\n\n0. Import\n```\nfrom shakkala import Shakkala\n```\n\n1. Create Shakkala object\n```\nsh = Shakkala()\n```\nOR for advanced usage:\n```\nsh = Shakkala(version={version_num})\n```\n2. Prepare input\n```\ninput_text = \"فإن لم يكونا كذلك أتى بما يقتضيه الحال وهذا أولى\"\ninput_int = sh.prepare_input(input_text)\n```\n3. Call the neural network\n```\nmodel, graph = sh.get_model()\nlogits = model.predict(input_int)[0]\n```\n4. Predict output\n```\npredicted_harakat = sh.logits_to_text(logits)\nfinal_output = sh.get_final_text(input_text, predicted_harakat)\n```\nAvailable models:
\n\n- version_num=1: First test of the solution.\n- version_num=2: Main release version.\n- version_num=3: Some enhancements from version number 2.\n\nIt worth to try both version_num=2 and version_num=3.\n\n## Perfomance Tips\nShakkala built in object oriented way to load the model once into memory for faster prediction, to make sure you dont load it multiple times in your service or application follow the steps:\n- Load the model in global variable:\n```\nsh = Shakkala(folder_location, version={version_num})\nmodel, graph = sh.get_model()\n```\n- Then inside your request function or loop add:\n```\ninput_int = sh.prepare_input(input_text)\nlogits = model.predict(input_int)[0]\npredicted_harakat = sh.logits_to_text(logits)\nfinal_output = sh.get_final_text(input_text, predicted_harakat)\n```\n\n## Accuracy\nIn this beta version 2 accuracy reached up to 95% and in some data it reach more based on complexity and data disribution.\nThis beta version trained on more than million sentences with majority of historical Arabic data from books and **some of** available formed modern data in the internet.
\n\n### Prediction Example\nFor live demo based on Shakkala library click the [link](http://ahmadai.com/shakkala/)
\n\n| Real output | Predicted output |\n| ------------- | ---------------- |\n| فَإِنْ لَمْ يَكُونَا كَذَلِكَ أَتَى بِمَا يَقْتَضِيهِ الْحَالُ وَهَذَا أَوْلَى | فَإِنْ لَمْ يَكُونَا كَذَلِكَ أَتَى بِمَا يَقْتَضِيهِ الْحَالُ وَهَذَا أَوْلَى |\n| قَالَ الْإِسْنَوِيُّ وَسَوَاءٌ فِيمَا قَالُوهُ مَاتَ فِي حَيَاةِ أَبَوَيْهِ أَمْ لَا | قَالَ الْإِسْنَوِيُّ وَسَوَاءٌ فِيمَا قَالُوهُ مَاتَ فِي حَيَاةِ أَبَوَيْهِ أَمْ لَا |\n| طَابِعَةٌ ثُلَاثِيَّةُ الْأَبْعَاد | طَابِعَةٌ ثَلَاثِيَّةُ الْأَبْعَادِ |\n\n### Accuracy Enhancements \nThe model can be enhanced to reach more than 95% accuracy with following:
\n- Availability of more formed **modern** data to train the network. (because current version trained with mostly available historical Arabic data and some modern data)\n- Stack different models\n\n## References\n- A paper compare different arabic text diacritization models and show that shakkala is the best among available neural networks for this solution:\n[Arabic Text Diacritization Using Deep Neural Networks, 2019](https://arxiv.org/abs/1905.01965)\n\n## Citation\nFor academic work use\n```\nShakkala, Arabic text vocalization, Barqawi & Zerrouki\n```\nOR bibtex format\n```\n@misc{\n title={Shakkala, Arabic text vocalization},\n author={Barqawi, Zerrouki},\n url={https://github.com/Barqawiz/Shakkala},\n year={2017}\n}\n```\n\n## Contribution\n### Core Team\n1. Ahmad Barqawi: Neural Network Developer.
\n2. Taha Zerrouki: Mentor Data and Results.
\n### Contributors\n1. Zaid Farekh & propellerinc.me: Provide infrastructure and consultation support.
\n2. Mohammad Issam Aklik: Artist.
\n3. Brahim Sidi: Form new sentences.
\n4. Fadi Bakoura: Aggregate online content.
\n5. Ola Ghanem: Testing.
\n6. Ali Hamdi Ali Fadel: Contribute code.
\n\nLicense\n-------\nFree to use and distribute only mention the original project name Shakkala as base model.
\n\n The MIT License (MIT)\n\n Copyright (c) 2017 Shakkala Project\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n" }, { "path": "README.md", "content": "# Shakkala Project مشروع شكّالة\n[![PyPI version](https://badge.fury.io/py/shakkala.svg)](https://badge.fury.io/py/shakkala)\n\n\"Model\"\n\n## Introduction\nThe Shakkala project presents a recurrent neural network for Arabic text vocalization that automatically forms Arabic characters (تشكيل الحروف) to enhance text-to-speech systems. The model can also be used in other applications such as improving search results. In the beta version, the model was trained on over a million sentences, including a majority of historical Arabic data from books and some modern data from the internet. The accuracy of the model reached up to 95%, and in some data sets it achieved even higher levels of accuracy depending on complexity and data distribution. This innovative approach has the potential to significantly improve the quality of writing and text-to-speech systems for the Arabic language.\n\n## Requirements\n\n### Easy setup\nNo GitHub repository installation is needed for [pip](https://pypi.org/project/shakkala/) case:\n```\npip install shakkala\n```\n\n### Project setup\nExecute the source code from Github:
\n```\ncd requirements\npip install -r requirements.txt\ncd ..\n```\n\nNote: Shakkala has been tested with Tensorflow 2.9.3.
\n\n## Code Examples (How to)\nCheck full example in (demo.py) file.
\n\n0. Import\n```\nfrom shakkala import Shakkala\n```\n\n1. Create Shakkala object\n```\nsh = Shakkala()\n```\nOR for advanced usage:\n```\nsh = Shakkala(version={version_num})\n```\n2. Prepare input\n```\ninput_text = \"فإن لم يكونا كذلك أتى بما يقتضيه الحال وهذا أولى\"\ninput_int = sh.prepare_input(input_text)\n```\n3. Call the neural network\n```\nmodel, graph = sh.get_model()\nlogits = model.predict(input_int)[0]\n```\n4. Predict output\n```\npredicted_harakat = sh.logits_to_text(logits)\nfinal_output = sh.get_final_text(input_text, predicted_harakat)\n```\nAvailable models:
\n\n- version_num=1: First test of the solution.\n- version_num=2: Main release version.\n- version_num=3: Some enhancements from version number 2.\n\nIt worth to try both version_num=2 and version_num=3.\n\n### Demo run\nThe fastest way to start with Shakkala by running the demo from Github:\n```\npython demo.py\n```\n\n## Perfomance Tips\nShakkala built in object oriented way to load the model once into memory for faster prediction, to make sure you dont load it multiple times in your service or application follow the steps:\n- Load the model in global variable:\n```\nsh = Shakkala(folder_location, version={version_num})\nmodel, graph = sh.get_model()\n```\n- Then inside your request function or loop add:\n```\ninput_int = sh.prepare_input(input_text)\nlogits = model.predict(input_int)[0]\npredicted_harakat = sh.logits_to_text(logits)\nfinal_output = sh.get_final_text(input_text, predicted_harakat)\n```\n\n## Accuracy\nIn this beta version 2 accuracy reached up to 95% and in some data it reach more based on complexity and data disribution.\nThis beta version trained on more than million sentences with majority of historical Arabic data from books and **some of** available formed modern data in the internet.
\n\n\"history\"\n\n### Prediction Example\nFor live demo based on Shakkala library click the [link](http://ahmadai.com/shakkala/)
\n\n| Real output | Predicted output |\n| ------------- | ---------------- |\n| فَإِنْ لَمْ يَكُونَا كَذَلِكَ أَتَى بِمَا يَقْتَضِيهِ الْحَالُ وَهَذَا أَوْلَى | فَإِنْ لَمْ يَكُونَا كَذَلِكَ أَتَى بِمَا يَقْتَضِيهِ الْحَالُ وَهَذَا أَوْلَى |\n| قَالَ الْإِسْنَوِيُّ وَسَوَاءٌ فِيمَا قَالُوهُ مَاتَ فِي حَيَاةِ أَبَوَيْهِ أَمْ لَا | قَالَ الْإِسْنَوِيُّ وَسَوَاءٌ فِيمَا قَالُوهُ مَاتَ فِي حَيَاةِ أَبَوَيْهِ أَمْ لَا |\n| طَابِعَةٌ ثُلَاثِيَّةُ الْأَبْعَاد | طَابِعَةٌ ثَلَاثِيَّةُ الْأَبْعَادِ |\n\n### Accuracy Enhancements \nThe model can be enhanced to reach more than 95% accuracy with following:
\n- Availability of more formed **modern** data to train the network. (because current version trained with mostly available historical Arabic data and some modern data)\n- Stack different models\n\n## Model Design\n\"Model\"/\n\n## References\n- A paper compare different arabic text diacritization models and show that shakkala is the best among available neural networks for this solution:\n[Arabic Text Diacritization Using Deep Neural Networks, 2019](https://arxiv.org/abs/1905.01965)\n\n## Citation\nFor academic work use\n```\nShakkala, Arabic text vocalization, Barqawi & Zerrouki\n```\nOR bibtex format\n```\n@misc{\n title={Shakkala, Arabic text vocalization},\n author={Barqawi, Zerrouki},\n url={https://github.com/Barqawiz/Shakkala},\n year={2017}\n}\n```\n\n## Contribution\n### Core Team\n1. Ahmad Barqawi: Neural Network Developer.
\n2. Taha Zerrouki: Mentor Data and Results.
\n### Contributors\n1. Zaid Farekh & propellerinc.me: Provide infrastructure and consultation support.
\n2. Mohammad Issam Aklik: Artist.
\n3. Brahim Sidi: Form new sentences.
\n4. Fadi Bakoura: Aggregate online content.
\n5. Ola Ghanem: Testing.
\n6. Ali Hamdi Ali Fadel: Contribute code.
\n\nLicense\n-------\nFree to use and distribute only mention the original project name Shakkala as base model.
\n\n The MIT License (MIT)\n\n Copyright (c) 2017 Shakkala Project\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n" }, { "path": "requirements/publish_commands.txt", "content": "python setup.py sdist\npython setup.py bdist_wheel\ntwine upload dist/*\n" }, { "path": "requirements/requirements.txt", "content": "click==8.1.3\nh5py==3.8.0\nhtml5lib==1.1\nMarkdown==3.4.1\nnltk==3.6.6\nnumpy==1.24.1\noauthlib==3.2.2\nopt-einsum==3.3.0\npackaging==23.0\nregex==2022.10.31\nsix==1.16.0\ntensorflow==2.9.3\nurllib3==1.26.14\nwebencodings==0.5.1\nWerkzeug==2.2.2\nwrapt==1.14.1\n" }, { "path": "setup.py", "content": "from setuptools import setup, find_packages\n\nwith open(\"PIP_README.md\", \"r\") as fh:\n long_description = fh.read()\n\nsetup(\n name='shakkala',\n version='1.7',\n author='Ahmad Albarqawi',\n packages=find_packages(),\n include_package_data=True,\n url='https://ahmadai.com/shakkala/',\n data_files=[('dictionary', ['shakkala/dictionary/input_vocab_to_int.pickle',\n 'shakkala/dictionary/output_int_to_vocab.pickle']),\n ('model', ['shakkala/model/middle_model.h5',\n 'shakkala/model/second_model6.h5',\n 'shakkala/model/simple_model.h5'])],\n description=\"Deep learning for Arabic text Vocalization - التشكيل الالي للنصوص العربية\",\n long_description=long_description,\n long_description_content_type=\"text/markdown\",\n install_requires=[ 'tensorflow==2.9.3', 'h5py==3.8.0', 'nltk==3.6.6', 'numpy==1.24.1', 'click==8.1.3' ],\n)\n" }, { "path": "shakkala/Shakkala.py", "content": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nLicense\n-------\n The MIT License (MIT)\n\n Copyright (c) 2017 Tashkel Project\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\nCreated on Sat Dec 16 22:46:28 2017\n\n@author: Ahmad Barqawi\n\"\"\"\nfrom . import helper\nimport os\nimport tensorflow as tf\nfrom tensorflow.compat.v1.keras.models import Model\nfrom tensorflow.compat.v1.keras.models import load_model\nfrom tensorflow.compat.v1.keras.optimizers import Adam\nfrom tensorflow.compat.v1.keras.losses import sparse_categorical_crossentropy\nfrom tensorflow.compat.v1.keras.preprocessing.sequence import pad_sequences\nimport numpy as np\n\n\nclass Shakkala:\n\n # intial\n #max_sentence = 495\n\n def __init__(self, folder_location=None, version=3):\n\n if folder_location is None:\n folder_location = os.path.dirname(os.path.abspath(__file__))\n\n assert folder_location != None, \"model_location cant be empty, send location of keras model\"\n\n model_folder = os.path.join(folder_location, 'model')\n if version == 1:\n self.max_sentence = 495\n self.model_location = os.path.join(model_folder, ('simple_model' + '.h5'))\n elif version == 2:\n self.max_sentence = 315\n self.model_location = os.path.join(model_folder, ('middle_model' + '.h5'))\n elif version == 3:\n self.max_sentence = 315\n self.model_location = os.path.join(model_folder, ('second_model6' + '.h5'))\n\n dictionary_folder = os.path.join(folder_location, 'dictionary')\n input_vocab_to_int = helper.load_binary('input_vocab_to_int',dictionary_folder)\n output_int_to_vocab = helper.load_binary('output_int_to_vocab',dictionary_folder)\n\n self.dictionary = {\n \"input_vocab_to_int\":input_vocab_to_int,\n \"output_int_to_vocab\":output_int_to_vocab}\n\n # model\n def get_model(self):\n print('start load model')\n model = load_model(self.model_location)\n print('end load model')\n graph = tf.compat.v1.get_default_graph()\n\n return model, graph\n\n # input processing\n\n def prepare_input(self, input_sent):\n\n assert input_sent != None and len(input_sent) < self.max_sentence, \\\n \"max length for input_sent should be {} characters, you can split the sentence into multiple sentecens and call the function\".format(self.max_sentence)\n\n input_sent = [input_sent]\n\n return self.__preprocess(input_sent)\n\n def __preprocess(self, input_sent):\n\n input_vocab_to_int = self.dictionary[\"input_vocab_to_int\"]\n\n input_letters_ids = [[input_vocab_to_int.get(ch, input_vocab_to_int['']) for ch in sent] for sent in input_sent]\n\n input_letters_ids = self.__pad_size(input_letters_ids, self.max_sentence)\n\n return input_letters_ids\n\n # output processing\n\n def logits_to_text(self, logits):\n text = []\n for prediction in np.argmax(logits, 1):\n if self.dictionary['output_int_to_vocab'][prediction] == '':\n continue\n text.append(self.dictionary['output_int_to_vocab'][prediction])\n return text\n\n def get_final_text(self,input_sent, output_sent):\n return helper.combine_text_with_harakat(input_sent, output_sent)\n\n def clean_harakat(self, input_sent):\n return helper.clear_tashkel(input_sent)\n\n # common\n def __pad_size(self, x, length=None):\n return pad_sequences(x, maxlen=length, padding='post')\n" }, { "path": "shakkala/__init__.py", "content": "from .Shakkala import Shakkala\n" }, { "path": "shakkala/demo.py", "content": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\"\"\"\nExample code using Shakkala library\n\"\"\"\nimport os\nfrom Shakkala import Shakkala\n\nif __name__ == \"__main__\":\n input_text = \"فإن لم يكونا كذلك أتى بما يقتضيه الحال وهذا أولى\"\n\n folder_location = './'\n\n # create Shakkala object\n sh = Shakkala(folder_location, version=3)\n\n # prepare input\n input_int = sh.prepare_input(input_text)\n\n print(\"finished preparing input\")\n\n print(\"start with model\")\n\n model, graph = sh.get_model()\n\n # with graph.as_default():\n logits = model.predict(input_int)[0]\n\n print(\"prepare and print output\")\n predicted_harakat = sh.logits_to_text(logits)\n\n final_output = sh.get_final_text(input_text, predicted_harakat)\n print(final_output)\n\n print(\"finished successfully\")\n" }, { "path": "shakkala/helper.py", "content": "\"\"\"\nLicense\n-------\n The MIT License (MIT)\n\n Copyright (c) 2017 Tashkel Project\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\nCreated on Sat Dec 16 22:46:28 2017\n\n@author: Ahmad Barqawi\n\"\"\"\nimport os\nimport glob\nimport string\nimport re\nimport pickle\nfrom nltk.tokenize import sent_tokenize, word_tokenize\n\n#convert using chr(harakat[0])\nharakat = [1614,1615,1616,1618,1617,1611,1612,1613]\nconnector = 1617\n\n\ndef save_binary(data, file, folder):\n location = os.path.join(folder, (file+'.pickle') )\n with open(location, 'wb') as ff:\n pickle.dump(data, ff, protocol=pickle.HIGHEST_PROTOCOL)\n\ndef load_binary(file, folder):\n location = os.path.join(folder, (file+'.pickle') )\n with open(location, 'rb') as ff:\n data = pickle.load(ff)\n\n return data\n\ndef get_sentences(data):\n\n return [sent for line in re.split(\"[\\n,،]+\", data) if line for sent in sent_tokenize(line.strip()) if sent]\n #return [sent for line in data.split('\\n') if line for sent in sent_tokenize(line) if sent]\n\ndef clear_punctuations(text):\n text = \"\".join(c for c in text if c not in string.punctuation)\n return text\n\ndef clear_english_and_numbers(text):\n text = re.sub(r\"[a-zA-Z0-9٠-٩]\", \" \", text);\n return text\n\ndef is_tashkel(text):\n return any(ord(ch) in harakat for ch in text)\n\ndef clear_tashkel(text):\n text = \"\".join(c for c in text if ord(c) not in harakat)\n return text\n\ndef get_harakat():\n return \"\".join(chr(item)+\"|\" for item in harakat)[:-1]\n\ndef get_taskel(sentence):\n\n output = []\n current_haraka = \"\"\n for ch in reversed(sentence):\n\n if ord(ch) in harakat:\n if (current_haraka == \"\") or\\\n (ord(ch) == connector and chr(connector) not in current_haraka) or\\\n (chr(connector) == current_haraka):\n current_haraka += ch\n else:\n if current_haraka == \"\":\n current_haraka = \"ـ\"\n output.insert(0, current_haraka)\n current_haraka = \"\"\n return output\n\ndef combine_text_with_harakat(input_sent, output_sent):\n #print(\"input : \" , len(input_sent))\n #print(\"output : \" , len(output_sent))\n\n \"\"\"\n harakat_stack = Stack()\n temp_stack = Stack()\n #process harakat\n for character, haraka in zip(input_sent, output_sent):\n temp_stack = Stack()\n\n haraka = haraka.replace(\"\",\"\").replace(\"\",\"\").replace(\"ـ\",\"\")\n\n if (character == \" \" and haraka != \"\" and ord(haraka) == connector):\n combine = harakat_stack.pop()\n combine += haraka\n harakat_stack.push(combine)\n else:\n harakat_stack.push(haraka)\n \"\"\"\n\n #fix combine differences\n input_length = len(input_sent)\n output_length = len(output_sent) # harakat_stack.size()\n for index in range(0,(input_length-output_length)):\n output_sent.append(\"\")\n\n #combine with text\n text = \"\"\n for character, haraka in zip(input_sent, output_sent):\n if haraka == '' or haraka == 'ـ':\n haraka = ''\n text += character + \"\" + haraka\n\n return text\n\n\n\n\nclass Stack:\n def __init__(self):\n self.stack = []\n\n def isEmpty(self):\n return self.size() == 0\n\n def push(self, item):\n self.stack.append(item)\n\n def pop(self):\n return self.stack.pop()\n\n def peek(self):\n if self.size() == 0:\n return None\n else:\n return self.stack[len(self.stack)-1]\n\n def size(self):\n return len(self.stack)\n\n def to_array(self):\n return self.stack\n" } ]