Full Code of YZHANG1270/Aspect-Based-Sentiment-Analysis for AI

Repository: YZHANG1270/Aspect-Based-Sentiment-Analysis
Branch: master
Commit: e5505ab3fdcf
Files: 30
Total size: 20.9 MB

Directory structure:
gitextract_rxbagi6o/

├── README.md
├── ai_challenge_sentiment/
│   ├── code/
│   │   └── sentiment_analysis2018_baseline/
│   │       ├── README.md
│   │       ├── __init__.py
│   │       ├── config.py
│   │       ├── data_process.py
│   │       ├── main_predict.py
│   │       ├── main_train.py
│   │       ├── model.py
│   │       └── requirements.txt
│   ├── model.py
│   └── train.py
├── aspect_predict.py
├── config.json
├── data/
│   ├── aspect/
│   │   ├── aspect_svc_test.xlsx
│   │   └── aspect_svc_train.xlsx
│   ├── chinese/
│   │   ├── CH_CAME_SB1_TEST.xlsx
│   │   ├── CH_PHNS_SB1_TEST.xlsx
│   │   ├── Chinese_phones_training.xlsx
│   │   └── camera_training.xlsx
│   └── polarity/
│       └── polarity_docu.xlsx
├── polarity_predict.py
├── train/
│   ├── aspect_classifier.py
│   ├── model/
│   │   ├── bilstm.py
│   │   └── model.py
│   └── polarity_classifier.py
└── utils/
    ├── __init__.py
    ├── baidu_tagging.py
    ├── data_process.py
    ├── grammar.py
    └── utils.py

================================================
FILE CONTENTS
================================================

================================================
FILE: README.md
================================================
# ABSA

Aspect Based Sentiment Analysis

虽说是基于观点的分析，但也是基于句子层的分析，因为需要按句子进行分析。

![](https://github.com/YZHANG1270/Aspect-Based-Sentiment-Analysis/blob/master/img/absa.png?raw=true)




##### 概念参考

- ABSA refer presentation [[ppt](https://www.iaria.org/conferences2016/filesHUSO16/OrpheeDeClercq_Keynote_ABSA.pdf)]
- 阿里云的商品评价解析 [[link](https://help.aliyun.com/document_detail/64231.html?spm=5176.12095382.1232858.4.739e3b24xUnvbZ)]

| 参数名         | 值                                                           |
| -------------- | ------------------------------------------------------------ |
| textPolarity   | 整条文本情感极性：正、中、负，text字段输入非法时返回-100     |
| textIntensity  | 整条文本情感程度(取值范围[-1,1]，越大代表越正向，越小代表越负向，接近0代表中性) |
| aspectItem     | 属性情感列表，每个元素是一个json字段                         |
| aspectCategory | 属性类别                                                     |
| aspectIndex    | 属性词所在的起始位置，终结位置                               |
| aspectTerm     | 属性词                                                       |
| opinionTerm    | 情感词                                                       |
| aspectPolarity | 属性片段极性（正、中、负）                                   |



##### Task Process

1. 按句 提取 属性词
2. 按句 提取 情感词
3. 属性词所在起始位置，终止位置
4. 属性词 -> EA分类
5. 情感词 -> 极性分类
6. 整条文本的感情极性（正、负、中） 及其概率值



##### Done Tasks

根据现有数据集，实际完成的任务

- [x] 按句进行 EA 分类
- [x] 按句进行情感极性分析



##### To do

- [ ] 观点过滤：文字噪音处理、虚假评论、水军、广告、不含观点、无意义文本
- [ ] negation 否定处理



##### SemEval ABSA

- NLP的 SemEval 论文合辑 [[ACL](https://www.aclweb.org/anthology/)]
- SemEval - 2014 - ABSA [[competition](http://alt.qcri.org/semeval2014/task4/)] [[data](http://alt.qcri.org/semeval2014/task4/index.php?id=data-and-tools)] 
- SemEval - 2015 - ABSA [[competition](http://alt.qcri.org/semeval2015/task12/)] [[data](http://alt.qcri.org/semeval2015/task12/index.php?id=data-and-tools)] [[paper](https://www.aclweb.org/anthology/S15-2082)] 
- SemEval - 2016 - ABSA [[competition](http://alt.qcri.org/semeval2016/task5/)] [[data](http://alt.qcri.org/semeval2016/task5/index.php?id=data-and-tools)] [[guideline](http://alt.qcri.org/semeval2016/task5/data/uploads/absa2016_annotationguidelines.pdf)] [[paper](https://www.aclweb.org/anthology/S16-1002)]
- bonus: CodaLab Competitions [[intro](https://www.hse.ru/data/2017/05/31/1171931089/CodaLabCompetitions.pdf)] 



##### 可参考的GitHub项目

数据集基本都基于 2014-2016 SemEval 比赛

- [data: self data] [Unsupervised-Aspect-Extraction](https://github.com/ruidan/Unsupervised-Aspect-Extraction) 
- [data: SemEval-2016] [aspect-extraction](https://github.com/soujanyaporia/aspect-extraction) 
- [data: SemEval-2015] [AspectBasedSentimentAnalysis](https://github.com/yardstick17/AspectBasedSentimentAnalysis) 跑了下这个项目，其中结合了语法分析和机器学习，按照语法规则抽取的属性词。代码嵌套逻辑比较强，不建议套用。
- [data: SemEval-2016] [Review_aspect_extraction](https://github.com/yafangy/Review_aspect_extraction) 
- [data: SemEval-2014, 2016] [DE-CNN](https://github.com/howardhsu/DE-CNN) 
- [data: SemEval-2015] [Coupled-Multi-layer-Attentions](https://github.com/happywwy/Coupled-Multi-layer-Attentions) 
- [data: SemEval-2016 laptop] [mem_absa](https://github.com/ganeshjawahar/mem_absa) 
- [data: SemEval-2014] [ABSA-PyTorch](https://github.com/songyouwei/ABSA-PyTorch) 
- [data: SemEval-2014, 2016] [Attention_Based_LSTM_AspectBased_SA](https://github.com/gangeshwark/Attention_Based_LSTM_AspectBased_SA) 
- [data: SemEval-2014] [ABSA_Keras](https://github.com/AlexYangLi/ABSA_Keras) 利用了tensorflow hub，适用hub时出现了版本问题未跑通。
- [data: SemEval-2016] [ABSA](https://github.com/LingxB/ABSA/tree/master/Data/SemEval) 

 

##### paper

- Deep Learning for Aspect-Based Sentiment Analysis [[paper](https://cs224d.stanford.edu/reports/WangBo.pdf)]
- Fine-grained Opinion Mining with Recurrent Neural Networks and Word Embeddings [[paper](https://www.aclweb.org/anthology/D15-1168)]
- Encoding Conversation Context for Neural Keyphrase Extraction from Microblog Posts [[paper](https://ai.tencent.com/ailab/media/publications/naacl2018/Encoding_Conversation_Context_for_Neural_Keyphrase_Extraction_from_Microblog_Posts.pdf)]
- End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF [[paper](https://arxiv.org/pdf/1603.01354.pdf)]
- [2012] 用户评论中的标签抽取以及排序 [[paper](http://lipiji.com/docs/li2011opinion.pdf)] 



##### 数据集

###### 中文

- AI-Challenge [[data](https://drive.google.com/file/d/1OInXRx_OmIJgK3ZdoFZnmqUi0rGfOaQo/view)] 
- SemEval ABSA 2016 [[data](http://alt.qcri.org/semeval2016/task5/index.php?id=data-and-tools)] 


###### 英文

- Amazon product data [[data](http://jmcauley.ucsd.edu/data/amazon/)]
- Web data: Amazon reviews [[data](https://snap.stanford.edu/data/web-Amazon.html)]
- Amazon Fine Food Reviews [[kaggle](https://www.kaggle.com/snap/amazon-fine-food-reviews)]
- SemEval ABSA



#### 优化方向

##### 字/词/句 文本嵌入Embedding

###### 中文

- Chinese Word Vectors [[github](https://github.com/Embedding/Chinese-Word-Vectors)] 
- nlp_chinese_corpus [[github](https://github.com/brightmart/nlp_chinese_corpus)] 
- 泛化语料、专业语料、向量化时，如何整合，还是两者独立向量化





ABSA书的目录，可以学习逻辑

#### ABSA Book Outline

1. Introduction
2. Aspect-Based Sentiment Analysis (ABSA)
   - 2.1. The three tasks of ABSA
   - 2.2. Domain and benchmark datasets
   - 2.3. Previous approaches to ABSA tasks
   - 2.4. Evaluation measures of ABSA tasks
3. Deep Learning for ABSA
   - 3.1. Multiple layers of DNN
   - 3.2. Initialization of input vectors
     - 3.2.1. Word embeddings vectors
     - 3.2.2. Featuring vectors
     - 3.2.3. Part-Of-Speech (POS) and chunk tags
     - 3.2.4. Commonsense knowledge
   - 3.3. Training process of DNNs
   - 3.4. Convolutional Neural Network Model (CNN)
     - 3.4.1. Architecture
     - 3.4.2. Application in consumer review domain
   - 3.5. Recurrent Neural Network Models (RNN)
     - 3.5.1. Computation of RNN models
     - 3.5.2. Bidirectional RNN
     - 3.5.3. Attention mechanism and memory networks
     - 3.5.4. Application in the consumer review domain
     - 3.5.5. Application in targeted sentiment analysis
   - 3.6. Recursive Neural Network Model (RecNN)
     - 3.6.1. Architecture
     - 3.6.2. Application
   - 3.7. Hybrid models
4. Comparison of performance on benchmark datasets
   - 4.1. Opinion target extraction
   - 4.2. Aspect category detection
   - 4.3. Sentiment polarity of aspect-based consumer reviews
   - 4.4. Sentiment polarity of targeted text
5. Challenges
   - 5.1. Domain adaptation
   - 5.2. Multilingual application
   - 5.3. Technical requirements
   - 5.4. Linguistic complications
6. Conclusion
7. Appendix: List of Abbreviations
8. References


================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/README.md
================================================

AI Challenger Sentiment Analysis Baseline
=========================================
功能描述
---
本工程主要用于为参赛者提供一个baseline，方便参赛者快速上手比赛，主要功能涵盖完成比赛的全流程，如数据读取、分词、特征提取、模型定义以及封装、
模型训练、模型验证、模型存储以及模型预测等。baseline仅是一个简单的参考，希望参赛者能够充分发挥自己的想象，构建在该任务上更加强大的模型。

开发环境
---
* 主要依赖工具包以及版本，详情见requirements.txt

项目结构
---
* src/config.py 项目配置信息模块，主要包括文件读取或存储路径信息
* src/data_process.py 数据处理模块，主要包括数据的读取以及处理等功能
* src/model.py 模型定义模块，主要包括模型的定义以及使用封装
* src/main_train.py 模型训练模块，模型训练流程包括 数据读取、分词、特征提取、模型训练、模型验证、模型存储等步骤
* src/main_predict.py 模型预测模块，模型预测流程包括 数据和模型的读取、分词、模型预测、预测结果存储等步骤 


使用方法
---
* 配置 在config.py中配置好文件存储路径
* 训练 运行nohup python main_train.py -mn your_model_name & 训练模型并保存，同时通过日志可以得到验证集的F1_score指标
* 预测 运行nohup python main_predict.py -mn your_model_name $ 通过加载上一步的模型，在测试集上做预测



================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/__init__.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-



================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/data_process.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-

import pandas as pd
import jieba


# 加载数据
def load_data_from_csv(file_name, header=0, encoding="utf-8"):

    data_df = pd.read_csv(file_name, header=header, encoding=encoding)

    return data_df


# 分词
def seg_words(contents):
    contents_segs = list()
    for content in contents:
        segs = jieba.lcut(content)
        contents_segs.append(" ".join(segs))

    return contents_segs


================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/main_predict.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-

from data_process import seg_words, load_data_from_csv
import config
import logging
import argparse
from sklearn.externals import joblib

logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname)s] <%(processName)s> (%(threadName)s) %(message)s')
logger = logging.getLogger(__name__)

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('-mn', '--model_name', type=str, nargs='?',
                        help='the name of model')

    args = parser.parse_args()
    model_name = args.model_name
    if not model_name:
        model_name = "model_dict.pkl"

    # load data
    logger.info("start load data")
    test_data_df = load_data_from_csv(config.test_data_path)

    # load model
    logger.info("start load model")
    classifier_dict = joblib.load(config.model_save_path + model_name)

    columns = test_data_df.columns.tolist()
    # seg words
    logger.info("start seg test data")
    content_test = test_data_df.iloc[:, 1]
    content_test = seg_words(content_test)
    logger.info("complete seg test data")

    # model predict
    logger.info("start predict test data")
    for column in columns[2:]:
        test_data_df[column] = classifier_dict[column].predict(content_test)
        logger.info("compete %s predict" % column)

    test_data_df.to_csv(config.test_data_predict_out_path, encoding="utf_8_sig", index=False)
    logger.info("compete predict test data")


================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/main_train.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-

from data_process import load_data_from_csv, seg_words
from model import TextClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
import config
import logging
import numpy as np
from sklearn.externals import joblib
import os
import argparse

logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname)s] <%(processName)s> (%(threadName)s) %(message)s')
logger = logging.getLogger(__name__)

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('-mn', '--model_name', type=str, nargs='?',
                        help='the name of model')

    args = parser.parse_args()
    model_name = args.model_name
    if not model_name:
        model_name = "model_dict.pkl"

    # load train data
    logger.info("start load data")
    train_data_df = load_data_from_csv(config.train_data_path)
    validate_data_df = load_data_from_csv(config.validate_data_path)

    content_train = train_data_df.iloc[:, 1]

    logger.info("start seg train data")
    content_train = seg_words(content_train)
    logger.info("complete seg train data")

    columns = train_data_df.columns.values.tolist()

    logger.info("start train feature extraction")
    vectorizer_tfidf = TfidfVectorizer(analyzer='word', ngram_range=(1, 5), min_df=5, norm='l2')
    vectorizer_tfidf.fit(content_train)
    logger.info("complete train feature extraction models")
    logger.info("vocab shape: %s" % np.shape(vectorizer_tfidf.vocabulary_.keys()))

    # model train
    logger.info("start train model")
    classifier_dict = dict()
    for column in columns[2:]:
        label_train = train_data_df[column]
        text_classifier = TextClassifier(vectorizer=vectorizer_tfidf)
        logger.info("start train %s model" % column)
        text_classifier.fit(content_train, label_train)
        logger.info("complete train %s model" % column)
        classifier_dict[column] = text_classifier

    logger.info("complete train model")

    # validate model
    content_validate = validate_data_df.iloc[:, 1]

    logger.info("start seg validate data")
    content_validate = seg_words(content_validate)
    logger.info("complete seg validate data")

    logger.info("start validate model")
    f1_score_dict = dict()
    for column in columns[2:]:
        label_validate = validate_data_df[column]
        text_classifier = classifier_dict[column]
        f1_score = text_classifier.get_f1_score(content_validate, label_validate)
        f1_score_dict[column] = f1_score

    f1_score = np.mean(list(f1_score_dict.values()))
    str_score = "\n"
    for column in columns[2:]:
        str_score = str_score + column + ":" + str(f1_score_dict[column]) + "\n"

    logger.info("f1_scores: %s\n" % str_score)
    logger.info("f1_score: %s" % f1_score)
    logger.info("complete validate model")

    # save model
    logger.info("start save model")
    model_save_path = config.model_save_path
    if not os.path.exists(model_save_path):
        os.makedirs(model_save_path)

    joblib.dump(classifier_dict, model_save_path + model_name)
    logger.info("complete save model")




================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/model.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-

from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import f1_score
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname)s] <%(processName)s> (%(threadName)s) %(message)s')
logger = logging.getLogger(__name__)


class TextClassifier():

    def __init__(self, vectorizer, classifier=MultinomialNB()):
        classifier = SVC(kernel="rbf")
        # classifier = SVC(kernel="linear")
        self.classifier = classifier
        self.vectorizer = vectorizer

    def features(self, x):
        return self.vectorizer.transform(x)

    def fit(self, x, y):

        self.classifier.fit(self.features(x), y)

    def predict(self, x):

        return self.classifier.predict(self.features(x))

    def score(self, x, y):
        return self.classifier.score(self.features(x), y)

    def get_f1_score(self, x, y):
        return f1_score(y, self.predict(x), average='macro')





================================================
FILE: ai_challenge_sentiment/code/sentiment_analysis2018_baseline/requirements.txt
================================================
python==2.7.13
numpy==1.13.1
pandas==0.20.3
jieba==0.39
sklearn==0.19.2


================================================
FILE: ai_challenge_sentiment/model.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-

from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import f1_score
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname)s] <%(processName)s> (%(threadName)s) %(message)s')
logger = logging.getLogger(__name__)


class TextClassifier():

    def __init__(self, vectorizer, classifier=MultinomialNB()):
        classifier = SVC(kernel="rbf")
        # classifier = SVC(kernel="linear")
        self.classifier = classifier
        self.vectorizer = vectorizer

    def features(self, x):
        return self.vectorizer.transform(x)

    def fit(self, x, y):

        self.classifier.fit(self.features(x), y)

    def predict(self, x):

        return self.classifier.predict(self.features(x))

    def score(self, x, y):
        return self.classifier.score(self.features(x), y)

    def get_f1_score(self, x, y):
        return f1_score(y, self.predict(x), average='macro')





================================================
FILE: ai_challenge_sentiment/train.py
================================================
# -*- coding: utf-8 -*-
"""
Spyder Editor

This is a temporary script file.
"""


import os
os.chdir("C:/Users/LUMI/Desktop/sentiment")

import pandas as pd
import jieba
from model import TextClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
from sklearn.externals import joblib


def seg_words(contents):
    contents_segs = list()
    for content in contents:
        segs = jieba.lcut(content)
        contents_segs.append(" ".join(segs))

    return contents_segs

# load train data
train_data_df = pd.read_csv('data/train/train.csv')
validate_data_df = pd.read_csv('data/validation/validation.csv')

content_train = train_data_df.iloc[:, 1]
content_train = seg_words(content_train)

columns = train_data_df.columns.values.tolist()

vectorizer_tfidf = TfidfVectorizer(analyzer='word', ngram_range=(1, 5), min_df=5, norm='l2')
vectorizer_tfidf.fit(content_train)

# model train
classifier_dict = dict()
for column in columns[2:]:
    label_train = train_data_df[column]
    text_classifier = TextClassifier(vectorizer=vectorizer_tfidf)
    text_classifier.fit(content_train, label_train)
    classifier_dict[column] = text_classifier


# validate model
content_validate = validate_data_df.iloc[:, 1]

content_validate = seg_words(content_validate)


f1_score_dict = dict()
for column in columns[2:]:
    label_validate = validate_data_df[column]
    text_classifier = classifier_dict[column]
    f1_score = text_classifier.get_f1_score(content_validate, label_validate)
    f1_score_dict[column] = f1_score

f1_score = np.mean(list(f1_score_dict.values()))
str_score = "\n"
for column in columns[2:]:
    str_score = str_score + column + ":" + str(f1_score_dict[column]) + "\n"

# save model
joblib.dump(classifier_dict, model_save_path + model_name)



================================================
FILE: aspect_predict.py
================================================
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

import os
from sklearn.externals import joblib

from utils.utils import delimiter
from utils.data_process import seg_words,load_aspect_list

class AspectPredict(object):
    def __init__(self):
        path_delimiter = delimiter()
        path_absa = os.path.abspath('.')

        # config
        model_name = 'aspect_svc' # todo: add to config
        path_config = path_absa + path_delimiter + 'config.json'

        # load model
        path_model = path_absa + path_delimiter + 'model' + path_delimiter + '{}.mdl'.format(model_name)
        self.model = joblib.load(path_model)

        # load aspect list
        self.aspect_list = load_aspect_list(path_config)


    def predict(self, text):
        # 1. generate result
        result = dict()
        result['text'] = text
        result['aspectCategory'] = []

        # 2. seg words
        content_test = seg_words([text])

        # 3. predict
        all_result = dict()
        for column in self.aspect_list:
            all_result[column] = self.model[column].predict(content_test)[0]
            if all_result[column]>0.5:
                result['aspectCategory'].append(column)
        result['all_result'] = all_result

        print('PREDICT RESULT:',result)
        print('PREDICT ASPECT:', result['aspectCategory'])
        return result

if __name__=="__main__":
    aspect = AspectPredict()
    aspect.predict('这块屏幕不错')

================================================
FILE: config.json
================================================
{"aspect_list": ["HARDWARE#USABILITY", "BATTERY#USABILITY", "HARDWARE#QUALITY", "MEMORY#GENERAL", "OS#PRICE", "MULTIMEDIA_DEVICES#QUALITY", "MULTIMEDIA_DEVICES#OPERATION_PERFORMANCE", "PORTS#DESIGN_FEATURES", "MULTIMEDIA_DEVICES#USABILITY", "OS#GENERAL", "SUPPORT#MISCELLANEOUS", "KEYBOARD#GENERAL", "POWER_SUPPLY#OPERATION_PERFORMANCE", "PHONE#QUALITY", "MEMORY#DESIGN_FEATURES", "CPU#USABILITY", "OS#CONNECTIVITY", "SOFTWARE#MISCELLANEOUS", "CPU#OPERATION_PERFORMANCE", "KEYBOARD#USABILITY", "PORTS#USABILITY", "KEYBOARD#QUALITY", "HARD_DISC#QUALITY", "MULTIMEDIA_DEVICES#CONNECTIVITY", "SOFTWARE#OPERATION_PERFORMANCE", "MEMORY#USABILITY", "PHONE#CONNECTIVITY", "DISPLAY#OPERATION_PERFORMANCE", "PHONE#DESIGN_FEATURES", "KEYBOARD#OPERATION_PERFORMANCE", "HARDWARE#OPERATION_PERFORMANCE", "POWER_SUPPLY#CONNECTIVITY", "PHONE#USABILITY", "OS#QUALITY", "BATTERY#OPERATION_PERFORMANCE", "HARDWARE#CONNECTIVITY", "POWER_SUPPLY#QUALITY", "HARD_DISC#OPERATION_PERFORMANCE", "SUPPORT#QUALITY", "PHONE#OPERATION_PERFORMANCE", "CPU#GENERAL", "SUPPORT#USABILITY", "DISPLAY#QUALITY", "OS#DESIGN_FEATURES", "POWER_SUPPLY#USABILITY", "HARDWARE#DESIGN_FEATURES", "CPU#QUALITY", "PHONE#MISCELLANEOUS", "SOFTWARE#QUALITY", "OS#OPERATION_PERFORMANCE", "WARRANTY#OPERATION_PERFORMANCE", "PHONE#GENERAL", "PHONE#PRICE", "MULTIMEDIA_DEVICES#GENERAL", "PORTS#OPERATION_PERFORMANCE", "POWER_SUPPLY#GENERAL", "KEYBOARD#DESIGN_FEATURES", "MEMORY#QUALITY", "SOFTWARE#USABILITY", "DISPLAY#DESIGN_FEATURES", "BATTERY#QUALITY", "PORTS#CONNECTIVITY", "PORTS#QUALITY", "HARDWARE#GENERAL", "OS#USABILITY", "SOFTWARE#GENERAL", "DISPLAY#USABILITY", "DISPLAY#GENERAL", "MULTIMEDIA_DEVICES#DESIGN_FEATURES", "BATTERY#DESIGN_FEATURES", "OTHERS", "SOFTWARE#CONNECTIVITY", "SOFTWARE#DESIGN_FEATURES"]}

================================================
FILE: data/polarity/polarity_docu.xlsx
================================================
[File too large to display: 20.9 MB]

================================================
FILE: polarity_predict.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

import os
from sklearn.externals import joblib

from utils.utils import delimiter
from utils.grammar import chinese_only
from utils.data_process import seg_words, gen_text_vec

class PolarityClassifier(object):
    """
    text classification
    """
    def __init__(self):
        # config
        model_name = 'polarity_doc' # doc-based

        # path
        path_delimiter = delimiter()
        if 'absa' in os.path.abspath('.').split(path_delimiter):
            path_absa = os.path.abspath('.')
        else:
            # 被调用路径=path_comment
            path_absa = os.path.abspath('.') + path_delimiter + 'train' \
                        + path_delimiter + 'sentiment' + path_delimiter + 'absa'

        # model path
        path_model_dir = path_absa + path_delimiter + 'model'

        # load tokenizer
        path_tokenizer = path_model_dir + path_delimiter + '{}.tk'.format(model_name)
        self.tokenizer = joblib.load(path_tokenizer)

        # load model
        path_model = path_model_dir + path_delimiter + '{}.mdl'.format(model_name)
        self.model = joblib.load(path_model)
        self.model._make_predict_function()

    def predict(self, comment):

        # 1. chinese only
        cmt = chinese_only([comment])

        # 2. jieba token
        cmt = seg_words(cmt)[0]

        # 3. gen word vector
        _cmt = gen_text_vec(self.tokenizer, cmt, maxlen = 200)

        # token observation
        # split_tokens = []
        # for token in str(_cmt).split(" "):
        #     if token.isdigit():
        #         split_tokens.append(token)
        # print("len(split_tokens):{}".format(len(split_tokens)))

        # 4. predict
        neg_prob = self.model.predict(_cmt)[0][0]
        # neg_prob = (neg_prob > 0.5)

        # 5. json result output
        result = {'items':[{'negative_prob': 0,'sentiment': 0}], 'log_id': '', 'text': ''}
        result['items'][0]['negative_prob'] = neg_prob
        result['items'][0]['sentiment'] = int(round(neg_prob))  # 1表示差评；0表示好评
        result['text'] = comment
        print("SENTIMENT RESULT: ",result)
        return result


if __name__=="__main__":
    t = PolarityClassifier()
    t.predict('这块电池好看')


================================================
FILE: train/aspect_classifier.py
================================================
# -*- coding: utf-8 -*-
__author__ = 'ZhangYi'

import os
import ast
import json
import pandas as pd
import numpy as np
from sklearn.externals import joblib
from sklearn.feature_extraction.text import TfidfVectorizer

from train.model.model import TextClassifier
from utils.utils import delimiter
from utils.data_process import nan_to_others,category_transpose,seg_words,load_aspect_list

class AspectClassifier(object):
    """
    Aspect(=EA) Classifier Train Part
    """
    def __init__(self):
        path_delimiter = delimiter()
        path_absa = os.path.abspath('..')

        # config
        task_tag = 'aspect_'
        model_name = task_tag + 'svc'

        # config path
        self.path_config = path_absa + path_delimiter + 'config.json'

        # model path
        self.model_path = path_absa + path_delimiter + 'model' + path_delimiter + '{}.mdl'.format(model_name)

        # data path
        self.path_data = path_absa + path_delimiter +'data'
        self.path_data_ch = path_absa + path_delimiter +'data' + path_delimiter + 'chinese' + path_delimiter
        self.path_train_df = self.path_data + path_delimiter + 'aspect' + path_delimiter + '{}_train.xlsx'.format(model_name)
        self.path_test_df = self.path_data + path_delimiter + 'aspect' + path_delimiter + '{}_test.xlsx'.format(model_name)

    def data_process(self):
        if os.path.isfile(self.path_train_df) \
                and os.path.isfile(self.path_test_df) \
                and os.path.isfile(self.path_config):

            train_df = pd.read_excel(self.path_train_df)
            test_df = pd.read_excel(self.path_test_df)
            self.category_list = load_aspect_list(self.path_config)

        else:
            # 1. load data
            train = pd.read_excel(self.path_data_ch+'Chinese_phones_training.xlsx')
            test = pd.read_excel(self.path_data_ch+'CH_PHNS_SB1_TEST.xlsx')

            # 2. mark NaN as 'OTHERS'
            _data = []
            for data in [train, test]:
                df = nan_to_others(data)
                _data.append(df)

            # 3. generate category list
            self.category_list = list(set(_data[0]['category']))  # len = 73

            # 4. save category list to config
            cate_dict = {'aspect_list':self.category_list}
            with open(self.path_config, "w") as f:
                f.write(json.dumps(cate_dict))
            f.close()

            # 5. generate df by category transpose
            all_data = []
            for d in _data:
                df = category_transpose(d, self.category_list)
                all_data.append(df)

            # 6. save data
            train_df, test_df = all_data[0], all_data[1]
            train_df.to_excel(self.path_train_df, index=False)
            test_df.to_excel(self.path_test_df, index=False)

        return train_df, test_df

    def train(self, train_df):
        content_train = seg_words(train_df['text'])
        vectorizer_tfidf = TfidfVectorizer(analyzer='word', ngram_range=(1, 5), min_df=5, norm='l2')
        vectorizer_tfidf.fit(content_train)

        # model train
        classifier_dict = dict()
        for column in self.category_list:
            print(column)
            label_train = train_df[column]
            text_classifier = TextClassifier(vectorizer=vectorizer_tfidf)
            text_classifier.fit(content_train, label_train)
            classifier_dict[column] = text_classifier

        # save model
        if os.path.isfile(self.model_path):
            pass
        else:
            joblib.dump(classifier_dict, self.model_path)

    def test(self, test_df):
        classifier = joblib.load(self.model_path)
        content_test = seg_words(test_df['text'])

        f1_score_dict = dict()
        for column in self.category_list:
            label_validate = test_df[column]
            text_classifier = classifier[column]
            f1_score = text_classifier.get_f1_score(content_test, label_validate)
            f1_score_dict[column] = f1_score

        f1_score = np.mean(list(f1_score_dict.values()))
        print('F1-SCORE-DICT: ', f1_score_dict)
        print('MEAN OF F1-SCORE-DICT: ', f1_score)

        return f1_score_dict


if __name__=="__main__":
    aspect = AspectClassifier()
    train_df, test_df = aspect.data_process()

    aspect.train(train_df)
    aspect.test(test_df)

================================================
FILE: train/model/bilstm.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

from sklearn.metrics import accuracy_score, f1_score, confusion_matrix
from keras.models import Sequential
from keras.layers import Dense, LSTM, Embedding, Dropout,Bidirectional, GlobalMaxPool1D


class BiLSTM():
    def __init__(self, max_features, embed_size):
        model = Sequential()
        model.add(Embedding(max_features, embed_size))
        model.add(Bidirectional(LSTM(32, return_sequences=True)))
        model.add(GlobalMaxPool1D())
        model.add(Dense(20, activation="relu"))
        model.add(Dropout(0.05))
        model.add(Dense(1, activation="sigmoid"))
        model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

        self.classifier = model

    def fit(self, x, y, batch_size, epochs, validation_split):
        self.classifier.fit(x,y, batch_size=batch_size, epochs=epochs, validation_split=0.2)

    def predict(self, x):
        return self.classifier.predict(x)

    def evaluate(self, y_true, y_pred):
        acc = accuracy_score(y_pred, y_true)
        f1 = f1_score(y_pred, y_true)
        cfs_matrix = confusion_matrix(y_pred, y_true)
        print('Accuracy Score:', acc)
        print('F1-score: {0}'.format(f1))
        print('Confusion matrix:\n', cfs_matrix)

        return acc, f1, cfs_matrix

    def _make_predict_function(self):
        self.classifier._make_predict_function()

================================================
FILE: train/model/model.py
================================================
#!/user/bin/env python
# -*- coding:utf-8 -*-
__author__ = 'ZhangYi'

from sklearn.svm import SVC
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import f1_score
import logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s [%(levelname)s] <%(processName)s> (%(threadName)s) %(message)s')
logger = logging.getLogger(__name__)


class TextClassifier():

    def __init__(self, vectorizer, classifier=MultinomialNB()):
        classifier = SVC(kernel="rbf")
        # classifier = SVC(kernel="linear")
        self.classifier = classifier
        self.vectorizer = vectorizer

    def features(self, x):
        return self.vectorizer.transform(x)

    def fit(self, x, y):

        self.classifier.fit(self.features(x), y)

    def predict(self, x):

        return self.classifier.predict(self.features(x))

    def score(self, x, y):
        return self.classifier.score(self.features(x), y)

    def get_f1_score(self, x, y):
        return f1_score(y, self.predict(x), average='macro')





================================================
FILE: train/polarity_classifier.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

import os
import pandas as pd
from sklearn.externals import joblib
from sklearn.model_selection import train_test_split
from keras.preprocessing.text import Tokenizer


from train.model.bilstm import BiLSTM
from utils.utils import delimiter
from utils.grammar import chinese_only
from utils.data_process import merge_excel,seg_words,remove_empty_row,gen_text_vec

class PolarityClassifier(object):
    """
    train sentiment model and generate model file
    """
    def __init__(self):
        path_delimiter = delimiter()
        path_absa = os.path.abspath('..')

        # config
        self.maxlen = 200          # doc word length
        task_tag = 'polarity_'
        model_name = task_tag + 'docu'

        # model path
        path_model = path_absa + path_delimiter + 'model'
        self.model_path = path_model + path_delimiter + '{}.mdl'.format(model_name)
        self.path_tokenizer = path_model + path_delimiter + '{}.tk'.format(model_name)

        # data path
        path_data_doc_level = path_delimiter.join(path_absa.split(path_delimiter)[:-2]) + path_delimiter + "data" \
                              + path_delimiter + 'sentiment' + path_delimiter + 'document_level'
        self.path_train_data = path_data_doc_level + path_delimiter + 'train_data'

        self.path_data = path_absa + path_delimiter + 'data'
        self.path_corpus = self.path_data + path_delimiter + 'polarity' + path_delimiter + '{}.xlsx'.format(model_name)

        # generate tokenizer
        self.data = self.data_process()
        self.tokenizer = self.gen_tokenizer(self.data['cmt_split'])

    def data_process(self):
        if os.path.isfile(self.path_corpus):
            data = pd.read_excel(self.path_corpus)

        else:
            # 1. merge data
            data = merge_excel(self.path_train_data)

            # 2. Chinese character only
            data['cmt_zh'] = chinese_only(data['comment_content'])

            # 3. jieba token for dictionary
            data['cmt_split'] = seg_words(data['cmt_zh'])

            # 4. remove empty comment
            data = remove_empty_row(data, 'cmt_split')

            # 5. save data
            data.to_excel(self.path_corpus)
        return data

    def gen_tokenizer(self, cut_corpus_list):
        if os.path.isfile(self.path_tokenizer):
            tokenizer = joblib.load(self.path_tokenizer)
        else:
            tokenizer = Tokenizer()
            tokenizer.fit_on_texts(cut_corpus_list.astype(str))
            joblib.dump(tokenizer, self.path_tokenizer)
        return tokenizer

    def gen_train_test(self, x, y):
        X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=1)
        return X_train, X_test, y_train, y_test

    def train(self, X_train, y_train):
        embed_size = 256
        max_features = 66000  # dictionary size
        classifier = BiLSTM(max_features, embed_size)

        epochs = 2
        batch_size = 100
        X_tr = gen_text_vec(self.tokenizer, X_train, self.maxlen)
        classifier.fit(X_tr, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.2)

        # save model
        if os.path.isfile(self.model_path):
            print('model exist already')
        else:
            joblib.dump(classifier, self.model_path)

    def test(self, X_test, y_test):
        X_te = gen_text_vec(self.tokenizer, X_test, self.maxlen)

        # load model
        model = joblib.load(self.model_path)
        pred_prob = model.predict(X_te)

        # pred = (pred_prob > 0.65)
        pred = [int(round(i[0])) for i in pred_prob]
        y_test = [int(i) for i in y_test]

        # evaluate
        eval = model.evaluate(y_test, pred)
        return eval


    def batch_predict(self, batch_cmt_df):

        # 1. chinese only
        batch_cmt_df['cmt_zh'] = chinese_only(batch_cmt_df['comment_content'])

        # 2. token cut
        batch_cmt_df['cmt_split'] = seg_words(batch_cmt_df['cmt_zh'])

        # 暂时没有remove empty环节

        # 3. predict
        self.test(batch_cmt_df['cmt_split'],batch_cmt_df['label'])

        # # save result
        # result = pd.DataFrame(np.array([self.X_test,self.y_test,pred]).T,columns=['comment_zh','GroundTruth','bilstm'])
        # result.to_excel('data/sentiment/result_.xlsx')


if __name__=="__main__":
    pc = PolarityClassifier()
    data = pc.data
    X_train, X_test, y_train, y_test = pc.gen_train_test(data['cmt_split'], data['label'])
    pc.train(X_train, y_train)
    pc.test(X_test, y_test)

================================================
FILE: utils/__init__.py
================================================


================================================
FILE: utils/baidu_tagging.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from  aip  import  AipNlp
import  pandas  as  pd
import  time

"""  你的  APPID  AK  SK  """
APP_ID  =  '155934'
API_KEY  =  'PBW2w1dveS7x3YcKSZW0V7'
SECRET_KEY  =  'AOE75EWZqeI6kM7Kesq8i6FzQruDI'
client  =  AipNlp(APP_ID,  API_KEY,  SECRET_KEY)

# 请求文件
source_file  =  "请求文件路径"
source_df  =  pd.read_excel(source_file)


comments  =  []
neg_probs  =  []
pos_probs  =  []
confidences  =  []
sentiments  =  []
complete_count  =  0
#  请求错误统计
err_count  =  0
err_comment  =  []
start_time  =  time.time()
#  循环请求
i  =  0
while  i  <  len(source_df):
        comment  =  source_df["comment_content"][i]
        try:
                query_result  =  client.sentimentClassify(comment[:1024])
        except  Exception  as  e:
                print("query_result:{}".format(query_result))
                print("#######请求过程存在问题#######")
                err_count  +=  1
                err_comment.append(comment)
                i  +=  1
                continue
        try:
                result  =  query_result['items'][0]
                neg_prob  =  result['negative_prob']
                pos_prob  =  result['positive_prob']
                confidence  =  result['confidence']
                sentiment  =  result['sentiment']
        except  KeyError  as  e:
                print("#######请求QPS限制#######")
                print("i={}".format(i))
                continue
        i  +=  1
        comments.append(comment)
        neg_probs.append(neg_prob)
        pos_probs.append(pos_prob)
        confidences.append(confidence)
        sentiments.append(sentiment)
        complete_count  +=  1
        print("总共：{}条".format(len(source_df)))
        print("请求完成:  {}条".format(complete_count))
        print("完成进度：{}%".format(round(complete_count  /  len(source_df)  *  100,  2)))
        cost_mins  =  (time.time()  -  start_time)  /  60
        print("累计用时：{}分钟".format(round(cost_mins,  2)))
        avg_query_time  =  complete_count  /  cost_mins
        #  print("每条请求平均用时：{}".format(avg_query_time))
        left_mins  =  (len(source_df)  -  complete_count  -  err_count)  /  avg_query_time
        print("预计还需：{}分钟".format(round(left_mins,  2)))
        print("\n")

print("所有请求完成！")
print("请求总数量：{}".format(len(source_df)))
print("请求过程中存在问题的数量：{}".format(err_count))



#  保存结果

# 请求成功的结果保存
desti_df  =  pd.DataFrame()
desti_df["comment"]  =  comments
desti_df["neg_probs"]  =  neg_probs
desti_df["pos_probs"]  =  pos_probs
desti_df["confidences"]  =  confidences
desti_df["sentiments"]  =  sentiments
desti_file  =  "请求结果保存路径"
desti_df.to_excel(desti_file, engine='xlsxwriter')

# 请求失败的结果保存
err_df  =  pd.DataFrame()
err_file  =  "请求结果报错保存路径"
err_df["comment"] = err_comment
err_df.to_excel(err_file, engine='xlsxwriter') # 如果请求接口里有奇怪字符，保存文件时就使用, engine='xlsxwriter'

================================================
FILE: utils/data_process.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

import ast
import jieba
import itertools
import pandas as pd
import numpy as np
from keras.preprocessing.sequence import pad_sequences

# mark NaN as 'OTHERS'
def nan_to_others(df):
    new_cate = []
    new_polarity = []

    # dataframe必须含有列：['text', 'category', 'polarity']
    for idx, i in enumerate(df['polarity']):
        if i in ['negative', 'positive', 'neutral', 'conflict']:
            new_cate.append(df['category'][idx])
            new_polarity.append(i)
        else:
            new_cate.append('OTHERS')
            new_polarity.append('OTHERS')
    _df = pd.DataFrame(np.array([df['text'], new_cate, new_polarity]).T, columns=['text', 'category', 'polarity'])
    return _df

# tokenize
def seg_words(contents):
    contents_segs = list()
    for content in contents:
        segs = jieba.lcut(content)
        contents_segs.append(" ".join(segs))
    return contents_segs

# get text vector
def gen_text_vec(tokenizer, cut_corpus_list, maxlen):
    text_vec = tokenizer.texts_to_sequences(cut_corpus_list)
    t_vec = pad_sequences(text_vec, maxlen=maxlen)
    return t_vec

# category transpose
def category_transpose(df, category_list):
    for i in category_list:
        l_ist = []
        # dataframe必须含有列：['category']
        for cate in df['category']:
            if cate == i:
                l_ist.append(1)
            else:
                l_ist.append(0)
        df[i] = l_ist
    return df

# load config: aspect_list
def load_aspect_list(path_config):
    # only one param in config: aspect_list
    a = 0
    with open(path_config, "r", encoding='utf-8') as f:
        for i in f:
            category_list = ast.literal_eval(i)['aspect_list']
            a = a + 1
            if a == 1:
                break
    f.close()
    return category_list

# merge excel
def merge_excel(path_data_dir):
    cmt_l = []
    scr_l = []

    # 被merge的df都必须有['comment_content', 'label']
    data_source = ['/2019-04-12_lock_comment_jd_spider_baidu_sentiment.xlsx', \
                   '/20190329_train_lock_comments_document_level_with_label.xls', \
                   '/all_comments_document_level_without_lock_comments.xls', \
                   '/bad_comments_in_forum_mi.com_youpin.xls']
    for i in data_source:
        path_data = path_data_dir + i
        _data = pd.read_excel(path_data)

        cmt_l.append(_data['comment_content'])
        scr_l.append(_data['label'])

    comment = list(itertools.chain.from_iterable(cmt_l))
    score = list(itertools.chain.from_iterable(scr_l))

    data = pd.DataFrame(np.array([comment, score]).T, columns=['comment_content', 'label'])
    return data

# remove row by column with empty value
def remove_empty_row(df, column_name):
    row_to_delete = []
    for idx, i in enumerate(df[column_name]):
        if not bool(i):
            row_to_delete.append(idx)
    df = df.drop(df.index[row_to_delete])
    return df.reset_index(drop=True)

================================================
FILE: utils/grammar.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

import re

def chinese_only(txt_list):
    cmt_zh = []
    for cmt in txt_list:
        line = cmt.strip()
        p2 = re.compile(u'[^\u4e00-\u9fa5]')
        zh = " ".join(p2.split(line)).strip()
        cmt_zh.append(",".join(zh.split()))
    return cmt_zh

================================================
FILE: utils/utils.py
================================================
# -*- coding: utf-8 -*-

__author__ = 'ZhangYi'

import sys

def delimiter():
    path_delimiter = '/'
    if 'win' in sys.platform:
        path_delimiter = '\\'
    return path_delimiter