Full Code of zhongbiaodev/py-mysql-elasticsearch-sync for AI

Repository: zhongbiaodev/py-mysql-elasticsearch-sync
Branch: master
Commit: a2a076e00a73
Files: 10
Total size: 40.5 KB

Directory structure:
gitextract_re_gm3op/

├── .gitignore
├── LICENSE
├── MANIFEST.in
├── README.md
├── README_CN.md
├── es_sync/
│   └── __init__.py
├── requirements.txt
├── setup.py
├── src/
│   └── __init__.py
└── upstart.conf

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
docs/_build/

# PyBuilder
target/

*.yaml

# virtualenv
venv/
ENV/

# Pycharm settings
*.idea/

================================================
FILE: LICENSE
================================================
The MIT License (MIT)

Copyright (c) by Windfarer

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



================================================
FILE: MANIFEST.in
================================================
include README.md LICENSE
recursive-include src *.yaml

================================================
FILE: README.md
================================================
# py-mysql-elasticsearch-sync
Simple and fast MySQL to Elasticsearch sync tool, written in Python.

[中文文档](https://github.com/zhongbiaodev/py-mysql-elasticsearch-sync/blob/master/README_CN.md)

## Introduction
This tool helps you to initialize MySQL dump table to Elasticsearch by parsing mysqldump, then incremental sync MySQL table to Elasticsearch by processing MySQL Binlog.
Also, during the binlog syncing, this tool will save the binlog sync position, so that it is easy to recover after this tool being shutdown for any reason.

## Installation
By following these steps.

##### 1. ibxml2 and libxslt
This tool depends on python lxml package, so that you should install  the lxml's dependecies correctly, the libxml2 and libxslt are required.

For example, in CentOS:

```
sudo yum install libxml2 libxml2-devel libxslt libxslt-devel
```

Or in Debian/Ubuntu:

```
sudo apt-get install libxml2-dev libxslt-dev python-dev
```

See [lxml Installation](http://lxml.de/installation.html) for more infomation.
##### 2. mysqldump
And then, mysqldump is required in the machine where this tool will be run on it.(and the mysql server must enable binlog)


##### 3. this tool
Then install this tool

```
pip install py-mysql-elasticsearch-sync
```

## Configuration
There is a [sample config](https://github.com/zhongbiaodev/py-mysql-elasticsearch-sync/blob/master/es_sync/sample.yaml) file in repo, you can start by editing it.

## Running
Simply run command

```
es-sync path/to/your/config.yaml
```
and the tool will dump your data as stream to sync, when dump is over, it will start to sync binlog.

The latest synced binlog file and position are recorded in your info file which is configured in your config file. You can restart dump step by remove it, or you can change sync position by edit it.

Or if you  but want to load it from your own dumpfile. You should dump your table first as xml format(by adding ```-X```option to your mysqldump command) 

then

```
es-sync path/to/your/config.yaml --fromfile
```
to start sync, when xml sync is over, it will also start binlog sync.

## Deployment
We provide an [upstart script]((https://github.com/zhongbiaodev/py-mysql-elasticsearch-sync/blob/master/upstart.conf)) to help you deploy this tool, you can edit it for your own condition, besides, you can deploy it in your own way.

## MultiTable Supporting
Now Multi-table is supported through setting tables in config file, the first table is master as default and the others are slave.

Master table and slave tables must use the same primary key, which is defined via _id.

Table has higher priority than tables.

## TODO
- [ ] MultiIndex Supporting


================================================
FILE: README_CN.md
================================================
# py-mysql-elasticsearch-sync
一个从MySQL向Elasticsearch同步数据的工具，使用Python实现。

## 简介
在第一次初始化数据时，本工具解析mysqldump导出的数据，并导入ES中，在后续增量更新中，解析binlog的数据，对ES中的数据进行同步。在binlog同步阶段，支持断点恢复，因此无需担心意外中断的问题。

## 安装

##### 1. ibxml2 和 libxslt
本工具基于lxml库，因此需要安装它的依赖的libxml2和libxslt

在CentOS中:

```
sudo yum install libxml2 libxml2-devel libxslt libxslt-devel
```

在Debian/Ubuntu中:

```
sudo apt-get install libxml2-dev libxslt-dev python-dev
```

查看[lxml Installation](http://lxml.de/installation.html)来获取更多相关信息

##### 2. mysqldump
在运行本工具的机器上需要有mysqldump，并且mysql服务器需要开启binlog功能。


##### 3. 本工具
安装本工具

```
pip install py-mysql-elasticsearch-sync
```

## 配置
你可以通过修改[配置文件示例](https://github.com/zhongbiaodev/py-mysql-elasticsearch-sync/blob/master/es_sync/sample.yaml)来编写自己的配置文件

## 运行
运行命令

```
es-sync path/to/your/config.yaml
```
工具将开始执行mysqldump并解析流进行同步，当dump结束后，将启动binlog同步

最近一次binlog同步位置记录在一个文件中，这个文件的路径在config文件中配置过。

你可以删除记录文件来从头进行binlog同步，或者修改文件里的内容，来从特定位置开始同步。


你也可以把自己从mysql导出的xml文件同步进ES中(在mysqldump的命令中加上参数```-X```即可导出xml) 

然后执行

```
es-sync path/to/your/config.yaml --fromfile
```
启动从xml导入，当从xml导入完毕后，它会开始同步binlog

## 服务管理
我们写了一个[upstart脚本](https://github.com/zhongbiaodev/py-mysql-elasticsearch-sync/blob/master/upstart.conf)来管理本工具的运行，你也可以用你自己的方式进行部署运行

## 多表支持
你可以在config文件中配置tables以支持多表，默认tables中第一张表为主表，其余表为从表。

主表和从表主键必须相同，均为_id字段。

当同时设置table和tables时，table优先级较高。

## TODO
- [ ] 多索引支持


================================================
FILE: es_sync/__init__.py
================================================
from __future__ import print_function, unicode_literals
from future.builtins import str, range
import sys

PY2 = sys.version_info[0] == 2

if PY2:
    import os
    DEVNULL = open(os.devnull, 'wb')
else:
    from subprocess import DEVNULL


def encode_in_py2(s):
    if PY2:
        return s.encode('utf-8')
    return s

import os.path
import yaml
import signal
import requests
import subprocess
import json
import logging
import shlex
import datetime
import decimal
from lxml.etree import iterparse
from functools import reduce
from pymysqlreplication import BinLogStreamReader
from pymysqlreplication.row_event import DeleteRowsEvent, UpdateRowsEvent, WriteRowsEvent
from pymysqlreplication.event import RotateEvent, XidEvent

__version__ = '0.4.2'


# The magic spell for removing invalid characters in xml stream.
REMOVE_INVALID_PIPE = r'tr -d "\00\01\02\03\04\05\06\07\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37"'

DEFAULT_BULKSIZE = 100
DEFAULT_BINLOG_BULKSIZE = 1


class ElasticSync(object):
    table_structure = {}
    log_file = None
    log_pos = None

    @property
    def is_binlog_sync(self):
        rv = bool(self.log_file and self.log_pos)
        return rv

    def __init__(self):
        try:
            self.config = yaml.load(open(sys.argv[1]))
        except IndexError:
            print('Error: not specify config file')
            exit(1)

        mysql = self.config.get('mysql')
        if mysql.get('table'):
            self.tables = [mysql.get('table')]
            self.dump_cmd = 'mysqldump -h {host} -P {port} -u {user} --password={password} {db} {table} ' \
                        '--default-character-set=utf8 -X --opt --quick'.format(**mysql)
        elif mysql.get('tables'):
            self.tables = mysql.get('tables')
            mysql.update({
                'tables': ' '.join(mysql.get('tables'))
            })
            self.dump_cmd = 'mysqldump -h {host} -P {port} -u {user} --password={password} --database {db} --tables {tables} ' \
                        '--default-character-set=utf8 -X --opt --quick'.format(**mysql)
        else:
            print('Error: must specify either table or tables')
            exit(1)
        self.master = self.tables[0]  # use the first table as master
        self.current_table = None

        self.binlog_conf = dict(
            [(key, self.config['mysql'][key]) for key in ['host', 'port', 'user', 'password', 'db']]
        )

        self.endpoint = 'http://{host}:{port}/{index}/{type}/_bulk'.format(
            host=self.config['elastic']['host'],
            port=self.config['elastic']['port'],
            index=self.config['elastic']['index'],
            type=self.config['elastic']['type']
        )  # todo: supporting multi-index

        self.mapping = self.config.get('mapping') or {}
        if self.mapping.get('_id'):
            self.id_key = self.mapping.pop('_id')
        else:
            self.id_key = None

        self.ignoring = self.config.get('ignoring') or []

        record_path = self.config['binlog_sync']['record_file']
        if os.path.isfile(record_path):
            with open(record_path, 'r') as f:
                record = yaml.load(f)
                self.log_file = record.get('log_file')
                self.log_pos = record.get('log_pos')

        self.bulk_size = self.config.get('elastic').get('bulk_size') or DEFAULT_BULKSIZE
        self.binlog_bulk_size = self.config.get('elastic').get('binlog_bulk_size') or DEFAULT_BINLOG_BULKSIZE

        self._init_logging()
        self._force_commit = False

    def _init_logging(self):
        logging.basicConfig(filename=self.config['logging']['file'],
                            level=logging.INFO,
                            format='[%(levelname)s] - %(filename)s[line:%(lineno)d] - %(asctime)s %(message)s')
        self.logger = logging.getLogger(__name__)
        logging.getLogger("requests").setLevel(logging.WARNING)  # disable requests info logging

        def cleanup(*args):
            self.logger.info('Received stop signal')
            self.logger.info('Shutdown')
            sys.exit(0)

        signal.signal(signal.SIGINT, cleanup)
        signal.signal(signal.SIGTERM, cleanup)

    def _post_to_es(self, data):
        """
        send post requests to es restful api
        """
        resp = requests.post(self.endpoint, data=data)
        if resp.json().get('errors'):  # a boolean to figure error occurs
            for item in resp.json()['items']:
                if list(item.values())[0].get('error'):
                    logging.error(item)
        else:
            self._save_binlog_record()

    def _bulker(self, bulk_size):
        """
        Example:
            u = bulker()
            u.send(None)  #for generator initialize
            u.send(json_str)  # input json item
            u.send(another_json_str)  # input json item
            ...
            u.send(None) force finish bulk and post
        """
        while True:
            data = ""
            for i in range(bulk_size):
                item = yield
                if item:
                    data = data + item + "\n"
                else:
                    break
                if self._force_commit:
                    break
            # print(data)
            print('-'*10)
            if data:
                self._post_to_es(data)

            self._force_commit = False

    def _updater(self, data):
        """
        encapsulation of bulker
        """
        if self.is_binlog_sync:
            u = self._bulker(bulk_size=self.binlog_bulk_size)
        else:
            u = self._bulker(bulk_size=self.bulk_size)

        u.send(None)  # push the generator to first yield
        for item in data:
            u.send(item)
        u.send(None)  # tell the generator it's the end

    def _json_serializer(self, obj):
        """
        format the object which json not supported
        """
        if isinstance(obj, datetime.datetime) or isinstance(obj, datetime.date):
            return obj.isoformat()
        elif isinstance(obj, decimal.Decimal):
            return str(obj)
        raise TypeError('Type not serializable for obj {obj}'.format(obj=obj))

    def _processor(self, data):
        """
        The action must be one of the following:
        create
            Create a document only if the document does not already exist.
        index
            Create a new document or replace an existing document.
        update
            Do a partial update on a document.
        delete
            Delete a document.
        """
        for item in data:
            if self.id_key:
                action_content = {'_id': item['doc'][self.id_key]}
            else:
                action_content = {}
            for field in self.ignoring:
                try:
                    item['doc'].pop(field)
                except KeyError:
                    pass
            meta = json.dumps({item['action']: action_content})
            if item['action'] == 'index':
                body = json.dumps(item['doc'], default=self._json_serializer)
                rv = meta + '\n' + body
            elif item['action'] == 'update':
                body = json.dumps({'doc': item['doc']}, default=self._json_serializer)
                rv = meta + '\n' + body
            elif item['action'] == 'delete':
                rv = meta + '\n'
            elif item['action'] == 'create':
                body = json.dumps(item['doc'], default=self._json_serializer)
                rv = meta + '\n' + body
            else:
                logging.error('unknown action type in doc')
                raise TypeError('unknown action type in doc')
            yield rv

    def _mapper(self, data):
        """
        mapping old key to new key
        """
        for item in data:
            if self.mapping:
                for k, v in self.mapping.items():
                    try:
                        item['doc'][k] = item['doc'][v]
                        del item['doc'][v]
                    except KeyError:
                        continue
            # print(doc)
            yield item

    def _formatter(self, data):
        """
        format every field from xml, according to parsed table structure
        """
        for item in data:
            for field, serializer in self.table_structure.items():
                if field in item['doc'] and item['doc'][field]:
                    try:
                        item['doc'][field] = serializer(item['doc'][field])
                    except ValueError as e:
                        self.logger.error(
                            "Error occurred during format, ErrorMessage:{msg}, ErrorItem:{item}".format(
                            msg=str(e),
                            item=str(item)))
                        item['doc'][field] = None
                    except TypeError as e:
                        item['doc'][field] = None
            # print(item)
            yield item

    def _binlog_loader(self):
        """
        read row from binlog
        """
        if self.is_binlog_sync:
            resume_stream = True
            logging.info("Resume from binlog_file: {file}  binlog_pos: {pos}".format(file=self.log_file,
                                                                                     pos=self.log_pos))
        else:
            resume_stream = False

        stream = BinLogStreamReader(connection_settings=self.binlog_conf,
                                    server_id=self.config['mysql']['server_id'],
                                    only_events=[DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent, RotateEvent, XidEvent],
                                    only_tables=self.tables,
                                    resume_stream=resume_stream,
                                    blocking=True,
                                    log_file=self.log_file,
                                    log_pos=self.log_pos)
        for binlogevent in stream:
            self.log_file = stream.log_file
            self.log_pos = stream.log_pos

            # RotateEvent to update binlog record when no related table changed
            if isinstance(binlogevent, RotateEvent):
                self._save_binlog_record()
                continue

            if isinstance(binlogevent, XidEvent):  # event_type == 16
                self._force_commit = True
                continue

            for row in binlogevent.rows:
                if isinstance(binlogevent, DeleteRowsEvent):
                    if binlogevent.table == self.master:
                        rv = {
                            'action': 'delete',
                            'doc': row['values']
                        }
                    else:
                        rv = {
                            'action': 'update',
                            'doc': {k: row['values'][k] if self.id_key and self.id_key == k else None for k in row['values']}
                        }
                elif isinstance(binlogevent, UpdateRowsEvent):
                    rv = {
                        'action': 'update',
                        'doc': row['after_values']
                    }
                elif isinstance(binlogevent, WriteRowsEvent):
                    if binlogevent.table == self.master:
                        rv = {
                                'action': 'create',
                                'doc': row['values']
                            }
                    else:
                        rv = {
                                'action': 'update',
                                'doc': row['values']
                            }
                else:
                    logging.error('unknown action type in binlog')
                    raise TypeError('unknown action type in binlog')
                yield rv
                # print(rv)
        stream.close()
        raise IOError('mysql connection closed')

    def _parse_table_structure(self, data):
        """
        parse the table structure
        """
        for item in data.iter():
            if item.tag == 'field':
                field = item.attrib.get('Field')
                type = item.attrib.get('Type')
                if 'int' in type:
                    serializer = int
                elif 'float' in type:
                    serializer = float
                elif 'datetime' in type:
                    if '(' in type:
                        serializer = lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
                    else:
                        serializer = lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
                elif 'char' in type:
                    serializer = str
                elif 'text' in type:
                    serializer = str
                else:
                    serializer = str
                self.table_structure[field] = serializer

    def _parse_and_remove(self, f, path):
        """
        snippet from python cookbook, for parsing large xml file
        """
        path_parts = path.split('/')
        doc = iterparse(f, ('start', 'end'), recover=False, encoding='utf-8', huge_tree=True)
        # Skip the root element
        next(doc)
        tag_stack = []
        elem_stack = []
        for event, elem in doc:
            if event == 'start':
                if elem.tag == 'table_data':
                    self.current_table = elem.attrib['name']
                tag_stack.append(elem.tag)
                elem_stack.append(elem)
            elif event == 'end':
                if tag_stack == ['database', 'table_data']:
                    self.current_table = None
                if tag_stack == path_parts:
                    yield elem
                    elem_stack[-2].remove(elem)
                if tag_stack == ['database', 'table_structure']:
                # dirty hack for getting the tables structure
                    self._parse_table_structure(elem)
                    elem_stack[-2].remove(elem)
                try:
                    tag_stack.pop()
                    elem_stack.pop()
                except IndexError:
                    pass

    def _xml_parser(self, f_obj):
        """
        parse mysqldump XML streaming, convert every item to dict object.
        'database/table_data/row'
        """
        for row in self._parse_and_remove(f_obj, 'database/table_data/row'):
            doc = {}
            for field in row.iter(tag='field'):
                k = field.attrib.get('name')
                v = field.text
                doc[k] = v
            if not self.current_table or self.current_table == self.master:
                yield {'action': 'create', 'doc': doc}
            else:
                yield {'action': 'update', 'doc': doc}

    def _save_binlog_record(self):
        if self.is_binlog_sync:
            with open(self.config['binlog_sync']['record_file'], 'w') as f:
                logging.info("Sync binlog_file: {file}  binlog_pos: {pos}".format(
                    file=self.log_file,
                    pos=self.log_pos)
                )
                yaml.safe_dump({"log_file": self.log_file,
                                "log_pos": self.log_pos},
                               f,
                               default_flow_style=False)

    def _xml_dump_loader(self):
        mysqldump = subprocess.Popen(
            shlex.split(encode_in_py2(self.dump_cmd)),
            stdout=subprocess.PIPE,
            stderr=DEVNULL,
            close_fds=True)

        remove_invalid_pipe = subprocess.Popen(
            shlex.split(encode_in_py2(REMOVE_INVALID_PIPE)),
            stdin=mysqldump.stdout,
            stdout=subprocess.PIPE,
            stderr=DEVNULL,
            close_fds=True)

        return remove_invalid_pipe.stdout

    def _xml_file_loader(self, filename):
        f = open(filename, 'rb')  # bytes required

        remove_invalid_pipe = subprocess.Popen(
            shlex.split(encode_in_py2(REMOVE_INVALID_PIPE)),
            stdin=f,
            stdout=subprocess.PIPE,
            stderr=DEVNULL,
            close_fds=True)
        return remove_invalid_pipe.stdout

    def _send_email(self, title, content):
        """
        send notification email
        """
        if not self.config.get('email'):
            return

        import smtplib
        from email.mime.text import MIMEText

        msg = MIMEText(content)
        msg['Subject'] = title
        msg['From'] = self.config['email']['from']['username']
        msg['To'] = ', '.join(self.config['email']['to'])

        # Send the message via our own SMTP server.
        s = smtplib.SMTP()
        s.connect(self.config['email']['from']['host'])
        s.login(user=self.config['email']['from']['username'],
                password=self.config['email']['from']['password'])
        s.sendmail(msg['From'], msg['To'], msg=msg.as_string())
        s.quit()

    def _sync_from_stream(self):
        logging.info("Start to dump from stream")
        docs = reduce(lambda x, y: y(x), [self._xml_parser, 
                                          self._formatter,
                                          self._mapper, 
                                          self._processor], 
                      self._xml_dump_loader())
        self._updater(docs)
        logging.info("Dump success")

    def _sync_from_file(self):
        logging.info("Start to dump from xml file")
        logging.info("Filename: {}".format(self.config['xml_file']['filename']))
        docs = reduce(lambda x, y: y(x), [self._xml_parser, 
                                          self._formatter, 
                                          self._mapper, 
                                          self._processor],
                      self._xml_file_loader(self.config['xml_file']['filename']))
        self._updater(docs)
        logging.info("Dump success")

    def _sync_from_binlog(self):
        logging.info("Start to sync binlog")
        docs = reduce(lambda x, y: y(x), [self._mapper,
                                          self._processor],
                      self._binlog_loader())
        self._updater(docs)

    def run(self):
        """
        workflow:
        1. sync dump data
        2. sync binlog
        """
        try:
            if not self.is_binlog_sync:
                if len(sys.argv) > 2 and sys.argv[2] == '--fromfile':
                    self._sync_from_file()
                else:
                    self._sync_from_stream()
            self._sync_from_binlog()
        except Exception:
            import traceback
            logging.error(traceback.format_exc())
            self._send_email('es sync error', traceback.format_exc())
            raise


def start():
    instance = ElasticSync()
    instance.run()

if __name__ == '__main__':
    start()


================================================
FILE: requirements.txt
================================================
PyMySQL==0.6.7
mysql-replication>=0.8
requests>=2.9.1
PyYAML>=3.11
lxml>=3.5.0
future>=0.15.2 #for py2 compat


================================================
FILE: setup.py
================================================
from setuptools import setup, find_packages
import es_sync

setup(
    name='py-mysql-elasticsearch-sync',
    version=es_sync.__version__,
    packages=find_packages(),
    url='https://github.com/zhongbiaodev/py-mysql-elasticsearch-sync',
    license='MIT',
    author='Windfarer',
    author_email='windfarer@gmail.com',
    description='MySQL to Elasticsearch sync tool',
    install_requires=[
        'PyMySQL==0.6.7',
        'mysql-replication==0.9',
        'requests==2.9.1',
        'PyYAML==3.11',
        'lxml==3.5.0',
        'future==0.15.2'
    ],
    entry_points={
        'console_scripts': [
            'es-sync=es_sync:start',
        ]
    },
    include_package_data=True
)

================================================
FILE: src/__init__.py
================================================
from __future__ import print_function, unicode_literals
from future.builtins import str, range
import sys
PY2 = sys.version_info[0] == 2

if PY2:
    import os
    DEVNULL = open(os.devnull, 'wb')
else:
    from subprocess import DEVNULL
def encode_in_py2(s):
    if PY2:
        return s.encode('utf-8')
    return s

import os.path
import yaml
import signal
import requests
import subprocess
import json
import logging
import shlex
from datetime import datetime
from lxml.etree import iterparse
from functools import reduce
from pymysqlreplication import BinLogStreamReader
from pymysqlreplication.row_event import DeleteRowsEvent, UpdateRowsEvent, WriteRowsEvent

__version__ = '0.3.3.1'


# The magic spell for removing invalid characters in xml stream.
REMOVE_INVALID_PIPE = r'tr -d "\00\01\02\03\04\05\06\07\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37"'

DEFAULT_BULKSIZE = 100
DEFAULT_BINLOG_BULKSIZE = 1


class ElasticSync(object):
    table_structure = {}
    log_file = None
    log_pos = None

    @property
    def is_binlog_sync(self):
        rv = bool(self.log_file and self.log_pos)
        return rv

    def __init__(self):
        try:
            self.config = yaml.load(open(sys.argv[1]))
        except IndexError:
            print('Error: not specify config file')
            exit(1)

        self.dump_cmd = 'mysqldump -h {host} -P {port} -u {user} --password={password} {db} {table} ' \
                        '--default-character-set=utf8 -X'.format(**self.config['mysql'])

        self.binlog_conf = dict(
            [(key, self.config['mysql'][key]) for key in ['host', 'port', 'user', 'password', 'db']]
        )

        self.endpoint = 'http://{host}:{port}/{index}/{type}/_bulk'.format(
            host=self.config['elastic']['host'],
            port=self.config['elastic']['port'],
            index=self.config['elastic']['index'],
            type=self.config['elastic']['type']
        )  # todo: supporting multi-index

        self.mapping = self.config.get('mapping') or {}
        if self.mapping.get('_id'):
            self.id_key = self.mapping.pop('_id')
        else:
            self.id_key = None

        record_path = self.config['binlog_sync']['record_file']
        if os.path.isfile(record_path):
            with open(record_path, 'r') as f:
                record = yaml.load(f)
                self.log_file = record.get('log_file')
                self.log_pos = record.get('log_pos')

        self.bulk_size = self.config.get('elastic').get('bulk_size') or DEFAULT_BULKSIZE
        self.binlog_bulk_size = self.config.get('elastic').get('binlog_bulk_size') or DEFAULT_BINLOG_BULKSIZE

        self._init_logging()

    def _init_logging(self):
        logging.basicConfig(filename=self.config['logging']['file'],
                            level=logging.INFO,
                            format='[%(levelname)s] %(asctime)s %(message)s')
        self.logger = logging.getLogger(__name__)
        logging.getLogger("requests").setLevel(logging.WARNING)  # disable requests info logging

        def cleanup(*args):
            self.logger.info('Received stop signal')
            self.logger.info('Shutdown')
            sys.exit(0)

        signal.signal(signal.SIGINT, cleanup)
        signal.signal(signal.SIGTERM, cleanup)

    def _post_to_es(self, data):
        """
        send post requests to es restful api
        """
        resp = requests.post(self.endpoint, data=data)
        if resp.json().get('errors'):  # a boolean to figure error occurs
            for item in resp.json()['items']:
                if list(item.values())[0].get('error'):
                    logging.error(item)
        else:
            self._save_binlog_record()

    def _bulker(self, bulk_size):
        """
        Example:
            u = bulker()
            u.send(None)  #for generator initialize
            u.send(json_str)  # input json item
            u.send(another_json_str)  # input json item
            ...
            u.send(None) force finish bulk and post
        """
        while True:
            data = ""
            for i in range(bulk_size):
                item = yield
                if item:
                    data = data + item + "\n"
                else:
                    break
            # print(data)
            print('-'*10)
            if data:
                self._post_to_es(data)

    def _updater(self, data):
        """
        encapsulation of bulker
        """
        if self.is_binlog_sync:
                u = self._bulker(bulk_size=self.binlog_bulk_size)
        else:
                u = self._bulker(bulk_size=self.bulk_size)

        u.send(None)  # push the generator to first yield
        for item in data:
            u.send(item)
        u.send(None)  # tell the generator it's the end

    def _json_serializer(self, obj):
        """
        format the object which json not supported
        """
        if isinstance(obj, datetime):
            return obj.isoformat()
        raise TypeError('Type not serializable')

    def _processor(self, data):
        """
        The action must be one of the following:
        create
            Create a document only if the document does not already exist.
        index
            Create a new document or replace an existing document.
        update
            Do a partial update on a document.
        delete
            Delete a document.
        """
        for item in data:
            if self.id_key:
                action_content = {'_id': item['doc'][self.id_key]}
            else:
                action_content = {}
            meta = json.dumps({item['action']: action_content})
            if item['action'] == 'index':
                body = json.dumps(item['doc'], default=self._json_serializer)
                rv = meta + '\n' + body
            elif item['action'] == 'update':
                body = json.dumps({'doc': item['doc']}, default=self._json_serializer)
                rv = meta + '\n' + body
            elif item['action'] == 'delete':
                rv = meta + '\n'
            elif item['action'] == 'create':
                body = json.dumps(item['doc'], default=self._json_serializer)
                rv = meta + '\n' + body
            else:
                logging.error('unknown action type in doc')
                raise TypeError('unknown action type in doc')
            yield rv

    def _mapper(self, data):
        """
        mapping old key to new key
        """
        for item in data:
            if self.mapping:
                for k, v in self.mapping.items():
                    item['doc'][k] = item['doc'][v]
                    del item['doc'][v]
            # print(doc)
            yield item

    def _formatter(self, data):
        """
        format every field from xml, according to parsed table structure
        """
        for item in data:
            for field, serializer in self.table_structure.items():
                if item['doc'][field]:
                    try:
                        item['doc'][field] = serializer(item['doc'][field])
                    except ValueError as e:
                        self.logger.error("Error occurred during format, ErrorMessage:{msg}, ErrorItem:{item}".format(
                            msg=str(e),
                            item=str(item)))
                        item['doc'][field] = None
            # print(item)
            yield item

    def _binlog_loader(self):
        """
        read row from binlog
        """
        if self.is_binlog_sync:
            resume_stream = True
            logging.info("Resume from binlog_file: {file}  binlog_pos: {pos}".format(file=self.log_file,
                                                                                     pos=self.log_pos))
        else:
            resume_stream = False

        stream = BinLogStreamReader(connection_settings=self.binlog_conf,
                                    server_id=self.config['mysql']['server_id'],
                                    only_events=[DeleteRowsEvent, WriteRowsEvent, UpdateRowsEvent],
                                    only_tables=[self.config['mysql']['table']],
                                    resume_stream=resume_stream,
                                    blocking=True,
                                    log_file=self.log_file,
                                    log_pos=self.log_pos)
        for binlogevent in stream:
            self.log_file = stream.log_file
            self.log_pos = stream.log_pos
            for row in binlogevent.rows:
                if isinstance(binlogevent, DeleteRowsEvent):
                    rv = {
                        'action': 'delete',
                        'doc': row['values']
                    }
                elif isinstance(binlogevent, UpdateRowsEvent):
                    rv = {
                        'action': 'update',
                        'doc': row['after_values']
                    }
                elif isinstance(binlogevent, WriteRowsEvent):
                    rv = {
                        'action': 'index',
                        'doc': row['values']
                    }
                else:
                    logging.error('unknown action type in binlog')
                    raise TypeError('unknown action type in binlog')
                yield rv
                # print(rv)
        stream.close()
        raise IOError('mysql connection closed')

    def _parse_table_structure(self, data):
        """
        parse the table structure
        """
        for item in data.iter():
            if item.tag == 'field':
                field = item.attrib.get('Field')
                type = item.attrib.get('Type')
                if 'int' in type:
                    serializer = int
                elif 'float' in type:
                    serializer = float
                elif 'datetime' in type:
                    if '(' in type:
                        serializer = lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
                    else:
                        serializer = lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
                elif 'char' in type:
                    serializer = str
                elif 'text' in type:
                    serializer = str
                else:
                    serializer = str
                self.table_structure[field] = serializer

    def _parse_and_remove(self, f, path):
        """
        snippet from python cookbook, for parsing large xml file
        """
        path_parts = path.split('/')
        doc = iterparse(f, ('start', 'end'), recover=False, encoding='utf-8', huge_tree=True)
        # Skip the root element
        next(doc)
        tag_stack = []
        elem_stack = []
        for event, elem in doc:
            if event == 'start':
                tag_stack.append(elem.tag)
                elem_stack.append(elem)
            elif event == 'end':
                if tag_stack == path_parts:
                    yield elem
                    elem_stack[-2].remove(elem)
                if tag_stack == ['database', 'table_structure']:  # dirty hack for getting the tables structure
                    self._parse_table_structure(elem)
                    elem_stack[-2].remove(elem)
                try:
                    tag_stack.pop()
                    elem_stack.pop()
                except IndexError:
                    pass

    def _xml_parser(self, f_obj):
        """
        parse mysqldump XML streaming, convert every item to dict object. 'database/table_data/row'
        """
        for row in self._parse_and_remove(f_obj, 'database/table_data/row'):
            doc = {}
            for field in row.iter(tag='field'):
                k = field.attrib.get('name')
                v = field.text
                doc[k] = v
            yield {'action': 'index', 'doc': doc}

    def _save_binlog_record(self):
        if self.is_binlog_sync:
            with open(self.config['binlog_sync']['record_file'], 'w') as f:
                logging.info("Sync binlog_file: {file}  binlog_pos: {pos}".format(
                    file=self.log_file,
                    pos=self.log_pos)
                )
                yaml.safe_dump({"log_file": self.log_file, "log_pos": self.log_pos}, f, default_flow_style=False)

    def _xml_dump_loader(self):
        mysqldump = subprocess.Popen(
            shlex.split(encode_in_py2(self.dump_cmd)),
            stdout=subprocess.PIPE,
            stderr=DEVNULL,
            close_fds=True)

        remove_invalid_pipe = subprocess.Popen(
            shlex.split(encode_in_py2(REMOVE_INVALID_PIPE)),
            stdin=mysqldump.stdout,
            stdout=subprocess.PIPE,
            stderr=DEVNULL,
            close_fds=True)

        return remove_invalid_pipe.stdout

    def _xml_file_loader(self, filename):
        f = open(filename, 'rb')  # bytes required
        return f

    def _send_email(self, title, content):
        """
        send notification email
        """
        if not self.config.get('email'):
            return

        import smtplib
        from email.mime.text import MIMEText

        msg = MIMEText(content)
        msg['Subject'] = title
        msg['From'] = self.config['email']['from']['username']
        msg['To'] = ', '.join(self.config['email']['to'])

        # Send the message via our own SMTP server.
        s = smtplib.SMTP()
        s.connect(self.config['email']['from']['host'])
        s.login(user=self.config['email']['from']['username'],
                password=self.config['email']['from']['password'])
        s.sendmail(msg['From'], msg['To'], msg=msg.as_string())
        s.quit()

    def _sync_from_stream(self):
        logging.info("Start to dump from stream")
        docs = reduce(lambda x, y: y(x), [self._xml_parser, 
                                          self._formatter,
                                          self._mapper, 
                                          self._processor], 
                      self._xml_dump_loader())
        self._updater(docs)
        logging.info("Dump success")

    def _sync_from_file(self):
        logging.info("Start to dump from xml file")
        logging.info("Filename: {}".format(self.config['xml_file']['filename']))
        docs = reduce(lambda x, y: y(x), [self._xml_parser, 
                                          self._formatter, 
                                          self._mapper, 
                                          self._processor],
                      self._xml_file_loader(self.config['xml_file']['filename']))
        self._updater(docs)
        logging.info("Dump success")

    def _sync_from_binlog(self):
        logging.info("Start to sync binlog")
        docs = reduce(lambda x, y: y(x), [self._mapper, self._processor], self._binlog_loader())
        self._updater(docs)

    def run(self):
        """
        workflow:
        1. sync dump data
        2. sync binlog
        """
        try:
            if not self.is_binlog_sync:
                if len(sys.argv) > 2 and sys.argv[2] == '--fromfile':
                    self._sync_from_file()
                else:
                    self._sync_from_stream()
            self._sync_from_binlog()
        except Exception:
            import traceback
            logging.error(traceback.format_exc())
            self._send_email('es sync error', traceback.format_exc())
            raise


def start():
    instance = ElasticSync()
    instance.run()

if __name__ == '__main__':
    start()


================================================
FILE: upstart.conf
================================================
description 'es sync'
start on runlevel [2345]
stop on runlevel [06]

respawn
normal exit 0

chdir <PATH_TO_CONFIG>

script
    es-sync config.yaml
end script