![\"img\"](\"https://user-images.githubusercontent.com/35942268/147061994-f0d5a3ec-2d5f-4d72-af1c-139289547f25.png\")
\n\n一个基于Canal实现ES文档增量更新的高性能轻量框架
\n\n ![](\"https://www.oscs1024.com/platform/badge/WGrape/esupdater.svg?size=small\"/){kind=icon}
\n ![](\"https://img.shields.io/badge/php-7.0+-blue.svg\"){kind=icon}
\n ![\"GitHub](\"https://img.shields.io/github/v/release/wgrape/esupdater\")
![](\"https://img.shields.io/badge/version-2.x-blue.svg\"){kind=icon}
\n ![\"Docker](\"https://img.shields.io/docker/pulls/lvsid/phpkafka\")
![](\"https://app.travis-ci.com/WGrape/esupdater.svg?branch=master\")
\n ![](\"https://img.shields.io/badge/unitest-100%25-yellow.svg\"){kind=icon}
\n ![](\"https://img.shields.io/badge/license-MIT-green.svg\"){kind=icon}
\n ![](\"https://img.shields.io/badge/doc-中文-red.svg\"){kind=icon}
\n
![\"Architecture\"](\"https://user-images.githubusercontent.com/35942268/145793762-a23899d6-c162-4527-ae72-643edc80bb18.png\")
\n\n### 1、轻量级框架\n无论安装使用，还是代码设计，整个框架都非常轻量，优雅的完成数据二次处理和ES增量更新。\n\n### 2、全面容器化\n为解决各种依赖安装的复杂麻烦问题，已实现全面容器化，只需一条命令就可以轻松安装、部署、和维护。\n\n### 3、事件驱动化\n基于框架内部的事件驱动设计，可以轻松地注册不同数据表的变更事件和回调，优雅地实现增量更新。\n\n### 4、插件化扩展\n在不影响框架内部运行的前提下，支持插件化扩展，实现对内部行为的自定义扩展。\n\n### 5、高性能消费\n通过一个```Consumer```进程和多个```Worker```进程的一对多通信模型，最少提高10倍的吞吐量，实现高性能消费。\n\n## 二、快速上手\n> 预计只需要 **3分钟** 即可完成 ！\n\n以下操作中会依赖Docker，所以请先安装并启动它。如果只是试用则强烈建议你全程使用在线Docker网站，按如下步骤安装即可，非常方便。\n\n### 1、获取项目\n通过```git clone```或下载Release包即可获取项目，如果出错请参考[获取过程帮助](doc/HELP.md#12)文档。\n\n```bash\ngit clone https://github.com/WGrape/esupdater\ncd esupdater\n```\n\n### 2、开始安装\n执行```install```目录下的```install.sh```安装脚本时，需要传递如下参数以实现[设置环境变量](./doc/APPLICATION.md#3)。如果出错请参考[安装过程帮助](doc/HELP.md#13)文档。\n\n- ```your_local_ip``` ：本机IP参数，通过```ifconfig```查看，通常为192.168开头，而不是127.0.0.1\n\n```bash\ncd install\nbash install.sh ${your_local_ip}\ncd ..\n```\n\n### 3、运行项目\n安装成功后，执行根目录下的```start.sh```启动脚本即可。如果出错请参考[运行过程帮助](doc/HELP.md#3)文档。\n\n```bash\nbash start.sh\n\n# 查看日志输出\ntail -f /home/log/esupdater/debug.log.20220111\n```\n\n### 4、测试运行\n在另一个窗口进入```kafkaContainer```容器中，按如下操作启动```Kafka生产者```\n\n```bash\ndocker exec -it kafkaContainer /bin/bash\ncd /opt/kafka/\n\n# 启动时可能会出现warn, 忽略即可\n./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic default_topic\n```\n\n![\"img1\"](\"https://user-images.githubusercontent.com/35942268/148804272-b00483a9-3861-4aab-8b2f-aee963784694.png\")
\n\n启动成功后会进入一个生产消息的命令行，发送任意消息后，查看上一步日志中的输出，如果出现如下类似日志则说明服务已经成功运行 ！\n\n![\"img2\"](\"https://user-images.githubusercontent.com/35942268/148806227-25af15b9-5609-4de3-ac13-96fc83c7c99b.png\")
\n\n## 三、业务接入\n如果需要在你的业务中接入此项目，请参考[应用接入文档](./doc/APPLICATION.md)。\n\n## 四、扩展列表\n基于插件化扩展开发，项目提供了一系列开箱即用的扩展。\n\n### 1、AutoGenerateCallback\n一个自动生成```Handler```和```Service```的事件回调模块的扩展。具体使用见[使用介绍](./plugin/autogeneratecallback/README.md)\n\n\n## 五、关于项目\n\n### 1、深入了解\n如果想要深入了解本项目，在 [doc目录](./doc) 下提供了如下丰富完善的项目文档，欢迎阅读。\n\n- [APPLICATION](doc/APPLICATION.md) ：帮助你快速在业务中接入此项目\n- [HOWTOCODE](doc/HOWTOCODE.md) ：更深的了解项目，包括架构设计、底层原理\n- [HELP](doc/HELP.md) ：解决安装和部署过程中问题的帮助手册，包括镜像制作帮助、容器部署帮助等\n\n### 2、参与项目\n项目源码设计简单易懂，如有更好的想法，可参考[如何贡献](doc/CONTRIBUTING.md)文档，期待提出宝贵的 [Pull request](https://github.com/WGrape/esupdater/pulls) 。\n\n如果在了解和使用过程中，有任何疑问，也欢迎提出宝贵的 [Issue](https://github.com/WGrape/esupdater/issues/new) 。\n\n开源不易，如果支持本项目 **欢迎Star ！** 以激励维护和更新的动力。\n\n## 六、贡献列表\n所有对本项目有过重要贡献的用户，会收录在此贡献者列表中。\n\n- 感谢 [sick-cat](https://github.com/sick-cat) 提出的Issue ：[启动配置](https://github.com/WGrape/esupdater/issues/41)\n- 感谢 [onser3](https://github.com/onser3) 提出的Issue ：[自动生成handler和service层](https://github.com/WGrape/esupdater/issues/44)\n"
},
{
"path": "bootstrap.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n// PHP Configuration.\ndate_default_timezone_set('Asia/Shanghai');\n\n// Define path constants.\nconst ROOT_PATH = __DIR__ . '/';\nconst APP_PATH = ROOT_PATH . 'app/';\nconst CONFIG_PATH = ROOT_PATH . 'config/';\nconst FRAMEWORK_PATH = ROOT_PATH . 'framework/';\nconst RUNTIME_PATH = ROOT_PATH . 'runtime/';\nconst VENDOR_PATH = ROOT_PATH . 'vendor/';\n\n// Define file constants.\nconst RUNTIME_ESUPDATER_CONSUMER_PID_FILE = RUNTIME_PATH . 'esupdater-consumer.pid';\nconst RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE = RUNTIME_PATH . 'esupdater-consumer.status';\nconst RUNTIME_IGNORE_ERROR_TEMP_FILE = RUNTIME_PATH . 'ignore-error.temp';\nconst CREATE_WORKER_LOG_FILE = RUNTIME_PATH . 'create-worker.log';\nconst RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX = 'esupdater-worker-';\nconst COMPOSER_AUTOLOAD_FILE = VENDOR_PATH . 'autoload.php';\nconst ENVIRONMENT_FILE = ROOT_PATH . '.env';\n\n// Define data constants.\nconst DEFAULT_PID = 0;\n\n// Load config files.\ninclude_once CONFIG_PATH . 'consumer.php';\ninclude_once CONFIG_PATH . 'db.php';\ninclude_once CONFIG_PATH . 'es.php';\ninclude_once CONFIG_PATH . 'log.php';\ninclude_once CONFIG_PATH . 'event.php';\n\n/**\n * Include composer autoload file or register autoload.\n */\nif (!file_exists(COMPOSER_AUTOLOAD_FILE)) {\n function autoloadCallback(string $classname)\n {\n $classname = str_replace('\\\\', '/', $classname);\n\n $file = ROOT_PATH . \"{$classname}.php\";\n if (file_exists($file)) {\n include_once $file;\n } else {\n echo 'class file' . $classname . 'not found!';\n }\n }\n\n spl_autoload_register(\"autoloadCallback\", true, true);\n} else {\n include_once COMPOSER_AUTOLOAD_FILE;\n}\n\n/**\n * Register shutdown callback.\n */\nfunction shutdownCallback()\n{\n $manager = new \\framework\\Manager();\n\n // Delete the files of consumer process if it exited without delete files.\n if ($manager->isConsumerProcess()) {\n if (file_exists(RUNTIME_ESUPDATER_CONSUMER_PID_FILE)) {\n unlink(RUNTIME_ESUPDATER_CONSUMER_PID_FILE);\n }\n if (file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE)) {\n unlink(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);\n }\n }\n\n // Delete the pid file of worker process if it exited without delete files.\n $workerPIDFile = $manager->isWorkerProcess();\n if ($workerPIDFile !== false && file_exists($workerPIDFile)) {\n unlink($workerPIDFile);\n }\n}\n\nregister_shutdown_function('shutdownCallback');\n\n/**\n * Register exception callback.\n *\n * @param Throwable $exception\n */\nfunction exception_handler(Throwable $exception = null)\n{\n // do something.\n}\n\nset_exception_handler('exception_handler');\n\n/**\n * Register error callback.\n *\n * @param int $errNo\n *\n * @param string $errMessage\n *\n * @param string $errFile\n *\n * @param int $errLine\n */\nfunction error_handler(int $errNo, string $errMessage, string $errFile, int $errLine)\n{\n // do something.\n}\n\nset_error_handler('error_handler');\n"
},
{
"path": "composer.json",
"content": "{\n \"name\": \"wgrape/esupdater\",\n \"description\": \"A high-performance lightweight framework of PHP to achieve incremental update of ES documents.\",\n \"license\": \"MIT\",\n \"authors\": [\n {\n \"name\": \"wgrape\",\n \"email\": \"wgrapeu@gmail.com\"\n }\n ],\n \"require\": {},\n \"autoload\": {\n \"psr-4\": {\n \"app\\\\\": \"app/\",\n \"framework\\\\\": \"framework/\",\n \"test\\\\\": \"test/\"\n }\n }\n}\n"
},
{
"path": "config/consumer.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n$consumer = [\n 'check_status_interval_seconds' => 2,\n 'broker_list_string' => '',\n 'partition' => 0,\n 'timeout_millisecond' => 2 * 1000,\n 'group_id' => 'default_group',\n 'topic' => 'default_topic',\n 'max_worker_count' => 10,\n];\n"
},
{
"path": "config/db.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n$db = [\n 'database' => [\n 'host' => '',\n 'port' => 3306,\n 'username' => '',\n 'password' => '',\n 'database' => '',\n 'charset' => 'utf8mb4',\n ]\n];\n"
},
{
"path": "config/es.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n$es = [\n 'host' => '',\n 'port' => '',\n 'user_password' => '',\n 'doc_type' => '_doc'\n];\n"
},
{
"path": "config/event.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n$event = [\n\n // You can choose the autoCallback, it's very simple.\n // 'alpha.user' => '\\app\\alpha\\user\\UserHandler',\n\n // You can also choose the manualCallback, it's a bit complicated but powerful.\n 'alpha.user' => [\n 'onInsert' => [\n 'callback' => function ($parsedCanalData) {\n return (new \\app\\alpha\\user\\UserHandler)->onInsert($parsedCanalData);\n },\n ],\n 'onUpdate' => [\n 'filter' => function ($parsedCanalData) {\n // Return false if you need skip this kind of canal data.\n if (!isset($parsedCanalData['data'][0]) || $parsedCanalData['data'][0]['id'] < 10000000) {\n return false;\n }\n\n // Return the filtered canal data.\n $parsedCanalData['data'][0]['name'] .= '_filtered';\n return $parsedCanalData;\n },\n 'callback' => function (array $parsedCanalData) {\n return (new \\app\\alpha\\user\\UserHandler)->onUpdate($parsedCanalData);\n },\n 'finally' => function ($filterResult, $callbackResult) {\n $filterSuccess = $filterResult ? 'success' : 'failed';\n $callbackSuccess = $callbackResult ? 'success' : 'failed';\n \\framework\\Logger::logInfo(\"Work finally: alpha.user.onInsert.filter is {$filterSuccess}, alpha.user.onInsert.callback is {$callbackSuccess}\");\n },\n ],\n 'onDelete' => [\n 'callback' => function (array $parsedCanalData) {\n return (new \\app\\alpha\\user\\UserHandler)->onDelete($parsedCanalData);\n },\n ],\n ],\n];\n"
},
{
"path": "config/log.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n$log = [\n 'debug' => '/home/log/esupdater/debug.log',\n 'info' => '/home/log/esupdater/info.log',\n 'slow' => [\n 'millisecond' => 500,\n 'path' => '/home/log/esupdater/slow.log',\n ],\n 'warning' => '/home/log/esupdater/warning.log',\n 'error' => '/home/log/esupdater/error.log',\n 'fatal' => '/home/log/esupdater/fatal.log',\n];\n"
},
{
"path": "config/test.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n$test = [\n 'testcases_directory' => 'test/testcases/',\n];\n"
},
{
"path": "doc/APPLICATION.md",
"content": "# 业务接入文档\n\n- [一、快速接入](#1)\n- [1、修改配置](#11)\n- [2、创建应用](#12)\n- [3、创建事件回调](#13)\n- [4、注册事件回调](#14)\n- [5、部署项目](#15)\n- [二、应用配置](#2)\n- [1、消费配置](#21)\n- [2、数据库配置](#22)\n- [3、ES配置](#23)\n- [4、日志配置](#24)\n- [5、事件配置](#25)\n- [6、单测配置](#26)\n- [三、系统变量](#3)\n- [四、部署管理](#4)\n- [1、容器化部署](#41)\n- [2、运行时配置](#42)\n- [五、单元测试](#5)\n- [1、手动测试](#51)\n- [2、自动测试](#52)\n- [3、添加用例](#53)\n- [4、测试报告](#54)\n\n## 一、快速接入\n\n### 1、修改配置\n只需要修改 [consumer.php](./config/consumer.php) 配置文件中的```broker_list_string```、```group_id```、```topic```这三个必须的配置项即可， 否则无法正常消费数据。\n\n其他非必须的配置请参考[应用配置](#2)文档。\n\n### 2、创建应用\n\n在```/app/```目录下，创建一个以业务为命名规范的应用名称，如```/app/alpha/```。\n\n### 3、创建事件回调\n在上一步中创建的应用目录下，创建一个```Handler```事件回调类\n\n- [/app/alpha/user/UserHandler.php](./app/alpha/user/UserHandler.php) ：作用类似 ```Controller```\n\n如果需要在事件回调中做大量复杂的业务操作，可以创建一个对应的```Service```业务处理类 ：\n\n- [/app/alpha/user/UserService.php](./app/alpha/user/UserService.php) ：作用类似 ```Service```\n\n建议无论业务是否复杂，都把业务放在```Service```中操作。\n\n> 1、在业务Service中可以自由的调用```common```应用下的```DBService```、```ESService```等服务\n>\n> 2、如果业务更复杂，可以考虑在应用目录下设计属于自己的业务分层，如```daos```、```services```等\n> \n> 3、你可以直接选择使用 [AutoGenerateCallback扩展](/plugin/autogeneratecallback/) 实现事件回调模块的自动创建，免去手动操作此步的过程 ！\n\n### 4、注册事件回调\n在```/config/event.php```配置文件中添加一个新的键值对，表示当```数据库.数据表```出现变更事件时，由对应的```事件Handler```响应处理。\n\n```php\n$event = [\n // 当alpha数据库中的user表发生INSERT/UPDATE/DELETE事件时,\n // 系统会自动创建\\app\\alpha\\user\\UserHandler事件回调类,\n // 并根据不同的事件类型调用不同的方法, 如INSERT事件则调用回调类的onInsert()方法\n 'alpha.user' => '\\app\\alpha\\user\\UserHandler',\n];\n```\n\n除此之外，框架还支持更加强大的事件注册和驱动机制，如果需要请参考[高级事件配置](#251)。\n\n### 5、部署项目\n至此业务接入部分已经完成，参考 [部署管理](#3) 部分部署代码即可。\n\n## 二、应用配置\n\n### 1、消费配置\n\n配置文件 ```/config/consumer.php```，设置Kafka的消费配置\n\n```php\n 2,\n // broker服务器列表,如果多个则以逗号分割，如192.168.0.18:9092,192.168.0.18:9093\n 'broker_list_string' => '192.168.0.18:9092',\n // 消费分区\n 'partition' => 0,\n // 消费超时时间, 单位毫秒\n 'timeout_millisecond' => 2 * 1000,\n // 消费组id\n 'group_id' => '',\n // 消费主题\n 'topic' => '',\n // worker的最大进程数\n 'max_worker_count' => 10,\n];\n```\n\n### 2、数据库配置\n配置文件 ```/config/db.php```，设置访问数据库的配置\n\n```php\n [\n 'host' => '数据库地址',\n 'port' => 3306,\n 'username' => '用户名',\n 'password' => '密码',\n 'database' => '数据库',\n 'charset' => 'utf8mb4',\n ]\n];\n```\n\n### 3、ES配置\n配置文件 ```/config/es.php```，设置访问ES的配置\n\n```php\n 'ES服务host',\n 'port' => 'ES服务端口',\n 'user_password' => 'ES服务凭证',\n 'doc_type' => '_doc'\n];\n```\n\n### 4、日志配置\n\n> 在```/start.sh```启动脚本中，```docker run -v ...``` 会把容器中配置的日志目录挂载到本机相应目录中\n\n配置文件 ```/config/log.php```，配置了不同日志级别的文件路径，如下所示\n\n```php\n '/home/log/esupdater/debug.log',\n 'info' => '/home/log/esupdater/info.log',\n 'slow' => [\n 'millisecond' => 500, // work进程处理耗时超过500ms则记录慢日志\n 'path' => '/home/log/esupdater/slow.log',\n ],\n 'warning' => '/home/log/esupdater/warning.log',\n 'error' => '/home/log/esupdater/error.log',\n 'fatal' => '/home/log/esupdater/fatal.log',\n];\n```\n\n### 5、事件配置\n配置文件 ```/config/event.php```，如下所示\n\n- Key ：```数据库名.表名```\n- Value : ```Handler```\n\n表示当此数据表的数据更新时，由对应的```Handler```处理\n\n```php\n '\\app\\alpha\\user\\UserHandler',\n];\n```\n\n#### (1) 高级事件配置\n上面的这种Key所对应的Value为字符串的配置方式，是一种简单的自动回调配置。 如果Value是Map时，就会使用高级事件配置。\n\n这个Map会再次以如```onInsert```、```onUpdate```、```onDelete```不同的事件为key，value则由以下几种回调函数组成，分别为 ：\n\n- ```filter``` 过滤器 \\[可选\\] ：实现对Canal数据的过滤处理、对事件回调的拦截\n- ```callback``` 事件回调 \\[可选\\] ：实现事件的回调处理\n- ```finally``` 末尾执行 \\[可选\\] ：实现事件的兜底处理，可用于统计数据、记录日志等\n\n关于高级事件配置可以参考 [高级配置示例](../config/event.php) 。\n\n### 6、单测配置\n配置文件 ```config/test.php```，如下所示\n\n```php\n 'test/testcases/',\n];\n```\n\n## 三、系统变量\n在```/.env```文件中记录了服务所需要的所有系统变量，在执行```install.sh```安装脚本时完成系统变量的设置，并由```/framework/Environment.php```类解析并处理。\n\n## 四、部署管理\n\n### 1、容器化部署\n\n如果部署过程中出错，请参考[容器部署帮助](HELP.md#3)文档。\n\n#### (1) 启动\n\n```bash\nbash ./start.sh\n```\n\n#### (2) 停止\n\n```bash\nbash ./stop.sh\n```\n\n#### (3) 重启\n\n```bash\nbash ./restart.sh\n```\n\n### 2、运行时配置\n可以在```/start.sh```脚本中执行```docker run```时设置```核心数```、```目录挂载```等参数，请自定义修改。\n\n如果需要设置更多的容器参数，可以参考[官方文档](https://docs.docker.com/config/containers/resource_constraints/) 。\n\n| Id | 配置名称 | 配置参数 | 参数值 | 默认值 | 释义 |\n| --- | :----: | :----: | :---: | :---: | :---: |\n| 1 | 核心数 | --cpus | float | 1.5 | 设置允许的最大核心数 |\n| 2 | CPU核心集 | --cpuset-cpus | int | 未设置 | 设置允许执行的CPU核心 |\n| 3 | 内存核心集 | --cpuset-mems | int | 未设置 | 设置使用哪些核心的内存 |\n| 4 | 目录挂载 | -v | string | /home/log/esupdater | 设置容器挂载的目录 |\n\n## 五、单元测试\n根目录下的```/test```目录是单元测试目录，其中有一个```/test/run.php```入口文件，它会自动执行 [testcases_directory](HOWTOCODE.md#36) 目录下所有的测试用例。\n\n### 1、手动测试\n```bash\nphp test/run.php\n```\n\n### 2、自动测试\n```bash\ncp test/prepare-commit-msg ./.git/hooks\nchmod +x .git/hooks/prepare-commit-msg\n\n# 此后提交代码会自动执行单元测试，只有单测成功才会允许提交代码\ngit add .\ngit commit -m \"add: xxx\"\n```\n\n如下图实际使用中，每次Commit代码会自动执行测试。\n\n![\"img\"](\"https://user-images.githubusercontent.com/35942268/152677495-1aae134b-93b2-443f-b5cf-8daa719f35f6.png\")
\n\n### 3、添加用例\n在```test/testcases/app```目录下，先创建应用目录（如```alpha```），然后在此目录下以```Test*```开头创建单测文件即可，具体内容可参考 [TestUserService](../test/testcases/app/alpha/TestUserService.php) 单测文件\n\n### 4、测试报告\n在测试运行结束后，会自动生成一个测试报告```/test/report/index.html```文件，点击这里查看报告\n"
},
{
"path": "doc/CHANGELOG.md",
"content": "### 2、版本 ：v2.0.5\n发布日期 ：2021-01-11\n\n#### 修复\n- [重大更新更容易上手-添加系统变量机制](https://github.com/WGrape/esupdater/commit/551c5585d878ec3ebe72c142cb294aa37c64ed29)\n- [完善启动脚本中判断是否已启动的逻辑](https://github.com/WGrape/esupdater/commit/495a0bf15ed8dbe696bccd41c1c2095ab10eea7d)\n\n#### 特性\n- [完善重要更新-简化你的上手流程](https://github.com/WGrape/esupdater/commit/2a430f008f0f604624ea0fa6fbd07954b2e4f152)\n- [重大更新更容易上手-添加系统变量机制](https://github.com/WGrape/esupdater/commit/551c5585d878ec3ebe72c142cb294aa37c64ed29)\n- [内置自动创建kafka容器的脚本](https://github.com/WGrape/esupdater/commit/49f6327cab60102172d071cce84105f6978200b3)\n\n#### 完善\n- 多个文档的完善优化\n\n\n### 1、版本 ：v2.0.0-beta\n发布日期 ：2021-01-04\n\n#### 修复\n- 修复制作镜像失败时误提示成功的bug ：[PR](https://github.com/WGrape/esupdater/pull/37)\n\n#### 特性\n- 支持composer ：[PR](https://github.com/WGrape/esupdater/pull/37)\n\n#### 完善\n- 完善文档\n"
},
{
"path": "doc/CONTRIBUTING.md",
"content": "### 目录\n- [1、如何报告问题](#1)\n- [2、如何提交PR](#2)\n- [3、如何理解项目](#3)\n- [4、代码提交规约](#4)\n- [1、单测通过](#41)\n- [2、commit message 规范](#42)\n- [3、分支管理](#43)\n- [5、打包Release](#5)\n- [6、版本对比](#6)\n- [7、项目数据](#7)\n\n## 1、如何报告问题\n如果在了解和使用过程中，有任何疑问，非常欢迎提出宝贵的 [Issue](https://github.com/WGrape/esupdater/issues/new)\n\n## 2、如何提交PR\nPR的提交不限制范围，如代码、文档等修改均在允许范围内，可 [参考这里](https://github.com/WGrape/esupdater/commit/186e229308463aa745c6b1cbfd02f77bc62ab9d4) 的PR提交\n\n## 3、如何理解项目\n在[HOWTOCODE](HOWTOCODE.md)文档中介绍了详细的实现原理和设计，帮助你了解项目\n\n## 4、代码提交规约\n在提交代码前，至少需要做到以下几项\n\n### (1) 单测通过\n整个项目的单元测试必须通过\n\n### (2) commit message 规范\n规范使用如```fix: 修复Logger中记录日志时间错误的bug```这种组合的提交规范\n- fix: 修复bug相关\n- doc: 文档完善相关\n- refactor: 重大功能重构\n- feat: 新功能、新组件等\n- test: 新增测试或测试相关的修改\n- style: 调整代码格式等对功能和性能无较大影响的修改\n- chore: 构建过程或辅助工具的变动，如dockerfile的修改\n\n### (3) 分支管理\n```v1```版本的开发提交到```v1.x```分支，```v2```版本的开发提交到```v2.x```分支，且```CI```检查通过\n\n### (4) 提交内容注释\n对于重要代码部分，请以评论的方式写清楚原因，可以参考 [test: 添加环境变量的测试用例](https://github.com/WGrape/esupdater/commit/f9e4b4fe867889f398f3ec175af0d5dfc16de4a0) 、[feat: 支持Composer和修复制作镜像失败时误提示成功的bug](https://github.com/WGrape/esupdater/pull/37/files#r800161416)\n\n## 5、打包Release\n基于```v1.x```和```v2.x```分支分别打包不同的Release版本。\n\n## 6、版本对比\n\n### (1) Composer\n| 主版本号 | Composer | 优势 | 劣势 |\n| --- | :----: | :----: | :----: |\n| v1.x | 不支持 | 不需要安装Composer也可以用 | 可能无法正常使用外部依赖 |\n| v2.x | 支持 | 可以方便的调用外部依赖 | 本地开发时需要安装Composer |\n\n## 7、项目数据\n![](\"https://starchart.cc/WGrape/esupdater.svg\")
\n"
},
{
"path": "doc/HELP.md",
"content": "### 目录\n- [一、安装过程帮助](#1)\n- [1、Git命令不存在](#11)\n- [2、无法正常Clone](#12)\n- [3、运行安装脚本出错](#13)\n- [4、Windows系统如何安装](#14)\n- [5、out of capacity错误](#15)\n- [二、镜像制作帮助](#2)\n- [1、Docker命令不存在](#21)\n- [2、无法连接Docker](#22)\n- [3、pecl.php.net更新失败](#23)\n- [三、容器部署帮助](#3)\n- [1、phpkafka镜像不存在](#31)\n- [2、/home/log/esupdater/目录不存在或无权限写](#32)\n- [3、KafkaConsumer创建失败](#33)\n- [4、Consumer highLevelConsuming fetch timeout](#34)\n- [四、版本选择](#4)\n\n## 一、安装过程帮助\n请先通过 ```git clone``` 或 [下载Release包](https://github.com/WGrape/esupdater/releases) 的方式获取项目\n\n### 1、Git命令不存在\n检查git是否已正常安装，查看 [如何安装Git](https://git-scm.com/book/zh/v2/%E8%B5%B7%E6%AD%A5-%E5%AE%89%E8%A3%85-Git)\n\n### 2、无法正常Clone\n针对于常见无法正常Clone的问题，有如下几种解决方案\n\n- 尝试使用```https```方式进行clone\n- 检查网络和网速是否正常，或使用 [esupdater国内版仓库](https://gitee.com/WGrape/esupdater)\n\n### 3、运行安装脚本出错\n如果获取项目已经成功，但是在运行```install.sh```安装脚本阶段出错的话，有如下几种解决方案\n\n- 制作镜像过程出错 ：参考 [镜像制作帮助](#2) 文档\n\n- 提示```Error response from daemon: Get \"https://registry-1.docker.io/v2/\": EOF``` 错误 ：检查网络连接，关闭网络代理即可\n\n### 4、Windows系统如何安装\n目前暂不支持直接在Windows系统上操作，可以选择在Linux虚拟机、Docker环境中安装，如使用 在线Docker网站\n\n### 5、out of capacity错误\n\nWe are really sorry but we are out of capacity and cannot create your session at the moment. Please try again later.\n\n访问 ```https://labs.play-with-docker.com/``` 时如果出现上述错误，暂无解决方案，需要多尝试几次。\n\n\n## 二、镜像制作帮助\n在```install/image```目录中已提供了开箱可用的```phpkafka```镜像文件，只需要简单的执行```bash make.sh```命令即可快速生成```phpkafka```镜像。\n\n自带的```install/image/Dockerfile```镜像文件，已经过多台Unix机器上的多次测试，均可以顺利的成功制作。但是不排除在特殊情况下会存在制作失败的情况，下面会总结出常见的错误和解决方案。\n\n如果还无法解决，可以直接使用已经推到 [Docker Hub](https://hub.docker.com/repository/docker/lvsid/phpkafka) 上的```lvsid/phpkafka:v1.0``` ，修改方式如下 ：\n\n```bash\n# 打开根目录下的Dockerfile文件\ncd esupdater\nvi Dockerfile\n# 把 FROM phpkafka 替换为 FROM lvsid/phpkafka:v1.0 即可\n```\n\n### 1、Docker命令不存在\n安装镜像必须依赖于```Docker```，所以请务必成功安装```Docker```，否则无法创建镜像。\n\n### 2、无法连接Docker\n\n#### (1) 错误提示\n```text\nCannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n```\n\n#### (2) 错误原因\n本地Docker服务未启动\n\n#### (3) 解决方案\n开启本地Docker服务即可\n\n### 3、pecl.php.net更新失败\n\n#### (1) 错误提示\n```text\nUpdating channel \"pecl.php.net\"\nChannel \"pecl.php.net\" is not responding over http://, failed with message: File http://pecl.php.net:80/channel.xml not valid (redirected but no location)\nTrying channel \"pecl.php.net\" over https:// instead\nCannot retrieve channel.xml for channel \"pecl.php.net\" (File https://pecl.php.net:443/channel.xml not valid (redirected but no location))\n```\n\n#### (2) 错误原因\n网络异常，无法正常连接```pecl.php.net```\n\n#### (3) 解决方案\n检查网络是否正常或关掉网络代理\n\n## 三、容器部署帮助\n\n### 1、phpkafka镜像不存在\n> pull access denied for phpkafka, repository does not exist ... ...\n\n出现这种错误是因为跳过了安装步骤，直接执行部署操作导致的。\n\n由于容器化部署方案依赖于```phpkafka```镜像，所以如果提示此镜像不存在，请先参考[快速使用-开始安装](../README.md#22)文档执行安装操作，或直接手动执行```cd image && bash make.sh```完成镜像的制作。\n\n### 2、/home/log/esupdater/目录不存在或无权限写\n由于容器默认会把目录挂载到宿主机的 ```/home/log/esupdater/``` 相同目录下，所以请确保宿主机有此目录和写入权限\n\n或者也可以选择修改[容器的运行时配置](APPLICATION.md#32)中的```目录挂载```，修改方式如下\n\n```bash\nvi start.sh\n\n# 替换以下内容\ndocker run --cpus=1.5 --name esupdaterContainer -d -v {你的宿主机目录}:/home/log/esupdater/ esupdater\n```\n\n### 3、KafkaConsumer创建失败\n> Consumer failed to new KafkaConsumer: \"group.id\" must be configured\n\n如果在```fatal.log```中出现```KafkaConsumer```创建失败的报错，请检查```consumer.php```中的```kafka```服务配置是否可以正常连接\n\n### 4、Consumer highLevelConsuming fetch timeout\n重新启动后可能会报一段时间的```Consumer highLevelConsuming fetch timeout```问题，持续约为2~5秒。\n\n原因 ：重启后需要重新连接```kafka```消费数据，在第一次连接时需要建立TCP和一些额外资源等，所以导致耗时相对较长。\n\n## 四、版本选择\n\n项目版本号规则为```主版本```-```次版本```-```修订号```，其中主版本主要做重大功能升级，次版本主要做性能和功能优化，修订号则做问题修复和完善。\n\n所以```次版本```和```修订号```建议选择最新稳定版本的 [Release包](https://github.com/WGrape/esupdater/releases) ，```主版本```则根据以下对比信息选择合适的即可，可以查看更详细的 [版本对比](doc/CONTRIBUTING.md#5) 信息。\n\n| 主版本号 | Composer |\n| --- | :----: |\n| v1.x | 不支持 |\n| v2.x | 支持 |"
},
{
"path": "doc/HOWTOCODE.md",
"content": "### 目录\n- [一、架构设计](#1)\n- [1、基于Canal](#11) \n- [2、ES文档更新](#12)\n- [3、完整架构](#13)\n- [二、底层原理](#2)\n- [1、生命周期](#21)\n- [2、命令执行](#22)\n- [3、binlog数据处理过程](#23)\n- [4、文件目录规范](#24)\n- [5、程序设计规范](#25)\n- [三、部署过程](#3)\n- [四、参考文档](#4)\n\n## 一、架构设计\n\n### 1、基于Canal\nCanal提供了数据库增量订阅与消费的功能，不需要业务代码的侵入和依赖，通过读取MQ，即可获取到数据库的增量更新\n\n### 2、ES文档更新\n对于数据源为数据库（如MySQL）的ES文档更新，主要有全量更新和增量更新两种方案\n\n- 全量更新 ：脚本全量查询数据库，统一写入至ES中\n\n- 增量更新 ：双写或读取```binlog```，实现ES的增量更新\n\nESUpdater就是读取```binlog```，实现ES文档增量更新的一种解决方案\n\n### 3、完整架构\nESUpdater提供了从消费Kafka中的数据库增量数据，到ES文档增量更新的一个完整业务框架，方便业务的扩展。\n\n- ```Consumer``` 进程 ：订阅Kafka队列，实时获取数据库的增量变更\n- ```Worker``` 进程 ：操作业务逻辑，将数据更新至ES文档\n\n![](\"https://user-images.githubusercontent.com/35942268/147027126-1df83ddf-8698-44dd-a988-5499f7eeb063.png\")
\n\n## 二、底层原理\nESUpdater的核心由```Consumer```进程和```Worker```进程组成，其中根目录下的```/esupdater.php```为入口文件\n\n### 1、生命周期\n```Consumer```进程和```Worker```进程的生命周期都是由命令控制\n\n#### (1) Consumer \n```Consumer```进程由```php esupdater.php start```命令启动，由```php esupdater.php stop```命令停止\n\n#### (2) Worker\n当```Consumer```进程从Kafka中拿到消息后，会通过```exec```的方式执行```php esupdater work```命令，以启动一个新的PHP进程，即```Worker```进程。\n\n```Worker```进程会分为后台和非后台两种执行方式，使用哪种执行方式取决于当前```Worker```进程的数量，如果少于配置的```max_worker_count```会使用后台执行的方式，否则使用非后台执行的方式。通过这种方式可以在加快消费速度的同时，保证稳定性。\n\n所以Worker进程的启动完全由```Consumer```控制，如果想要停止```Worker```进程，必须先停止```Consumer```进程，然后等待```Worker```进程正常执行结束即可\n\n### 2、命令执行\n\n#### (1) start\n当使用```php esupdater.php start```命令时，会启动一个进程，这个进程会以阻塞主进程的方式订阅Kafka消息，所以这个进程叫做```Consumer```进程\n\n```Consumer```进程启动后会先在```/runtime```目录下写```/runtime/esupdater-consumer.pid```文件和```esupdater-consumer.status```文件，分别记录进程它的进程ID和消费状态```start```。\n\n在```Consumer```进程消费kafka消息的同时，会每隔配置的```check_status_interval_seconds```时间检测一次消费状态（```esupdater-consumer.status```文件），当消费状态变为```stop```时，进程会停止消费，此时```Consumer```进程会完全结束。\n\n#### (2) stop\n当使用```php esupdater.php stop```命令时，会启动一个进程，这个进程会向```/runtime/esupdater-consumer.status```文件中写入```stop```指定。\n\n然后每隔一秒钟就会检测```Consumer```进程和```Worker```进程是否都已经完全结束，如果已经检测10秒钟还未完全结束就会通知停止失败，否则停止成功。\n\n#### (3) work\n当```Consumer```进程使用```php esupdater work```命令启动```Worker```进程时，```Worker```进程会记录下```/runtime/esupdater-worker-{pid}.pid```进程ID文件，只有当结束后才会删除此文件。\n\n### 3、binlog数据处理过程\n处理过程为```binlog => canalData => urlencode(canalData)```，可以参考文件 [/framework/Canal.php](../framework/Canal.php)\n\n1. Canal将```binlog```数据解析为```json```格式并投递至kafka\n2. Consumer进程消费kafka，使用```urlencode```方式编码获取到的消息数据\n3. Consumer进程把编码后的消息数据，传递至Worker进程\n4. Worker进程再依次拆解数据即可\n\n### 4、文件目录规范\n\n#### (1) 目录结构\n- ```app```目录 ：应用目录\n- ```config```目录 ：项目的唯一配置入口\n- ```doc```目录 ：项目文档目录\n- ```framework```目录 ：项目的核心框架目录\n- ```install```目录 ：安装目录\n- ```runtime```目录 ：服务运行时产生的中间文件目录，如PID文件，但不包括日志文件。设计思想基于[/proc/](https://en.wikipedia.org/wiki/Procfs)\n- ```test```目录 ：单元测试目录 \n- ```/```目录 ：根目录下存放所有上述目录，和必要的一级文件如```.gitignore```文件\n\n#### (2) 文件规范\n- ```shell```脚本不能省略```.sh```后缀，且统一以```bash xxx.sh```的方式执行\n- 文档统一以大写英文命名，如```README.md``` / ```HELP.md```\n\n### 5、程序设计规范\n关于设计规范可以参考文章 [漫谈编程之编程规范](https://github.com/WGrape/Blog/issues/25)\n\n- 调用类的时候使用命名空间前缀，不使用在头部声明```use```的方式\n\n## 三、部署过程\n\n> 容器化部署方案依赖于```phpkafka```镜像，所以请确保```phpkafka```镜像已经生成。为了避免重复构建耗时，建议把```phpkafka```镜像推到Docker远程仓库中。\n\n容器构建主要通过根目录下的```/Dockerfile```镜像文件，它会基于```phpkafka```镜像构建一个新的镜像，名为```esupdater```。\n\n### 1、启动\n当执行如下命令时，会使用```/Dockerfile```文件创建```esupdater```镜像，并创建```esupdaterContainer```容器，最后通过在容器中执行```php esupdater.php start```命令实现服务的启动\n\n```bash\nbash ./start.sh\n```\n\n启动成功后，除命令行输出```Start success```外，在宿主机```/home/log/esupdater/info.log.{date}```日志中会输出启动日志，如下图所示\n\n![\"img\"](\"https://user-images.githubusercontent.com/35942268/147385923-80cb29e5-225b-4c83-8637-2513d3e17a1d.png\")
\n\n### 2、停止\n当执行以下命令时，会先在容器中执行```php esupdater.php stop```命令，等待容器内```Consumer```进程和```Worker```进程全部停止后，删除镜像和容器\n\n```bash\nbash ./stop.sh\n```\n\n停止成功后，除命令行输出```Stop success```外，同样的在宿主机```/home/log/esupdater/info.log.{date}```日志中会输出停止成功日志，如下图所示\n\n![\"img\"](\"https://user-images.githubusercontent.com/35942268/147386373-dd4b66ff-60b8-43ab-8c5a-f03148258f27.png\")
\n\n### 3、重启\n当执行以下命令时，会先执行```bash stop.sh```命令，再执行```bash start.sh```命令，以防止出现重复启动的问题\n\n```bash\nbash ./restart.sh\n```\n\n## 四、参考文档\n\n- 有关```php-rdkafka```的配置可以 [参考文档](https://github.com/arnaud-lb/php-rdkafka)\n- 有关```librdkafka```的配置可以 [参考文档](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)\n- 有关```PHP Kafka```类的使用可以 [参考文档](https://arnaud.le-blanc.net/php-rdkafka-doc/phpdoc/class.rdkafka-consumertopic.html)\n"
},
{
"path": "doc/README.md",
"content": "### What is this directory\nIt's the directory of different docs.\n"
},
{
"path": "esupdater.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\ninclude_once 'bootstrap.php';\n\n$command = strtolower(isset($argv[1]) ? $argv[1] : '');\nif (empty($command)) {\n echo \"Command empty!\\n\";\n return;\n}\n\n$manager = new \\framework\\Manager();\nswitch ($command) {\n case \"start\":\n $manager->commandStart();\n break;\n case \"stop\":\n $success = $manager->commandStop();\n echo \"{$success}\\n\";\n break;\n case \"work\":\n $canalData = isset($argv[2]) ? $argv[2] : \"\";\n if (empty($canalData)) {\n return;\n }\n $success = $manager->commandWork($canalData);\n echo \"{$success}\\n\";\n break;\n default:\n echo \"Not support command: {$command}\\n\";\n return;\n}\n"
},
{
"path": "framework/Canal.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Canal\n{\n /**\n * Encode the canal data to string.\n *\n * @param string $unEncodedCanalData the json format data in kafka queue (canal put it into kafka)\n *\n * @return string\n */\n public function encode(string $unEncodedCanalData): string\n {\n return urlencode($unEncodedCanalData);\n }\n\n /**\n * Decode the canal data to string.\n *\n * @param string $encodedCanalData the urlencoded canal data\n *\n * @return string\n */\n public function decode(string $encodedCanalData): string\n {\n return urldecode($encodedCanalData);\n }\n\n /**\n * Parse the canal data to array.\n *\n * @param string $encodedCanalData the urlencoded canal data\n *\n * @return array\n */\n public function parse(string $encodedCanalData): array\n {\n return json_decode($this->decode($encodedCanalData), true);\n }\n\n /**\n * Check the canal data format.\n *\n * @param array $parsedCanalData\n *\n * @return bool\n */\n public function checkParsedCanalData(array $parsedCanalData): bool\n {\n if (empty($parsedCanalData) || empty($parsedCanalData['data'])) {\n return false;\n }\n if (!isset($parsedCanalData['database']) || !isset($parsedCanalData['table']) || !isset($parsedCanalData['type'])) {\n return false;\n }\n if (!isset($parsedCanalData['id']) || !isset($parsedCanalData['ts'])) {\n return false;\n }\n return true;\n }\n}\n"
},
{
"path": "framework/Consumer.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Consumer\n{\n const TIMER_MARK = 'consume';\n const CONSUMER_EXIT_WITH_EMPTY_STRING = '';\n const START_FLAG_STRING = 'start';\n const STOP_FLAG_STRING = 'stop';\n\n private $checkStatusIntervalSeconds;\n private $brokerListString;\n private $partition;\n private $timeoutMillisecond;\n private $groupId;\n private $topic;\n private $maxWorkerCount;\n\n public function __construct($consumer)\n {\n $this->checkStatusIntervalSeconds = (isset($consumer['check_status_interval_seconds']) && !empty($consumer['check_status_interval_seconds'])) ? $consumer['check_status_interval_seconds'] : 2;\n $this->brokerListString = (isset($consumer['broker_list_string']) && !empty($consumer['broker_list_string'])) ? $consumer['broker_list_string'] : ($this->getLocalIP() . ':9092');\n $this->partition = (isset($consumer['partition']) && !empty($consumer['partition'])) ? $consumer['partition'] : 0;\n $this->timeoutMillisecond = (isset($consumer['timeout_millisecond']) && !empty($consumer['timeout_millisecond'])) ? $consumer['timeout_millisecond'] : 2 * 1000;\n $this->groupId = (isset($consumer['group_id']) && !empty($consumer['group_id'])) ? $consumer['group_id'] : 'default_group';\n $this->topic = (isset($consumer['topic']) && !empty($consumer['topic'])) ? $consumer['topic'] : 'default_topic';\n $this->maxWorkerCount = (isset($consumer['max_worker_count']) && !empty($consumer['max_worker_count'])) ? $consumer['max_worker_count'] : 100;\n }\n\n /**\n * Get localIP by ifconfig command\n * @DEPRECATED: It would get error local IP in docker container.\n * @return string\n */\n public function getLocalIPByIfconfig(): string\n {\n $localIP = '127.0.0.1';\n $command = \"ifconfig -a | grep inet | grep -v 127.0.0.1 | grep -v inet6 | grep '192.168' | head -n 1 | awk '{print $2}' | tr -d \\\"addr:\\\"\";\n exec($command, $output);\n if (isset($output[0])) {\n $localIP = $output[0];\n if (stripos($localIP, '192.168') !== 0) {\n $localIP = '127.0.0.1';\n }\n }\n return $localIP;\n }\n\n /**\n * Get local host IP\n * @return string\n */\n public function getLocalIP(): string\n {\n $localIP = \\framework\\Environment::getSystemVariable('ESUPDATER_LOCAL_IP');\n if (empty($localIP)) {\n $localIP = '127.0.0.1';\n }\n return $localIP;\n }\n\n /**\n * Whether need to stop consume or not\n * @return bool\n */\n public function isNeedStop(): bool\n {\n return file_get_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE) === self::STOP_FLAG_STRING;\n }\n\n /**\n * Whether need to check status or not\n * @return bool\n */\n public function isNeedCheckStatus(): bool\n {\n $millisecond = \\framework\\Timer::elapsed(self::TIMER_MARK);\n return ($millisecond / 1000) >= $this->checkStatusIntervalSeconds;\n }\n\n /**\n * Get property\n * @param string $propertyName\n * @return string | false | array | int | mixed\n */\n public function getProperty(string $propertyName)\n {\n return property_exists($this, $propertyName) ? $this->$propertyName : '';\n }\n\n /**\n * DEPRECATED: low level consuming, but now it's not available\n * @param false $onlyForTest\n * @return string\n */\n public function lowLevelConsuming($onlyForTest = false): string\n {\n // Create consumer config object\n $consumerConfigObject = new \\RdKafka\\Conf();\n $consumerConfigObject->set('group.id', $this->groupId);\n\n // Create topic config object\n $topicConfigObject = new \\RdKafka\\TopicConf();\n $topicConfigObject->set('auto.offset.reset', 'smallest');\n\n // Create consumer object\n $consumerObject = new \\RdKafka\\Consumer($consumerConfigObject);\n $consumerObject->addBrokers($this->brokerListString);\n\n // Create topic object\n $topicObject = $consumerObject->newTopic($this->topic, $topicConfigObject);\n\n // Consume start from last offset\n $topicObject->consumeStart($this->partition, RD_KAFKA_OFFSET_STORED);\n\n while (true) {\n $message = $topicObject->consume($this->partition, $this->timeoutMillisecond);\n switch ($message->err) {\n case RD_KAFKA_RESP_ERR_NO_ERROR:\n if (is_null($message) || empty($message->payload)) {\n echo \"Message is null or payload is empty\\n\";\n continue;\n }\n var_dump($message);\n break;\n case RD_KAFKA_RESP_ERR__PARTITION_EOF:\n echo \"No more messages\\n\";\n break;\n case RD_KAFKA_RESP_ERR__TIMED_OUT:\n echo \"Timeout\\n\";\n break;\n default:\n echo \"Unknown message error\\n\";\n break;\n }\n }\n }\n\n /**\n * High level consuming\n * @param false $onlyForTest\n * @return string\n */\n public function highLevelConsuming($onlyForTest = false): string\n {\n $canal = new Canal();\n $manager = new Manager();\n\n // Create topic config object\n $topicConfigObject = new \\RdKafka\\TopicConf();\n $topicConfigObject->set('request.required.acks', true);\n $topicConfigObject->set('auto.commit.interval.ms', 100);\n $topicConfigObject->set('auto.offset.reset', 'smallest');\n\n // Create consumer config object\n $consumerConfigObject = new \\RdKafka\\Conf();\n $consumerConfigObject->setRebalanceCb(function (\\RdKafka\\KafkaConsumer $kafka, $err, array $partitions = null) {\n switch ($err) {\n case RD_KAFKA_RESP_ERR__ASSIGN_PARTITIONS:\n $kafka->assign($partitions);\n break;\n case RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS:\n $kafka->assign(NULL);\n break;\n default:\n Logger::logError(\"Consumer setRebalanceCb occurs exception: {$err}\");\n }\n });\n $consumerConfigObject->set('group.id', $this->groupId);\n $consumerConfigObject->set('metadata.broker.list', $this->brokerListString);\n $consumerConfigObject->setDefaultTopicConf($topicConfigObject);\n\n // Create consumer object\n try {\n $consumerObject = new \\RdKafka\\KafkaConsumer($consumerConfigObject);\n $consumerObject->subscribe([$this->topic]);\n } catch (\\Throwable $e) {\n Logger::logFatal(\"Consumer failed to new KafkaConsumer: \" . $e->getMessage());\n return self::CONSUMER_EXIT_WITH_EMPTY_STRING;\n }\n\n // Timer start\n \\framework\\Timer::start(self::TIMER_MARK);\n\n // Consuming loop\n while (true) {\n // Check consume status\n if ($this->isNeedCheckStatus() && $this->isNeedStop()) {\n Logger::logInfo('Consumer need to stop');\n return self::CONSUMER_EXIT_WITH_EMPTY_STRING;\n }\n\n // Fetch message\n $message = $consumerObject->consume($this->timeoutMillisecond);\n switch ($message->err) {\n case RD_KAFKA_RESP_ERR_NO_ERROR:\n if (is_null($message) || empty($message->payload)) {\n Logger::logDebug('Consumer fetch message is null or payload is empty');\n continue;\n }\n if ($onlyForTest) {\n return $message->payload;\n }\n\n $logFile = CREATE_WORKER_LOG_FILE;\n $canalData = $canal->encode($message->payload);\n $count = $manager->getRunningWorkersCount();\n if ($count !== false && $count < ($this->maxWorkerCount - 1)) {\n Logger::logDebug(\"Consumer handle message: create non-block worker process, current count is {$count}\");\n exec(\"nohup php esupdater.php work {$canalData} >> {$logFile} &\");\n } else {\n Logger::logDebug(\"Consumer handle message: create block worker process, current count is {$count}\");\n exec(\"php esupdater.php work {$canalData} >> {$logFile}\");\n }\n break;\n case RD_KAFKA_RESP_ERR__PARTITION_EOF:\n Logger::logDebug('Consumer highLevelConsuming fetch no more messages');\n break;\n case RD_KAFKA_RESP_ERR__TIMED_OUT:\n Logger::logDebug('Consumer highLevelConsuming fetch timeout');\n break;\n default:\n Logger::logError('Consumer highLevelConsuming catch unknown message error');\n break;\n }\n }\n }\n}\n"
},
{
"path": "framework/Environment.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Environment\n{\n /**\n * Variable container.\n * @var array\n */\n public static $variableContainer = [];\n\n /**\n * Get system variable.\n * @param $variable\n * @return string\n */\n public static function getSystemVariable($variable): string\n {\n $result = '';\n if (empty(self::$variableContainer)) {\n self::parseEnvFile();\n }\n if (isset(self::$variableContainer[$variable])) {\n $result = self::$variableContainer[$variable];\n }\n return $result;\n }\n\n /**\n * Parse the .env file.\n */\n public static function parseEnvFile()\n {\n $file = ENVIRONMENT_FILE;\n $handler = fopen($file, 'r+');\n while (!feof($handler)) {\n $content = trim(fgets($handler));\n $slices = explode('=', $content);\n if (count($slices) === 2) {\n self::$variableContainer[$slices[0]] = $slices[1];\n }\n }\n }\n}\n"
},
{
"path": "framework/Listener.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Listener\n{\n /**\n * Database event type: insert.\n */\n const TYPE_INSERT = 'INSERT';\n\n /**\n * Database event type: update.\n */\n const TYPE_UPDATE = 'UPDATE';\n\n /**\n * Database event type: delete.\n */\n const TYPE_DELETE = 'DELETE';\n\n /**\n * Timer mark of work\n */\n const TIMER_MARK = 'work';\n\n /**\n * Find and\n * @param string $canalData\n */\n public function dispatch(string $canalData)\n {\n // Timer start.\n Timer::start(self::TIMER_MARK);\n\n // Parse and check canal data.\n $canalParser = new Canal();\n $parsedCanalData = $canalParser->parse($canalData);\n Logger::setLogIdByParsedCanalData($parsedCanalData);\n if (!$canalParser->checkParsedCanalData($parsedCanalData)) {\n Logger::logError(\"Check canal data error\");\n return;\n }\n\n // Check the key is valid.\n global $event;\n $database = $parsedCanalData['database'];\n $table = $parsedCanalData['table'];\n $key = \"{$database}.{$table}\";\n if (!isset($event[$key])) {\n Logger::logError(\"Not found the valid event config for key: key={$key}\");\n return;\n }\n\n $onWhichEvent = false;\n switch ($parsedCanalData['type']) {\n case self::TYPE_INSERT:\n $onWhichEvent = 'onInsert';\n break;\n case self::TYPE_UPDATE:\n $onWhichEvent = 'onUpdate';\n break;\n case self::TYPE_DELETE:\n $onWhichEvent = 'onDelete';\n break;\n }\n\n $isAutoCallback = is_string($event[$key]);\n if ($isAutoCallback) {\n $this->autoCallback($key, $onWhichEvent, $parsedCanalData);\n } else {\n $this->manualCallback($key, $onWhichEvent, $parsedCanalData);\n }\n\n global $log;\n $cost = Timer::elapsed(self::TIMER_MARK);\n if (isset($log['slow']['millisecond']) && $cost > $log['slow']['millisecond']) {\n Logger::logSlow(\"Work slow: key={$key}, onWhichEvent={$onWhichEvent}, cost={$cost}ms\");\n }\n }\n\n /**\n * Auto callback\n * @param string $key\n * @param string $onWhichEvent\n * @param array $parsedCanalData\n */\n public function autoCallback(string $key, string $onWhichEvent, array $parsedCanalData)\n {\n global $event;\n if ($onWhichEvent === false || !method_exists($event[$key], $onWhichEvent)) {\n Logger::logError(\"Not found the valid auto callback: key={$key}, type={$parsedCanalData['type']}\");\n return;\n }\n\n $handler = new $event[$key];\n $handler->$onWhichEvent($parsedCanalData);\n }\n\n /**\n * Manual callback\n * @param string $key\n * @param string $onWhichEvent\n * @param array $parsedCanalData\n */\n public function manualCallback(string $key, string $onWhichEvent, array $parsedCanalData)\n {\n global $event;\n if ($onWhichEvent === false || !isset($event[$key][$onWhichEvent]) || !is_array($event[$key][$onWhichEvent])) {\n Logger::logError(\"Not found the valid manual callback: key={$key}, type={$parsedCanalData['type']}\");\n return;\n }\n\n $filterResult = true;\n if (isset($event[$key][$onWhichEvent]['filter']) && is_callable($event[$key][$onWhichEvent]['filter'])) {\n $filterResult = $event[$key][$onWhichEvent]['filter']($parsedCanalData);\n if ($filterResult) {\n $parsedCanalData = $filterResult;\n }\n }\n $callbackResult = false;\n if ($filterResult && isset($event[$key][$onWhichEvent]['callback']) && is_callable($event[$key][$onWhichEvent]['callback'])) {\n $callbackResult = $event[$key][$onWhichEvent]['callback']($parsedCanalData);\n }\n if (isset($event[$key][$onWhichEvent]['finally']) && is_callable($event[$key][$onWhichEvent]['finally'])) {\n $event[$key][$onWhichEvent]['finally']($filterResult, $callbackResult);\n }\n }\n}\n\n"
},
{
"path": "framework/Logger.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Logger\n{\n /**\n * Support log level.\n */\n const LEVEL_DEBUG = 'debug';\n const LEVEL_INFO = 'info';\n const LEVEL_SLOW = 'slow';\n const LEVEL_WARNING = 'warning';\n const LEVEL_ERROR = 'error';\n const LEVEL_FATAL = 'fatal';\n\n /**\n * The unique id of log.\n *\n * @var string\n */\n private static $logId;\n\n /**\n * The formula of logId.\n *\n * @var string\n */\n private static $formula;\n\n /**\n * Setup the logId.\n *\n * @param string $logIdParam\n *\n * @param string $formulaParam\n */\n public static function setLogId(string $logIdParam, string $formulaParam)\n {\n self::$logId = $logIdParam;\n self::$formula = $formulaParam;\n }\n\n /**\n * Setup the logId according the parsed canal data.\n *\n * @param array $parsedCanalData\n */\n public static function setLogIdByParsedCanalData(array $parsedCanalData)\n {\n $database = isset($parsedCanalData['database']) ? $parsedCanalData['database'] : '';\n $table = isset($parsedCanalData['table']) ? $parsedCanalData['table'] : '';\n $type = isset($parsedCanalData['type']) ? $parsedCanalData['type'] : '';\n $id = isset($parsedCanalData['id']) ? $parsedCanalData['id'] : 0;\n $ts = isset($parsedCanalData['ts']) ? $parsedCanalData['ts'] : 0;\n\n $formula = self::$formula = \"{$database}+{$table}+{$type}+{$id}+{$ts}\";\n self::$logId = md5($formula);\n }\n\n /**\n * Return the debug data.\n *\n * @return array[]\n */\n public static function returnDumpData(): array\n {\n $result = [\n 'runtime_files' => [],\n ];\n\n $handle = opendir(RUNTIME_PATH);\n while ($handle && ($file = readdir($handle)) !== false) {\n if (!is_file($file)) {\n continue;\n }\n $result['runtime_files'][] = $file;\n }\n closedir($handle);\n\n return $result;\n }\n\n /**\n * Write log in debug mode.\n *\n * @param string $data the message to write\n */\n public static function logDebug(string $data)\n {\n self::write(self::LEVEL_DEBUG, $data);\n }\n\n /**\n * Write log in info mode.\n *\n * @param string $data the message to write\n */\n public static function logInfo(string $data)\n {\n self::write(self::LEVEL_INFO, $data);\n }\n\n /**\n * Write log in slow mode.\n *\n * @param string $data the message to write\n */\n public static function logSlow(string $data)\n {\n self::write(self::LEVEL_SLOW, $data);\n }\n\n /**\n * Write log in warning mode.\n *\n * @param string $data the message to write\n */\n public static function logWarning(string $data)\n {\n self::write(self::LEVEL_WARNING, $data);\n }\n\n /**\n * Write log in error mode.\n *\n * @param string $data the message to write\n */\n public static function logError(string $data)\n {\n self::write(self::LEVEL_ERROR, $data);\n }\n\n /**\n * Write log in fatal mode.\n *\n * @param string $data the message to write\n */\n public static function logFatal(string $data)\n {\n self::write(self::LEVEL_FATAL, $data);\n }\n\n /**\n * Get the path of log file in different level mode.\n *\n * @param $logLevel\n *\n * @return string\n */\n public static function getLogFilePath($logLevel): string\n {\n global $log;\n $date = date('Ymd');\n if ($logLevel === self::LEVEL_SLOW) {\n return \"{$log[$logLevel]['path']}.{$date}\";\n }\n return \"{$log[$logLevel]}.{$date}\";\n }\n\n /**\n * The common method for writing log.\n *\n * @param $level\n *\n * @param $data\n */\n public static function write($level, $data)\n {\n $logId = self::$logId;\n $formula = self::$formula;\n $datetime = date('Y-m-d H:i:s');\n\n $header = \"{$datetime} | logid = {$logId} = {$formula}\";\n $body = $data;\n $footer = \"\";\n if ($level === self::LEVEL_FATAL) {\n $dumpData = self::returnDumpData();\n $footer .= \"----------Dump is data as follows----------\\n\";\n $footer .= (\"runtime_files: \" . implode(', ', $dumpData['runtime_files']));\n }\n $content = \"{$header}\\n{$body}\\n{$footer}\\n\";\n\n $file = self::getLogFilePath($level);\n file_put_contents($file, $content, FILE_APPEND | LOCK_EX);\n }\n}\n\n"
},
{
"path": "framework/Manager.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Manager\n{\n const COMMAND_WORK_SUCCESS = 'success';\n const COMMAND_STOP_SUCCESS = 'success';\n const COMMAND_STOP_FAILED = 'failed';\n\n /**\n * Get running workers count.\n *\n * @notice You must do something when this function return false\n *\n * @return false|int\n */\n public function getRunningWorkersCount()\n {\n $handle = opendir(RUNTIME_PATH);\n if (!$handle) {\n return false;\n }\n\n $count = 0;\n while (($file = readdir($handle)) !== false) {\n if (strpos($file, RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX) === 0) {\n ++$count;\n }\n }\n closedir($handle);\n return $count;\n }\n\n /**\n * Start consumer and blocking.\n */\n public function startConsumerAndBlocking()\n {\n file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, Consumer::START_FLAG_STRING);\n file_put_contents(RUNTIME_ESUPDATER_CONSUMER_PID_FILE, intval(getmypid()));\n\n global $consumer;\n (new Consumer($consumer))->highLevelConsuming();\n }\n\n /**\n * Stop consumer by IPC(InterProcess Communication): Shared File.\n */\n public function stopConsumerByIPC()\n {\n // Consumer process may not been created, so communicate only when consumer was created, or will be mistaken for stop failed\n if (file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE)) {\n file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, Consumer::STOP_FLAG_STRING);\n }\n }\n\n /**\n * Whether consumer was stopped or not.\n *\n * @return bool\n */\n public function isConsumerStopped(): bool\n {\n return !file_exists(RUNTIME_ESUPDATER_CONSUMER_PID_FILE) && !file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);\n }\n\n /**\n * Whether all workers were stopped or not.\n *\n * @return bool\n */\n public function isWorkersStopped(): bool\n {\n $count = $this->getRunningWorkersCount();\n if ($count === false || $count > 0) {\n return false;\n }\n return true;\n }\n\n /**\n * Whether the current process is consumer process.\n */\n public function isConsumerProcess()\n {\n $pid = getmypid();\n $consumerPIDFile = RUNTIME_ESUPDATER_CONSUMER_PID_FILE;\n if (file_exists($consumerPIDFile) && file_get_contents($consumerPIDFile) == $pid) {\n return $consumerPIDFile;\n }\n return false;\n }\n\n /**\n * Whether the current process is worker process.\n */\n public function isWorkerProcess()\n {\n $pid = getmypid();\n $workerPIDFile = RUNTIME_PATH . RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX . $pid . \".pid\";\n if (file_exists($workerPIDFile) && file_get_contents($workerPIDFile) == $pid) {\n return $workerPIDFile;\n }\n return false;\n }\n\n /**\n * Command: start.\n */\n public function commandStart()\n {\n $formula = \"start+\" . date('Y-m-d H:i:s');\n $logId = md5($formula);\n Logger::setLogId($logId, $formula);\n Logger::logInfo('Start esupdater');\n\n // Start consumer and blocking ...\n $this->startConsumerAndBlocking();\n\n // After blocking, it means consumer is stopped, so remove consumer runtime files now\n unlink(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);\n unlink(RUNTIME_ESUPDATER_CONSUMER_PID_FILE);\n }\n\n /**\n * Command: stop.\n *\n * @return string\n */\n public function commandStop(): string\n {\n $formula = \"stop+\" . date('Y-m-d H:i:s');\n $logId = md5($formula);\n Logger::setLogId($logId, $formula);\n Logger::logInfo('Stop esupdater');\n\n // Stop consumer by IPC(InterProcess Communication)\n $this->stopConsumerByIPC();\n\n // Wait consumer and all workers were stopped, the max wait time is 10 seconds\n $maxWaitSecond = 10;\n $startTimestamp = time();\n while (true) {\n if ($this->isConsumerStopped() && $this->isWorkersStopped()) {\n Logger::logInfo('Stop esupdater successfully');\n return self::COMMAND_STOP_SUCCESS;\n }\n if ((time() - $startTimestamp) > $maxWaitSecond) {\n Logger::logFatal('Failed to stop esupdater');\n return self::COMMAND_STOP_FAILED;\n }\n sleep(1);\n }\n }\n\n /**\n * Command: work.\n *\n * @param string $canalData\n *\n * @return string\n */\n public function commandWork(string $canalData): string\n {\n $pid = getmypid();\n $workerPIDFile = RUNTIME_PATH . RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX . $pid . \".pid\";\n file_put_contents($workerPIDFile, intval($pid));\n\n (new \\framework\\Listener())->dispatch($canalData);\n\n unlink($workerPIDFile);\n return self::COMMAND_WORK_SUCCESS;\n }\n}\n"
},
{
"path": "framework/Timer.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace framework;\n\nclass Timer\n{\n /**\n * Store the benchmark of time.\n *\n * @var array\n */\n public static $container = [];\n\n /**\n * Return nowSecondTimestampWithMicrosecond.\n *\n * @return float\n */\n public static function getNowSecondTimestampWithMicrosecond(): float\n {\n list($microsecond, $secondTimestamp) = explode(' ', microtime());\n return ((float)$microsecond + (float)$secondTimestamp);\n }\n\n /**\n * Timer start and return nowSecondTimestampWithMicrosecond.\n *\n * @param string $mark\n *\n * @return float\n */\n public static function start(string $mark): float\n {\n self::$container[$mark] = $nowSecondTimestampWithMicrosecond = self::getNowSecondTimestampWithMicrosecond();\n return $nowSecondTimestampWithMicrosecond;\n }\n\n /**\n * Return elapsed milliseconds from start.\n *\n * @param string $mark\n *\n * @return int\n */\n public static function elapsed(string $mark): int\n {\n if (!isset(self::$container[$mark])) {\n return 0;\n }\n\n $startSecondTimestampWithMicrosecond = self::$container[$mark];\n $nowSecondTimestampWithMicrosecond = self::getNowSecondTimestampWithMicrosecond();\n return (int)(($nowSecondTimestampWithMicrosecond - $startSecondTimestampWithMicrosecond) * 1000);\n }\n}"
},
{
"path": "install/README.md",
"content": "### What is this directory\nIt's the installation directory, you just run ```bash install.sh``` would help you to install automatically.\n"
},
{
"path": "install/container/kafka.sh",
"content": "#!/usr/bin/env bash\n\n# Pull images first.\ndocker pull wurstmeister/zookeeper\ndocker pull wurstmeister/kafka\n\n# Start zookeeper.\ncontainerCount=0\nfor file in $(docker container ls -f name=zookeeperContainer -q)\ndo\n ((containerCount++))\ndone\nif [ $containerCount -ne 0 ]; then\n echo -e \"Start zookeeperContainer already, is skipped.\"\nelse\n docker run -d --name zookeeperContainer -p 2181:2181 -t wurstmeister/zookeeper\nfi\n\n# Start kafka server.\n# use ifconfig command to get the local ip: 192.168.x.x not 127.0.0.1\nlocalIP=$(ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|grep '192.168' |awk '{print $2}'|tr -d \"addr:\")\ncontainerCount=0\nfor file in $(docker container ls -f name=kafkaContainer -q)\ndo\n ((containerCount++))\ndone\nif [ $containerCount -ne 0 ]; then\n echo -e \"Start kafkaContainer already, is skipped.\"\nelse\n docker run -d --name kafkaContainer -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=${localIP}:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${localIP}:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka\nfi\n\n# Login kafka server container\n# docker exec -it kafkaContainer /bin/bash\n# cd /opt/kafka/\n# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic default_topic\n# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic default_topic --from-beginning\n"
},
{
"path": "install/image/Dockerfile",
"content": "# This dockerfile will make images of phpkafka.\n\n# Must use version 7.0, or it would be error.\nFROM php:7.0-fpm\n\n# Use homeland image.\nRUN mv /etc/apt/sources.list /etc/apt/sources.list.bak \\\n && echo 'deb http://mirrors.163.com/debian/ stretch main non-free contrib' > /etc/apt/sources.list \\\n && echo 'deb http://mirrors.163.com/debian/ stretch-updates main non-free contrib' >> /etc/apt/sources.list \\\n && echo 'deb http://mirrors.163.com/debian-security/ stretch/updates main non-free contrib' >> /etc/apt/sources.list \\\n && apt-get update\n\n# Disable commandline interactive.\nENV DEBIAN_FRONTEND noninteractive\n\n# Install git.\n# -y is input yes automatically.\nRUN apt-get update \\\n && apt-get -y install git\n\n# Install vim.\nRUN apt-get update \\\n && apt-get -y install vim\n\n# Install composer\nRUN curl -sS https://getcomposer.org/installer | php \\\n && mv composer.phar /usr/local/bin/composer\n\n# Install pecl.\nRUN apt-get update \\\n && apt-get -y install autoconf \\\n && apt-get -y install libz-dev\n\n# Install libkakfa.\nRUN apt-get update \\\n && apt-get -y install librdkafka-dev=0.9.3-1\n\n\n# Install the php kafka extension.\n# Need librdkafka-dev=0.9.3-1 / rdkafka-3.0.0 this version will work, or it would be error.\n# Check rdkafka versions: https://pecl.php.net/package/rdkafka\n# Check librdkafka versions: apt search librdkafka-dev\nRUN pecl channel-update pecl.php.net \\\n && pecl install rdkafka-3.0.0 \\\n && docker-php-ext-enable rdkafka\n\n# Install and enable mysqli extension\nRUN docker-php-ext-install mysqli && docker-php-ext-enable mysqli\n"
},
{
"path": "install/image/README.md",
"content": "### What is this directory\nIt's a docker image directory, you just run ```bash make.sh``` would help you to make a ```php-kafka``` image \n"
},
{
"path": "install/image/make.sh",
"content": "#!/usr/bin/env bash\ndocker build -t phpkafka .\nif [ $? -ne 0 ]; then\n echo -e \"\"\n echo -e \">>>>>>>>Make image failure<<<<<<<<\"\n exit 1\nelse\n echo -e \"\"\n echo -e \"========Make image success========\"\n docker images\nfi\n\n# Push to docker repository\n# docker login\n# input your name and password\n# docker tag adf2495d561e lvsid/phpkafka:v1.0\n# docker push lvsid/phpkafka:v1.0\n"
},
{
"path": "install/install.sh",
"content": "#!/usr/bin/env bash\n\n# Check params.\nif [ ! -n \"$1\" ] ;then\n echo -e \"Please input the localIP param.\"\n exit 1\nelse\n export ESUPDATER_LOCAL_IP=$1\n systemVariables=\"ESUPDATER_LOCAL_IP=$1\\n\"\n echo -e $systemVariables > ../.env\nfi\n\n# The part of you must do.\n# 1. Make image\ncd image && bash make.sh && cd ..\n\n# The part of you could do.\n# 1. Run kafka container\ncd container && bash kafka.sh && cd ..\n"
},
{
"path": "plugin/README.md",
"content": "### What is this directory\nIt's a plugin directory, all plugins were stored here.\n"
},
{
"path": "plugin/autogeneratecallback/README.md",
"content": "## AutoGenerateCallback\n一个自动生成```Handler```和```Service```的事件回调模块的扩展。\n\n### 1、如何使用\n- $namespace 参数 ：新增模块的命名空间，如 ```app\\alpha\\account```\n- $moduleName 参数 ：新增的模块名称，如 ```Account```\n\n```shell\nphp plugin/autogeneratecallback/autogeneratecallback.php {$namespace} {$moduleName}\n```\n\n#### 注意事项\n\n- 命名空间中的``` \\ ```符号（首位不需要）需要转义，所以别忘记输入两次``` \\\\ ```，如```app\\alpha\\account```\n- 模块名称使用大驼峰命名，如```MyProfile```\n\n### 2、使用示例\n项目中 [account](/app/alpha/account) 模块下的文件即是通过如下命令自动生成而来的。\n\n```shell\n php plugin/autogeneratecallback/autogeneratecallback.php app\\\\alpha\\\\account Account\n```\n\n![\"img\"](\"https://user-images.githubusercontent.com/35942268/154846773-73e8bc1b-97e0-4d59-be18-23ebaf123c50.png\")
\n\n### 3、实现原理\n基于 [模板替换](https://www.google.com/search?q=%E6%A8%A1%E6%9D%BF%E6%9B%BF%E6%8D%A2) 原理， 先在模板文件 [handler.template](./handler.template) 和 [service.template](./service.template) 中定义如```{{变量}}```此类的```占位符```，再使用正则匹配把```占位符```替换为目标文本。\n"
},
{
"path": "plugin/autogeneratecallback/autogeneratecallback.php",
"content": " $pluginDirectory . 'handler.template',\n 'php_file' => \"{$namespaceDirectory}/{$moduleName}Handler.php\",\n ],\n [\n 'template_file' => $pluginDirectory . 'service.template',\n 'php_file' => \"{$namespaceDirectory}/{$moduleName}Service.php\",\n ],\n];\nforeach ($configList as $item) {\n if (file_exists($item['php_file'])) {\n echo \"The file already exist: {$item['php_file']}\\n\";\n continue;\n }\n if (!is_dir($namespaceDirectory)) {\n mkdir($namespaceDirectory);\n }\n\n $handle = fopen($item['template_file'], 'r');\n while (!feof($handle)) {\n $content = fgets($handle);\n if (preg_match($namespacePattern, $content) > 0) {\n $content = str_replace($namespacePattern, $namespace, $content);\n }\n if (preg_match($moduleNamePattern, $content) > 0) {\n $content = str_replace($moduleNamePattern, $moduleName, $content);\n }\n file_put_contents($item['php_file'], $content, FILE_APPEND);\n }\n}\n"
},
{
"path": "plugin/autogeneratecallback/handler.template",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace {{namespace}};\n\nclass {{moduleName}}Handler\n{\n /**\n * The event callback when table of {{moduleName}} trigger insert event.\n * @param array $parsedCanalData\n * @return bool\n */\n public function onInsert(array $parsedCanalData): bool\n {\n return (new {{moduleName}}Service())->doInsert($parsedCanalData);\n }\n\n /**\n * The event callback when table of {{moduleName}} trigger update event.\n * @param array $parsedCanalData\n * @return bool\n */\n public function onUpdate(array $parsedCanalData): bool\n {\n return (new {{moduleName}}Service())->doUpdate($parsedCanalData);\n }\n\n /**\n * The event callback when table of {{moduleName}} trigger delete event.\n * @param array $parsedCanalData\n * @return bool\n */\n public function onDelete(array $parsedCanalData): bool\n {\n return (new {{moduleName}}Service())->doDelete($parsedCanalData);\n }\n}\n"
},
{
"path": "plugin/autogeneratecallback/service.template",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace {{namespace}};\n\nclass {{moduleName}}Service\n{\n /**\n * Update different indexes when insert.\n * @param array $parsedCanalData\n * @return bool\n */\n public function doInsert(array $parsedCanalData): bool\n {\n $success = true;\n\n if (!$this->updateAIndexWhenInsert($parsedCanalData)) {\n $success = false;\n }\n\n if (!$this->updateBIndexWhenInsert($parsedCanalData)) {\n $success = false;\n }\n\n // update some other indexes...\n\n return $success;\n }\n\n /**\n * Update different indexes when update.\n * @param array $parsedCanalData\n * @return bool\n */\n public function doUpdate(array $parsedCanalData): bool\n {\n $success = true;\n\n if (!$this->updateAIndexWhenUpdate($parsedCanalData)) {\n $success = false;\n }\n\n if (!$this->updateBIndexWhenUpdate($parsedCanalData)) {\n $success = false;\n }\n\n // update some other indexes...\n\n return $success;\n }\n\n /**\n * Update different indexes when delete.\n * @param array $parsedCanalData\n * @return bool\n */\n public function doDelete(array $parsedCanalData): bool\n {\n $success = true;\n\n if (!$this->updateAIndexWhenDelete($parsedCanalData)) {\n $success = false;\n }\n\n if (!$this->updateBIndexWhenDelete($parsedCanalData)) {\n $success = false;\n }\n\n // update some other indexes...\n\n return $success;\n }\n\n /**\n * Update a index: event of insert\n * @param array $parsedCanalData\n * @return bool\n */\n public function updateAIndexWhenInsert(array $parsedCanalData): bool\n {\n return true;\n }\n\n /**\n * Update b index: event of insert\n * @param array $parsedCanalData\n * @return bool\n */\n public function updateBIndexWhenInsert(array $parsedCanalData): bool\n {\n return true;\n }\n\n /**\n * Update a index: event of update\n * @param array $parsedCanalData\n * @return bool\n */\n public function updateAIndexWhenUpdate(array $parsedCanalData): bool\n {\n return true;\n }\n\n /**\n * Update b index: event of update\n * @param array $parsedCanalData\n * @return bool\n */\n public function updateBIndexWhenUpdate(array $parsedCanalData): bool\n {\n return true;\n }\n\n /**\n * Update a index: event of delete\n * @param array $parsedCanalData\n * @return bool\n */\n public function updateAIndexWhenDelete(array $parsedCanalData): bool\n {\n return true;\n }\n\n /**\n * Update b index: event of delete\n * @param array $parsedCanalData\n * @return bool\n */\n public function updateBIndexWhenDelete(array $parsedCanalData): bool\n {\n return true;\n }\n}\n"
},
{
"path": "plugin/loader.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\n// PHP Configuration.\ndate_default_timezone_set('Asia/Shanghai');\n\nconst PLUGIN_PATH = __DIR__ . '/';\nconst ROOT_PATH = PLUGIN_PATH . '../';\n"
},
{
"path": "restart.sh",
"content": "#!/usr/bin/env bash\nbash stop.sh && bash start.sh\n"
},
{
"path": "start.sh",
"content": "#!/usr/bin/env bash\n# Prevent start repeatedly\ncontainerCount=0\nfor file in $(docker container ls -f name=esupdaterContainer -q)\ndo\n ((containerCount++))\ndone\nif [ $containerCount -ne 0 ]; then\n echo -e \">>>>>>>>Start failure: please run stop.sh first<<<<<<<<\"\n exit 1\nfi\n\n# Pull without password\ngit config --global credential.helper store\ngit pull\n\n# Build esupdater image\ndocker build -t esupdater .\nif [ $? -ne 0 ]; then\n echo -e \">>>>>>>>Start failure: failed to build<<<<<<<<\"\n exit 1\nfi\n\n# Run container\n# docker run --cpuset-cpus=\"0,1\" --cpus=1.5 --cpuset-mems=\"2,3\" --name {ContainerName} -d -v {LocalPath:ContainerPath} {imageName}\ndocker run --cpus=1.5 --name esupdaterContainer -d -v /home/log/esupdater/:/home/log/esupdater/ esupdater\nif [ $? -ne 0 ]; then\n echo -e \">>>>>>>>Start failure: failed to run<<<<<<<<\"\n exit 1\nfi\n\n# Exec command\ndocker exec -d esupdaterContainer php esupdater.php start\nif [ $? -ne 0 ]; then\n echo -e \">>>>>>>>Start failure: failed to exec<<<<<<<<\"\n exit 1\nelse\n echo -e \"========Start success========\"\nfi\n"
},
{
"path": "stop.sh",
"content": "#!/usr/bin/env bash\n# Exec command: php esupdater.php stop\n# Use exec command with -i argument would run and return result synchronously: docker exec -i esupdaterContainer php -r \"var_dump(123);\"\ndocker exec -i esupdaterContainer php esupdater.php stop\n\n# Stop and remove container\ndocker stop esupdaterContainer\ndocker container rm esupdaterContainer\n\n# Remove image\ndocker rmi esupdater\n\nif [ $? -ne 0 ]; then\n echo -e \">>>>>>>>Stop failure<<<<<<<<\"\n exit 1\nelse\n echo -e \"========Stop success========\"\nfi\n"
},
{
"path": "test/BaseTest.php",
"content": "\n * @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence\n */\n\nnamespace test;\n\nclass BaseTest\n{\n /**\n * Test success.\n *\n * @return bool\n */\n protected function success(): bool\n {\n $debugTrace = debug_backtrace();\n $fileShortName = $this->getCallerFileName($debugTrace);\n $functionName = $this->getCallerFunctionName($debugTrace);\n echo \"Test Success: {$fileShortName} -> {$functionName}\\n\";\n return true;\n }\n\n /**\n * Test failed.\n * @param string $err\n *\n * @return bool\n */\n protected function failed($err = \"\"): bool\n {\n $debugTrace = debug_backtrace();\n $fileShortName = $this->getCallerFileName($debugTrace);\n $functionName = $this->getCallerFunctionName($debugTrace);\n echo \"Test Failed: {$fileShortName} -> {$functionName}\\n\";\n if (!empty($err)) {\n echo \"$err\\n\";\n }\n return false;\n }\n\n /**\n * Return text with success color.\n *\n * @param $text\n *\n * @return string\n */\n public function decorateSuccessText($text): string\n {\n return \"\\033[32m{$text}\\033[0m\";\n }\n\n /**\n * Return text with failed color.\n *\n * @param $text\n *\n * @return string\n */\n public function decorateFailedText($text): string\n {\n return \"\\033[31;4m{$text}\\033[0m\";\n }\n\n /**\n * Get file name of caller.\n *\n * @param array $debugTrace the data of debug_backtrace() return\n *\n * @return string\n */\n protected function getCallerFileName(array $debugTrace): string\n {\n if (empty($debugTrace)) {\n return '';\n }\n $fileFullName = $debugTrace[0]['file'];\n $sliceList = explode('/test/', $fileFullName);\n $fileShortName = '';\n if (isset($sliceList[1])) {\n $fileShortName = \"/test/\" . $sliceList[1];\n }\n return $fileShortName;\n }\n\n /**\n * Get function name of caller.\n *\n * @param array $debugTrace the data of debug_backtrace() return\n *\n * @return string\n */\n protected function getCallerFunctionName(array $debugTrace): string\n {\n if (!isset($debugTrace[1])) {\n return '';\n }\n return $debugTrace[1]['function'];\n }\n}"
},
{
"path": "test/README.md",
"content": "### What is this directory\nIt's a unit test directory, all testcases are stored here, includes ```/app/alpha/```, ```/app/common```，```/framework``` and so on.\n"
},
{
"path": "test/prepare-commit-msg",
"content": "#!/bin/sh\n\n# Run test\nresult=`php test/run.php | grep 'Test Failed'`\nif [ -n \"$result\" ]; then\n echo \"Sorry, you would not commit if you failed the test \"\n exit 1\nfi"
},
{
"path": "test/report/index.css",
"content": "#Container {\n text-align: center;\n}\n\n.section{\n margin-top: 30px;\n}\n\n#table-1 thead, #table-1 tr {\n border-top-width: 1px;\n border-top-style: solid;\n border-top-color: rgb(230, 189, 189);\n}\n\n#table-1 {\n width: 700px;\n margin: 20px auto;\n border-bottom-width: 1px;\n border-bottom-style: solid;\n border-bottom-color: rgb(230, 189, 189);\n}\n\n/* Padding and font style */\n#table-1 td, #table-1 th {\n padding: 5px 10px;\n font-size: 12px;\n font-family: Verdana;\n color: rgb(177, 106, 104);\n}\n\n/* Alternating background colors */\n#table-1 tr:nth-child(even) {\n background: rgb(238, 211, 210)\n}\n\n#table-1 tr:nth-child(odd) {\n background: #FFF\n}\n\n#table-2 thead, #table-2 tr {\n border-top-width: 1px;\n border-top-style: solid;\n border-top-color: rgb(235, 242, 224);\n}\n#table-2 {\n width: 700px;\n margin: 20px auto;\n border-bottom-width: 1px;\n border-bottom-style: solid;\n border-bottom-color: rgb(235, 242, 224);\n}\n\n/* Padding and font style */\n#table-2 td, #table-2 th {\n padding: 5px 10px;\n font-size: 12px;\n font-family: Verdana;\n color: rgb(149, 170, 109);\n}\n\n/* Alternating background colors */\n#table-2 tr:nth-child(even) {\n background: rgb(230, 238, 214)\n}\n#table-2 tr:nth-child(odd) {\n background: #FFF\n}"
},
{
"path": "test/report/index.html",
"content": "| Id | TestClass | TestMethod | TestResult |
|---|---|---|---|
| 1 | test\\testcases\\app\\common\\TestESService | testIsSuccess | success |
| 2 | test\\testcases\\app\\common\\TestESService | testIsNeedToUpdate | success |
| 3 | test\\testcases\\app\\alpha\\TestUserService | testGetUserId | success |
| 4 | test\\testcases\\framework\\TestConsumer | testConstruct | success |
| 5 | test\\testcases\\framework\\TestConsumer | testIsNeedStop | success |
| 6 | test\\testcases\\framework\\TestConsumer | testIsNeedCheckStatus | success |
| 7 | test\\testcases\\framework\\TestConsumer | testHighLevelConsuming | success |
| 8 | test\\testcases\\framework\\TestTimer | testElapsed | success |
| 9 | test\\testcases\\framework\\TestCanal | testCheckParsedCanalData | success |
| 10 | test\\testcases\\framework\\TestCommand | testWork | success |
| 11 | test\\testcases\\framework\\TestManager | testGetRunningWorkersCount | success |
| 12 | test\\testcases\\framework\\TestManager | testStopConsumerByIPC | success |
| 13 | test\\testcases\\framework\\TestManager | testIsConsumerStopped | success |
| 14 | test\\testcases\\framework\\TestManager | testIsWorkersStopped | success |
| 15 | test\\testcases\\framework\\TestManager | testIsConsumerProcess | success |
| 16 | test\\testcases\\framework\\TestManager | testIsWorkerProcess | success |
| 17 | test\\testcases\\framework\\TestEnvironment | testParseEnvFile | success |
| Id | TestClass | TestMethod | TestResult |
|---|