Repository: WGrape/esupdater
Branch: v2.x
Commit: a550e67a6ece
Files: 58
Total size: 100.8 KB
Directory structure:
gitextract_gfvoa9pt/
├── .editorconfig
├── .github/
│ ├── CODE_OF_CONDUCT.md
│ └── workflows/
│ └── release.yml
├── .gitignore
├── .travis.yml
├── CHANGELOG.md
├── Dockerfile
├── LICENSE
├── README.md
├── bootstrap.php
├── composer.json
├── config/
│ ├── consumer.php
│ ├── db.php
│ ├── es.php
│ ├── event.php
│ ├── log.php
│ └── test.php
├── doc/
│ ├── APPLICATION.md
│ ├── CHANGELOG.md
│ ├── CONTRIBUTING.md
│ ├── HELP.md
│ ├── HOWTOCODE.md
│ └── README.md
├── esupdater.php
├── framework/
│ ├── Canal.php
│ ├── Consumer.php
│ ├── Environment.php
│ ├── Listener.php
│ ├── Logger.php
│ ├── Manager.php
│ └── Timer.php
├── install/
│ ├── README.md
│ ├── container/
│ │ └── kafka.sh
│ ├── image/
│ │ ├── Dockerfile
│ │ ├── README.md
│ │ └── make.sh
│ └── install.sh
├── plugin/
│ ├── README.md
│ ├── autogeneratecallback/
│ │ ├── README.md
│ │ ├── autogeneratecallback.php
│ │ ├── handler.template
│ │ └── service.template
│ └── loader.php
├── restart.sh
├── start.sh
├── stop.sh
└── test/
├── BaseTest.php
├── README.md
├── prepare-commit-msg
├── report/
│ ├── index.css
│ └── index.html
├── run.php
└── testcases/
└── framework/
├── TestCanal.php
├── TestCommand.php
├── TestConsumer.php
├── TestEnvironment.php
├── TestManager.php
└── TestTimer.php
================================================
FILE CONTENTS
================================================
================================================
FILE: .editorconfig
================================================
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 4
trim_trailing_whitespace = true
[*.md]
insert_final_newline = false
trim_trailing_whitespace = false
================================================
FILE: .github/CODE_OF_CONDUCT.md
================================================
# Contributor Code of Conduct
As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.
We are committed to making participation in this project a harassment-free experience for everyone, regardless of the level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, age, or religion.
Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.
This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/)
================================================
FILE: .github/workflows/release.yml
================================================
on:
push:
branches:
- v2.x
name: Release
jobs:
release-please:
runs-on: ubuntu-latest
steps:
- uses: GoogleCloudPlatform/release-please-action@v3
id: release
with:
token: ${{ secrets.RELEASE_TOKEN }}
release-type: node
package-name: posture
changelog-types: '[{"type": "types", "section":"Types", "hidden": false},{"type": "revert", "section":"Reverts", "hidden": false},{"type": "feat", "section": "Features", "hidden": false},{"type": "fix", "section": "Bug Fixes", "hidden": false},{"type": "improvement", "section": "Feature Improvements", "hidden": false},{"type": "docs", "section":"Docs", "hidden": false},{"type": "doc", "section":"Docs", "hidden": false},{"type": "style", "section":"Styling", "hidden": false},{"type": "refactor", "section":"Code Refactoring", "hidden": false},{"type": "perf", "section":"Performance Improvements", "hidden": false},{"type": "test", "section":"Tests", "hidden": false},{"type": "build", "section":"Build System", "hidden": false},{"type": "ci", "section":"CI", "hidden":false}]'
================================================
FILE: .gitignore
================================================
.git
.idea
logs
.DS_Store
composer.lock
# Ignore directories of /app except common and alpha.
app/*
!app/common/
!app/alpha/
# Ignore directories of /test/testcases/app except common and alpha.
test/testcases/app/*
!test/testcases/app/common/
!test/testcases/app/alpha/
# Ignore runtime directory.
runtime/
# Ignore vendor directory.
vendor/
================================================
FILE: .travis.yml
================================================
# CI was triggered at pull request
language: php
php:
- 7.0
- nightly
script:
- sudo mkdir -p /home/log/esupdater/
- sudo chmod -R 777 /home/log/
- composer install
- php test/run.php
branches:
only:
- v1.x
- v2.x
================================================
FILE: CHANGELOG.md
================================================
# Changelog
## [2.1.1](https://github.com/WGrape/esupdater/compare/v2.1.0...v2.1.1) (2022-09-19)
### Docs
* fix version ([cc7f225](https://github.com/WGrape/esupdater/commit/cc7f225357253fbaa48028e9192666a6ebe89602))
## [2.1.0](https://github.com/WGrape/esupdater/compare/v2.0.5...v2.1.0) (2022-09-19)
### Features
* phpkafka镜像推到DockerHub中提供甚至无需安装都即可使用的镜像 ([d6f56cd](https://github.com/WGrape/esupdater/commit/d6f56cdd7e5086c15a51abe9cccf86844a9cfc82))
* 添加插件扩展功能-完成第一个扩展autogeneratecallback ([e1408c5](https://github.com/WGrape/esupdater/commit/e1408c5b8aa9c9dfa3bc89092026974806197069))
### Bug Fixes
* 修复事件中未获取到返回值的问题 ([480487a](https://github.com/WGrape/esupdater/commit/480487a1897f93cbff68d81d07ffe879d6fb5a57))
* 修复注释错误 ([2a1d121](https://github.com/WGrape/esupdater/commit/2a1d121d5a96d533887e498c4de9b64100d3f61a))
### Tests
* 添加环境变量的测试用例 ([f9e4b4f](https://github.com/WGrape/esupdater/commit/f9e4b4fe867889f398f3ec175af0d5dfc16de4a0))
### Docs
* QUESTION文档移至issues ([af874e5](https://github.com/WGrape/esupdater/commit/af874e5c5fe335ca34db04d92864c0fc72a73db4))
* README添加贡献者列表 ([3dec42a](https://github.com/WGrape/esupdater/commit/3dec42a2f874b45b5c3fd2f3a5d19d8507d2941b))
* 优化文档 ([995a288](https://github.com/WGrape/esupdater/commit/995a2889364551768b142b996b3f7d9172a531db))
* 修改文档描述 ([649e50a](https://github.com/WGrape/esupdater/commit/649e50a74780cf02d27788d1daec160b239b8246))
* 完善README文档 ([2192ef4](https://github.com/WGrape/esupdater/commit/2192ef4eb8abb13ff5e8c88243e614c05840e218))
* 完善业务接入文档和贡献文档 ([58042b0](https://github.com/WGrape/esupdater/commit/58042b07ab6c25f7ca042fe09f6c440631f91055))
* 完善文档-强调esupdater对数据的二次处理功能 ([35e0feb](https://github.com/WGrape/esupdater/commit/35e0feb782d294fcc96dc73216815a72954f3aa1))
* 添加runtime目录可类比为/proc目录的介绍 ([63469dc](https://github.com/WGrape/esupdater/commit/63469dc6fe67773022154cd3dcb07f47ffa8163d))
================================================
FILE: Dockerfile
================================================
FROM phpkafka
# If you failed to install phpkafka, you can use the lvsid/phpkafka:v1.0 in dockerhub: https://hub.docker.com/repository/docker/lvsid/phpkafka
# FROM lvsid/phpkafka:v1.0
WORKDIR /dist
COPY . /dist/
RUN mkdir -p /home/log/esupdater \
&& composer install --quiet
# Do not run start command here, because it means the container is equal consumer process,
# once the consumer was stopped, the container would exit,
# so the workers would not stopped safely.
# CMD ["php", "/dist/esupdater.php", "start"]
================================================
FILE: LICENSE
================================================
The MIT License (MIT)
Copyright (c) 2021 WGrape
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
<div align="center" >
<img width="200" alt="img" src="https://user-images.githubusercontent.com/35942268/147061994-f0d5a3ec-2d5f-4d72-af1c-139289547f25.png">
</div>
<div align="center">
<p>一个基于Canal实现ES文档增量更新的高性能轻量框架</p>
</div>
<p align="center">
<a href="https://www.oscs1024.com/project/oscs/WGrape/esupdater?ref=badge_small" alt="OSCS Status"><img src="https://www.oscs1024.com/platform/badge/WGrape/esupdater.svg?size=small"/></a>
<img src="https://img.shields.io/badge/php-7.0+-blue.svg">
<img alt="GitHub release (latest by date)" src="https://img.shields.io/github/v/release/wgrape/esupdater">
<img src="https://img.shields.io/badge/version-2.x-blue.svg">
<img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/lvsid/phpkafka">
<a href="https://app.travis-ci.com/github/WGrape/esupdater"><img src="https://app.travis-ci.com/WGrape/esupdater.svg?branch=master"><a>
<a href="https://wgrape.github.io/esupdater/report.html"><img src="https://img.shields.io/badge/unitest-100%25-yellow.svg"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green.svg"></a>
<a href="doc/HOWTOCODE.md"><img src="https://img.shields.io/badge/doc-中文-red.svg"></a>
</p>
- [一、介绍](#1)
- [1、轻量级框架](#11)
- [2、全面容器化](#12)
- [3、事件驱动化](#13)
- [4、插件化扩展](#14)
- [5、高性能消费](#15)
- [二、快速上手](#2)
- [三、业务接入](#3)
- [四、扩展列表](#4)
- [五、关于项目](#5)
- [1、深入了解](#51)
- [2、欢迎参与](#52)
- [六、贡献列表](#6)
## <span id="1">一、介绍</span>
ESUpdater是一个基于Canal实现ES文档增量更新的高性能轻量框架。基于以下优势，可以让你快速上手和使用。
<img width="900" alt="Architecture" src="https://user-images.githubusercontent.com/35942268/145793762-a23899d6-c162-4527-ae72-643edc80bb18.png">
### <span id="11">1、轻量级框架</span>
无论安装使用，还是代码设计，整个框架都非常轻量，优雅的完成数据二次处理和ES增量更新。
### <span id="12">2、全面容器化</span>
为解决各种依赖安装的复杂麻烦问题，已实现全面容器化，只需一条命令就可以轻松安装、部署、和维护。
### <span id="13">3、事件驱动化</span>
基于框架内部的事件驱动设计，可以轻松地注册不同数据表的变更事件和回调，优雅地实现增量更新。
### <span id="14">4、插件化扩展</span>
在不影响框架内部运行的前提下，支持插件化扩展，实现对内部行为的自定义扩展。
### <span id="15">5、高性能消费</span>
通过一个```Consumer```进程和多个```Worker```进程的一对多通信模型，最少提高10倍的吞吐量，实现高性能消费。
## <span id="2">二、快速上手</span>
> 预计只需要 **3分钟** 即可完成 ！
以下操作中会依赖Docker，所以请先安装并启动它。如果只是试用则强烈建议你全程使用<a href="https://labs.play-with-docker.com/">在线Docker网站</a>，按如下步骤安装即可，非常方便。
### <span id="21">1、获取项目</span>
通过```git clone```或下载Release包即可获取项目，如果出错请参考[获取过程帮助](doc/HELP.md#12)文档。
```bash
git clone https://github.com/WGrape/esupdater
cd esupdater
```
### <span id="22">2、开始安装</span>
执行```install```目录下的```install.sh```安装脚本时，需要传递如下参数以实现[设置环境变量](./doc/APPLICATION.md#3)。如果出错请参考[安装过程帮助](doc/HELP.md#13)文档。
- ```your_local_ip``` ：本机IP参数，通过```ifconfig```查看，通常为192.168开头，而不是127.0.0.1
```bash
cd install
bash install.sh ${your_local_ip}
cd ..
```
### <span id="24">3、运行项目</span>
安装成功后，执行根目录下的```start.sh```启动脚本即可。如果出错请参考[运行过程帮助](doc/HELP.md#3)文档。
```bash
bash start.sh
# 查看日志输出
tail -f /home/log/esupdater/debug.log.20220111
```
### <span id="25">4、测试运行</span>
在另一个窗口进入```kafkaContainer```容器中，按如下操作启动```Kafka生产者```
```bash
docker exec -it kafkaContainer /bin/bash
cd /opt/kafka/
# 启动时可能会出现warn, 忽略即可
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic default_topic
```
<img width="843" alt="img1" src="https://user-images.githubusercontent.com/35942268/148804272-b00483a9-3861-4aab-8b2f-aee963784694.png">
启动成功后会进入一个生产消息的命令行，发送任意消息后，查看上一步日志中的输出，如果出现如下类似日志则说明服务已经成功运行 ！
<img width="823" alt="img2" src="https://user-images.githubusercontent.com/35942268/148806227-25af15b9-5609-4de3-ac13-96fc83c7c99b.png">
## <span id="3">三、业务接入</span>
如果需要在你的业务中接入此项目，请参考[应用接入文档](./doc/APPLICATION.md)。
## <span id="4">四、扩展列表</span>
基于插件化扩展开发，项目提供了一系列开箱即用的扩展。
### 1、AutoGenerateCallback
一个自动生成```Handler```和```Service```的事件回调模块的扩展。具体使用见[使用介绍](./plugin/autogeneratecallback/README.md)
## <span id="5">五、关于项目</span>
### <span id="51">1、深入了解</span>
如果想要深入了解本项目，在 [doc目录](./doc) 下提供了如下丰富完善的项目文档，欢迎阅读。
- [APPLICATION](doc/APPLICATION.md) ：帮助你快速在业务中接入此项目
- [HOWTOCODE](doc/HOWTOCODE.md) ：更深的了解项目，包括架构设计、底层原理
- [HELP](doc/HELP.md) ：解决安装和部署过程中问题的帮助手册，包括镜像制作帮助、容器部署帮助等
### <span id="52">2、参与项目</span>
项目源码设计简单易懂，如有更好的想法，可参考[如何贡献](doc/CONTRIBUTING.md)文档，期待提出宝贵的 [Pull request](https://github.com/WGrape/esupdater/pulls) 。
如果在了解和使用过程中，有任何疑问，也欢迎提出宝贵的 [Issue](https://github.com/WGrape/esupdater/issues/new) 。
开源不易，如果支持本项目 **欢迎Star ！** 以激励维护和更新的动力。
## <span id="6">六、贡献列表</span>
所有对本项目有过重要贡献的用户，会收录在此贡献者列表中。
- 感谢 [sick-cat](https://github.com/sick-cat) 提出的Issue ：[启动配置](https://github.com/WGrape/esupdater/issues/41)
- 感谢 [onser3](https://github.com/onser3) 提出的Issue ：[自动生成handler和service层](https://github.com/WGrape/esupdater/issues/44)
================================================
FILE: bootstrap.php
================================================
<?php
/**
* The bootloader file of esupdater, test and so on.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
// PHP Configuration.
date_default_timezone_set('Asia/Shanghai');
// Define path constants.
const ROOT_PATH = __DIR__ . '/';
const APP_PATH = ROOT_PATH . 'app/';
const CONFIG_PATH = ROOT_PATH . 'config/';
const FRAMEWORK_PATH = ROOT_PATH . 'framework/';
const RUNTIME_PATH = ROOT_PATH . 'runtime/';
const VENDOR_PATH = ROOT_PATH . 'vendor/';
// Define file constants.
const RUNTIME_ESUPDATER_CONSUMER_PID_FILE = RUNTIME_PATH . 'esupdater-consumer.pid';
const RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE = RUNTIME_PATH . 'esupdater-consumer.status';
const RUNTIME_IGNORE_ERROR_TEMP_FILE = RUNTIME_PATH . 'ignore-error.temp';
const CREATE_WORKER_LOG_FILE = RUNTIME_PATH . 'create-worker.log';
const RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX = 'esupdater-worker-';
const COMPOSER_AUTOLOAD_FILE = VENDOR_PATH . 'autoload.php';
const ENVIRONMENT_FILE = ROOT_PATH . '.env';
// Define data constants.
const DEFAULT_PID = 0;
// Load config files.
include_once CONFIG_PATH . 'consumer.php';
include_once CONFIG_PATH . 'db.php';
include_once CONFIG_PATH . 'es.php';
include_once CONFIG_PATH . 'log.php';
include_once CONFIG_PATH . 'event.php';
/**
* Include composer autoload file or register autoload.
*/
if (!file_exists(COMPOSER_AUTOLOAD_FILE)) {
function autoloadCallback(string $classname)
{
$classname = str_replace('\\', '/', $classname);
$file = ROOT_PATH . "{$classname}.php";
if (file_exists($file)) {
include_once $file;
} else {
echo 'class file' . $classname . 'not found!';
}
}
spl_autoload_register("autoloadCallback", true, true);
} else {
include_once COMPOSER_AUTOLOAD_FILE;
}
/**
* Register shutdown callback.
*/
function shutdownCallback()
{
$manager = new \framework\Manager();
// Delete the files of consumer process if it exited without delete files.
if ($manager->isConsumerProcess()) {
if (file_exists(RUNTIME_ESUPDATER_CONSUMER_PID_FILE)) {
unlink(RUNTIME_ESUPDATER_CONSUMER_PID_FILE);
}
if (file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE)) {
unlink(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);
}
}
// Delete the pid file of worker process if it exited without delete files.
$workerPIDFile = $manager->isWorkerProcess();
if ($workerPIDFile !== false && file_exists($workerPIDFile)) {
unlink($workerPIDFile);
}
}
register_shutdown_function('shutdownCallback');
/**
* Register exception callback.
*
* @param Throwable $exception
*/
function exception_handler(Throwable $exception = null)
{
// do something.
}
set_exception_handler('exception_handler');
/**
* Register error callback.
*
* @param int $errNo
*
* @param string $errMessage
*
* @param string $errFile
*
* @param int $errLine
*/
function error_handler(int $errNo, string $errMessage, string $errFile, int $errLine)
{
// do something.
}
set_error_handler('error_handler');
================================================
FILE: composer.json
================================================
{
"name": "wgrape/esupdater",
"description": "A high-performance lightweight framework of PHP to achieve incremental update of ES documents.",
"license": "MIT",
"authors": [
{
"name": "wgrape",
"email": "wgrapeu@gmail.com"
}
],
"require": {},
"autoload": {
"psr-4": {
"app\\": "app/",
"framework\\": "framework/",
"test\\": "test/"
}
}
}
================================================
FILE: config/consumer.php
================================================
<?php
/**
* The kafka consuming configuration.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
$consumer = [
'check_status_interval_seconds' => 2,
'broker_list_string' => '',
'partition' => 0,
'timeout_millisecond' => 2 * 1000,
'group_id' => 'default_group',
'topic' => 'default_topic',
'max_worker_count' => 10,
];
================================================
FILE: config/db.php
================================================
<?php
/**
* The mysql querying configuration.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
$db = [
'database' => [
'host' => '',
'port' => 3306,
'username' => '',
'password' => '',
'database' => '',
'charset' => 'utf8mb4',
]
];
================================================
FILE: config/es.php
================================================
<?php
/**
* The elasticsearch configuration.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
$es = [
'host' => '',
'port' => '',
'user_password' => '',
'doc_type' => '_doc'
];
================================================
FILE: config/event.php
================================================
<?php
/**
* The event registering configuration, you can choose autoCallback or manualCallback.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
$event = [
// You can choose the autoCallback, it's very simple.
// 'alpha.user' => '\app\alpha\user\UserHandler',
// You can also choose the manualCallback, it's a bit complicated but powerful.
'alpha.user' => [
'onInsert' => [
'callback' => function ($parsedCanalData) {
return (new \app\alpha\user\UserHandler)->onInsert($parsedCanalData);
},
],
'onUpdate' => [
'filter' => function ($parsedCanalData) {
// Return false if you need skip this kind of canal data.
if (!isset($parsedCanalData['data'][0]) || $parsedCanalData['data'][0]['id'] < 10000000) {
return false;
}
// Return the filtered canal data.
$parsedCanalData['data'][0]['name'] .= '_filtered';
return $parsedCanalData;
},
'callback' => function (array $parsedCanalData) {
return (new \app\alpha\user\UserHandler)->onUpdate($parsedCanalData);
},
'finally' => function ($filterResult, $callbackResult) {
$filterSuccess = $filterResult ? 'success' : 'failed';
$callbackSuccess = $callbackResult ? 'success' : 'failed';
\framework\Logger::logInfo("Work finally: alpha.user.onInsert.filter is {$filterSuccess}, alpha.user.onInsert.callback is {$callbackSuccess}");
},
],
'onDelete' => [
'callback' => function (array $parsedCanalData) {
return (new \app\alpha\user\UserHandler)->onDelete($parsedCanalData);
},
],
],
];
================================================
FILE: config/log.php
================================================
<?php
/**
* The log outputting configuration.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
$log = [
'debug' => '/home/log/esupdater/debug.log',
'info' => '/home/log/esupdater/info.log',
'slow' => [
'millisecond' => 500,
'path' => '/home/log/esupdater/slow.log',
],
'warning' => '/home/log/esupdater/warning.log',
'error' => '/home/log/esupdater/error.log',
'fatal' => '/home/log/esupdater/fatal.log',
];
================================================
FILE: config/test.php
================================================
<?php
/**
* The testing configuration.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
$test = [
'testcases_directory' => 'test/testcases/',
];
================================================
FILE: doc/APPLICATION.md
================================================
# 业务接入文档
- [一、快速接入](#1)
- [1、修改配置](#11)
- [2、创建应用](#12)
- [3、创建事件回调](#13)
- [4、注册事件回调](#14)
- [5、部署项目](#15)
- [二、应用配置](#2)
- [1、消费配置](#21)
- [2、数据库配置](#22)
- [3、ES配置](#23)
- [4、日志配置](#24)
- [5、事件配置](#25)
- [6、单测配置](#26)
- [三、系统变量](#3)
- [四、部署管理](#4)
- [1、容器化部署](#41)
- [2、运行时配置](#42)
- [五、单元测试](#5)
- [1、手动测试](#51)
- [2、自动测试](#52)
- [3、添加用例](#53)
- [4、测试报告](#54)
## 一、快速接入
### <span id="11">1、修改配置</span>
只需要修改 [consumer.php](./config/consumer.php) 配置文件中的```broker_list_string```、```group_id```、```topic```这三个必须的配置项即可， 否则无法正常消费数据。
其他非必须的配置请参考[应用配置](#2)文档。
### <span id="12">2、创建应用</span>
在```/app/```目录下，创建一个以业务为命名规范的应用名称，如```/app/alpha/```。
### <span id="13">3、创建事件回调</span>
在上一步中创建的应用目录下，创建一个```Handler```事件回调类
- [/app/alpha/user/UserHandler.php](./app/alpha/user/UserHandler.php) ：作用类似 ```Controller```
如果需要在事件回调中做大量复杂的业务操作，可以创建一个对应的```Service```业务处理类 ：
- [/app/alpha/user/UserService.php](./app/alpha/user/UserService.php) ：作用类似 ```Service```
建议无论业务是否复杂，都把业务放在```Service```中操作。
> 1、在业务Service中可以自由的调用```common```应用下的```DBService```、```ESService```等服务
>
> 2、如果业务更复杂，可以考虑在应用目录下设计属于自己的业务分层，如```daos```、```services```等
>
> 3、你可以直接选择使用 [AutoGenerateCallback扩展](/plugin/autogeneratecallback/) 实现事件回调模块的自动创建，免去手动操作此步的过程 ！
### <span id="14">4、注册事件回调</span>
在```/config/event.php```配置文件中添加一个新的键值对，表示当```数据库.数据表```出现变更事件时，由对应的```事件Handler```响应处理。
```php
$event = [
// 当alpha数据库中的user表发生INSERT/UPDATE/DELETE事件时,
// 系统会自动创建\app\alpha\user\UserHandler事件回调类,
// 并根据不同的事件类型调用不同的方法, 如INSERT事件则调用回调类的onInsert()方法
'alpha.user' => '\app\alpha\user\UserHandler',
];
```
除此之外，框架还支持更加强大的事件注册和驱动机制，如果需要请参考[高级事件配置](#251)。
### <span id="15">5、部署项目</span>
至此业务接入部分已经完成，参考 [部署管理](#3) 部分部署代码即可。
## 二、应用配置
### <span id="21">1、消费配置</span>
配置文件 ```/config/consumer.php```，设置Kafka的消费配置
```php
<?php
$consumer = [
// 检测消费状态的触发数, 单位为秒
'check_status_interval_seconds' => 2,
// broker服务器列表,如果多个则以逗号分割，如192.168.0.18:9092,192.168.0.18:9093
'broker_list_string' => '192.168.0.18:9092',
// 消费分区
'partition' => 0,
// 消费超时时间, 单位毫秒
'timeout_millisecond' => 2 * 1000,
// 消费组id
'group_id' => '',
// 消费主题
'topic' => '',
// worker的最大进程数
'max_worker_count' => 10,
];
```
### <span id="22">2、数据库配置</span>
配置文件 ```/config/db.php```，设置访问数据库的配置
```php
<?php
$db = [
'database' => [
'host' => '数据库地址',
'port' => 3306,
'username' => '用户名',
'password' => '密码',
'database' => '数据库',
'charset' => 'utf8mb4',
]
];
```
### <span id="23">3、ES配置</span>
配置文件 ```/config/es.php```，设置访问ES的配置
```php
<?php
$es = [
'host' => 'ES服务host',
'port' => 'ES服务端口',
'user_password' => 'ES服务凭证',
'doc_type' => '_doc'
];
```
### <span id="24">4、日志配置</span>
> 在```/start.sh```启动脚本中，```docker run -v ...``` 会把容器中配置的日志目录挂载到本机相应目录中
配置文件 ```/config/log.php```，配置了不同日志级别的文件路径，如下所示
```php
<?php
$log = [
'debug' => '/home/log/esupdater/debug.log',
'info' => '/home/log/esupdater/info.log',
'slow' => [
'millisecond' => 500, // work进程处理耗时超过500ms则记录慢日志
'path' => '/home/log/esupdater/slow.log',
],
'warning' => '/home/log/esupdater/warning.log',
'error' => '/home/log/esupdater/error.log',
'fatal' => '/home/log/esupdater/fatal.log',
];
```
### <span id="25">5、事件配置</span>
配置文件 ```/config/event.php```，如下所示
- Key ：```数据库名.表名```
- Value : ```Handler```
表示当此数据表的数据更新时，由对应的```Handler```处理
```php
<?php
$event = [
'alpha.user' => '\app\alpha\user\UserHandler',
];
```
#### <span id="251">(1) 高级事件配置</span>
上面的这种Key所对应的Value为字符串的配置方式，是一种简单的自动回调配置。 如果Value是Map时，就会使用高级事件配置。
这个Map会再次以如```onInsert```、```onUpdate```、```onDelete```不同的事件为key，value则由以下几种回调函数组成，分别为 ：
- ```filter``` 过滤器 \[可选\] ：实现对Canal数据的过滤处理、对事件回调的拦截
- ```callback``` 事件回调 \[可选\] ：实现事件的回调处理
- ```finally``` 末尾执行 \[可选\] ：实现事件的兜底处理，可用于统计数据、记录日志等
关于高级事件配置可以参考 [高级配置示例](../config/event.php) 。
### <span id="26">6、单测配置</span>
配置文件 ```config/test.php```，如下所示
```php
<?php
$test = [
// 所有单元测试用例所在的统一目录
'testcases_directory' => 'test/testcases/',
];
```
## <span id="3">三、系统变量</span>
在```/.env```文件中记录了服务所需要的所有系统变量，在执行```install.sh```安装脚本时完成系统变量的设置，并由```/framework/Environment.php```类解析并处理。
## <span id="4">四、部署管理</span>
### <span id="41">1、容器化部署</span>
如果部署过程中出错，请参考[容器部署帮助](HELP.md#3)文档。
#### <span id="411">(1) 启动</span>
```bash
bash ./start.sh
```
#### <span id="412">(2) 停止</span>
```bash
bash ./stop.sh
```
#### <span id="413">(3) 重启</span>
```bash
bash ./restart.sh
```
### <span id="42">2、运行时配置</span>
可以在```/start.sh```脚本中执行```docker run```时设置```核心数```、```目录挂载```等参数，请自定义修改。
如果需要设置更多的容器参数，可以参考[官方文档](https://docs.docker.com/config/containers/resource_constraints/) 。
| Id | 配置名称 | 配置参数 | 参数值 | 默认值 | 释义 |
| --- | :----: | :----: | :---: | :---: | :---: |
| 1 | 核心数 | --cpus | float | 1.5 | 设置允许的最大核心数 |
| 2 | CPU核心集 | --cpuset-cpus | int | 未设置 | 设置允许执行的CPU核心 |
| 3 | 内存核心集 | --cpuset-mems | int | 未设置 | 设置使用哪些核心的内存 |
| 4 | 目录挂载 | -v | string | /home/log/esupdater | 设置容器挂载的目录 |
## <span id="5">五、单元测试</span>
根目录下的```/test```目录是单元测试目录，其中有一个```/test/run.php```入口文件，它会自动执行 [testcases_directory](HOWTOCODE.md#36) 目录下所有的测试用例。
### <span id="51">1、手动测试</span>
```bash
php test/run.php
```
### <span id="52">2、自动测试</span>
```bash
cp test/prepare-commit-msg ./.git/hooks
chmod +x .git/hooks/prepare-commit-msg
# 此后提交代码会自动执行单元测试，只有单测成功才会允许提交代码
git add .
git commit -m "add: xxx"
```
如下图实际使用中，每次Commit代码会自动执行测试。
<img width="500" alt="img" src="https://user-images.githubusercontent.com/35942268/152677495-1aae134b-93b2-443f-b5cf-8daa719f35f6.png">
### <span id="53">3、添加用例</span>
在```test/testcases/app```目录下，先创建应用目录（如```alpha```），然后在此目录下以```Test*```开头创建单测文件即可，具体内容可参考 [TestUserService](../test/testcases/app/alpha/TestUserService.php) 单测文件
### <span id="54">4、测试报告</span>
在测试运行结束后，会自动生成一个测试报告```/test/report/index.html```文件，<a href="https://wgrape.github.io/esupdater/report.html">点击这里</a>查看报告
================================================
FILE: doc/CHANGELOG.md
================================================
### 2、版本 ：v2.0.5
发布日期 ：2021-01-11
#### 修复
- [重大更新更容易上手-添加系统变量机制](https://github.com/WGrape/esupdater/commit/551c5585d878ec3ebe72c142cb294aa37c64ed29)
- [完善启动脚本中判断是否已启动的逻辑](https://github.com/WGrape/esupdater/commit/495a0bf15ed8dbe696bccd41c1c2095ab10eea7d)
#### 特性
- [完善重要更新-简化你的上手流程](https://github.com/WGrape/esupdater/commit/2a430f008f0f604624ea0fa6fbd07954b2e4f152)
- [重大更新更容易上手-添加系统变量机制](https://github.com/WGrape/esupdater/commit/551c5585d878ec3ebe72c142cb294aa37c64ed29)
- [内置自动创建kafka容器的脚本](https://github.com/WGrape/esupdater/commit/49f6327cab60102172d071cce84105f6978200b3)
#### 完善
- 多个文档的完善优化
### 1、版本 ：v2.0.0-beta
发布日期 ：2021-01-04
#### 修复
- 修复制作镜像失败时误提示成功的bug ：[PR](https://github.com/WGrape/esupdater/pull/37)
#### 特性
- 支持composer ：[PR](https://github.com/WGrape/esupdater/pull/37)
#### 完善
- 完善文档
================================================
FILE: doc/CONTRIBUTING.md
================================================
### 目录
- [1、如何报告问题](#1)
- [2、如何提交PR](#2)
- [3、如何理解项目](#3)
- [4、代码提交规约](#4)
- [1、单测通过](#41)
- [2、commit message 规范](#42)
- [3、分支管理](#43)
- [5、打包Release](#5)
- [6、版本对比](#6)
- [7、项目数据](#7)
## <span id="1">1、如何报告问题</span>
如果在了解和使用过程中，有任何疑问，非常欢迎提出宝贵的 [Issue](https://github.com/WGrape/esupdater/issues/new)
## <span id="2">2、如何提交PR</span>
PR的提交不限制范围，如代码、文档等修改均在允许范围内，可 [参考这里](https://github.com/WGrape/esupdater/commit/186e229308463aa745c6b1cbfd02f77bc62ab9d4) 的PR提交
## <span id="3">3、如何理解项目</span>
在[HOWTOCODE](HOWTOCODE.md)文档中介绍了详细的实现原理和设计，帮助你了解项目
## <span id="4">4、代码提交规约</span>
在提交代码前，至少需要做到以下几项
### <span id="41">(1) 单测通过</span>
整个项目的单元测试必须通过
### <span id="42">(2) commit message 规范</span>
规范使用如```fix: 修复Logger中记录日志时间错误的bug```这种组合的提交规范
- fix: 修复bug相关
- doc: 文档完善相关
- refactor: 重大功能重构
- feat: 新功能、新组件等
- test: 新增测试或测试相关的修改
- style: 调整代码格式等对功能和性能无较大影响的修改
- chore: 构建过程或辅助工具的变动，如dockerfile的修改
### <span id="43">(3) 分支管理</span>
```v1```版本的开发提交到```v1.x```分支，```v2```版本的开发提交到```v2.x```分支，且```CI```检查通过
### <span id="43">(4) 提交内容注释</span>
对于重要代码部分，请以评论的方式写清楚原因，可以参考 [test: 添加环境变量的测试用例](https://github.com/WGrape/esupdater/commit/f9e4b4fe867889f398f3ec175af0d5dfc16de4a0) 、[feat: 支持Composer和修复制作镜像失败时误提示成功的bug](https://github.com/WGrape/esupdater/pull/37/files#r800161416)
## <span id="5">5、打包Release</span>
基于```v1.x```和```v2.x```分支分别打包不同的Release版本。
## <span id="6">6、版本对比</span>
### (1) Composer
| 主版本号 | Composer | 优势 | 劣势 |
| --- | :----: | :----: | :----: |
| v1.x | 不支持 | 不需要安装Composer也可以用 | 可能无法正常使用外部依赖 |
| v2.x | 支持 | 可以方便的调用外部依赖 | 本地开发时需要安装Composer |
## <span id="7">7、项目数据</span>
<a href="https://starchart.cc/WGrape/esupdater"><img src="https://starchart.cc/WGrape/esupdater.svg" width="700"></a>
================================================
FILE: doc/HELP.md
================================================
### 目录
- [一、安装过程帮助](#1)
- [1、Git命令不存在](#11)
- [2、无法正常Clone](#12)
- [3、运行安装脚本出错](#13)
- [4、Windows系统如何安装](#14)
- [5、out of capacity错误](#15)
- [二、镜像制作帮助](#2)
- [1、Docker命令不存在](#21)
- [2、无法连接Docker](#22)
- [3、pecl.php.net更新失败](#23)
- [三、容器部署帮助](#3)
- [1、phpkafka镜像不存在](#31)
- [2、/home/log/esupdater/目录不存在或无权限写](#32)
- [3、KafkaConsumer创建失败](#33)
- [4、Consumer highLevelConsuming fetch timeout](#34)
- [四、版本选择](#4)
## <span id="1">一、安装过程帮助</span>
请先通过 ```git clone``` 或 [下载Release包](https://github.com/WGrape/esupdater/releases) 的方式获取项目
### <span id="11">1、Git命令不存在</span>
检查git是否已正常安装，查看 [如何安装Git](https://git-scm.com/book/zh/v2/%E8%B5%B7%E6%AD%A5-%E5%AE%89%E8%A3%85-Git)
### <span id="12">2、无法正常Clone</span>
针对于常见无法正常Clone的问题，有如下几种解决方案
- 尝试使用```https```方式进行clone
- 检查网络和网速是否正常，或使用 [esupdater国内版仓库](https://gitee.com/WGrape/esupdater)
### <span id="13">3、运行安装脚本出错</span>
如果获取项目已经成功，但是在运行```install.sh```安装脚本阶段出错的话，有如下几种解决方案
- 制作镜像过程出错 ：参考 [镜像制作帮助](#2) 文档
- 提示```Error response from daemon: Get "https://registry-1.docker.io/v2/": EOF``` 错误 ：检查网络连接，关闭网络代理即可
### <span id="14">4、Windows系统如何安装</span>
目前暂不支持直接在Windows系统上操作，可以选择在Linux虚拟机、Docker环境中安装，如使用 <a href="https://labs.play-with-docker.com/">在线Docker网站</a>
### <span id="15">5、out of capacity错误</span>
We are really sorry but we are out of capacity and cannot create your session at the moment. Please try again later.
访问 ```https://labs.play-with-docker.com/``` 时如果出现上述错误，暂无解决方案，需要多尝试几次。
## <span id="2">二、镜像制作帮助</span>
在```install/image```目录中已提供了开箱可用的```phpkafka```镜像文件，只需要简单的执行```bash make.sh```命令即可快速生成```phpkafka```镜像。
自带的```install/image/Dockerfile```镜像文件，已经过多台Unix机器上的多次测试，均可以顺利的成功制作。但是不排除在特殊情况下会存在制作失败的情况，下面会总结出常见的错误和解决方案。
如果还无法解决，可以直接使用已经推到 [Docker Hub](https://hub.docker.com/repository/docker/lvsid/phpkafka) 上的```lvsid/phpkafka:v1.0``` ，修改方式如下 ：
```bash
# 打开根目录下的Dockerfile文件
cd esupdater
vi Dockerfile
# 把 FROM phpkafka 替换为 FROM lvsid/phpkafka:v1.0 即可
```
### <span id="21">1、Docker命令不存在</span>
安装镜像必须依赖于```Docker```，所以请务必成功安装```Docker```，否则无法创建镜像。
### <span id="22">2、无法连接Docker</span>
#### (1) 错误提示
```text
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
```
#### (2) 错误原因
本地Docker服务未启动
#### (3) 解决方案
开启本地Docker服务即可
### <span id="23">3、pecl.php.net更新失败</span>
#### (1) 错误提示
```text
Updating channel "pecl.php.net"
Channel "pecl.php.net" is not responding over http://, failed with message: File http://pecl.php.net:80/channel.xml not valid (redirected but no location)
Trying channel "pecl.php.net" over https:// instead
Cannot retrieve channel.xml for channel "pecl.php.net" (File https://pecl.php.net:443/channel.xml not valid (redirected but no location))
```
#### (2) 错误原因
网络异常，无法正常连接```pecl.php.net```
#### (3) 解决方案
检查网络是否正常或关掉网络代理
## <span id="3">三、容器部署帮助</span>
### <span id="31">1、phpkafka镜像不存在</span>
> pull access denied for phpkafka, repository does not exist ... ...
出现这种错误是因为跳过了安装步骤，直接执行部署操作导致的。
由于容器化部署方案依赖于```phpkafka```镜像，所以如果提示此镜像不存在，请先参考[快速使用-开始安装](../README.md#22)文档执行安装操作，或直接手动执行```cd image && bash make.sh```完成镜像的制作。
### <span id="32">2、/home/log/esupdater/目录不存在或无权限写</span>
由于容器默认会把目录挂载到宿主机的 ```/home/log/esupdater/``` 相同目录下，所以请确保宿主机有此目录和写入权限
或者也可以选择修改[容器的运行时配置](APPLICATION.md#32)中的```目录挂载```，修改方式如下
```bash
vi start.sh
# 替换以下内容
docker run --cpus=1.5 --name esupdaterContainer -d -v {你的宿主机目录}:/home/log/esupdater/ esupdater
```
### <span id="33">3、KafkaConsumer创建失败</span>
> Consumer failed to new KafkaConsumer: "group.id" must be configured
如果在```fatal.log```中出现```KafkaConsumer```创建失败的报错，请检查```consumer.php```中的```kafka```服务配置是否可以正常连接
### <span id="34">4、Consumer highLevelConsuming fetch timeout</span>
重新启动后可能会报一段时间的```Consumer highLevelConsuming fetch timeout```问题，持续约为2~5秒。
原因 ：重启后需要重新连接```kafka```消费数据，在第一次连接时需要建立TCP和一些额外资源等，所以导致耗时相对较长。
## <span id="4">四、版本选择</span>
项目版本号规则为```主版本```-```次版本```-```修订号```，其中主版本主要做重大功能升级，次版本主要做性能和功能优化，修订号则做问题修复和完善。
所以```次版本```和```修订号```建议选择最新稳定版本的 [Release包](https://github.com/WGrape/esupdater/releases) ，```主版本```则根据以下对比信息选择合适的即可，可以查看更详细的 [版本对比](doc/CONTRIBUTING.md#5) 信息。
| 主版本号 | Composer |
| --- | :----: |
| v1.x | 不支持 |
| v2.x | 支持 |
================================================
FILE: doc/HOWTOCODE.md
================================================
### 目录
- [一、架构设计](#1)
- [1、基于Canal](#11)
- [2、ES文档更新](#12)
- [3、完整架构](#13)
- [二、底层原理](#2)
- [1、生命周期](#21)
- [2、命令执行](#22)
- [3、binlog数据处理过程](#23)
- [4、文件目录规范](#24)
- [5、程序设计规范](#25)
- [三、部署过程](#3)
- [四、参考文档](#4)
## <span id="1">一、架构设计</span>
### <span id="11">1、基于Canal</span>
Canal提供了数据库增量订阅与消费的功能，不需要业务代码的侵入和依赖，通过读取MQ，即可获取到数据库的增量更新
### <span id="12">2、ES文档更新</span>
对于数据源为数据库（如MySQL）的ES文档更新，主要有全量更新和增量更新两种方案
- 全量更新 ：脚本全量查询数据库，统一写入至ES中
- 增量更新 ：双写或读取```binlog```，实现ES的增量更新
ESUpdater就是读取```binlog```，实现ES文档增量更新的一种解决方案
### <span id="13">3、完整架构</span>
ESUpdater提供了从消费Kafka中的数据库增量数据，到ES文档增量更新的一个完整业务框架，方便业务的扩展。
- ```Consumer``` 进程 ：订阅Kafka队列，实时获取数据库的增量变更
- ```Worker``` 进程 ：操作业务逻辑，将数据更新至ES文档
<img src="https://user-images.githubusercontent.com/35942268/147027126-1df83ddf-8698-44dd-a988-5499f7eeb063.png" width="625">
## <span id="2">二、底层原理</span>
ESUpdater的核心由```Consumer```进程和```Worker```进程组成，其中根目录下的```/esupdater.php```为入口文件
### <span id="21">1、生命周期</span>
```Consumer```进程和```Worker```进程的生命周期都是由命令控制
#### <span id="211">(1) Consumer</span>
```Consumer```进程由```php esupdater.php start```命令启动，由```php esupdater.php stop```命令停止
#### <span id="212">(2) Worker</span>
当```Consumer```进程从Kafka中拿到消息后，会通过```exec```的方式执行```php esupdater work```命令，以启动一个新的PHP进程，即```Worker```进程。
```Worker```进程会分为后台和非后台两种执行方式，使用哪种执行方式取决于当前```Worker```进程的数量，如果少于配置的```max_worker_count```会使用后台执行的方式，否则使用非后台执行的方式。通过这种方式可以在加快消费速度的同时，保证稳定性。
所以Worker进程的启动完全由```Consumer```控制，如果想要停止```Worker```进程，必须先停止```Consumer```进程，然后等待```Worker```进程正常执行结束即可
### <span id="22">2、命令执行</span>
#### <span id="221">(1) start</span>
当使用```php esupdater.php start```命令时，会启动一个进程，这个进程会以阻塞主进程的方式订阅Kafka消息，所以这个进程叫做```Consumer```进程
```Consumer```进程启动后会先在```/runtime```目录下写```/runtime/esupdater-consumer.pid```文件和```esupdater-consumer.status```文件，分别记录进程它的进程ID和消费状态```start```。
在```Consumer```进程消费kafka消息的同时，会每隔配置的```check_status_interval_seconds```时间检测一次消费状态（```esupdater-consumer.status```文件），当消费状态变为```stop```时，进程会停止消费，此时```Consumer```进程会完全结束。
#### <span id="222">(2) stop</span>
当使用```php esupdater.php stop```命令时，会启动一个进程，这个进程会向```/runtime/esupdater-consumer.status```文件中写入```stop```指定。
然后每隔一秒钟就会检测```Consumer```进程和```Worker```进程是否都已经完全结束，如果已经检测10秒钟还未完全结束就会通知停止失败，否则停止成功。
#### <span id="223">(3) work</span>
当```Consumer```进程使用```php esupdater work```命令启动```Worker```进程时，```Worker```进程会记录下```/runtime/esupdater-worker-{pid}.pid```进程ID文件，只有当结束后才会删除此文件。
### <span id="23">3、binlog数据处理过程</span>
处理过程为```binlog => canalData => urlencode(canalData)```，可以参考文件 [/framework/Canal.php](../framework/Canal.php)
1. Canal将```binlog```数据解析为```json```格式并投递至kafka
2. Consumer进程消费kafka，使用```urlencode```方式编码获取到的消息数据
3. Consumer进程把编码后的消息数据，传递至Worker进程
4. Worker进程再依次拆解数据即可
### <span id="24">4、文件目录规范</span>
#### <span id="241">(1) 目录结构
- ```app```目录 ：应用目录
- ```config```目录 ：项目的唯一配置入口
- ```doc```目录 ：项目文档目录
- ```framework```目录 ：项目的核心框架目录
- ```install```目录 ：安装目录
- ```runtime```目录 ：服务运行时产生的中间文件目录，如PID文件，但不包括日志文件。设计思想基于[/proc/](https://en.wikipedia.org/wiki/Procfs)
- ```test```目录 ：单元测试目录
- ```/```目录 ：根目录下存放所有上述目录，和必要的一级文件如```.gitignore```文件
#### <span id="242">(2) 文件规范
- ```shell```脚本不能省略```.sh```后缀，且统一以```bash xxx.sh```的方式执行
- 文档统一以大写英文命名，如```README.md``` / ```HELP.md```
### <span id="25">5、程序设计规范</span>
关于设计规范可以参考文章 [漫谈编程之编程规范](https://github.com/WGrape/Blog/issues/25)
- 调用类的时候使用命名空间前缀，不使用在头部声明```use```的方式
## <span id="3">三、部署过程</span>
> 容器化部署方案依赖于```phpkafka```镜像，所以请确保```phpkafka```镜像已经生成。为了避免重复构建耗时，建议把```phpkafka```镜像推到Docker远程仓库中。
容器构建主要通过根目录下的```/Dockerfile```镜像文件，它会基于```phpkafka```镜像构建一个新的镜像，名为```esupdater```。
### <span id="31">1、启动</span>
当执行如下命令时，会使用```/Dockerfile```文件创建```esupdater```镜像，并创建```esupdaterContainer```容器，最后通过在容器中执行```php esupdater.php start```命令实现服务的启动
```bash
bash ./start.sh
```
启动成功后，除命令行输出```Start success```外，在宿主机```/home/log/esupdater/info.log.{date}```日志中会输出启动日志，如下图所示
<img width="700" alt="img" src="https://user-images.githubusercontent.com/35942268/147385923-80cb29e5-225b-4c83-8637-2513d3e17a1d.png">
### <span id="32">2、停止</span>
当执行以下命令时，会先在容器中执行```php esupdater.php stop```命令，等待容器内```Consumer```进程和```Worker```进程全部停止后，删除镜像和容器
```bash
bash ./stop.sh
```
停止成功后，除命令行输出```Stop success```外，同样的在宿主机```/home/log/esupdater/info.log.{date}```日志中会输出停止成功日志，如下图所示
<img width="700" alt="img" src="https://user-images.githubusercontent.com/35942268/147386373-dd4b66ff-60b8-43ab-8c5a-f03148258f27.png">
### <span id="33">3、重启</span>
当执行以下命令时，会先执行```bash stop.sh```命令，再执行```bash start.sh```命令，以防止出现重复启动的问题
```bash
bash ./restart.sh
```
## <span id="4">四、参考文档</span>
- 有关```php-rdkafka```的配置可以 [参考文档](https://github.com/arnaud-lb/php-rdkafka)
- 有关```librdkafka```的配置可以 [参考文档](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)
- 有关```PHP Kafka```类的使用可以 [参考文档](https://arnaud.le-blanc.net/php-rdkafka-doc/phpdoc/class.rdkafka-consumertopic.html)
================================================
FILE: doc/README.md
================================================
### What is this directory
It's the directory of different docs.
================================================
FILE: esupdater.php
================================================
<?php
/**
* The main file of esupdater.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
include_once 'bootstrap.php';
$command = strtolower(isset($argv[1]) ? $argv[1] : '');
if (empty($command)) {
echo "Command empty!\n";
return;
}
$manager = new \framework\Manager();
switch ($command) {
case "start":
$manager->commandStart();
break;
case "stop":
$success = $manager->commandStop();
echo "{$success}\n";
break;
case "work":
$canalData = isset($argv[2]) ? $argv[2] : "";
if (empty($canalData)) {
return;
}
$success = $manager->commandWork($canalData);
echo "{$success}\n";
break;
default:
echo "Not support command: {$command}\n";
return;
}
================================================
FILE: framework/Canal.php
================================================
<?php
/**
* The common usages of canal.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Canal
{
/**
* Encode the canal data to string.
*
* @param string $unEncodedCanalData the json format data in kafka queue (canal put it into kafka)
*
* @return string
*/
public function encode(string $unEncodedCanalData): string
{
return urlencode($unEncodedCanalData);
}
/**
* Decode the canal data to string.
*
* @param string $encodedCanalData the urlencoded canal data
*
* @return string
*/
public function decode(string $encodedCanalData): string
{
return urldecode($encodedCanalData);
}
/**
* Parse the canal data to array.
*
* @param string $encodedCanalData the urlencoded canal data
*
* @return array
*/
public function parse(string $encodedCanalData): array
{
return json_decode($this->decode($encodedCanalData), true);
}
/**
* Check the canal data format.
*
* @param array $parsedCanalData
*
* @return bool
*/
public function checkParsedCanalData(array $parsedCanalData): bool
{
if (empty($parsedCanalData) || empty($parsedCanalData['data'])) {
return false;
}
if (!isset($parsedCanalData['database']) || !isset($parsedCanalData['table']) || !isset($parsedCanalData['type'])) {
return false;
}
if (!isset($parsedCanalData['id']) || !isset($parsedCanalData['ts'])) {
return false;
}
return true;
}
}
================================================
FILE: framework/Consumer.php
================================================
<?php
/**
* The consumer process.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Consumer
{
const TIMER_MARK = 'consume';
const CONSUMER_EXIT_WITH_EMPTY_STRING = '';
const START_FLAG_STRING = 'start';
const STOP_FLAG_STRING = 'stop';
private $checkStatusIntervalSeconds;
private $brokerListString;
private $partition;
private $timeoutMillisecond;
private $groupId;
private $topic;
private $maxWorkerCount;
public function __construct($consumer)
{
$this->checkStatusIntervalSeconds = (isset($consumer['check_status_interval_seconds']) && !empty($consumer['check_status_interval_seconds'])) ? $consumer['check_status_interval_seconds'] : 2;
$this->brokerListString = (isset($consumer['broker_list_string']) && !empty($consumer['broker_list_string'])) ? $consumer['broker_list_string'] : ($this->getLocalIP() . ':9092');
$this->partition = (isset($consumer['partition']) && !empty($consumer['partition'])) ? $consumer['partition'] : 0;
$this->timeoutMillisecond = (isset($consumer['timeout_millisecond']) && !empty($consumer['timeout_millisecond'])) ? $consumer['timeout_millisecond'] : 2 * 1000;
$this->groupId = (isset($consumer['group_id']) && !empty($consumer['group_id'])) ? $consumer['group_id'] : 'default_group';
$this->topic = (isset($consumer['topic']) && !empty($consumer['topic'])) ? $consumer['topic'] : 'default_topic';
$this->maxWorkerCount = (isset($consumer['max_worker_count']) && !empty($consumer['max_worker_count'])) ? $consumer['max_worker_count'] : 100;
}
/**
* Get localIP by ifconfig command
* @DEPRECATED: It would get error local IP in docker container.
* @return string
*/
public function getLocalIPByIfconfig(): string
{
$localIP = '127.0.0.1';
$command = "ifconfig -a | grep inet | grep -v 127.0.0.1 | grep -v inet6 | grep '192.168' | head -n 1 | awk '{print $2}' | tr -d \"addr:\"";
exec($command, $output);
if (isset($output[0])) {
$localIP = $output[0];
if (stripos($localIP, '192.168') !== 0) {
$localIP = '127.0.0.1';
}
}
return $localIP;
}
/**
* Get local host IP
* @return string
*/
public function getLocalIP(): string
{
$localIP = \framework\Environment::getSystemVariable('ESUPDATER_LOCAL_IP');
if (empty($localIP)) {
$localIP = '127.0.0.1';
}
return $localIP;
}
/**
* Whether need to stop consume or not
* @return bool
*/
public function isNeedStop(): bool
{
return file_get_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE) === self::STOP_FLAG_STRING;
}
/**
* Whether need to check status or not
* @return bool
*/
public function isNeedCheckStatus(): bool
{
$millisecond = \framework\Timer::elapsed(self::TIMER_MARK);
return ($millisecond / 1000) >= $this->checkStatusIntervalSeconds;
}
/**
* Get property
* @param string $propertyName
* @return string | false | array | int | mixed
*/
public function getProperty(string $propertyName)
{
return property_exists($this, $propertyName) ? $this->$propertyName : '';
}
/**
* DEPRECATED: low level consuming, but now it's not available
* @param false $onlyForTest
* @return string
*/
public function lowLevelConsuming($onlyForTest = false): string
{
// Create consumer config object
$consumerConfigObject = new \RdKafka\Conf();
$consumerConfigObject->set('group.id', $this->groupId);
// Create topic config object
$topicConfigObject = new \RdKafka\TopicConf();
$topicConfigObject->set('auto.offset.reset', 'smallest');
// Create consumer object
$consumerObject = new \RdKafka\Consumer($consumerConfigObject);
$consumerObject->addBrokers($this->brokerListString);
// Create topic object
$topicObject = $consumerObject->newTopic($this->topic, $topicConfigObject);
// Consume start from last offset
$topicObject->consumeStart($this->partition, RD_KAFKA_OFFSET_STORED);
while (true) {
$message = $topicObject->consume($this->partition, $this->timeoutMillisecond);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
if (is_null($message) || empty($message->payload)) {
echo "Message is null or payload is empty\n";
continue;
}
var_dump($message);
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
echo "No more messages\n";
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
echo "Timeout\n";
break;
default:
echo "Unknown message error\n";
break;
}
}
}
/**
* High level consuming
* @param false $onlyForTest
* @return string
*/
public function highLevelConsuming($onlyForTest = false): string
{
$canal = new Canal();
$manager = new Manager();
// Create topic config object
$topicConfigObject = new \RdKafka\TopicConf();
$topicConfigObject->set('request.required.acks', true);
$topicConfigObject->set('auto.commit.interval.ms', 100);
$topicConfigObject->set('auto.offset.reset', 'smallest');
// Create consumer config object
$consumerConfigObject = new \RdKafka\Conf();
$consumerConfigObject->setRebalanceCb(function (\RdKafka\KafkaConsumer $kafka, $err, array $partitions = null) {
switch ($err) {
case RD_KAFKA_RESP_ERR__ASSIGN_PARTITIONS:
$kafka->assign($partitions);
break;
case RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS:
$kafka->assign(NULL);
break;
default:
Logger::logError("Consumer setRebalanceCb occurs exception: {$err}");
}
});
$consumerConfigObject->set('group.id', $this->groupId);
$consumerConfigObject->set('metadata.broker.list', $this->brokerListString);
$consumerConfigObject->setDefaultTopicConf($topicConfigObject);
// Create consumer object
try {
$consumerObject = new \RdKafka\KafkaConsumer($consumerConfigObject);
$consumerObject->subscribe([$this->topic]);
} catch (\Throwable $e) {
Logger::logFatal("Consumer failed to new KafkaConsumer: " . $e->getMessage());
return self::CONSUMER_EXIT_WITH_EMPTY_STRING;
}
// Timer start
\framework\Timer::start(self::TIMER_MARK);
// Consuming loop
while (true) {
// Check consume status
if ($this->isNeedCheckStatus() && $this->isNeedStop()) {
Logger::logInfo('Consumer need to stop');
return self::CONSUMER_EXIT_WITH_EMPTY_STRING;
}
// Fetch message
$message = $consumerObject->consume($this->timeoutMillisecond);
switch ($message->err) {
case RD_KAFKA_RESP_ERR_NO_ERROR:
if (is_null($message) || empty($message->payload)) {
Logger::logDebug('Consumer fetch message is null or payload is empty');
continue;
}
if ($onlyForTest) {
return $message->payload;
}
$logFile = CREATE_WORKER_LOG_FILE;
$canalData = $canal->encode($message->payload);
$count = $manager->getRunningWorkersCount();
if ($count !== false && $count < ($this->maxWorkerCount - 1)) {
Logger::logDebug("Consumer handle message: create non-block worker process, current count is {$count}");
exec("nohup php esupdater.php work {$canalData} >> {$logFile} &");
} else {
Logger::logDebug("Consumer handle message: create block worker process, current count is {$count}");
exec("php esupdater.php work {$canalData} >> {$logFile}");
}
break;
case RD_KAFKA_RESP_ERR__PARTITION_EOF:
Logger::logDebug('Consumer highLevelConsuming fetch no more messages');
break;
case RD_KAFKA_RESP_ERR__TIMED_OUT:
Logger::logDebug('Consumer highLevelConsuming fetch timeout');
break;
default:
Logger::logError('Consumer highLevelConsuming catch unknown message error');
break;
}
}
}
}
================================================
FILE: framework/Environment.php
================================================
<?php
/**
* The manager of different environment variables.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Environment
{
/**
* Variable container.
* @var array
*/
public static $variableContainer = [];
/**
* Get system variable.
* @param $variable
* @return string
*/
public static function getSystemVariable($variable): string
{
$result = '';
if (empty(self::$variableContainer)) {
self::parseEnvFile();
}
if (isset(self::$variableContainer[$variable])) {
$result = self::$variableContainer[$variable];
}
return $result;
}
/**
* Parse the .env file.
*/
public static function parseEnvFile()
{
$file = ENVIRONMENT_FILE;
$handler = fopen($file, 'r+');
while (!feof($handler)) {
$content = trim(fgets($handler));
$slices = explode('=', $content);
if (count($slices) === 2) {
self::$variableContainer[$slices[0]] = $slices[1];
}
}
}
}
================================================
FILE: framework/Listener.php
================================================
<?php
/**
* This is an event listener for calling(dispatching) handler when insert/update/delete event of database is triggered.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Listener
{
/**
* Database event type: insert.
*/
const TYPE_INSERT = 'INSERT';
/**
* Database event type: update.
*/
const TYPE_UPDATE = 'UPDATE';
/**
* Database event type: delete.
*/
const TYPE_DELETE = 'DELETE';
/**
* Timer mark of work
*/
const TIMER_MARK = 'work';
/**
* Find and
* @param string $canalData
*/
public function dispatch(string $canalData)
{
// Timer start.
Timer::start(self::TIMER_MARK);
// Parse and check canal data.
$canalParser = new Canal();
$parsedCanalData = $canalParser->parse($canalData);
Logger::setLogIdByParsedCanalData($parsedCanalData);
if (!$canalParser->checkParsedCanalData($parsedCanalData)) {
Logger::logError("Check canal data error");
return;
}
// Check the key is valid.
global $event;
$database = $parsedCanalData['database'];
$table = $parsedCanalData['table'];
$key = "{$database}.{$table}";
if (!isset($event[$key])) {
Logger::logError("Not found the valid event config for key: key={$key}");
return;
}
$onWhichEvent = false;
switch ($parsedCanalData['type']) {
case self::TYPE_INSERT:
$onWhichEvent = 'onInsert';
break;
case self::TYPE_UPDATE:
$onWhichEvent = 'onUpdate';
break;
case self::TYPE_DELETE:
$onWhichEvent = 'onDelete';
break;
}
$isAutoCallback = is_string($event[$key]);
if ($isAutoCallback) {
$this->autoCallback($key, $onWhichEvent, $parsedCanalData);
} else {
$this->manualCallback($key, $onWhichEvent, $parsedCanalData);
}
global $log;
$cost = Timer::elapsed(self::TIMER_MARK);
if (isset($log['slow']['millisecond']) && $cost > $log['slow']['millisecond']) {
Logger::logSlow("Work slow: key={$key}, onWhichEvent={$onWhichEvent}, cost={$cost}ms");
}
}
/**
* Auto callback
* @param string $key
* @param string $onWhichEvent
* @param array $parsedCanalData
*/
public function autoCallback(string $key, string $onWhichEvent, array $parsedCanalData)
{
global $event;
if ($onWhichEvent === false || !method_exists($event[$key], $onWhichEvent)) {
Logger::logError("Not found the valid auto callback: key={$key}, type={$parsedCanalData['type']}");
return;
}
$handler = new $event[$key];
$handler->$onWhichEvent($parsedCanalData);
}
/**
* Manual callback
* @param string $key
* @param string $onWhichEvent
* @param array $parsedCanalData
*/
public function manualCallback(string $key, string $onWhichEvent, array $parsedCanalData)
{
global $event;
if ($onWhichEvent === false || !isset($event[$key][$onWhichEvent]) || !is_array($event[$key][$onWhichEvent])) {
Logger::logError("Not found the valid manual callback: key={$key}, type={$parsedCanalData['type']}");
return;
}
$filterResult = true;
if (isset($event[$key][$onWhichEvent]['filter']) && is_callable($event[$key][$onWhichEvent]['filter'])) {
$filterResult = $event[$key][$onWhichEvent]['filter']($parsedCanalData);
if ($filterResult) {
$parsedCanalData = $filterResult;
}
}
$callbackResult = false;
if ($filterResult && isset($event[$key][$onWhichEvent]['callback']) && is_callable($event[$key][$onWhichEvent]['callback'])) {
$callbackResult = $event[$key][$onWhichEvent]['callback']($parsedCanalData);
}
if (isset($event[$key][$onWhichEvent]['finally']) && is_callable($event[$key][$onWhichEvent]['finally'])) {
$event[$key][$onWhichEvent]['finally']($filterResult, $callbackResult);
}
}
}
================================================
FILE: framework/Logger.php
================================================
<?php
/**
* This is the only one logger when you need to output logs.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Logger
{
/**
* Support log level.
*/
const LEVEL_DEBUG = 'debug';
const LEVEL_INFO = 'info';
const LEVEL_SLOW = 'slow';
const LEVEL_WARNING = 'warning';
const LEVEL_ERROR = 'error';
const LEVEL_FATAL = 'fatal';
/**
* The unique id of log.
*
* @var string
*/
private static $logId;
/**
* The formula of logId.
*
* @var string
*/
private static $formula;
/**
* Setup the logId.
*
* @param string $logIdParam
*
* @param string $formulaParam
*/
public static function setLogId(string $logIdParam, string $formulaParam)
{
self::$logId = $logIdParam;
self::$formula = $formulaParam;
}
/**
* Setup the logId according the parsed canal data.
*
* @param array $parsedCanalData
*/
public static function setLogIdByParsedCanalData(array $parsedCanalData)
{
$database = isset($parsedCanalData['database']) ? $parsedCanalData['database'] : '';
$table = isset($parsedCanalData['table']) ? $parsedCanalData['table'] : '';
$type = isset($parsedCanalData['type']) ? $parsedCanalData['type'] : '';
$id = isset($parsedCanalData['id']) ? $parsedCanalData['id'] : 0;
$ts = isset($parsedCanalData['ts']) ? $parsedCanalData['ts'] : 0;
$formula = self::$formula = "{$database}+{$table}+{$type}+{$id}+{$ts}";
self::$logId = md5($formula);
}
/**
* Return the debug data.
*
* @return array[]
*/
public static function returnDumpData(): array
{
$result = [
'runtime_files' => [],
];
$handle = opendir(RUNTIME_PATH);
while ($handle && ($file = readdir($handle)) !== false) {
if (!is_file($file)) {
continue;
}
$result['runtime_files'][] = $file;
}
closedir($handle);
return $result;
}
/**
* Write log in debug mode.
*
* @param string $data the message to write
*/
public static function logDebug(string $data)
{
self::write(self::LEVEL_DEBUG, $data);
}
/**
* Write log in info mode.
*
* @param string $data the message to write
*/
public static function logInfo(string $data)
{
self::write(self::LEVEL_INFO, $data);
}
/**
* Write log in slow mode.
*
* @param string $data the message to write
*/
public static function logSlow(string $data)
{
self::write(self::LEVEL_SLOW, $data);
}
/**
* Write log in warning mode.
*
* @param string $data the message to write
*/
public static function logWarning(string $data)
{
self::write(self::LEVEL_WARNING, $data);
}
/**
* Write log in error mode.
*
* @param string $data the message to write
*/
public static function logError(string $data)
{
self::write(self::LEVEL_ERROR, $data);
}
/**
* Write log in fatal mode.
*
* @param string $data the message to write
*/
public static function logFatal(string $data)
{
self::write(self::LEVEL_FATAL, $data);
}
/**
* Get the path of log file in different level mode.
*
* @param $logLevel
*
* @return string
*/
public static function getLogFilePath($logLevel): string
{
global $log;
$date = date('Ymd');
if ($logLevel === self::LEVEL_SLOW) {
return "{$log[$logLevel]['path']}.{$date}";
}
return "{$log[$logLevel]}.{$date}";
}
/**
* The common method for writing log.
*
* @param $level
*
* @param $data
*/
public static function write($level, $data)
{
$logId = self::$logId;
$formula = self::$formula;
$datetime = date('Y-m-d H:i:s');
$header = "{$datetime} | logid = {$logId} = {$formula}";
$body = $data;
$footer = "";
if ($level === self::LEVEL_FATAL) {
$dumpData = self::returnDumpData();
$footer .= "----------Dump is data as follows----------\n";
$footer .= ("runtime_files: " . implode(', ', $dumpData['runtime_files']));
}
$content = "{$header}\n{$body}\n{$footer}\n";
$file = self::getLogFilePath($level);
file_put_contents($file, $content, FILE_APPEND | LOCK_EX);
}
}
================================================
FILE: framework/Manager.php
================================================
<?php
/**
* It's a process manager, manage all processes, including worker and consumer.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Manager
{
const COMMAND_WORK_SUCCESS = 'success';
const COMMAND_STOP_SUCCESS = 'success';
const COMMAND_STOP_FAILED = 'failed';
/**
* Get running workers count.
*
* @notice You must do something when this function return false
*
* @return false|int
*/
public function getRunningWorkersCount()
{
$handle = opendir(RUNTIME_PATH);
if (!$handle) {
return false;
}
$count = 0;
while (($file = readdir($handle)) !== false) {
if (strpos($file, RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX) === 0) {
++$count;
}
}
closedir($handle);
return $count;
}
/**
* Start consumer and blocking.
*/
public function startConsumerAndBlocking()
{
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, Consumer::START_FLAG_STRING);
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_PID_FILE, intval(getmypid()));
global $consumer;
(new Consumer($consumer))->highLevelConsuming();
}
/**
* Stop consumer by IPC(InterProcess Communication): Shared File.
*/
public function stopConsumerByIPC()
{
// Consumer process may not been created, so communicate only when consumer was created, or will be mistaken for stop failed
if (file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE)) {
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, Consumer::STOP_FLAG_STRING);
}
}
/**
* Whether consumer was stopped or not.
*
* @return bool
*/
public function isConsumerStopped(): bool
{
return !file_exists(RUNTIME_ESUPDATER_CONSUMER_PID_FILE) && !file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);
}
/**
* Whether all workers were stopped or not.
*
* @return bool
*/
public function isWorkersStopped(): bool
{
$count = $this->getRunningWorkersCount();
if ($count === false || $count > 0) {
return false;
}
return true;
}
/**
* Whether the current process is consumer process.
*/
public function isConsumerProcess()
{
$pid = getmypid();
$consumerPIDFile = RUNTIME_ESUPDATER_CONSUMER_PID_FILE;
if (file_exists($consumerPIDFile) && file_get_contents($consumerPIDFile) == $pid) {
return $consumerPIDFile;
}
return false;
}
/**
* Whether the current process is worker process.
*/
public function isWorkerProcess()
{
$pid = getmypid();
$workerPIDFile = RUNTIME_PATH . RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX . $pid . ".pid";
if (file_exists($workerPIDFile) && file_get_contents($workerPIDFile) == $pid) {
return $workerPIDFile;
}
return false;
}
/**
* Command: start.
*/
public function commandStart()
{
$formula = "start+" . date('Y-m-d H:i:s');
$logId = md5($formula);
Logger::setLogId($logId, $formula);
Logger::logInfo('Start esupdater');
// Start consumer and blocking ...
$this->startConsumerAndBlocking();
// After blocking, it means consumer is stopped, so remove consumer runtime files now
unlink(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);
unlink(RUNTIME_ESUPDATER_CONSUMER_PID_FILE);
}
/**
* Command: stop.
*
* @return string
*/
public function commandStop(): string
{
$formula = "stop+" . date('Y-m-d H:i:s');
$logId = md5($formula);
Logger::setLogId($logId, $formula);
Logger::logInfo('Stop esupdater');
// Stop consumer by IPC(InterProcess Communication)
$this->stopConsumerByIPC();
// Wait consumer and all workers were stopped, the max wait time is 10 seconds
$maxWaitSecond = 10;
$startTimestamp = time();
while (true) {
if ($this->isConsumerStopped() && $this->isWorkersStopped()) {
Logger::logInfo('Stop esupdater successfully');
return self::COMMAND_STOP_SUCCESS;
}
if ((time() - $startTimestamp) > $maxWaitSecond) {
Logger::logFatal('Failed to stop esupdater');
return self::COMMAND_STOP_FAILED;
}
sleep(1);
}
}
/**
* Command: work.
*
* @param string $canalData
*
* @return string
*/
public function commandWork(string $canalData): string
{
$pid = getmypid();
$workerPIDFile = RUNTIME_PATH . RUNTIME_ESUPDATER_WORKER_PID_FILE_PREFIX . $pid . ".pid";
file_put_contents($workerPIDFile, intval($pid));
(new \framework\Listener())->dispatch($canalData);
unlink($workerPIDFile);
return self::COMMAND_WORK_SUCCESS;
}
}
================================================
FILE: framework/Timer.php
================================================
<?php
/**
* Do something when you need to timing.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace framework;
class Timer
{
/**
* Store the benchmark of time.
*
* @var array
*/
public static $container = [];
/**
* Return nowSecondTimestampWithMicrosecond.
*
* @return float
*/
public static function getNowSecondTimestampWithMicrosecond(): float
{
list($microsecond, $secondTimestamp) = explode(' ', microtime());
return ((float)$microsecond + (float)$secondTimestamp);
}
/**
* Timer start and return nowSecondTimestampWithMicrosecond.
*
* @param string $mark
*
* @return float
*/
public static function start(string $mark): float
{
self::$container[$mark] = $nowSecondTimestampWithMicrosecond = self::getNowSecondTimestampWithMicrosecond();
return $nowSecondTimestampWithMicrosecond;
}
/**
* Return elapsed milliseconds from start.
*
* @param string $mark
*
* @return int
*/
public static function elapsed(string $mark): int
{
if (!isset(self::$container[$mark])) {
return 0;
}
$startSecondTimestampWithMicrosecond = self::$container[$mark];
$nowSecondTimestampWithMicrosecond = self::getNowSecondTimestampWithMicrosecond();
return (int)(($nowSecondTimestampWithMicrosecond - $startSecondTimestampWithMicrosecond) * 1000);
}
}
================================================
FILE: install/README.md
================================================
### What is this directory
It's the installation directory, you just run ```bash install.sh``` would help you to install automatically.
================================================
FILE: install/container/kafka.sh
================================================
#!/usr/bin/env bash
# Pull images first.
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
# Start zookeeper.
containerCount=0
for file in $(docker container ls -f name=zookeeperContainer -q)
do
((containerCount++))
done
if [ $containerCount -ne 0 ]; then
echo -e "Start zookeeperContainer already, is skipped."
else
docker run -d --name zookeeperContainer -p 2181:2181 -t wurstmeister/zookeeper
fi
# Start kafka server.
# use ifconfig command to get the local ip: 192.168.x.x not 127.0.0.1
localIP=$(ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|grep '192.168' |awk '{print $2}'|tr -d "addr:")
containerCount=0
for file in $(docker container ls -f name=kafkaContainer -q)
do
((containerCount++))
done
if [ $containerCount -ne 0 ]; then
echo -e "Start kafkaContainer already, is skipped."
else
docker run -d --name kafkaContainer -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=${localIP}:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${localIP}:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka
fi
# Login kafka server container
# docker exec -it kafkaContainer /bin/bash
# cd /opt/kafka/
# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic default_topic
# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic default_topic --from-beginning
================================================
FILE: install/image/Dockerfile
================================================
# This dockerfile will make images of phpkafka.
# Must use version 7.0, or it would be error.
FROM php:7.0-fpm
# Use homeland image.
RUN mv /etc/apt/sources.list /etc/apt/sources.list.bak \
&& echo 'deb http://mirrors.163.com/debian/ stretch main non-free contrib' > /etc/apt/sources.list \
&& echo 'deb http://mirrors.163.com/debian/ stretch-updates main non-free contrib' >> /etc/apt/sources.list \
&& echo 'deb http://mirrors.163.com/debian-security/ stretch/updates main non-free contrib' >> /etc/apt/sources.list \
&& apt-get update
# Disable commandline interactive.
ENV DEBIAN_FRONTEND noninteractive
# Install git.
# -y is input yes automatically.
RUN apt-get update \
&& apt-get -y install git
# Install vim.
RUN apt-get update \
&& apt-get -y install vim
# Install composer
RUN curl -sS https://getcomposer.org/installer | php \
&& mv composer.phar /usr/local/bin/composer
# Install pecl.
RUN apt-get update \
&& apt-get -y install autoconf \
&& apt-get -y install libz-dev
# Install libkakfa.
RUN apt-get update \
&& apt-get -y install librdkafka-dev=0.9.3-1
# Install the php kafka extension.
# Need librdkafka-dev=0.9.3-1 / rdkafka-3.0.0 this version will work, or it would be error.
# Check rdkafka versions: https://pecl.php.net/package/rdkafka
# Check librdkafka versions: apt search librdkafka-dev
RUN pecl channel-update pecl.php.net \
&& pecl install rdkafka-3.0.0 \
&& docker-php-ext-enable rdkafka
# Install and enable mysqli extension
RUN docker-php-ext-install mysqli && docker-php-ext-enable mysqli
================================================
FILE: install/image/README.md
================================================
### What is this directory
It's a docker image directory, you just run ```bash make.sh``` would help you to make a ```php-kafka``` image
================================================
FILE: install/image/make.sh
================================================
#!/usr/bin/env bash
docker build -t phpkafka .
if [ $? -ne 0 ]; then
echo -e ""
echo -e ">>>>>>>>Make image failure<<<<<<<<"
exit 1
else
echo -e ""
echo -e "========Make image success========"
docker images
fi
# Push to docker repository
# docker login
# input your name and password
# docker tag adf2495d561e lvsid/phpkafka:v1.0
# docker push lvsid/phpkafka:v1.0
================================================
FILE: install/install.sh
================================================
#!/usr/bin/env bash
# Check params.
if [ ! -n "$1" ] ;then
echo -e "Please input the localIP param."
exit 1
else
export ESUPDATER_LOCAL_IP=$1
systemVariables="ESUPDATER_LOCAL_IP=$1\n"
echo -e $systemVariables > ../.env
fi
# The part of you must do.
# 1. Make image
cd image && bash make.sh && cd ..
# The part of you could do.
# 1. Run kafka container
cd container && bash kafka.sh && cd ..
================================================
FILE: plugin/README.md
================================================
### What is this directory
It's a plugin directory, all plugins were stored here.
================================================
FILE: plugin/autogeneratecallback/README.md
================================================
## AutoGenerateCallback
一个自动生成```Handler```和```Service```的事件回调模块的扩展。
### 1、如何使用
- $namespace 参数 ：新增模块的命名空间，如 ```app\alpha\account```
- $moduleName 参数 ：新增的模块名称，如 ```Account```
```shell
php plugin/autogeneratecallback/autogeneratecallback.php {$namespace} {$moduleName}
```
#### 注意事项
- 命名空间中的``` \ ```符号（首位不需要）需要转义，所以别忘记输入两次``` \\ ```，如```app\alpha\account```
- 模块名称使用大驼峰命名，如```MyProfile```
### 2、使用示例
项目中 [account](/app/alpha/account) 模块下的文件即是通过如下命令自动生成而来的。
```shell
php plugin/autogeneratecallback/autogeneratecallback.php app\\alpha\\account Account
```
<img width="720" alt="img" src="https://user-images.githubusercontent.com/35942268/154846773-73e8bc1b-97e0-4d59-be18-23ebaf123c50.png">
### 3、实现原理
基于 [模板替换](https://www.google.com/search?q=%E6%A8%A1%E6%9D%BF%E6%9B%BF%E6%8D%A2) 原理， 先在模板文件 [handler.template](./handler.template) 和 [service.template](./service.template) 中定义如```{{变量}}```此类的```占位符```，再使用正则匹配把```占位符```替换为目标文本。
================================================
FILE: plugin/autogeneratecallback/autogeneratecallback.php
================================================
<?php
require_once __DIR__ . "/../loader.php";
$pluginDirectory = PLUGIN_PATH . "autogeneratecallback/";
$namespace = isset($argv[1]) ? $argv[1] : '';
$moduleName = isset($argv[2]) ? $argv[2] : '';
if (strpos($namespace, '\\') === false) {
die("Error namespace\n");
}
if (strpos($moduleName, '\\') != false) {
die("Error moduleName\n");
}
$namespaceDirectory = ROOT_PATH . str_replace("\\", '/', $namespace);
$namespacePattern = '{{namespace}}';
$moduleNamePattern = '{{moduleName}}';
$configList = [
[
'template_file' => $pluginDirectory . 'handler.template',
'php_file' => "{$namespaceDirectory}/{$moduleName}Handler.php",
],
[
'template_file' => $pluginDirectory . 'service.template',
'php_file' => "{$namespaceDirectory}/{$moduleName}Service.php",
],
];
foreach ($configList as $item) {
if (file_exists($item['php_file'])) {
echo "The file already exist: {$item['php_file']}\n";
continue;
}
if (!is_dir($namespaceDirectory)) {
mkdir($namespaceDirectory);
}
$handle = fopen($item['template_file'], 'r');
while (!feof($handle)) {
$content = fgets($handle);
if (preg_match($namespacePattern, $content) > 0) {
$content = str_replace($namespacePattern, $namespace, $content);
}
if (preg_match($moduleNamePattern, $content) > 0) {
$content = str_replace($moduleNamePattern, $moduleName, $content);
}
file_put_contents($item['php_file'], $content, FILE_APPEND);
}
}
================================================
FILE: plugin/autogeneratecallback/handler.template
================================================
<?php
/**
* The handler of {{moduleName}} event module in alpha application.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace {{namespace}};
class {{moduleName}}Handler
{
/**
* The event callback when table of {{moduleName}} trigger insert event.
* @param array $parsedCanalData
* @return bool
*/
public function onInsert(array $parsedCanalData): bool
{
return (new {{moduleName}}Service())->doInsert($parsedCanalData);
}
/**
* The event callback when table of {{moduleName}} trigger update event.
* @param array $parsedCanalData
* @return bool
*/
public function onUpdate(array $parsedCanalData): bool
{
return (new {{moduleName}}Service())->doUpdate($parsedCanalData);
}
/**
* The event callback when table of {{moduleName}} trigger delete event.
* @param array $parsedCanalData
* @return bool
*/
public function onDelete(array $parsedCanalData): bool
{
return (new {{moduleName}}Service())->doDelete($parsedCanalData);
}
}
================================================
FILE: plugin/autogeneratecallback/service.template
================================================
<?php
/**
* The service of {{moduleName}} event module in alpha application.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace {{namespace}};
class {{moduleName}}Service
{
/**
* Update different indexes when insert.
* @param array $parsedCanalData
* @return bool
*/
public function doInsert(array $parsedCanalData): bool
{
$success = true;
if (!$this->updateAIndexWhenInsert($parsedCanalData)) {
$success = false;
}
if (!$this->updateBIndexWhenInsert($parsedCanalData)) {
$success = false;
}
// update some other indexes...
return $success;
}
/**
* Update different indexes when update.
* @param array $parsedCanalData
* @return bool
*/
public function doUpdate(array $parsedCanalData): bool
{
$success = true;
if (!$this->updateAIndexWhenUpdate($parsedCanalData)) {
$success = false;
}
if (!$this->updateBIndexWhenUpdate($parsedCanalData)) {
$success = false;
}
// update some other indexes...
return $success;
}
/**
* Update different indexes when delete.
* @param array $parsedCanalData
* @return bool
*/
public function doDelete(array $parsedCanalData): bool
{
$success = true;
if (!$this->updateAIndexWhenDelete($parsedCanalData)) {
$success = false;
}
if (!$this->updateBIndexWhenDelete($parsedCanalData)) {
$success = false;
}
// update some other indexes...
return $success;
}
/**
* Update a index: event of insert
* @param array $parsedCanalData
* @return bool
*/
public function updateAIndexWhenInsert(array $parsedCanalData): bool
{
return true;
}
/**
* Update b index: event of insert
* @param array $parsedCanalData
* @return bool
*/
public function updateBIndexWhenInsert(array $parsedCanalData): bool
{
return true;
}
/**
* Update a index: event of update
* @param array $parsedCanalData
* @return bool
*/
public function updateAIndexWhenUpdate(array $parsedCanalData): bool
{
return true;
}
/**
* Update b index: event of update
* @param array $parsedCanalData
* @return bool
*/
public function updateBIndexWhenUpdate(array $parsedCanalData): bool
{
return true;
}
/**
* Update a index: event of delete
* @param array $parsedCanalData
* @return bool
*/
public function updateAIndexWhenDelete(array $parsedCanalData): bool
{
return true;
}
/**
* Update b index: event of delete
* @param array $parsedCanalData
* @return bool
*/
public function updateBIndexWhenDelete(array $parsedCanalData): bool
{
return true;
}
}
================================================
FILE: plugin/loader.php
================================================
<?php
/**
* The loader file of esupdater plugin
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
// PHP Configuration.
date_default_timezone_set('Asia/Shanghai');
const PLUGIN_PATH = __DIR__ . '/';
const ROOT_PATH = PLUGIN_PATH . '../';
================================================
FILE: restart.sh
================================================
#!/usr/bin/env bash
bash stop.sh && bash start.sh
================================================
FILE: start.sh
================================================
#!/usr/bin/env bash
# Prevent start repeatedly
containerCount=0
for file in $(docker container ls -f name=esupdaterContainer -q)
do
((containerCount++))
done
if [ $containerCount -ne 0 ]; then
echo -e ">>>>>>>>Start failure: please run stop.sh first<<<<<<<<"
exit 1
fi
# Pull without password
git config --global credential.helper store
git pull
# Build esupdater image
docker build -t esupdater .
if [ $? -ne 0 ]; then
echo -e ">>>>>>>>Start failure: failed to build<<<<<<<<"
exit 1
fi
# Run container
# docker run --cpuset-cpus="0,1" --cpus=1.5 --cpuset-mems="2,3" --name {ContainerName} -d -v {LocalPath:ContainerPath} {imageName}
docker run --cpus=1.5 --name esupdaterContainer -d -v /home/log/esupdater/:/home/log/esupdater/ esupdater
if [ $? -ne 0 ]; then
echo -e ">>>>>>>>Start failure: failed to run<<<<<<<<"
exit 1
fi
# Exec command
docker exec -d esupdaterContainer php esupdater.php start
if [ $? -ne 0 ]; then
echo -e ">>>>>>>>Start failure: failed to exec<<<<<<<<"
exit 1
else
echo -e "========Start success========"
fi
================================================
FILE: stop.sh
================================================
#!/usr/bin/env bash
# Exec command: php esupdater.php stop
# Use exec command with -i argument would run and return result synchronously: docker exec -i esupdaterContainer php -r "var_dump(123);"
docker exec -i esupdaterContainer php esupdater.php stop
# Stop and remove container
docker stop esupdaterContainer
docker container rm esupdaterContainer
# Remove image
docker rmi esupdater
if [ $? -ne 0 ]; then
echo -e ">>>>>>>>Stop failure<<<<<<<<"
exit 1
else
echo -e "========Stop success========"
fi
================================================
FILE: test/BaseTest.php
================================================
<?php
/**
* The base unit test class.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test;
class BaseTest
{
/**
* Test success.
*
* @return bool
*/
protected function success(): bool
{
$debugTrace = debug_backtrace();
$fileShortName = $this->getCallerFileName($debugTrace);
$functionName = $this->getCallerFunctionName($debugTrace);
echo "Test Success: {$fileShortName} -> {$functionName}\n";
return true;
}
/**
* Test failed.
* @param string $err
*
* @return bool
*/
protected function failed($err = ""): bool
{
$debugTrace = debug_backtrace();
$fileShortName = $this->getCallerFileName($debugTrace);
$functionName = $this->getCallerFunctionName($debugTrace);
echo "Test Failed: {$fileShortName} -> {$functionName}\n";
if (!empty($err)) {
echo "$err\n";
}
return false;
}
/**
* Return text with success color.
*
* @param $text
*
* @return string
*/
public function decorateSuccessText($text): string
{
return "\033[32m{$text}\033[0m";
}
/**
* Return text with failed color.
*
* @param $text
*
* @return string
*/
public function decorateFailedText($text): string
{
return "\033[31;4m{$text}\033[0m";
}
/**
* Get file name of caller.
*
* @param array $debugTrace the data of debug_backtrace() return
*
* @return string
*/
protected function getCallerFileName(array $debugTrace): string
{
if (empty($debugTrace)) {
return '';
}
$fileFullName = $debugTrace[0]['file'];
$sliceList = explode('/test/', $fileFullName);
$fileShortName = '';
if (isset($sliceList[1])) {
$fileShortName = "/test/" . $sliceList[1];
}
return $fileShortName;
}
/**
* Get function name of caller.
*
* @param array $debugTrace the data of debug_backtrace() return
*
* @return string
*/
protected function getCallerFunctionName(array $debugTrace): string
{
if (!isset($debugTrace[1])) {
return '';
}
return $debugTrace[1]['function'];
}
}
================================================
FILE: test/README.md
================================================
### What is this directory
It's a unit test directory, all testcases are stored here, includes ```/app/alpha/```, ```/app/common```，```/framework``` and so on.
================================================
FILE: test/prepare-commit-msg
================================================
#!/bin/sh
# Run test
result=`php test/run.php | grep 'Test Failed'`
if [ -n "$result" ]; then
echo "Sorry, you would not commit if you failed the test "
exit 1
fi
================================================
FILE: test/report/index.css
================================================
#Container {
text-align: center;
}
.section{
margin-top: 30px;
}
#table-1 thead, #table-1 tr {
border-top-width: 1px;
border-top-style: solid;
border-top-color: rgb(230, 189, 189);
}
#table-1 {
width: 700px;
margin: 20px auto;
border-bottom-width: 1px;
border-bottom-style: solid;
border-bottom-color: rgb(230, 189, 189);
}
/* Padding and font style */
#table-1 td, #table-1 th {
padding: 5px 10px;
font-size: 12px;
font-family: Verdana;
color: rgb(177, 106, 104);
}
/* Alternating background colors */
#table-1 tr:nth-child(even) {
background: rgb(238, 211, 210)
}
#table-1 tr:nth-child(odd) {
background: #FFF
}
#table-2 thead, #table-2 tr {
border-top-width: 1px;
border-top-style: solid;
border-top-color: rgb(235, 242, 224);
}
#table-2 {
width: 700px;
margin: 20px auto;
border-bottom-width: 1px;
border-bottom-style: solid;
border-bottom-color: rgb(235, 242, 224);
}
/* Padding and font style */
#table-2 td, #table-2 th {
padding: 5px 10px;
font-size: 12px;
font-family: Verdana;
color: rgb(149, 170, 109);
}
/* Alternating background colors */
#table-2 tr:nth-child(even) {
background: rgb(230, 238, 214)
}
#table-2 tr:nth-child(odd) {
background: #FFF
}
================================================
FILE: test/report/index.html
================================================
<html><head><link rel='stylesheet' href='index.css'></head><body><div id='Container'><div class='section'><h2>ESUpdater UnitTest</h2><table id='table-2'><tr><th>Id</th><th>TestClass</th><th>TestMethod</th><th>TestResult</th></tr><tr><td>1</td><td>test\testcases\app\common\TestESService</td><td>testIsSuccess</td><td>success</td></tr><tr><td>2</td><td>test\testcases\app\common\TestESService</td><td>testIsNeedToUpdate</td><td>success</td></tr><tr><td>3</td><td>test\testcases\app\alpha\TestUserService</td><td>testGetUserId</td><td>success</td></tr><tr><td>4</td><td>test\testcases\framework\TestConsumer</td><td>testConstruct</td><td>success</td></tr><tr><td>5</td><td>test\testcases\framework\TestConsumer</td><td>testIsNeedStop</td><td>success</td></tr><tr><td>6</td><td>test\testcases\framework\TestConsumer</td><td>testIsNeedCheckStatus</td><td>success</td></tr><tr><td>7</td><td>test\testcases\framework\TestConsumer</td><td>testHighLevelConsuming</td><td>success</td></tr><tr><td>8</td><td>test\testcases\framework\TestTimer</td><td>testElapsed</td><td>success</td></tr><tr><td>9</td><td>test\testcases\framework\TestCanal</td><td>testCheckParsedCanalData</td><td>success</td></tr><tr><td>10</td><td>test\testcases\framework\TestCommand</td><td>testWork</td><td>success</td></tr><tr><td>11</td><td>test\testcases\framework\TestManager</td><td>testGetRunningWorkersCount</td><td>success</td></tr><tr><td>12</td><td>test\testcases\framework\TestManager</td><td>testStopConsumerByIPC</td><td>success</td></tr><tr><td>13</td><td>test\testcases\framework\TestManager</td><td>testIsConsumerStopped</td><td>success</td></tr><tr><td>14</td><td>test\testcases\framework\TestManager</td><td>testIsWorkersStopped</td><td>success</td></tr><tr><td>15</td><td>test\testcases\framework\TestManager</td><td>testIsConsumerProcess</td><td>success</td></tr><tr><td>16</td><td>test\testcases\framework\TestManager</td><td>testIsWorkerProcess</td><td>success</td></tr><tr><td>17</td><td>test\testcases\framework\TestEnvironment</td><td>testParseEnvFile</td><td>success</td></tr></table></div></div></body></html>
================================================
FILE: test/run.php
================================================
<?php
/**
* The main file of esupdater unit test.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
include_once __DIR__ . '/../bootstrap.php';
include_once __DIR__ . '/../config/test.php';
class ProjectTest
{
/**
* The root directory.
*
* @var string
*/
protected $rootDirectory = "";
/**
* The directories need to test.
*
* @var array
*/
protected $testDirectories = [];
/**
* ProjectTest constructor.
*/
function __construct()
{
$this->rootDirectory = ROOT_PATH;
global $test;
if (isset($test['testcases_directory']) && !empty($test['testcases_directory'])) {
$this->findTestDirectories($test['testcases_directory']);
}
}
/**
* Find all directories need to test.
*
* @param $directory
*/
protected function findTestDirectories($directory)
{
$currentDirectory = "{$this->rootDirectory}{$directory}";
$handler = opendir($currentDirectory);
if ($handler === false) {
return;
}
while (false !== ($subDir = readdir($handler))) {
$path = $currentDirectory . $subDir;
if (is_dir($path) && !in_array($subDir, ['.', '..'])) {
$targetDirectory = $directory . $subDir . '/';
$this->testDirectories[] = $targetDirectory;
$this->findTestDirectories($targetDirectory);
}
}
}
/**
* The main function of unit test.
*
* @return array
*/
public function run(): array
{
$testResultMap = [];
foreach ($this->testDirectories as $directory) {
$path = "{$this->rootDirectory}/{$directory}";
if (!is_dir($path)) {
continue;
}
$handler = opendir($path);
while (false !== ($filename = readdir($handler))) {
if (!preg_match('/\.php$/', $filename)) {
continue;
}
$class = $directory . explode('.', $filename)[0];
$class = str_replace("/", "\\", $class);
$testObject = new $class();
try {
$reflectClass = new \ReflectionClass($class);
} catch (ReflectionException $e) {
exit(1);
}
$methodObjects = $reflectClass->getMethods();
foreach ($methodObjects as $methodObject) {
$method = $methodObject->getName();
$ownerClass = $methodObject->getDeclaringClass()->getName();
if ('TestBase' == $ownerClass || strpos($method, 'test') !== 0) {
continue;
}
if (!isset($testResultMap[$class])) {
$testResultMap[$class] = [];
}
if (!isset($testResultMap[$class][$method])) {
$testResultMap[$class][$method] = false;
}
if (!$testObject->$method()) {
echo "Unfortunately! You failed the test\n";
exit(1);
}
$testResultMap[$class][$method] = true;
}
}
}
echo "Congratulations! All testcases passed!\n";
return $testResultMap;
}
/**
* Output the test report of html format.
*
* @param array $testResultMap
*/
public function outputHTML(array $testResultMap)
{
$file = ROOT_PATH . "test/report/index.html";
$caseListHtml = '';
$id = 0;
foreach ($testResultMap as $testClass => $item) {
foreach ($item as $testMethod => $testResult) {
++$id;
$testResult = $testResult ? 'success' : 'failed';
$caseListHtml .= "<tr><td>{$id}</td><td>{$testClass}</td><td>{$testMethod}</td><td>{$testResult}</td></tr>";
}
}
$html = "<html><head><link rel='stylesheet' href='index.css'></head><body><div id='Container'><div class='section'><h2>ESUpdater UnitTest</h2><table id='table-2'><tr><th>Id</th><th>TestClass</th><th>TestMethod</th><th>TestResult</th></tr>" . $caseListHtml . "</table></div></div></body></html>";
file_put_contents($file, $html);
}
}
$projectTest = new ProjectTest();
$testResultMap = $projectTest->run();
$projectTest->outputHTML($testResultMap);
================================================
FILE: test/testcases/framework/TestCanal.php
================================================
<?php
/**
* The unit test class of Canal.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test\testcases\framework;
use test\BaseTest;
class TestCanal extends BaseTest
{
public function testCheckParsedCanalData(): bool
{
$caseList = [
[
'data' => [],
'except' => false,
],
[
'data' => [
'data' => [
[],
],
'database' => 'test',
'table' => '',
'type' => 'update',
'id' => 1,
'ts' => 1,
],
'except' => true,
],
];
$service = new \framework\Canal();
foreach ($caseList as $case) {
$data = $case['data'];
$except = $case['except'];
if ($except != $service->checkParsedCanalData($data)) {
return $this->failed();
}
}
return $this->success();
}
}
================================================
FILE: test/testcases/framework/TestCommand.php
================================================
<?php
/**
* The unit test class of Command.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test\testcases\framework;
use test\BaseTest;
class TestCommand extends BaseTest
{
public function testWork(): bool
{
$canalData = (new \framework\Canal())->encode('{"data":[{"userid":"20292","name":"jack"}],"database":"alpha","es":1639020016000,"id":4967056,"isDdl":false,"mysqlType":{"userid":"int(11)","name":"varchar(50)"},"old":null,"pkNames":["workid"],"sql":"","table":"user","ts":1639020017052,"type":"UPDATE"}');
exec("php esupdater.php work '{$canalData}'", $output);
if (!isset($output[0]) || $output[0] !== \framework\Manager::COMMAND_WORK_SUCCESS) {
return $this->failed();
}
return $this->success();
}
}
================================================
FILE: test/testcases/framework/TestConsumer.php
================================================
<?php
/**
* The unit test class of Consumer.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test\testcases\framework;
use test\BaseTest;
class TestConsumer extends BaseTest
{
/**
* Format snake string to camel string.
*
* @param $uncamelizedString
*
* @param string $separator
*
* @return string
*/
public function camelize($uncamelizedString, $separator = '_'): string
{
$uncamelizedString = $separator . str_replace($separator, " ", strtolower($uncamelizedString));
return ltrim(str_replace(" ", "", ucwords($uncamelizedString)), $separator);
}
/**
* Produce message.
*
* @param \framework\Consumer $consumerObject
*
* @param string $message
*
* @return bool
*/
public function produceMessage(\framework\Consumer $consumerObject, string $message): bool
{
// Create producer config object
$producerConfigObject = new \RdKafka\Conf();
// Create producer object
$producerObject = new \RdKafka\Producer($producerConfigObject);
$producerObject->addBrokers($consumerObject->getProperty('brokerListString'));
// Create topic object
$topicObject = $producerObject->newTopic($consumerObject->getProperty('topic'));
// Produce message and put it at buffer
$topicObject->produce(RD_KAFKA_PARTITION_UA, 0, $message);
// setup block time / millisecond / 0 is Non-blocking
$producerObject->poll(0);
// Produce message successfully by default
return true;
// Starting from 4.0, programs MUST call flush() before shutting down, otherwise some messages and callbacks may be lost.
// Push buffer and setup timeout / millisecond
// $result = $producerObject->flush(1000);
// Return false means unable to flush, messages might be lost!
// return RD_KAFKA_RESP_ERR_NO_ERROR === $result;
}
public function testConstruct(): bool
{
$consumer = [
'check_status_interval_seconds' => 3,
'broker_list_string' => '192.168.0.18:9002',
'partition' => 2,
'timeout_millisecond' => 200,
'group_id' => 'test_consume_group',
'topic' => 'test_topic',
'max_worker_count' => 5,
];
$consumerObject = new \framework\Consumer($consumer);
foreach ($consumer as $field => $value) {
$property = $this->camelize($field);
if ($consumer[$field] != $consumerObject->getProperty($property)) {
return $this->failed();
}
}
return $this->success();
}
public function testIsNeedStop(): bool
{
global $consumer;
$consumerObject = new \framework\Consumer($consumer);
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, 'start');
if ($consumerObject->isNeedStop()) {
return $this->failed();
}
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, 'stop');
if (!$consumerObject->isNeedStop()) {
return $this->failed();
}
// set default value
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, 'start');
return $this->success();
}
public function testIsNeedCheckStatus(): bool
{
$consumer = [
'check_status_interval_seconds' => 2,
];
$consumerObject = new \framework\Consumer($consumer);
\framework\Timer::start(\framework\Consumer::TIMER_MARK);
if ($consumerObject->isNeedCheckStatus()) {
return $this->failed();
}
sleep(1);
if ($consumerObject->isNeedCheckStatus()) {
return $this->failed();
}
sleep(2);
if (!$consumerObject->isNeedCheckStatus()) {
return $this->failed();
}
if (!$consumerObject->isNeedCheckStatus()) {
return $this->failed();
}
return $this->success();
}
public function testHighLevelConsuming(): bool
{
$pass = true;
if ($pass) {
return $this->success();
}
global $consumer;
$consumerObject = new \framework\Consumer($consumer);
$message = "test_message_" . rand(10000, 99999);
$this->produceMessage($consumerObject, $message);
$formula = "TestConsumer+testHighLevelConsuming+highLevelConsuming+" . date('Y-m-d H:i:s');
$logId = md5($formula);
\framework\Logger::setLogId($logId, $formula);
if ($message != $consumerObject->highLevelConsuming(true)) {
return $this->failed();
}
return $this->success();
}
}
================================================
FILE: test/testcases/framework/TestEnvironment.php
================================================
<?php
/**
* The unit test class of Environment.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test\testcases\framework;
use test\BaseTest;
class TestEnvironment extends BaseTest
{
public function testParseEnvFile(): bool
{
\framework\Environment::$variableContainer = [];
if (count(\framework\Environment::$variableContainer) != 0) {
return $this->failed();
}
\framework\Environment::parseEnvFile();
if (count(\framework\Environment::$variableContainer) != 1) {
return $this->failed();
}
if (\framework\Environment::getSystemVariable('ESUPDATER_LOCAL_IP') != '192.168.12.22') {
return $this->failed();
}
return $this->success();
}
}
================================================
FILE: test/testcases/framework/TestManager.php
================================================
<?php
/**
* The unit test class of Manager.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test\testcases\framework;
use test\BaseTest;
class TestManager extends BaseTest
{
public function testGetRunningWorkersCount(): bool
{
$manager = new \framework\Manager();
file_put_contents(RUNTIME_PATH . 'esupdater-worker-1.pid', 1);
file_put_contents(RUNTIME_PATH . 'esupdater-worker-2.pid', 2);
file_put_contents(RUNTIME_PATH . 'esupdater-worker-3.pid', 3);
file_put_contents(RUNTIME_PATH . 'esupdater-worker-4.pid', 4);
file_put_contents(RUNTIME_PATH . 'esupdater-worker-5.pid', 5);
if ($manager->getRunningWorkersCount() !== 5) {
return $this->failed();
}
unlink(RUNTIME_PATH . 'esupdater-worker-1.pid');
unlink(RUNTIME_PATH . 'esupdater-worker-2.pid');
unlink(RUNTIME_PATH . 'esupdater-worker-3.pid');
unlink(RUNTIME_PATH . 'esupdater-worker-4.pid');
unlink(RUNTIME_PATH . 'esupdater-worker-5.pid');
return $this->success();
}
public function testStopConsumerByIPC(): bool
{
$manager = new \framework\Manager();
$manager->stopConsumerByIPC();
if (file_exists(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE) && file_get_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE) !== \framework\Consumer::STOP_FLAG_STRING) {
return $this->failed();
}
unlink(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);
return $this->success();
}
public function testIsConsumerStopped(): bool
{
$manager = new \framework\Manager();
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE, \framework\Consumer::START_FLAG_STRING);
file_put_contents(RUNTIME_ESUPDATER_CONSUMER_PID_FILE, 123456);
if ($manager->isConsumerStopped()) {
return $this->failed();
}
unlink(RUNTIME_ESUPDATER_CONSUMER_STATUS_FILE);
unlink(RUNTIME_ESUPDATER_CONSUMER_PID_FILE);
if (!$manager->isConsumerStopped()) {
return $this->failed();
}
return $this->success();
}
public function testIsWorkersStopped(): bool
{
$manager = new \framework\Manager();
if (!$manager->isWorkersStopped()) {
return $this->failed();
}
file_put_contents(RUNTIME_PATH . 'esupdater-worker-1.pid', 1);
file_put_contents(RUNTIME_PATH . 'esupdater-worker-2.pid', 2);
if ($manager->isWorkersStopped()) {
return $this->failed();
}
unlink(RUNTIME_PATH . 'esupdater-worker-1.pid');
unlink(RUNTIME_PATH . 'esupdater-worker-2.pid');
if (!$manager->isWorkersStopped()) {
return $this->failed();
}
return $this->success();
}
public function testIsConsumerProcess(): bool
{
$manager = new \framework\Manager();
$pid = getmypid();
$file = RUNTIME_ESUPDATER_CONSUMER_PID_FILE;
file_put_contents($file, $pid);
if ($manager->isConsumerProcess()) {
unlink($file);
return $this->success();
}
unlink($file);
return $this->failed();
}
public function testIsWorkerProcess(): bool
{
$manager = new \framework\Manager();
$pid = getmypid();
$file = RUNTIME_PATH . "esupdater-worker-{$pid}.pid";
file_put_contents($file, $pid);
if ($manager->isWorkerProcess()) {
unlink($file);
return $this->success();
}
unlink($file);
return $this->failed();
}
}
================================================
FILE: test/testcases/framework/TestTimer.php
================================================
<?php
/**
* The unit test class of Timer.
*
* @author wgrape <https://github.com/WGrape>
* @license https://github.com/WGrape/esupdater/blob/master/LICENSE MIT Licence
*/
namespace test\testcases\framework;
use test\BaseTest;
class TestTimer extends BaseTest
{
public function testElapsed(): bool
{
\framework\Timer::start('test1');
\framework\Timer::start('test2');
sleep(1);
$elapsedTime1 = \framework\Timer::elapsed('test1');
$elapsedTime2 = \framework\Timer::elapsed('test2');
if (intval($elapsedTime1 / 1000) !== 1) {
return $this->failed();
}
if (intval($elapsedTime2 / 1000) !== 1) {
return $this->failed();
}
sleep(1);
$elapsedTime1 = \framework\Timer::elapsed('test1');
$elapsedTime2 = \framework\Timer::elapsed('test2');
if (intval($elapsedTime1 / 1000) !== 2) {
return $this->failed();
}
if (intval($elapsedTime2 / 1000) !== 2) {
return $this->failed();
}
return $this->success();
}
}
gitextract_gfvoa9pt/
├── .editorconfig
├── .github/
│ ├── CODE_OF_CONDUCT.md
│ └── workflows/
│ └── release.yml
├── .gitignore
├── .travis.yml
├── CHANGELOG.md
├── Dockerfile
├── LICENSE
├── README.md
├── bootstrap.php
├── composer.json
├── config/
│ ├── consumer.php
│ ├── db.php
│ ├── es.php
│ ├── event.php
│ ├── log.php
│ └── test.php
├── doc/
│ ├── APPLICATION.md
│ ├── CHANGELOG.md
│ ├── CONTRIBUTING.md
│ ├── HELP.md
│ ├── HOWTOCODE.md
│ └── README.md
├── esupdater.php
├── framework/
│ ├── Canal.php
│ ├── Consumer.php
│ ├── Environment.php
│ ├── Listener.php
│ ├── Logger.php
│ ├── Manager.php
│ └── Timer.php
├── install/
│ ├── README.md
│ ├── container/
│ │ └── kafka.sh
│ ├── image/
│ │ ├── Dockerfile
│ │ ├── README.md
│ │ └── make.sh
│ └── install.sh
├── plugin/
│ ├── README.md
│ ├── autogeneratecallback/
│ │ ├── README.md
│ │ ├── autogeneratecallback.php
│ │ ├── handler.template
│ │ └── service.template
│ └── loader.php
├── restart.sh
├── start.sh
├── stop.sh
└── test/
├── BaseTest.php
├── README.md
├── prepare-commit-msg
├── report/
│ ├── index.css
│ └── index.html
├── run.php
└── testcases/
└── framework/
├── TestCanal.php
├── TestCommand.php
├── TestConsumer.php
├── TestEnvironment.php
├── TestManager.php
└── TestTimer.php
SYMBOL INDEX (86 symbols across 16 files)
FILE: bootstrap.php
function autoloadCallback (line 43) | function autoloadCallback(string $classname)
function shutdownCallback (line 63) | function shutdownCallback()
function exception_handler (line 91) | function exception_handler(Throwable $exception = null)
function error_handler (line 109) | function error_handler(int $errNo, string $errMessage, string $errFile, ...
FILE: framework/Canal.php
class Canal (line 11) | class Canal
method encode (line 20) | public function encode(string $unEncodedCanalData): string
method decode (line 32) | public function decode(string $encodedCanalData): string
method parse (line 44) | public function parse(string $encodedCanalData): array
method checkParsedCanalData (line 56) | public function checkParsedCanalData(array $parsedCanalData): bool
FILE: framework/Consumer.php
class Consumer (line 11) | class Consumer
method __construct (line 26) | public function __construct($consumer)
method getLocalIPByIfconfig (line 42) | public function getLocalIPByIfconfig(): string
method getLocalIP (line 60) | public function getLocalIP(): string
method isNeedStop (line 73) | public function isNeedStop(): bool
method isNeedCheckStatus (line 82) | public function isNeedCheckStatus(): bool
method getProperty (line 93) | public function getProperty(string $propertyName)
method lowLevelConsuming (line 103) | public function lowLevelConsuming($onlyForTest = false): string
method highLevelConsuming (line 151) | public function highLevelConsuming($onlyForTest = false): string
FILE: framework/Environment.php
class Environment (line 11) | class Environment
method getSystemVariable (line 24) | public static function getSystemVariable($variable): string
method parseEnvFile (line 39) | public static function parseEnvFile()
FILE: framework/Listener.php
class Listener (line 11) | class Listener
method dispatch (line 37) | public function dispatch(string $canalData)
method autoCallback (line 94) | public function autoCallback(string $key, string $onWhichEvent, array ...
method manualCallback (line 112) | public function manualCallback(string $key, string $onWhichEvent, arra...
FILE: framework/Logger.php
class Logger (line 11) | class Logger
method setLogId (line 44) | public static function setLogId(string $logIdParam, string $formulaParam)
method setLogIdByParsedCanalData (line 55) | public static function setLogIdByParsedCanalData(array $parsedCanalData)
method returnDumpData (line 72) | public static function returnDumpData(): array
method logDebug (line 95) | public static function logDebug(string $data)
method logInfo (line 105) | public static function logInfo(string $data)
method logSlow (line 115) | public static function logSlow(string $data)
method logWarning (line 125) | public static function logWarning(string $data)
method logError (line 135) | public static function logError(string $data)
method logFatal (line 145) | public static function logFatal(string $data)
method getLogFilePath (line 157) | public static function getLogFilePath($logLevel): string
method write (line 174) | public static function write($level, $data)
FILE: framework/Manager.php
class Manager (line 11) | class Manager
method getRunningWorkersCount (line 24) | public function getRunningWorkersCount()
method startConsumerAndBlocking (line 44) | public function startConsumerAndBlocking()
method stopConsumerByIPC (line 56) | public function stopConsumerByIPC()
method isConsumerStopped (line 69) | public function isConsumerStopped(): bool
method isWorkersStopped (line 79) | public function isWorkersStopped(): bool
method isConsumerProcess (line 91) | public function isConsumerProcess()
method isWorkerProcess (line 104) | public function isWorkerProcess()
method commandStart (line 117) | public function commandStart()
method commandStop (line 137) | public function commandStop(): string
method commandWork (line 170) | public function commandWork(string $canalData): string
FILE: framework/Timer.php
class Timer (line 11) | class Timer
method getNowSecondTimestampWithMicrosecond (line 25) | public static function getNowSecondTimestampWithMicrosecond(): float
method start (line 38) | public static function start(string $mark): float
method elapsed (line 51) | public static function elapsed(string $mark): int
FILE: test/BaseTest.php
class BaseTest (line 11) | class BaseTest
method success (line 18) | protected function success(): bool
method failed (line 33) | protected function failed($err = ""): bool
method decorateSuccessText (line 52) | public function decorateSuccessText($text): string
method decorateFailedText (line 64) | public function decorateFailedText($text): string
method getCallerFileName (line 76) | protected function getCallerFileName(array $debugTrace): string
method getCallerFunctionName (line 97) | protected function getCallerFunctionName(array $debugTrace): string
FILE: test/run.php
class ProjectTest (line 12) | class ProjectTest
method __construct (line 31) | function __construct()
method findTestDirectories (line 45) | protected function findTestDirectories($directory)
method run (line 68) | public function run(): array
method outputHTML (line 122) | public function outputHTML(array $testResultMap)
FILE: test/testcases/framework/TestCanal.php
class TestCanal (line 14) | class TestCanal extends BaseTest
method testCheckParsedCanalData (line 16) | public function testCheckParsedCanalData(): bool
FILE: test/testcases/framework/TestCommand.php
class TestCommand (line 14) | class TestCommand extends BaseTest
method testWork (line 16) | public function testWork(): bool
FILE: test/testcases/framework/TestConsumer.php
class TestConsumer (line 14) | class TestConsumer extends BaseTest
method camelize (line 25) | public function camelize($uncamelizedString, $separator = '_'): string
method produceMessage (line 40) | public function produceMessage(\framework\Consumer $consumerObject, st...
method testConstruct (line 69) | public function testConstruct(): bool
method testIsNeedStop (line 91) | public function testIsNeedStop(): bool
method testIsNeedCheckStatus (line 111) | public function testIsNeedCheckStatus(): bool
method testHighLevelConsuming (line 137) | public function testHighLevelConsuming(): bool
FILE: test/testcases/framework/TestEnvironment.php
class TestEnvironment (line 14) | class TestEnvironment extends BaseTest
method testParseEnvFile (line 16) | public function testParseEnvFile(): bool
FILE: test/testcases/framework/TestManager.php
class TestManager (line 14) | class TestManager extends BaseTest
method testGetRunningWorkersCount (line 16) | public function testGetRunningWorkersCount(): bool
method testStopConsumerByIPC (line 35) | public function testStopConsumerByIPC(): bool
method testIsConsumerStopped (line 48) | public function testIsConsumerStopped(): bool
method testIsWorkersStopped (line 66) | public function testIsWorkersStopped(): bool
method testIsConsumerProcess (line 89) | public function testIsConsumerProcess(): bool
method testIsWorkerProcess (line 105) | public function testIsWorkerProcess(): bool
FILE: test/testcases/framework/TestTimer.php
class TestTimer (line 14) | class TestTimer extends BaseTest
method testElapsed (line 16) | public function testElapsed(): bool
Condensed preview — 58 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (122K chars).
[
{
"path": ".editorconfig",
"chars": 217,
"preview": "root = true\n\n[*]\ncharset = utf-8\nend_of_line = lf\ninsert_final_newline = true\nindent_style = space\nindent_size = 4\ntrim_"
},
{
"path": ".github/CODE_OF_CONDUCT.md",
"chars": 1426,
"preview": "# Contributor Code of Conduct\n\nAs contributors and maintainers of this project, we pledge to respect all people who cont"
},
{
"path": ".github/workflows/release.yml",
"chars": 1109,
"preview": "on:\n push:\n branches:\n - v2.x\n\nname: Release\n\njobs:\n release-please:\n runs-on: ubuntu-latest\n steps:\n "
},
{
"path": ".gitignore",
"chars": 346,
"preview": ".git\n.idea\nlogs\n.DS_Store\ncomposer.lock\n\n# Ignore directories of /app except common and alpha.\napp/*\n!app/common/\n!app/a"
},
{
"path": ".travis.yml",
"chars": 239,
"preview": "# CI was triggered at pull request\nlanguage: php\n\nphp:\n - 7.0\n - nightly\n\nscript:\n - sudo mkdir -p /home/log/esupdate"
},
{
"path": "CHANGELOG.md",
"chars": 1886,
"preview": "# Changelog\n\n## [2.1.1](https://github.com/WGrape/esupdater/compare/v2.1.0...v2.1.1) (2022-09-19)\n\n\n### Docs\n\n* fix vers"
},
{
"path": "Dockerfile",
"chars": 519,
"preview": "FROM phpkafka\n# If you failed to install phpkafka, you can use the lvsid/phpkafka:v1.0 in dockerhub: https://hub.docker."
},
{
"path": "LICENSE",
"chars": 1072,
"preview": "The MIT License (MIT)\n\nCopyright (c) 2021 WGrape\n\nPermission is hereby granted, free of charge, to any person obtaining "
},
{
"path": "README.md",
"chars": 4790,
"preview": "<div align=\"center\" >\n<img width=\"200\" alt=\"img\" src=\"https://user-images.githubusercontent.com/35942268/147061994-f0d5a"
},
{
"path": "bootstrap.php",
"chars": 3256,
"preview": "<?php\n/**\n * The bootloader file of esupdater, test and so on.\n *\n * @author wgrape <https://github.com/WGrape>\n * @lic"
},
{
"path": "composer.json",
"chars": 462,
"preview": "{\n \"name\": \"wgrape/esupdater\",\n \"description\": \"A high-performance lightweight framework of PHP to achieve increme"
},
{
"path": "config/consumer.php",
"chars": 531,
"preview": "<?php\n/**\n * The kafka consuming configuration.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://gi"
},
{
"path": "config/db.php",
"chars": 384,
"preview": "<?php\n/**\n * The mysql querying configuration.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://git"
},
{
"path": "config/es.php",
"chars": 303,
"preview": "<?php\n/**\n * The elasticsearch configuration.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://gith"
},
{
"path": "config/event.php",
"chars": 1919,
"preview": "<?php\n/**\n * The event registering configuration, you can choose autoCallback or manualCallback.\n *\n * @author wgrape <"
},
{
"path": "config/log.php",
"chars": 558,
"preview": "<?php\n/**\n * The log outputting configuration.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://git"
},
{
"path": "config/test.php",
"chars": 236,
"preview": "<?php\n/**\n * The testing configuration.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github.com"
},
{
"path": "doc/APPLICATION.md",
"chars": 6487,
"preview": "# 业务接入文档\n\n- [一、快速接入](#1)\n- [1、修改配置](#11)\n- [2、创建应用](#12)\n- &"
},
{
"path": "doc/CHANGELOG.md",
"chars": 818,
"preview": "### 2、版本 ：v2.0.5\n发布日期 ：2021-01-11\n\n#### 修复\n- [重大更新更容易上手-添加系统变量机制](https://github.com/WGrape/esupdater/commit/551c5585d87"
},
{
"path": "doc/CONTRIBUTING.md",
"chars": 1789,
"preview": "### 目录\n- [1、如何报告问题](#1)\n- [2、如何提交PR](#2)\n- [3、如何理解项目](#3)\n- [4、代码提交规约](#4)\n- [1、单测通过](#41)\n- &nb"
},
{
"path": "doc/HELP.md",
"chars": 4448,
"preview": "### 目录\n- [一、安装过程帮助](#1)\n- [1、Git命令不存在](#11)\n- [2、无法正常Clone](#12)\n-  "
},
{
"path": "doc/HOWTOCODE.md",
"chars": 5063,
"preview": "### 目录\n- [一、架构设计](#1)\n- [1、基于Canal](#11) \n- [2、ES文档更新](#12)\n- &nb"
},
{
"path": "doc/README.md",
"chars": 65,
"preview": "### What is this directory\nIt's the directory of different docs.\n"
},
{
"path": "esupdater.php",
"chars": 872,
"preview": "<?php\n/**\n * The main file of esupdater.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github.co"
},
{
"path": "framework/Canal.php",
"chars": 1716,
"preview": "<?php\n/**\n * The common usages of canal.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github.co"
},
{
"path": "framework/Consumer.php",
"chars": 9271,
"preview": "<?php\n/**\n * The consumer process.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github.com/WGra"
},
{
"path": "framework/Environment.php",
"chars": 1208,
"preview": "<?php\n/**\n * The manager of different environment variables.\n *\n * @author wgrape <https://github.com/WGrape>\n * @licen"
},
{
"path": "framework/Listener.php",
"chars": 4385,
"preview": "<?php\n/**\n * This is an event listener for calling(dispatching) handler when insert/update/delete event of database is t"
},
{
"path": "framework/Logger.php",
"chars": 4753,
"preview": "<?php\n/**\n * This is the only one logger when you need to output logs.\n *\n * @author wgrape <https://github.com/WGrape>"
},
{
"path": "framework/Manager.php",
"chars": 5210,
"preview": "<?php\n/**\n * It's a process manager, manage all processes, including worker and consumer.\n *\n * @author wgrape <https:/"
},
{
"path": "framework/Timer.php",
"chars": 1571,
"preview": "<?php\n/**\n * Do something when you need to timing.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https:/"
},
{
"path": "install/README.md",
"chars": 136,
"preview": "### What is this directory\nIt's the installation directory, you just run ```bash install.sh``` would help you to install"
},
{
"path": "install/container/kafka.sh",
"chars": 1361,
"preview": "#!/usr/bin/env bash\n\n# Pull images first.\ndocker pull wurstmeister/zookeeper\ndocker pull wurstmeister/kafka\n\n# Start zoo"
},
{
"path": "install/image/Dockerfile",
"chars": 1582,
"preview": "# This dockerfile will make images of phpkafka.\n\n# Must use version 7.0, or it would be error.\nFROM php:7.0-fpm\n\n# Use h"
},
{
"path": "install/image/README.md",
"chars": 138,
"preview": "### What is this directory\nIt's a docker image directory, you just run ```bash make.sh``` would help you to make a ```ph"
},
{
"path": "install/image/make.sh",
"chars": 377,
"preview": "#!/usr/bin/env bash\ndocker build -t phpkafka .\nif [ $? -ne 0 ]; then\n echo -e \"\"\n echo -e \">>>>>>>>Make image failure<"
},
{
"path": "install/install.sh",
"chars": 414,
"preview": "#!/usr/bin/env bash\n\n# Check params.\nif [ ! -n \"$1\" ] ;then\n echo -e \"Please input the localIP param.\"\n exit 1\nels"
},
{
"path": "plugin/README.md",
"chars": 82,
"preview": "### What is this directory\nIt's a plugin directory, all plugins were stored here.\n"
},
{
"path": "plugin/autogeneratecallback/README.md",
"chars": 937,
"preview": "## AutoGenerateCallback\n一个自动生成```Handler```和```Service```的事件回调模块的扩展。\n\n### 1、如何使用\n- $namespace 参数 ：新增模块的命名空间，如 ```app\\alp"
},
{
"path": "plugin/autogeneratecallback/autogeneratecallback.php",
"chars": 1576,
"preview": "<?php\n\nrequire_once __DIR__ . \"/../loader.php\";\n\n$pluginDirectory = PLUGIN_PATH . \"autogeneratecallback/\";\n$namespace "
},
{
"path": "plugin/autogeneratecallback/handler.template",
"chars": 1158,
"preview": "<?php\n/**\n * The handler of {{moduleName}} event module in alpha application.\n *\n * @author wgrape <https://github.com/"
},
{
"path": "plugin/autogeneratecallback/service.template",
"chars": 3060,
"preview": "<?php\n/**\n * The service of {{moduleName}} event module in alpha application.\n *\n * @author wgrape <https://github.com/"
},
{
"path": "plugin/loader.php",
"chars": 327,
"preview": "<?php\n/**\n * The loader file of esupdater plugin\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://g"
},
{
"path": "restart.sh",
"chars": 50,
"preview": "#!/usr/bin/env bash\nbash stop.sh && bash start.sh\n"
},
{
"path": "start.sh",
"chars": 1059,
"preview": "#!/usr/bin/env bash\n# Prevent start repeatedly\ncontainerCount=0\nfor file in $(docker container ls -f name=esupdaterConta"
},
{
"path": "stop.sh",
"chars": 511,
"preview": "#!/usr/bin/env bash\n# Exec command: php esupdater.php stop\n# Use exec command with -i argument would run and return resu"
},
{
"path": "test/BaseTest.php",
"chars": 2434,
"preview": "<?php\n/**\n * The base unit test class.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github.com/"
},
{
"path": "test/README.md",
"chars": 160,
"preview": "### What is this directory\nIt's a unit test directory, all testcases are stored here, includes ```/app/alpha/```, ```/ap"
},
{
"path": "test/prepare-commit-msg",
"chars": 171,
"preview": "#!/bin/sh\n\n# Run test\nresult=`php test/run.php | grep 'Test Failed'`\nif [ -n \"$result\" ]; then\n echo \"Sorry, you woul"
},
{
"path": "test/report/index.css",
"chars": 1291,
"preview": "#Container {\n text-align: center;\n}\n\n.section{\n margin-top: 30px;\n}\n\n#table-1 thead, #table-1 tr {\n border-top-"
},
{
"path": "test/report/index.html",
"chars": 2099,
"preview": "<html><head><link rel='stylesheet' href='index.css'></head><body><div id='Container'><div class='section'><h2>ESUpdater "
},
{
"path": "test/run.php",
"chars": 4661,
"preview": "<?php\n/**\n * The main file of esupdater unit test.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https:/"
},
{
"path": "test/testcases/framework/TestCanal.php",
"chars": 1187,
"preview": "<?php\n\n/**\n * The unit test class of Canal.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github"
},
{
"path": "test/testcases/framework/TestCommand.php",
"chars": 873,
"preview": "<?php\n\n/**\n * The unit test class of Command.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://gith"
},
{
"path": "test/testcases/framework/TestConsumer.php",
"chars": 4947,
"preview": "<?php\n\n/**\n * The unit test class of Consumer.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://git"
},
{
"path": "test/testcases/framework/TestEnvironment.php",
"chars": 856,
"preview": "<?php\n\n/**\n * The unit test class of Environment.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://"
},
{
"path": "test/testcases/framework/TestManager.php",
"chars": 3731,
"preview": "<?php\n\n/**\n * The unit test class of Manager.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://gith"
},
{
"path": "test/testcases/framework/TestTimer.php",
"chars": 1104,
"preview": "<?php\n\n/**\n * The unit test class of Timer.\n *\n * @author wgrape <https://github.com/WGrape>\n * @license https://github"
}
]
About this extraction
This page contains the full source code of the WGrape/esupdater GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 58 files (100.8 KB), approximately 31.7k tokens, and a symbol index with 86 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.