Spug
Spug是面向中小型企业设计的轻量级无Agent的自动化运维平台，整合了主机管理、主机批量执行、主机在线终端、应用发布部署、在线任务计划、配置中心、监控、报警等一系列功能。
- 公司官网：https://www.spug.cc
- 项目官网：https://ops.spug.cc
- 使用文档：https://ops.spug.cc/docs/about-spug/
## 演示环境
演示地址：https://demo.spug.cc
## 🔐免费通配符SSL证书
免费通配符，付费证书价格亲民，性价比超高，低于市场其他平台价格，免费专家一对一配置服务，购买流程简单快速，且支持7天无理由退款和开具发票。提供一键下载和SSL过期通知配置，免费申请：[https://ssl.spug.cc](https://ssl.spug.cc)
## 🔥推送助手
推送助手是一个集成了电话、短信、邮件、飞书、钉钉、微信、企业微信等多通道的消息推送平台，可以3分钟实现个人电话短信推送，点击体验：[https://push.spug.cc](https://push.spug.cc)
## 特性
- **批量执行**: 主机命令在线批量执行
- **在线终端**: 主机支持浏览器在线终端登录
- **文件管理**: 主机文件在线上传下载
- **任务计划**: 灵活的在线任务计划
- **发布部署**: 支持自定义发布部署流程
- **配置中心**: 支持KV、文本、json等格式的配置
- **监控中心**: 支持站点、端口、进程、自定义等监控
- **报警中心**: 支持短信、邮件、钉钉、微信等报警方式
- **优雅美观**: 基于 Ant Design 的UI界面
- **开源免费**: 前后端代码完全开源
## 环境
* Python 3.6+
* Django 2.2
* Node 12.14
* React 16.11
## 安装
[官方文档](https://ops.spug.cc/docs/install-docker)
更多使用帮助请参考： [使用文档](https://ops.spug.cc/docs/host-manage/)
## 推荐项目
[Yearning — MYSQL 开源SQL语句审核平台](https://github.com/cookieY/Yearning)
## 预览
### 主机管理
#### 主机在线终端
#### 文件在线上传下载
#### 主机批量执行
#### 应用发布
#### 监控报警
#### 角色权限
## 赞助
 UCloud 5 元/月云主机 |
 阿里云 2核心2G低至99元/年 |
 马哥教育 IT人高薪职业学院 |
|---|
## License & Copyright
[AGPL-3.0](https://opensource.org/licenses/AGPL-3.0)
================================================
FILE: docs/FQA.md
================================================
### install mysqlclient
```shell
# for centos 7
yum install mariadb-devel python3-devel gcc
pip install mysqlclient
```
================================================
FILE: docs/docker/Dockerfile
================================================
FROM centos:7.9.2009
ENV TZ=Asia/Shanghai
RUN yum install -y epel-release https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm && yum install -y --setopt=tsflags=nodocs nginx redis mariadb-devel python36 python36-devel openldap-devel supervisor git gcc wget unzip net-tools sshpass rsync sshfs && yum -y clean all --enablerepo='*'
RUN pip3 install --no-cache-dir --upgrade pip -i https://mirrors.aliyun.com/pypi/simple/
RUN pip3 install --no-cache-dir -i https://mirrors.aliyun.com/pypi/simple/ \
gunicorn \
mysqlclient \
cryptography==36.0.2 \
apscheduler==3.7.0 \
asgiref==3.2.10 \
Django==2.2.28 \
channels==2.3.1 \
channels_redis==2.4.1 \
paramiko==2.11.0 \
django-redis==4.10.0 \
requests==2.22.0 \
GitPython==3.0.8 \
python-ldap==3.4.0 \
openpyxl==3.0.3 \
user_agents==2.2.0
RUN localedef -c -i en_US -f UTF-8 en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV LC_ALL=en_US.UTF-8
RUN echo -e '\n# Source definitions\n. /etc/profile\n' >> /root/.bashrc
RUN mkdir -p /data/repos
COPY init_spug /usr/bin/
COPY nginx.conf /etc/nginx/
COPY ssh_config /etc/ssh/
COPY spug.ini /etc/supervisord.d/
COPY redis.conf /etc/
COPY entrypoint.sh /
VOLUME /data
EXPOSE 80
ENTRYPOINT ["/entrypoint.sh"]
================================================
FILE: docs/docker/docker-compose.yml
================================================
version: "3.3"
services:
db:
image: mariadb:10.8
container_name: spug-db
restart: always
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
volumes:
- /data/spug/mysql:/var/lib/mysql
environment:
- MYSQL_DATABASE=spug
- MYSQL_USER=spug
- MYSQL_PASSWORD=spug.cc
- MYSQL_ROOT_PASSWORD=spug.cc
spug:
image: openspug/spug-service
container_name: spug
privileged: true
restart: always
volumes:
- /data/spug/service:/data/spug
- /data/spug/repos:/data/repos
ports:
# 如果80端口被占用可替换为其他端口，例如: - "8000:80"
- "80:80"
environment:
- SPUG_DOCKER_VERSION=v3.2.4
- MYSQL_DATABASE=spug
- MYSQL_USER=spug
- MYSQL_PASSWORD=spug.cc
- MYSQL_HOST=db
- MYSQL_PORT=3306
depends_on:
- db
================================================
FILE: docs/docker/entrypoint.sh
================================================
#!/bin/bash
#
set -e
if [ -e /root/.bashrc ]; then
source /root/.bashrc
fi
if [ ! -d /data/spug/spug_api ]; then
git clone -b $SPUG_DOCKER_VERSION https://gitee.com/openspug/spug.git /data/spug
curl -o web.tar.gz https://cdn.spug.cc/spug/web_${SPUG_DOCKER_VERSION}.tar.gz
tar xf web.tar.gz -C /data/spug/spug_web/
rm -f web.tar.gz
SECRET_KEY=$(< /dev/urandom tr -dc '!@#%^.a-zA-Z0-9' | head -c50)
cat > /data/spug/spug_api/spug/overrides.py << EOF
import os
DEBUG = False
ALLOWED_HOSTS = ['127.0.0.1']
SECRET_KEY = '${SECRET_KEY}'
DATABASES = {
'default': {
'ATOMIC_REQUESTS': True,
'ENGINE': 'django.db.backends.mysql',
'NAME': os.environ.get('MYSQL_DATABASE'),
'USER': os.environ.get('MYSQL_USER'),
'PASSWORD': os.environ.get('MYSQL_PASSWORD'),
'HOST': os.environ.get('MYSQL_HOST'),
'PORT': os.environ.get('MYSQL_PORT'),
'OPTIONS': {
'charset': 'utf8mb4',
'sql_mode': 'STRICT_TRANS_TABLES',
}
}
}
EOF
fi
exec supervisord -c /etc/supervisord.conf
================================================
FILE: docs/docker/init_spug
================================================
#!/bin/bash
#
set -e
set -u
python3 /data/spug/spug_api/manage.py updatedb
python3 /data/spug/spug_api/manage.py user add -u $1 -p $2 -n 管理员 -s
================================================
FILE: docs/docker/nginx.conf
================================================
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /data/spug/spug_web/build;
client_max_body_size 0;
add_header X-Frame-Options SAMEORIGIN always;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_http_version 1.1;
gzip_comp_level 7;
gzip_types text/plain text/css text/javascript application/javascript application/json;
gzip_disable "MSIE [1-6]\.";
gzip_vary on;
location ^~ /api/ {
rewrite ^/api(.*) $1 break;
proxy_pass http://127.0.0.1:9001;
proxy_read_timeout 180s;
proxy_redirect off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location ^~ /api/ws/ {
rewrite ^/api(.*) $1 break;
proxy_pass http://127.0.0.1:9002;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location / {
try_files $uri /index.html;
}
}
}
================================================
FILE: docs/docker/redis.conf
================================================
# Redis configuration file example.
#
# Note that in order to read the configuration file, Redis must be
# started with the file path as first argument:
#
# ./redis-server /path/to/redis.conf
# Note on units: when memory size is needed, it is possible to specify
# it in the usual form of 1k 5GB 4M and so forth:
#
# 1k => 1000 bytes
# 1kb => 1024 bytes
# 1m => 1000000 bytes
# 1mb => 1024*1024 bytes
# 1g => 1000000000 bytes
# 1gb => 1024*1024*1024 bytes
#
# units are case insensitive so 1GB 1Gb 1gB are all the same.
################################## INCLUDES ###################################
# Include one or more other config files here. This is useful if you
# have a standard template that goes to all Redis servers but also need
# to customize a few per-server settings. Include files can include
# other files, so use this wisely.
#
# Notice option "include" won't be rewritten by command "CONFIG REWRITE"
# from admin or Redis Sentinel. Since Redis always uses the last processed
# line as value of a configuration directive, you'd better put includes
# at the beginning of this file to avoid overwriting config change at runtime.
#
# If instead you are interested in using includes to override configuration
# options, it is better to use include as the last line.
#
# include /path/to/local.conf
# include /path/to/other.conf
################################## NETWORK #####################################
# By default, if no "bind" configuration directive is specified, Redis listens
# for connections from all the network interfaces available on the server.
# It is possible to listen to just one or multiple selected interfaces using
# the "bind" configuration directive, followed by one or more IP addresses.
#
# Examples:
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
#
# ~~~ WARNING ~~~ If the computer running Redis is directly exposed to the
# internet, binding to all the interfaces is dangerous and will expose the
# instance to everybody on the internet. So by default we uncomment the
# following bind directive, that will force Redis to listen only into
# the IPv4 lookback interface address (this means Redis will be able to
# accept connections only from clients running into the same computer it
# is running).
#
# IF YOU ARE SURE YOU WANT YOUR INSTANCE TO LISTEN TO ALL THE INTERFACES
# JUST COMMENT THE FOLLOWING LINE.
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bind 127.0.0.1
# Protected mode is a layer of security protection, in order to avoid that
# Redis instances left open on the internet are accessed and exploited.
#
# When protected mode is on and if:
#
# 1) The server is not binding explicitly to a set of addresses using the
# "bind" directive.
# 2) No password is configured.
#
# The server only accepts connections from clients connecting from the
# IPv4 and IPv6 loopback addresses 127.0.0.1 and ::1, and from Unix domain
# sockets.
#
# By default protected mode is enabled. You should disable it only if
# you are sure you want clients from other hosts to connect to Redis
# even if no authentication is configured, nor a specific set of interfaces
# are explicitly listed using the "bind" directive.
protected-mode yes
# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379
# TCP listen() backlog.
#
# In high requests-per-second environments you need an high backlog in order
# to avoid slow clients connections issues. Note that the Linux kernel
# will silently truncate it to the value of /proc/sys/net/core/somaxconn so
# make sure to raise both the value of somaxconn and tcp_max_syn_backlog
# in order to get the desired effect.
tcp-backlog 511
# Unix socket.
#
# Specify the path for the Unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
#
# unixsocket /tmp/redis.sock
# unixsocketperm 700
# Close the connection after a client is idle for N seconds (0 to disable)
timeout 0
# TCP keepalive.
#
# If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
# of communication. This is useful for two reasons:
#
# 1) Detect dead peers.
# 2) Take the connection alive from the point of view of network
# equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 300 seconds, which is the new
# Redis default starting with Redis 3.2.1.
tcp-keepalive 300
################################# GENERAL #####################################
# By default Redis does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
daemonize no
# If you run Redis from upstart or systemd, Redis can interact with your
# supervision tree. Options:
# supervised no - no supervision interaction
# supervised upstart - signal upstart by putting Redis into SIGSTOP mode
# supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET
# supervised auto - detect upstart or systemd method based on
# UPSTART_JOB or NOTIFY_SOCKET environment variables
# Note: these supervision methods only signal "process is ready."
# They do not enable continuous liveness pings back to your supervisor.
supervised no
# If a pid file is specified, Redis writes it where specified at startup
# and removes it at exit.
#
# When the server runs non daemonized, no pid file is created if none is
# specified in the configuration. When the server is daemonized, the pid file
# is used even if not specified, defaulting to "/var/run/redis.pid".
#
# Creating a pid file is best effort: if Redis is not able to create it
# nothing bad happens, the server will start and run normally.
pidfile /var/run/redis_6379.pid
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
loglevel notice
# Specify the log file name. Also the empty string can be used to force
# Redis to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile /var/log/redis/redis.log
# To enable logging to the system logger, just set 'syslog-enabled' to yes,
# and optionally update the other syslog parameters to suit your needs.
# syslog-enabled no
# Specify the syslog identity.
# syslog-ident redis
# Specify the syslog facility. Must be USER or between LOCAL0-LOCAL7.
# syslog-facility local0
# Set the number of databases. The default database is DB 0, you can select
# a different one on a per-connection basis using SELECT where
# dbid is a number between 0 and 'databases'-1
databases 16
################################ SNAPSHOTTING ################################
#
# Save the DB on disk:
#
# save
#
# Will save the DB if both the given number of seconds and the given
# number of write operations against the DB occurred.
#
# In the example below the behaviour will be to save:
# after 900 sec (15 min) if at least 1 key changed
# after 300 sec (5 min) if at least 10 keys changed
# after 60 sec if at least 10000 keys changed
#
# Note: you can disable saving completely by commenting out all "save" lines.
#
# It is also possible to remove all the previously configured save
# points by adding a save directive with a single empty string argument
# like in the following example:
#
# save ""
save 900 1
save 300 10
save 60 10000
# By default Redis will stop accepting writes if RDB snapshots are enabled
# (at least one save point) and the latest background save failed.
# This will make the user aware (in a hard way) that data is not persisting
# on disk properly, otherwise chances are that no one will notice and some
# disaster will happen.
#
# If the background saving process will start working again Redis will
# automatically allow writes again.
#
# However if you have setup your proper monitoring of the Redis server
# and persistence, you may want to disable this feature so that Redis will
# continue to work as usual even if there are problems with disk,
# permissions, and so forth.
stop-writes-on-bgsave-error yes
# Compress string objects using LZF when dump .rdb databases?
# For default that's set to 'yes' as it's almost always a win.
# If you want to save some CPU in the saving child set it to 'no' but
# the dataset will likely be bigger if you have compressible values or keys.
rdbcompression yes
# Since version 5 of RDB a CRC64 checksum is placed at the end of the file.
# This makes the format more resistant to corruption but there is a performance
# hit to pay (around 10%) when saving and loading RDB files, so you can disable it
# for maximum performances.
#
# RDB files created with checksum disabled have a checksum of zero that will
# tell the loading code to skip the check.
rdbchecksum yes
# The filename where to dump the DB
dbfilename dump.rdb
# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /var/lib/redis
################################# REPLICATION #################################
# Master-Slave replication. Use slaveof to make a Redis instance a copy of
# another Redis server. A few things to understand ASAP about Redis replication.
#
# 1) Redis replication is asynchronous, but you can configure a master to
# stop accepting writes if it appears to be not connected with at least
# a given number of slaves.
# 2) Redis slaves are able to perform a partial resynchronization with the
# master if the replication link is lost for a relatively small amount of
# time. You may want to configure the replication backlog size (see the next
# sections of this file) with a sensible value depending on your needs.
# 3) Replication is automatic and does not need user intervention. After a
# network partition slaves automatically try to reconnect to masters
# and resynchronize with them.
#
# slaveof
# If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the slave to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the slave request.
#
# masterauth
# When a slave loses its connection with the master, or when the replication
# is still in progress, the slave can act in two different ways:
#
# 1) if slave-serve-stale-data is set to 'yes' (the default) the slave will
# still reply to client requests, possibly with out of date data, or the
# data set may just be empty if this is the first synchronization.
#
# 2) if slave-serve-stale-data is set to 'no' the slave will reply with
# an error "SYNC with master in progress" to all the kind of commands
# but to INFO and SLAVEOF.
#
slave-serve-stale-data yes
# You can configure a slave instance to accept writes or not. Writing against
# a slave instance may be useful to store some ephemeral data (because data
# written on a slave will be easily deleted after resync with the master) but
# may also cause problems if clients are writing to it because of a
# misconfiguration.
#
# Since Redis 2.6 by default slaves are read-only.
#
# Note: read only slaves are not designed to be exposed to untrusted clients
# on the internet. It's just a protection layer against misuse of the instance.
# Still a read only slave exports by default all the administrative commands
# such as CONFIG, DEBUG, and so forth. To a limited extent you can improve
# security of read only slaves using 'rename-command' to shadow all the
# administrative / dangerous commands.
slave-read-only yes
# Replication SYNC strategy: disk or socket.
#
# -------------------------------------------------------
# WARNING: DISKLESS REPLICATION IS EXPERIMENTAL CURRENTLY
# -------------------------------------------------------
#
# New slaves and reconnecting slaves that are not able to continue the replication
# process just receiving differences, need to do what is called a "full
# synchronization". An RDB file is transmitted from the master to the slaves.
# The transmission can happen in two different ways:
#
# 1) Disk-backed: The Redis master creates a new process that writes the RDB
# file on disk. Later the file is transferred by the parent
# process to the slaves incrementally.
# 2) Diskless: The Redis master creates a new process that directly writes the
# RDB file to slave sockets, without touching the disk at all.
#
# With disk-backed replication, while the RDB file is generated, more slaves
# can be queued and served with the RDB file as soon as the current child producing
# the RDB file finishes its work. With diskless replication instead once
# the transfer starts, new slaves arriving will be queued and a new transfer
# will start when the current one terminates.
#
# When diskless replication is used, the master waits a configurable amount of
# time (in seconds) before starting the transfer in the hope that multiple slaves
# will arrive and the transfer can be parallelized.
#
# With slow disks and fast (large bandwidth) networks, diskless replication
# works better.
repl-diskless-sync no
# When diskless replication is enabled, it is possible to configure the delay
# the server waits in order to spawn the child that transfers the RDB via socket
# to the slaves.
#
# This is important since once the transfer starts, it is not possible to serve
# new slaves arriving, that will be queued for the next RDB transfer, so the server
# waits a delay in order to let more slaves arrive.
#
# The delay is specified in seconds, and by default is 5 seconds. To disable
# it entirely just set it to 0 seconds and the transfer will start ASAP.
repl-diskless-sync-delay 5
# Slaves send PINGs to server in a predefined interval. It's possible to change
# this interval with the repl_ping_slave_period option. The default value is 10
# seconds.
#
# repl-ping-slave-period 10
# The following option sets the replication timeout for:
#
# 1) Bulk transfer I/O during SYNC, from the point of view of slave.
# 2) Master timeout from the point of view of slaves (data, pings).
# 3) Slave timeout from the point of view of masters (REPLCONF ACK pings).
#
# It is important to make sure that this value is greater than the value
# specified for repl-ping-slave-period otherwise a timeout will be detected
# every time there is low traffic between the master and the slave.
#
# repl-timeout 60
# Disable TCP_NODELAY on the slave socket after SYNC?
#
# If you select "yes" Redis will use a smaller number of TCP packets and
# less bandwidth to send data to slaves. But this can add a delay for
# the data to appear on the slave side, up to 40 milliseconds with
# Linux kernels using a default configuration.
#
# If you select "no" the delay for data to appear on the slave side will
# be reduced but more bandwidth will be used for replication.
#
# By default we optimize for low latency, but in very high traffic conditions
# or when the master and slaves are many hops away, turning this to "yes" may
# be a good idea.
repl-disable-tcp-nodelay no
# Set the replication backlog size. The backlog is a buffer that accumulates
# slave data when slaves are disconnected for some time, so that when a slave
# wants to reconnect again, often a full resync is not needed, but a partial
# resync is enough, just passing the portion of data the slave missed while
# disconnected.
#
# The bigger the replication backlog, the longer the time the slave can be
# disconnected and later be able to perform a partial resynchronization.
#
# The backlog is only allocated once there is at least a slave connected.
#
# repl-backlog-size 1mb
# After a master has no longer connected slaves for some time, the backlog
# will be freed. The following option configures the amount of seconds that
# need to elapse, starting from the time the last slave disconnected, for
# the backlog buffer to be freed.
#
# A value of 0 means to never release the backlog.
#
# repl-backlog-ttl 3600
# The slave priority is an integer number published by Redis in the INFO output.
# It is used by Redis Sentinel in order to select a slave to promote into a
# master if the master is no longer working correctly.
#
# A slave with a low priority number is considered better for promotion, so
# for instance if there are three slaves with priority 10, 100, 25 Sentinel will
# pick the one with priority 10, that is the lowest.
#
# However a special priority of 0 marks the slave as not able to perform the
# role of master, so a slave with priority of 0 will never be selected by
# Redis Sentinel for promotion.
#
# By default the priority is 100.
slave-priority 100
# It is possible for a master to stop accepting writes if there are less than
# N slaves connected, having a lag less or equal than M seconds.
#
# The N slaves need to be in "online" state.
#
# The lag in seconds, that must be <= the specified value, is calculated from
# the last ping received from the slave, that is usually sent every second.
#
# This option does not GUARANTEE that N replicas will accept the write, but
# will limit the window of exposure for lost writes in case not enough slaves
# are available, to the specified number of seconds.
#
# For example to require at least 3 slaves with a lag <= 10 seconds use:
#
# min-slaves-to-write 3
# min-slaves-max-lag 10
#
# Setting one or the other to 0 disables the feature.
#
# By default min-slaves-to-write is set to 0 (feature disabled) and
# min-slaves-max-lag is set to 10.
# A Redis master is able to list the address and port of the attached
# slaves in different ways. For example the "INFO replication" section
# offers this information, which is used, among other tools, by
# Redis Sentinel in order to discover slave instances.
# Another place where this info is available is in the output of the
# "ROLE" command of a masteer.
#
# The listed IP and address normally reported by a slave is obtained
# in the following way:
#
# IP: The address is auto detected by checking the peer address
# of the socket used by the slave to connect with the master.
#
# Port: The port is communicated by the slave during the replication
# handshake, and is normally the port that the slave is using to
# list for connections.
#
# However when port forwarding or Network Address Translation (NAT) is
# used, the slave may be actually reachable via different IP and port
# pairs. The following two options can be used by a slave in order to
# report to its master a specific set of IP and port, so that both INFO
# and ROLE will report those values.
#
# There is no need to use both the options if you need to override just
# the port or the IP address.
#
# slave-announce-ip 5.5.5.5
# slave-announce-port 1234
################################## SECURITY ###################################
# Require clients to issue AUTH before processing any other
# commands. This might be useful in environments in which you do not trust
# others with access to the host running redis-server.
#
# This should stay commented out for backward compatibility and because most
# people do not need auth (e.g. they run their own servers).
#
# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#
# requirepass foobared
# Command renaming.
#
# It is possible to change the name of dangerous commands in a shared
# environment. For instance the CONFIG command may be renamed into something
# hard to guess so that it will still be available for internal-use tools
# but not available for general clients.
#
# Example:
#
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
#
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
# rename-command CONFIG ""
#
# Please note that changing the name of commands that are logged into the
# AOF file or transmitted to slaves may cause problems.
################################### LIMITS ####################################
# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able to configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
#
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
#
# maxclients 10000
# Don't use more memory than the specified amount of bytes.
# When the memory limit is reached Redis will try to remove keys
# according to the eviction policy selected (see maxmemory-policy).
#
# If Redis can't remove keys according to the policy, or if the policy is
# set to 'noeviction', Redis will start to reply with errors to commands
# that would use more memory, like SET, LPUSH, and so on, and will continue
# to reply to read-only commands like GET.
#
# This option is usually useful when using Redis as an LRU cache, or to set
# a hard memory limit for an instance (using the 'noeviction' policy).
#
# WARNING: If you have slaves attached to an instance with maxmemory on,
# the size of the output buffers needed to feed the slaves are subtracted
# from the used memory count, so that network problems / resyncs will
# not trigger a loop where keys are evicted, and in turn the output
# buffer of slaves is full with DELs of keys evicted triggering the deletion
# of more keys, and so forth until the database is completely emptied.
#
# In short... if you have slaves attached it is suggested that you set a lower
# limit for maxmemory so that there is some free RAM on the system for slave
# output buffers (but this is not needed if the policy is 'noeviction').
#
maxmemory 2GB
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory
# is reached. You can select among five behaviors:
#
# volatile-lru -> remove the key with an expire set using an LRU algorithm
# allkeys-lru -> remove any key according to the LRU algorithm
# volatile-random -> remove a random key with an expire set
# allkeys-random -> remove a random key, any key
# volatile-ttl -> remove the key with the nearest expire time (minor TTL)
# noeviction -> don't expire at all, just return an error on write operations
#
# Note: with any of the above policies, Redis will return an error on write
# operations, when there are no suitable keys for eviction.
#
# At the date of writing these commands are: set setnx setex append
# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd
# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby
# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby
# getset mset msetnx exec sort
#
# The default is:
#
maxmemory-policy allkeys-lru
# LRU and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can tune it for speed or
# accuracy. For default Redis will check five keys and pick the one that was
# used less recently, you can change the sample size using the following
# configuration directive.
#
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.
#
# maxmemory-samples 5
############################## APPEND ONLY MODE ###############################
# By default Redis asynchronously dumps the dataset on disk. This mode is
# good enough in many applications, but an issue with the Redis process or
# a power outage may result into a few minutes of writes lost (depending on
# the configured save points).
#
# The Append Only File is an alternative persistence mode that provides
# much better durability. For instance using the default data fsync policy
# (see later in the config file) Redis can lose just one second of writes in a
# dramatic event like a server power outage, or a single write if something
# wrong with the Redis process itself happens, but the operating system is
# still running correctly.
#
# AOF and RDB persistence can be enabled at the same time without problems.
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
#
# Please check http://redis.io/topics/persistence for more information.
appendonly no
# The name of the append only file (default: "appendonly.aof")
appendfilename "appendonly.aof"
# The fsync() call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
#
# Redis supports three different modes:
#
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log. Slow, Safest.
# everysec: fsync only one time every second. Compromise.
#
# The default is "everysec", as that's usually the right compromise between
# speed and data safety. It's up to you to understand if you can relax this to
# "no" that will let the operating system flush the output buffer when
# it wants, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting),
# or on the contrary, use "always" that's very slow but a bit safer than
# everysec.
#
# More details please check the following article:
# http://antirez.com/post/redis-persistence-demystified.html
#
# If unsure, use "everysec".
# appendfsync always
appendfsync everysec
# appendfsync no
# When the AOF fsync policy is set to always or everysec, and a background
# saving process (a background save or AOF log background rewriting) is
# performing a lot of I/O against the disk, in some Linux configurations
# Redis may block too long on the fsync() call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2) call.
#
# In order to mitigate this problem it's possible to use the following option
# that will prevent fsync() from being called in the main process while a
# BGSAVE or BGREWRITEAOF is in progress.
#
# This means that while another child is saving, the durability of Redis is
# the same as "appendfsync none". In practical terms, this means that it is
# possible to lose up to 30 seconds of log in the worst scenario (with the
# default Linux settings).
#
# If you have latency problems turn this to "yes". Otherwise leave it as
# "no" that is the safest pick from the point of view of durability.
no-appendfsync-on-rewrite no
# Automatic rewrite of the append only file.
# Redis is able to automatically rewrite the log file implicitly calling
# BGREWRITEAOF when the AOF log size grows by the specified percentage.
#
# This is how it works: Redis remembers the size of the AOF file after the
# latest rewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used).
#
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even if the percentage increase
# is reached but it is still pretty small.
#
# Specify a percentage of zero in order to disable the automatic AOF
# rewrite feature.
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
# An AOF file may be found to be truncated at the end during the Redis
# startup process, when the AOF data gets loaded back into memory.
# This may happen when the system where Redis is running
# crashes, especially when an ext4 filesystem is mounted without the
# data=ordered option (however this can't happen when Redis itself
# crashes or aborts but the operating system still works correctly).
#
# Redis can either exit with an error when this happens, or load as much
# data as possible (the default now) and start if the AOF file is found
# to be truncated at the end. The following option controls this behavior.
#
# If aof-load-truncated is set to yes, a truncated AOF file is loaded and
# the Redis server starts emitting a log to inform the user of the event.
# Otherwise if the option is set to no, the server aborts with an error
# and refuses to start. When the option is set to no, the user requires
# to fix the AOF file using the "redis-check-aof" utility before to restart
# the server.
#
# Note that if the AOF file will be found to be corrupted in the middle
# the server will still exit with an error. This option only applies when
# Redis will try to read more data from the AOF file but not enough bytes
# will be found.
aof-load-truncated yes
################################ LUA SCRIPTING ###############################
# Max execution time of a Lua script in milliseconds.
#
# If the maximum execution time is reached Redis will log that a script is
# still in execution after the maximum allowed time and will start to
# reply to queries with an error.
#
# When a long running script exceeds the maximum execution time only the
# SCRIPT KILL and SHUTDOWN NOSAVE commands are available. The first can be
# used to stop a script that did not yet called write commands. The second
# is the only way to shut down the server in the case a write command was
# already issued by the script but the user doesn't want to wait for the natural
# termination of the script.
#
# Set it to 0 or a negative value for unlimited execution without warnings.
lua-time-limit 5000
################################ REDIS CLUSTER ###############################
#
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# WARNING EXPERIMENTAL: Redis Cluster is considered to be stable code, however
# in order to mark it as "mature" we need to wait for a non trivial percentage
# of users to deploy it in production.
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
# Normal Redis instances can't be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
#
# cluster-enabled yes
# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system do not have
# overlapping cluster configuration file names.
#
# cluster-config-file nodes-6379.conf
# Cluster node timeout is the amount of milliseconds a node must be unreachable
# for it to be considered in failure state.
# Most other internal time limits are multiple of the node timeout.
#
# cluster-node-timeout 15000
# A slave of a failing master will avoid to start a failover if its data
# looks too old.
#
# There is no simple way for a slave to actually have a exact measure of
# its "data age", so the following two checks are performed:
#
# 1) If there are multiple slaves able to failover, they exchange messages
# in order to try to give an advantage to the slave with the best
# replication offset (more data from the master processed).
# Slaves will try to get their rank by offset, and apply to the start
# of the failover a delay proportional to their rank.
#
# 2) Every single slave computes the time of the last interaction with
# its master. This can be the last ping or command received (if the master
# is still in the "connected" state), or the time that elapsed since the
# disconnection with the master (if the replication link is currently down).
# If the last interaction is too old, the slave will not try to failover
# at all.
#
# The point "2" can be tuned by user. Specifically a slave will not perform
# the failover if, since the last interaction with the master, the time
# elapsed is greater than:
#
# (node-timeout * slave-validity-factor) + repl-ping-slave-period
#
# So for example if node-timeout is 30 seconds, and the slave-validity-factor
# is 10, and assuming a default repl-ping-slave-period of 10 seconds, the
# slave will not try to failover if it was not able to talk with the master
# for longer than 310 seconds.
#
# A large slave-validity-factor may allow slaves with too old data to failover
# a master, while a too small value may prevent the cluster from being able to
# elect a slave at all.
#
# For maximum availability, it is possible to set the slave-validity-factor
# to a value of 0, which means, that slaves will always try to failover the
# master regardless of the last time they interacted with the master.
# (However they'll always try to apply a delay proportional to their
# offset rank).
#
# Zero is the only value able to guarantee that when all the partitions heal
# the cluster will always be able to continue.
#
# cluster-slave-validity-factor 10
# Cluster slaves are able to migrate to orphaned masters, that are masters
# that are left without working slaves. This improves the cluster ability
# to resist to failures as otherwise an orphaned master can't be failed over
# in case of failure if it has no working slaves.
#
# Slaves migrate to orphaned masters only if there are still at least a
# given number of other working slaves for their old master. This number
# is the "migration barrier". A migration barrier of 1 means that a slave
# will migrate only if there is at least 1 other working slave for its master
# and so forth. It usually reflects the number of slaves you want for every
# master in your cluster.
#
# Default is 1 (slaves migrate only if their masters remain with at least
# one slave). To disable migration just set it to a very large value.
# A value of 0 can be set but is useful only for debugging and dangerous
# in production.
#
# cluster-migration-barrier 1
# By default Redis Cluster nodes stop accepting queries if they detect there
# is at least an hash slot uncovered (no available node is serving it).
# This way if the cluster is partially down (for example a range of hash slots
# are no longer covered) all the cluster becomes, eventually, unavailable.
# It automatically returns available as soon as all the slots are covered again.
#
# However sometimes you want the subset of the cluster which is working,
# to continue to accept queries for the part of the key space that is still
# covered. In order to do so, just set the cluster-require-full-coverage
# option to no.
#
# cluster-require-full-coverage yes
# In order to setup your cluster make sure to read the documentation
# available at http://redis.io web site.
################################## SLOW LOG ###################################
# The Redis Slow Log is a system to log queries that exceeded a specified
# execution time. The execution time does not include the I/O operations
# like talking with the client, sending the reply and so forth,
# but just the time needed to actually execute the command (this is the only
# stage of command execution where the thread is blocked and can not serve
# other requests in the meantime).
#
# You can configure the slow log with two parameters: one tells Redis
# what is the execution time, in microseconds, to exceed in order for the
# command to get logged, and the other parameter is the length of the
# slow log. When a new command is logged the oldest one is removed from the
# queue of logged commands.
# The following time is expressed in microseconds, so 1000000 is equivalent
# to one second. Note that a negative number disables the slow log, while
# a value of zero forces the logging of every command.
slowlog-log-slower-than 10000
# There is no limit to this length. Just be aware that it will consume memory.
# You can reclaim memory used by the slow log with SLOWLOG RESET.
slowlog-max-len 128
################################ LATENCY MONITOR ##############################
# The Redis latency monitoring subsystem samples different operations
# at runtime in order to collect data related to possible sources of
# latency of a Redis instance.
#
# Via the LATENCY command this information is available to the user that can
# print graphs and obtain reports.
#
# The system only logs operations that were performed in a time equal or
# greater than the amount of milliseconds specified via the
# latency-monitor-threshold configuration directive. When its value is set
# to zero, the latency monitor is turned off.
#
# By default latency monitoring is disabled since it is mostly not needed
# if you don't have latency issues, and collecting data has a performance
# impact, that while very small, can be measured under big load. Latency
# monitoring can easily be enabled at runtime using the command
# "CONFIG SET latency-monitor-threshold " if needed.
latency-monitor-threshold 0
############################# EVENT NOTIFICATION ##############################
# Redis can notify Pub/Sub clients about events happening in the key space.
# This feature is documented at http://redis.io/topics/notifications
#
# For instance if keyspace events notification is enabled, and a client
# performs a DEL operation on key "foo" stored in the Database 0, two
# messages will be published via Pub/Sub:
#
# PUBLISH __keyspace@0__:foo del
# PUBLISH __keyevent@0__:del foo
#
# It is possible to select the events that Redis will notify among a set
# of classes. Every class is identified by a single character:
#
# K Keyspace events, published with __keyspace@__ prefix.
# E Keyevent events, published with __keyevent@__ prefix.
# g Generic commands (non-type specific) like DEL, EXPIRE, RENAME, ...
# $ String commands
# l List commands
# s Set commands
# h Hash commands
# z Sorted set commands
# x Expired events (events generated every time a key expires)
# e Evicted events (events generated when a key is evicted for maxmemory)
# A Alias for g$lshzxe, so that the "AKE" string means all the events.
#
# The "notify-keyspace-events" takes as argument a string that is composed
# of zero or multiple characters. The empty string means that notifications
# are disabled.
#
# Example: to enable list and generic events, from the point of view of the
# event name, use:
#
# notify-keyspace-events Elg
#
# Example 2: to get the stream of the expired keys subscribing to channel
# name __keyevent@0__:expired use:
#
# notify-keyspace-events Ex
#
# By default all notifications are disabled because most users don't need
# this feature and the feature has some overhead. Note that if you don't
# specify at least one of K or E, no events will be delivered.
notify-keyspace-events ""
############################### ADVANCED CONFIG ###############################
# Hashes are encoded using a memory efficient data structure when they have a
# small number of entries, and the biggest entry does not exceed a given
# threshold. These thresholds can be configured using the following directives.
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
# Lists are also encoded in a special way to save a lot of space.
# The number of entries allowed per internal list node can be specified
# as a fixed maximum size or a maximum number of elements.
# For a fixed maximum size, use -5 through -1, meaning:
# -5: max size: 64 Kb <-- not recommended for normal workloads
# -4: max size: 32 Kb <-- not recommended
# -3: max size: 16 Kb <-- probably not recommended
# -2: max size: 8 Kb <-- good
# -1: max size: 4 Kb <-- good
# Positive numbers mean store up to _exactly_ that number of elements
# per list node.
# The highest performing option is usually -2 (8 Kb size) or -1 (4 Kb size),
# but if your use case is unique, adjust the settings as necessary.
list-max-ziplist-size -2
# Lists may also be compressed.
# Compress depth is the number of quicklist ziplist nodes from *each* side of
# the list to *exclude* from compression. The head and tail of the list
# are always uncompressed for fast push/pop operations. Settings are:
# 0: disable all list compression
# 1: depth 1 means "don't start compressing until after 1 node into the list,
# going from either the head or tail"
# So: [head]->node->node->...->node->[tail]
# [head], [tail] will always be uncompressed; inner nodes will compress.
# 2: [head]->[next]->node->node->...->node->[prev]->[tail]
# 2 here means: don't compress head or head->next or tail->prev or tail,
# but compress all nodes between them.
# 3: [head]->[next]->[next]->node->node->...->node->[prev]->[prev]->[tail]
# etc.
list-compress-depth 0
# Sets have a special encoding in just one case: when a set is composed
# of just strings that happen to be integers in radix 10 in the range
# of 64 bit signed integers.
# The following configuration setting sets the limit in the size of the
# set in order to use this special memory saving encoding.
set-max-intset-entries 512
# Similarly to hashes and lists, sorted sets are also specially encoded in
# order to save a lot of space. This encoding is only used when the length and
# elements of a sorted set are below the following limits:
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
# HyperLogLog sparse representation bytes limit. The limit includes the
# 16 bytes header. When an HyperLogLog using the sparse representation crosses
# this limit, it is converted into the dense representation.
#
# A value greater than 16000 is totally useless, since at that point the
# dense representation is more memory efficient.
#
# The suggested value is ~ 3000 in order to have the benefits of
# the space efficient encoding without slowing down too much PFADD,
# which is O(N) with the sparse encoding. The value can be raised to
# ~ 10000 when CPU is not a concern, but space is, and the data set is
# composed of many HyperLogLogs with cardinality in the 0 - 15000 range.
hll-sparse-max-bytes 3000
# Active rehashing uses 1 millisecond every 100 milliseconds of CPU time in
# order to help rehashing the main Redis hash table (the one mapping top-level
# keys to values). The hash table implementation Redis uses (see dict.c)
# performs a lazy rehashing: the more operation you run into a hash table
# that is rehashing, the more rehashing "steps" are performed, so if the
# server is idle the rehashing is never complete and some more memory is used
# by the hash table.
#
# The default is to use this millisecond 10 times every second in order to
# actively rehash the main dictionaries, freeing memory when possible.
#
# If unsure:
# use "activerehashing no" if you have hard latency requirements and it is
# not a good thing in your environment that Redis can reply from time to time
# to queries with 2 milliseconds delay.
#
# use "activerehashing yes" if you don't have such hard requirements but
# want to free memory asap when possible.
activerehashing yes
# The client output buffer limits can be used to force disconnection of clients
# that are not reading data from the server fast enough for some reason (a
# common reason is that a Pub/Sub client can't consume messages as fast as the
# publisher can produce them).
#
# The limit can be set differently for the three different classes of clients:
#
# normal -> normal clients including MONITOR clients
# slave -> slave clients
# pubsub -> clients subscribed to at least one pubsub channel or pattern
#
# The syntax of every client-output-buffer-limit directive is the following:
#
# client-output-buffer-limit
#
# A client is immediately disconnected once the hard limit is reached, or if
# the soft limit is reached and remains reached for the specified number of
# seconds (continuously).
# So for instance if the hard limit is 32 megabytes and the soft limit is
# 16 megabytes / 10 seconds, the client will get disconnected immediately
# if the size of the output buffers reach 32 megabytes, but will also get
# disconnected if the client reaches 16 megabytes and continuously overcomes
# the limit for 10 seconds.
#
# By default normal clients are not limited because they don't receive data
# without asking (in a push way), but just after a request, so only
# asynchronous clients may create a scenario where data is requested faster
# than it can read.
#
# Instead there is a default limit for pubsub and slave clients, since
# subscribers and slaves receive data in a push fashion.
#
# Both the hard or the soft limit can be disabled by setting them to zero.
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
# Redis calls an internal function to perform many background tasks, like
# closing connections of clients in timeout, purging expired keys that are
# never requested, and so forth.
#
# Not all tasks are performed with the same frequency, but Redis checks for
# tasks to perform according to the specified "hz" value.
#
# By default "hz" is set to 10. Raising the value will use more CPU when
# Redis is idle, but at the same time will make Redis more responsive when
# there are many keys expiring at the same time, and timeouts may be
# handled with more precision.
#
# The range is between 1 and 500, however a value over 100 is usually not
# a good idea. Most users should use the default of 10 and raise this up to
# 100 only in environments where very low latency is required.
hz 10
# When a child rewrites the AOF file, if the following option is enabled
# the file will be fsync-ed every 32 MB of data generated. This is useful
# in order to commit the file to the disk more incrementally and avoid
# big latency spikes.
aof-rewrite-incremental-fsync yes
================================================
FILE: docs/docker/spug.ini
================================================
[supervisord]
nodaemon=true
[program:nginx]
command = nginx -g "daemon off;"
autostart = true
[program:redis]
command = redis-server /etc/redis.conf
autostart = true
[program:spug-api]
command = sh /data/spug/spug_api/tools/start-api.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/api.log
redirect_stderr = true
[program:spug-ws]
command = sh /data/spug/spug_api/tools/start-ws.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/ws.log
redirect_stderr = true
[program:spug-worker]
command = sh /data/spug/spug_api/tools/start-worker.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/worker.log
redirect_stderr = true
[program:spug-monitor]
command = sh /data/spug/spug_api/tools/start-monitor.sh
autostart = true
startsecs = 3
stdout_logfile = /data/spug/spug_api/logs/monitor.log
redirect_stderr = true
[program:spug-scheduler]
command = sh /data/spug/spug_api/tools/start-scheduler.sh
autostart = true
startsecs = 3
stdout_logfile = /data/spug/spug_api/logs/scheduler.log
redirect_stderr = true
================================================
FILE: docs/docker/ssh_config
================================================
Host *
StrictHostKeyChecking no
================================================
FILE: docs/install.sh
================================================
#!/bin/bash
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
set -e
function spug_banner() {
echo " ";
echo " #### ##### # # #### ";
echo "# # # # # # #";
echo " #### # # # # # ";
echo " # ##### # # # ###";
echo "# # # # # # #";
echo " #### # #### #### ";
echo " ";
}
function init_system_lib() {
source /etc/os-release
case $ID in
centos|fedora|rhel)
echo "开始安装/更新可能缺少的依赖: git mariadb-server mariadb-devel python3-devel gcc openldap-devel redis nginx supervisor python36"
yum install -y epel-release
yum install -y git mariadb-server mariadb-devel python3-devel gcc openldap-devel redis nginx supervisor python36
sed -i 's/ default_server//g' /etc/nginx/nginx.conf
MYSQL_CONF=/etc/my.cnf.d/spug.cnf
SUPERVISOR_CONF=/etc/supervisord.d/spug.ini
REDIS_SRV=redis
SUPERVISOR_SRV=supervisord
;;
debian|ubuntu|devuan)
echo "开始安装/更新可能缺少的依赖: git mariadb-server libmariadbd-dev python3-venv libsasl2-dev libldap2-dev redis-server nginx supervisor"
apt update
apt install -y git mariadb-server libmariadbd-dev python3-dev python3-venv libsasl2-dev libldap2-dev redis-server nginx supervisor
rm -f /etc/nginx/sites-enabled/default
MYSQL_CONF=/etc/mysql/conf.d/spug.cnf
SUPERVISOR_CONF=/etc/supervisor/conf.d/spug.conf
REDIS_SRV=redis-server
SUPERVISOR_SRV=supervisor
;;
*)
exit 1
;;
esac
}
function install_spug() {
echo "开始安装Spug..."
mkdir -p /data
cd /data
git clone --depth=1 https://gitee.com/openspug/spug.git
curl -o /tmp/web_latest.tar.gz https://spug.dev/installer/web_latest.tar.gz
tar xf /tmp/web_latest.tar.gz -C spug/spug_web/
cd spug/spug_api
python3 -m venv venv
source venv/bin/activate
pip install wheel -i https://pypi.doubanio.com/simple/
pip install gunicorn mysqlclient -i https://pypi.doubanio.com/simple/
pip install -r requirements.txt -i https://pypi.doubanio.com/simple/
}
function setup_conf() {
echo "开始配置Spug配置..."
# mysql conf
cat << EOF > $MYSQL_CONF
[mysqld]
bind-address=127.0.0.1
EOF
# spug conf
cat << EOF > spug/overrides.py
DEBUG = False
ALLOWED_HOSTS = ['127.0.0.1']
DATABASES = {
'default': {
'ATOMIC_REQUESTS': True,
'ENGINE': 'django.db.backends.mysql',
'NAME': 'spug',
'USER': 'spug',
'PASSWORD': 'spug.dev',
'HOST': '127.0.0.1',
'OPTIONS': {
'charset': 'utf8mb4',
'sql_mode': 'STRICT_TRANS_TABLES',
}
}
}
EOF
cat << EOF > $SUPERVISOR_CONF
[program:spug-api]
command = bash /data/spug/spug_api/tools/start-api.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/api.log
redirect_stderr = true
[program:spug-ws]
command = bash /data/spug/spug_api/tools/start-ws.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/ws.log
redirect_stderr = true
[program:spug-worker]
command = bash /data/spug/spug_api/tools/start-worker.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/worker.log
redirect_stderr = true
[program:spug-monitor]
command = bash /data/spug/spug_api/tools/start-monitor.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/monitor.log
redirect_stderr = true
[program:spug-scheduler]
command = bash /data/spug/spug_api/tools/start-scheduler.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/scheduler.log
redirect_stderr = true
EOF
cat << EOF > /etc/nginx/conf.d/spug.conf
server {
listen 80 default_server;
root /data/spug/spug_web/build/;
location ^~ /api/ {
rewrite ^/api(.*) \$1 break;
proxy_pass http://127.0.0.1:9001;
proxy_redirect off;
proxy_set_header X-Real-IP \$remote_addr;
}
location ^~ /api/ws/ {
rewrite ^/api(.*) \$1 break;
proxy_pass http://127.0.0.1:9002;
proxy_http_version 1.1;
proxy_set_header Upgrade \$http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Real-IP \$remote_addr;
}
error_page 404 /index.html;
}
EOF
systemctl start mariadb
systemctl enable mariadb
mysql -e "create database spug default character set utf8mb4 collate utf8mb4_unicode_ci;"
mysql -e "grant all on spug.* to spug@127.0.0.1 identified by 'spug.dev'"
mysql -e "flush privileges"
python manage.py initdb
python manage.py useradd -u admin -p spug.dev -s -n 管理员
systemctl enable nginx
systemctl enable $REDIS_SRV
systemctl enable $SUPERVISOR_SRV
systemctl restart nginx
systemctl start $REDIS_SRV
systemctl restart $SUPERVISOR_SRV
}
spug_banner
init_system_lib
install_spug
setup_conf
echo -e "\n\n\033[33m安全警告：默认的数据库和Redis服务并不安全，请确保其仅监听在127.0.0.1，推荐参考官网文档自行加固安全配置！\033[0m"
echo -e "\033[32m安装成功！\033[0m"
echo "默认管理员账户：admin 密码：spug.dev"
echo "默认数据库用户：spug 密码：spug.dev"
================================================
FILE: spug_api/.gitignore
================================================
*.pyc
/venv/
__pycache__/
/.idea/
/db.sqlite3
migrations/
/access.log
/repos/*
/logs/*
================================================
FILE: spug_api/apps/account/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/account/history.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from libs.mixins import AdminView
from libs import json_response
from apps.account.models import History
class HistoryView(AdminView):
def get(self, request):
histories = History.objects.all()
return json_response(histories)
================================================
FILE: spug_api/apps/account/management/commands/set.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from apps.setting.utils import AppSetting
class Command(BaseCommand):
help = '系统设置'
def add_arguments(self, parser):
parser.add_argument('target', type=str, help='设置对象')
parser.add_argument('value', type=str, help='设置值')
def echo_success(self, msg):
self.stdout.write(self.style.SUCCESS(msg))
def echo_error(self, msg):
self.stderr.write(self.style.ERROR(msg))
def print_help(self, *args):
message = '''
系统设置命令用法：
set mfa disable 禁用登录MFA
'''
self.stdout.write(message)
def handle(self, *args, **options):
target = options['target']
if target == 'mfa':
if options['value'] != 'disable':
return self.echo_error(f'mfa设置，不支持的值【{options["value"]}】')
AppSetting.set('MFA', {'enable': False})
self.echo_success('MFA已禁用')
else:
self.echo_error('未识别的操作')
self.print_help()
================================================
FILE: spug_api/apps/account/management/commands/update.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from django.conf import settings
import subprocess
import requests
import os
class Command(BaseCommand):
help = '升级Spug版本'
def handle(self, *args, **options):
version, is_repair = settings.SPUG_VERSION, False
res = requests.get(f'https://api.spug.cc/apis/release/latest/?version={version}').json()
if res['error']:
return self.stderr.write(self.style.ERROR(f'获取新版本失败：{res["error"]}'))
if not res['data']['has_new']:
self.stdout.write(res['data']['extra'])
is_repair = True
answer = input(f'\r\n当前已是最新版本，是否要进行修复性更新[y|n]？')
else:
version = res['data']['version']
self.stdout.write(res['data']['content'])
self.stdout.write('\r\n')
self.stdout.write(res['data']['extra'])
answer = input(f'\r\n发现新版本 {version} 是否更新[y|n]？')
if answer.lower() != 'y':
return
# update web
web_dir = os.path.join(settings.BASE_DIR, '../spug_web')
commands = [
f'curl -o /tmp/spug_web.tar.gz https://cdn.spug.cc/spug/web_{version}.tar.gz',
f'rm -rf {web_dir}/build',
f'tar xf /tmp/spug_web.tar.gz -C {web_dir}'
]
task = subprocess.Popen(' && '.join(commands), shell=True)
if task.wait() != 0:
return self.stderr.write(self.style.ERROR('获取更新失败，排除网络问题后请附带输出内容至官方论坛反馈。'))
# update api
commands = [
f'cd {settings.BASE_DIR}',
f'git fetch origin refs/tags/{version}:refs/tags/{version} --no-tags',
f'git checkout {version}'
]
if is_repair:
commands.insert(1, f'git tag -d {version}')
task = subprocess.Popen(' && '.join(commands), shell=True)
if task.wait() != 0:
return self.stderr.write(self.style.ERROR('获取更新失败，排除网络问题后请附带输出内容至官方论坛反馈。'))
# update dep
commands = [
f'cd {settings.BASE_DIR}',
'pip3 install -r requirements.txt -i https://pypi.doubanio.com/simple/'
]
task = subprocess.Popen(' && '.join(commands), shell=True)
if task.wait() != 0:
return self.stderr.write(self.style.ERROR('更新依赖包失败，排除网络问题后请附带输出内容至官方论坛反馈。'))
# update db
apps = [x.split('.')[-1] for x in settings.INSTALLED_APPS if x.startswith('apps.')]
commands = [
f'cd {settings.BASE_DIR}',
f'python3 ./manage.py makemigrations ' + ' '.join(apps),
f'python3 ./manage.py migrate',
f'python3 ./tools/migrate.py {settings.SPUG_VERSION}'
]
task = subprocess.Popen(' && '.join(commands), shell=True)
if task.wait() != 0:
return self.stderr.write(self.style.ERROR('更新表结构失败，请附带输出内容至官方论坛反馈。'))
self.stdout.write(self.style.SUCCESS('''升级成功，请自行重启服务，如果通过官方文档安装一般重启命令为
Docker: docker restart spug
Centos: systemctl restart supervisord
Ubuntu: systemctl restart supervisor
'''))
self.stderr.write(self.style.WARNING(f'最后别忘了刷新浏览器，确保系统设置/关于里的api与web版本一致哦～'))
================================================
FILE: spug_api/apps/account/management/commands/updatedb.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from django.core.management import execute_from_command_line
from django.conf import settings
class Command(BaseCommand):
help = '初始化/更新数据库'
def handle(self, *args, **options):
args = ['manage.py', 'makemigrations']
apps = [x.split('.')[-1] for x in settings.INSTALLED_APPS if x.startswith('apps.')]
execute_from_command_line(args + apps)
execute_from_command_line(['manage.py', 'migrate'])
self.stdout.write(self.style.SUCCESS('初始化/更新成功'))
================================================
FILE: spug_api/apps/account/management/commands/user.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from django.core.cache import cache
from apps.account.models import User
class Command(BaseCommand):
help = '账户管理'
def add_arguments(self, parser):
parser.add_argument('action', type=str, help='执行动作')
parser.add_argument('-u', required=False, help='账户名称')
parser.add_argument('-p', required=False, help='账户密码')
parser.add_argument('-n', required=False, help='账户昵称')
parser.add_argument('-s', default=False, action='store_true', help='是否是超级用户（默认否）')
def echo_success(self, msg):
self.stdout.write(self.style.SUCCESS(msg))
def echo_error(self, msg):
self.stderr.write(self.style.ERROR(msg))
def print_help(self, *args):
message = '''
账户管理命令用法：
user add 创建账户，例如：user add -u admin -p 123 -n 管理员 -s
user reset 重置账户密码，例如：user reset -u admin -p 123
user enable 启用被禁用的账户，例如：user enable -u admin
'''
self.stdout.write(message)
def handle(self, *args, **options):
action = options['action']
if action == 'add':
if not all((options['u'], options['p'], options['n'])):
self.echo_error('缺少参数')
self.print_help()
elif User.objects.filter(username=options['u'], deleted_by_id__isnull=True).exists():
self.echo_error(f'已存在登录名为【{options["u"]}】的用户')
else:
User.objects.create(
username=options['u'],
nickname=options['n'],
password_hash=User.make_password(options['p']),
is_supper=options['s'],
)
self.echo_success('创建用户成功')
elif action == 'enable':
if not options['u']:
self.echo_error('缺少参数')
self.print_help()
user = User.objects.filter(username=options['u'], deleted_by_id__isnull=True).first()
if user:
user.is_active = True
user.save()
cache.delete(user.username)
self.echo_success('账户已启用')
elif action == 'reset':
if not all((options['u'], options['p'])):
self.echo_error('缺少参数')
self.print_help()
user = User.objects.filter(username=options['u'], deleted_by_id__isnull=True).first()
if not user:
return self.echo_error(f'未找到登录名为【{options["u"]}】的账户')
user.password_hash = User.make_password(options['p'])
user.save()
cache.delete(user.username)
self.echo_success('账户密码已重置')
else:
self.echo_error('未识别的操作')
self.print_help()
================================================
FILE: spug_api/apps/account/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from django.core.cache import cache
from libs import ModelMixin, human_datetime
from django.contrib.auth.hashers import make_password, check_password
import json
class User(models.Model, ModelMixin):
username = models.CharField(max_length=100)
nickname = models.CharField(max_length=100)
password_hash = models.CharField(max_length=100) # hashed password
type = models.CharField(max_length=20, default='default')
is_supper = models.BooleanField(default=False)
is_active = models.BooleanField(default=True)
access_token = models.CharField(max_length=32)
token_expired = models.IntegerField(null=True)
last_login = models.CharField(max_length=20)
last_ip = models.CharField(max_length=50)
wx_token = models.CharField(max_length=50, null=True)
roles = models.ManyToManyField('Role', db_table='user_role_rel')
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey('User', models.PROTECT, related_name='+', null=True)
deleted_at = models.CharField(max_length=20, null=True)
deleted_by = models.ForeignKey('User', models.PROTECT, related_name='+', null=True)
@staticmethod
def make_password(plain_password: str) -> str:
return make_password(plain_password, hasher='pbkdf2_sha256')
def verify_password(self, plain_password: str) -> bool:
return check_password(plain_password, self.password_hash)
def get_perms_cache(self):
return cache.get(f'perms_{self.id}', set())
def set_perms_cache(self, value=None):
cache.set(f'perms_{self.id}', value or set())
@property
def page_perms(self):
data = self.get_perms_cache()
if data:
return data
for item in self.roles.all():
if item.page_perms:
perms = json.loads(item.page_perms)
for m, v in perms.items():
for p, d in v.items():
data.update(f'{m}.{p}.{x}' for x in d)
self.set_perms_cache(data)
return data
@property
def deploy_perms(self):
data = {'apps': set(), 'envs': set()}
for item in self.roles.all():
if item.deploy_perms:
perms = json.loads(item.deploy_perms)
data['apps'].update(perms.get('apps', []))
data['envs'].update(perms.get('envs', []))
data['apps'].update(x.id for x in self.app_set.all())
return data
@property
def group_perms(self):
data = set()
for item in self.roles.all():
if item.group_perms:
data.update(json.loads(item.group_perms))
return list(data)
def has_perms(self, codes):
if self.is_supper:
return True
return self.page_perms.intersection(codes)
def __repr__(self):
return '' % self.username
class Meta:
db_table = 'users'
ordering = ('-id',)
class Role(models.Model, ModelMixin):
name = models.CharField(max_length=50)
desc = models.CharField(max_length=255, null=True)
page_perms = models.TextField(null=True)
deploy_perms = models.TextField(null=True)
group_perms = models.TextField(null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, on_delete=models.PROTECT, related_name='+')
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['page_perms'] = json.loads(self.page_perms) if self.page_perms else {}
tmp['deploy_perms'] = json.loads(self.deploy_perms) if self.deploy_perms else {}
tmp['group_perms'] = json.loads(self.group_perms) if self.group_perms else []
tmp['used'] = self.user_set.filter(deleted_by_id__isnull=True).count()
return tmp
def add_deploy_perm(self, target, value):
perms = {'apps': [], 'envs': []}
if self.deploy_perms:
perms.update(json.loads(self.deploy_perms))
perms[target].append(value)
self.deploy_perms = json.dumps(perms)
self.save()
def clear_perms_cache(self):
for item in self.user_set.all():
item.set_perms_cache()
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'roles'
ordering = ('-id',)
class History(models.Model, ModelMixin):
username = models.CharField(max_length=100, null=True)
type = models.CharField(max_length=20, default='default')
ip = models.CharField(max_length=50)
agent = models.CharField(max_length=255, null=True)
message = models.CharField(max_length=255, null=True)
is_success = models.BooleanField(default=True)
created_at = models.CharField(max_length=20, default=human_datetime)
class Meta:
db_table = 'login_histories'
ordering = ('-id',)
================================================
FILE: spug_api/apps/account/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.conf.urls import url
from apps.account.views import *
from apps.account.history import *
urlpatterns = [
url(r'^login/$', login),
url(r'^logout/$', logout),
url(r'^user/$', UserView.as_view()),
url(r'^role/$', RoleView.as_view()),
url(r'^self/$', SelfView.as_view()),
url(r'^login/history/$', HistoryView.as_view())
]
================================================
FILE: spug_api/apps/account/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apps.host.models import Group
import re
def get_host_perms(user):
ids = sub_ids = set(user.group_perms)
while sub_ids:
sub_ids = [x.id for x in Group.objects.filter(parent_id__in=sub_ids)]
ids.update(sub_ids)
return set(x.host_id for x in Group.hosts.through.objects.filter(group_id__in=ids))
def has_host_perm(user, target):
if user.is_supper:
return True
host_ids = get_host_perms(user)
if isinstance(target, (list, set, tuple)):
return set(target).issubset(host_ids)
return int(target) in host_ids
def verify_password(password):
if len(password) < 8:
return False
if not all(map(lambda x: re.findall(x, password), ['[0-9]', '[a-z]', '[A-Z]'])):
return False
return True
================================================
FILE: spug_api/apps/account/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.cache import cache
from django.conf import settings
from libs.mixins import AdminView, View
from libs import JsonParser, Argument, human_datetime, json_response
from libs.utils import get_request_real_ip, generate_random_str
from libs.push import send_login_code
from apps.account.models import User, Role, History
from apps.setting.utils import AppSetting
from apps.account.utils import verify_password
from libs.ldap import LDAP
from functools import partial
import user_agents
import ipaddress
import time
import uuid
import json
class UserView(AdminView):
def get(self, request):
users = []
for u in User.objects.filter(deleted_by_id__isnull=True):
tmp = u.to_dict(excludes=('access_token', 'password_hash'))
tmp['role_ids'] = [x.id for x in u.roles.all()]
tmp['password'] = '******'
users.append(tmp)
return json_response(users)
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('username', help='请输入登录名'),
Argument('password', help='请输入密码'),
Argument('nickname', help='请输入姓名'),
Argument('role_ids', type=list, default=[]),
Argument('wx_token', required=False),
).parse(request.body)
if error is None:
user = User.objects.filter(username=form.username, deleted_by_id__isnull=True).first()
if user and (not form.id or form.id != user.id):
return json_response(error=f'已存在登录名为【{form.username}】的用户')
role_ids, password = form.pop('role_ids'), form.pop('password')
if form.id:
user = User.objects.get(pk=form.id)
user.update_by_dict(form)
else:
if not verify_password(password):
return json_response(error='请设置至少8位包含数字、小写和大写字母的新密码')
user = User.objects.create(
password_hash=User.make_password(password),
created_by=request.user,
**form
)
user.roles.set(role_ids)
user.set_perms_cache()
return json_response(error=error)
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('password', required=False),
Argument('is_active', type=bool, required=False),
).parse(request.body)
if error is None:
user = User.objects.get(pk=form.id)
if form.password:
if not verify_password(form.password):
return json_response(error='请设置至少8位包含数字、小写和大写字母的新密码')
user.token_expired = 0
user.password_hash = User.make_password(form.pop('password'))
if form.is_active is not None:
user.is_active = form.is_active
cache.delete(user.username)
user.save()
return json_response(error=error)
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
user = User.objects.filter(pk=form.id).first()
if user:
if user.type == 'ldap':
return json_response(error='ldap账户无法删除，请使用禁用功能来禁止该账户访问系统')
if user.id == request.user.id:
return json_response(error='无法删除当前登录账户')
user.is_active = True
user.deleted_at = human_datetime()
user.deleted_by = request.user
user.roles.clear()
user.save()
return json_response(error=error)
class RoleView(AdminView):
def get(self, request):
roles = Role.objects.all()
return json_response(roles)
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入角色名称'),
Argument('desc', required=False)
).parse(request.body)
if error is None:
if form.id:
Role.objects.filter(pk=form.id).update(**form)
else:
Role.objects.create(created_by=request.user, **form)
return json_response(error=error)
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('page_perms', type=dict, required=False),
Argument('deploy_perms', type=dict, required=False),
Argument('group_perms', type=list, required=False)
).parse(request.body)
if error is None:
role = Role.objects.filter(pk=form.pop('id')).first()
if not role:
return json_response(error='未找到指定角色')
if form.page_perms is not None:
role.page_perms = json.dumps(form.page_perms)
role.clear_perms_cache()
if form.deploy_perms is not None:
role.deploy_perms = json.dumps(form.deploy_perms)
if form.group_perms is not None:
role.group_perms = json.dumps(form.group_perms)
role.user_set.update(token_expired=0)
role.save()
return json_response(error=error)
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
role = Role.objects.get(pk=form.id)
if role.user_set.exists():
return json_response(error='已有用户使用了该角色，请解除关联后再尝试删除')
role.delete()
return json_response(error=error)
class SelfView(View):
def get(self, request):
data = request.user.to_dict(selects=('nickname', 'wx_token'))
return json_response(data)
def patch(self, request):
form, error = JsonParser(
Argument('old_password', required=False),
Argument('new_password', required=False),
Argument('nickname', required=False, help='请输入昵称'),
Argument('wx_token', required=False),
).parse(request.body)
if error is None:
if form.old_password and form.new_password:
if request.user.type == 'ldap':
return json_response(error='LDAP账户无法修改密码')
if not verify_password(form.new_password):
return json_response(error='请设置至少8位包含数字、小写和大写字母的新密码')
if request.user.verify_password(form.old_password):
request.user.password_hash = User.make_password(form.new_password)
request.user.token_expired = 0
request.user.save()
return json_response()
else:
return json_response(error='原密码错误，请重新输入')
if form.nickname is not None:
request.user.nickname = form.nickname
if form.wx_token is not None:
request.user.wx_token = form.wx_token
request.user.save()
return json_response(error=error)
def login(request):
form, error = JsonParser(
Argument('username', help='请输入用户名'),
Argument('password', help='请输入密码'),
Argument('captcha', required=False),
Argument('type', required=False)
).parse(request.body)
if error is None:
handle_response = partial(handle_login_record, request, form.username, form.type)
user = User.objects.filter(username=form.username, type=form.type).first()
if user and not user.is_active:
return handle_response(error="账户已被系统禁用")
if form.type == 'ldap':
config = AppSetting.get_default('ldap_service')
if not config:
return handle_response(error='请在系统设置中配置LDAP后再尝试通过该方式登录')
ldap = LDAP(**config)
is_success, message = ldap.valid_user(form.username, form.password)
if is_success:
if not user:
user = User.objects.create(username=form.username, nickname=form.username, type=form.type)
return handle_user_info(handle_response, request, user, form.captcha)
elif message:
return handle_response(error=message)
else:
if user and user.deleted_by is None:
if user.verify_password(form.password):
return handle_user_info(handle_response, request, user, form.captcha)
value = cache.get_or_set(form.username, 0, 86400)
if value >= 3:
if user and user.is_active:
user.is_active = False
user.save()
return handle_response(error='账户已被系统禁用')
cache.set(form.username, value + 1, 86400)
return handle_response(error="用户名或密码错误，连续多次错误账户将会被禁用")
return json_response(error=error)
def handle_login_record(request, username, login_type, error=None):
x_real_ip = get_request_real_ip(request.headers)
user_agent = user_agents.parse(request.headers.get('User-Agent'))
History.objects.create(
username=username,
type=login_type,
ip=x_real_ip,
agent=user_agent,
is_success=False if error else True,
message=error
)
if error:
return json_response(error=error)
def handle_user_info(handle_response, request, user, captcha):
cache.delete(user.username)
key = f'{user.username}:code'
if captcha:
code = cache.get(key)
if not code:
return handle_response(error='验证码已失效，请重新获取')
if code != captcha:
ttl = cache.ttl(key)
cache.expire(key, ttl - 100)
return handle_response(error='验证码错误')
cache.delete(key)
else:
mfa = AppSetting.get_default('MFA', {'enable': False})
if mfa['enable']:
if not user.wx_token:
return handle_response(error='已启用登录双重认证，但您的账户未配置推送标识，请联系管理员')
spug_push_key = AppSetting.get_default('spug_push_key')
if not spug_push_key:
return handle_response(error='已启用登录双重认证，但系统未配置推送服务，请联系管理员')
code = generate_random_str(6)
send_login_code(spug_push_key, user.wx_token, code)
cache.set(key, code, 300)
return json_response({'required_mfa': True})
handle_response()
x_real_ip = get_request_real_ip(request.headers)
token_isvalid = user.access_token and len(user.access_token) == 32 and user.token_expired >= time.time()
user.access_token = user.access_token if token_isvalid else uuid.uuid4().hex
user.token_expired = time.time() + settings.TOKEN_TTL
user.last_login = human_datetime()
user.last_ip = x_real_ip
user.save()
verify_ip = AppSetting.get_default('verify_ip', True)
return json_response({
'id': user.id,
'access_token': user.access_token,
'nickname': user.nickname,
'is_supper': user.is_supper,
'has_real_ip': x_real_ip and ipaddress.ip_address(x_real_ip).is_global if verify_ip else True,
'permissions': [] if user.is_supper else list(user.page_perms)
})
def logout(request):
request.user.token_expired = 0
request.user.save()
return json_response()
================================================
FILE: spug_api/apps/alarm/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/alarm/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs import ModelMixin, human_datetime
from apps.account.models import User
import json
class Alarm(models.Model, ModelMixin):
MODES = (
('1', '微信'),
('2', '短信'),
('3', '钉钉'),
('4', '邮件'),
('5', '企业微信'),
('6', '电话'),
('7', '飞书'),
)
STATUS = (
('1', '报警发生'),
('2', '故障恢复'),
)
name = models.CharField(max_length=50)
type = models.CharField(max_length=50)
target = models.CharField(max_length=100)
notify_mode = models.CharField(max_length=255)
notify_grp = models.CharField(max_length=255)
status = models.CharField(max_length=2, choices=STATUS)
duration = models.CharField(max_length=50)
created_at = models.CharField(max_length=20, default=human_datetime)
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['notify_mode'] = ','.join(dict(self.MODES)[x] for x in json.loads(self.notify_mode))
tmp['notify_grp'] = json.loads(self.notify_grp)
tmp['status_alias'] = self.get_status_display()
return tmp
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'alarms'
ordering = ('-id',)
class Group(models.Model, ModelMixin):
name = models.CharField(max_length=50)
desc = models.CharField(max_length=255, null=True)
contacts = models.TextField(null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['contacts'] = json.loads(self.contacts)
return tmp
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'alarm_groups'
ordering = ('-id',)
class Contact(models.Model, ModelMixin):
name = models.CharField(max_length=50)
phone = models.CharField(max_length=20, null=True)
email = models.CharField(max_length=255, null=True)
ding = models.CharField(max_length=255, null=True)
wx_token = models.CharField(max_length=255, null=True)
qy_wx = models.CharField(max_length=255, null=True)
feishu = models.CharField(max_length=255, null=True)
secret = models.TextField(null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'alarm_contacts'
ordering = ('-id',)
================================================
FILE: spug_api/apps/alarm/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('alarm/', AlarmView.as_view()),
path('group/', GroupView.as_view()),
path('contact/', ContactView.as_view()),
path('test/', handle_test),
]
================================================
FILE: spug_api/apps/alarm/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from libs import json_response, JsonParser, Argument, auth
from libs.spug import Notification
from libs.push import get_contacts
from apps.alarm.models import Alarm, Group, Contact
from apps.monitor.models import Detection
from apps.setting.utils import AppSetting
import json
class AlarmView(View):
@auth('alarm.alarm.view')
def get(self, request):
alarms = Alarm.objects.all()
return json_response(alarms)
class GroupView(View):
@auth('alarm.group.view|monitor.monitor.add|monitor.monitor.edit|alarm.alarm.view')
def get(self, request):
groups = Group.objects.all()
return json_response(groups)
@auth('alarm.group.add|alarm.group.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入组名'),
Argument('contacts', type=list, help='请选择联系人'),
Argument('desc', required=False)
).parse(request.body)
if error is None:
form.contacts = json.dumps(form.contacts)
if form.id:
Group.objects.filter(pk=form.id).update(**form)
else:
form.created_by = request.user
Group.objects.create(**form)
return json_response(error=error)
@auth('alarm.group.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
detection = Detection.objects.filter(notify_grp__regex=fr'[^0-9]{form.id}[^0-9]').first()
if detection:
return json_response(error=f'监控任务【{detection.name}】正在使用该报警组，请解除关联后再尝试删除该联系组')
Group.objects.filter(pk=form.id).delete()
return json_response(error=error)
class ContactView(View):
@auth('alarm.contact.view|alarm.group.view|schedule.schedule.add|schedule.schedule.edit')
def get(self, request):
form, error = JsonParser(
Argument('with_push', required=False),
Argument('only_push', required=False),
).parse(request.GET)
if error is None:
response = []
if form.with_push or form.only_push:
push_key = AppSetting.get_default('spug_push_key')
if push_key:
response = get_contacts(push_key)
if form.only_push:
return json_response(response)
for item in Contact.objects.all():
response.append(item.to_dict())
return json_response(response)
return json_response(error=error)
@auth('alarm.contact.add|alarm.contact.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入联系人姓名'),
Argument('phone', required=False),
Argument('email', required=False),
Argument('ding', required=False),
Argument('wx_token', required=False),
Argument('qy_wx', required=False),
Argument('feishu', required=False),
Argument('secret', required=False),
).parse(request.body)
if error is None:
if form.id:
Contact.objects.filter(pk=form.id).update(**form)
else:
form.created_by = request.user
Contact.objects.create(**form)
return json_response(error=error)
@auth('alarm.contact.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
group = Group.objects.filter(contacts__contains=f'\"{form.id}\"').first()
if group:
return json_response(error=f'报警联系组【{group.name}】包含此联系人，请解除关联后再尝试删除该联系人')
Contact.objects.filter(pk=form.id).delete()
return json_response(error=error)
@auth('alarm.contact.add|alarm.contact.edit')
def handle_test(request):
form, error = JsonParser(
Argument('mode', help='参数错误'),
Argument('value', help='参数错误')
).parse(request.body)
if error is None:
notify = Notification(None, '1', 'https://spug.cc', 'Spug官网（测试）', '这是一条测试告警信息', None)
if form.mode == '3':
notify.monitor_by_dd([(form.value, None)])
elif form.mode == '4':
notify.monitor_by_email([form.value])
elif form.mode == '5':
notify.monitor_by_qy_wx([form.value])
elif form.mode == '7':
notify.monitor_by_fs([(form.value, None)])
return json_response(error=error)
================================================
FILE: spug_api/apps/apis/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/apis/config.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.http.response import HttpResponse
from django_redis import get_redis_connection
from apps.config.models import Environment
from apps.app.models import App
from apps.setting.utils import AppSetting
from apps.config.utils import compose_configs
import json
def get_configs(request):
app, env_id, no_prefix = _parse_params(request)
if not app or not env_id:
return HttpResponse('Invalid params', status=400)
configs = compose_configs(app, env_id, no_prefix)
fmt = request.GET.get('format', 'kv')
if fmt == 'kv':
return _kv_response(configs)
elif fmt == 'env':
return _env_response(configs)
elif fmt == 'json':
return _json_response(configs)
else:
return HttpResponse('Unsupported output format', status=400)
def _kv_response(data):
output = ''
for k, v in sorted(data.items()):
output += f'{k} = {v}\r\n'
return HttpResponse(output, content_type='text/plain; charset=utf-8')
def _env_response(data):
output = ''
for k, v in sorted(data.items()):
output += f'{k}={v}\n'
return HttpResponse(output, content_type='text/plain; charset=utf-8')
def _json_response(data):
data = dict(sorted(data.items()))
return HttpResponse(json.dumps(data), content_type='application/json')
def _parse_params(request):
app, env_id = None, None
api_token = request.GET.get('apiToken')
if api_token:
rds = get_redis_connection()
content = rds.get(api_token)
if content:
app_id, env_id = content.decode().split(',')
app = App.objects.filter(pk=app_id).first()
else:
api_key = AppSetting.get_default('api_key')
if api_key and request.GET.get('apiKey') == api_key:
app_key = request.GET.get('app')
env_key = request.GET.get('env')
if app_key and env_key:
app = App.objects.filter(key=app_key).first()
env = Environment.objects.filter(key=env_key).first()
if env:
env_id = env.id
return app, env_id, request.GET.get('noPrefix')
================================================
FILE: spug_api/apps/apis/deploy.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.http.response import HttpResponseBadRequest, HttpResponseForbidden, HttpResponse
from apps.setting.utils import AppSetting
from apps.deploy.models import Deploy, DeployRequest
from apps.repository.models import Repository
from apps.deploy.utils import dispatch as deploy_dispatch
from libs.utils import human_datetime
from threading import Thread
import hashlib
import hmac
import json
def auto_deploy(request, deploy_id, kind):
repo, body = _parse_request(request)
if not repo:
return HttpResponseForbidden()
try:
_, _kind, ref = body['ref'].split('/', 2)
if kind == 'branch' and _kind == 'heads':
commit_id = body['after']
if commit_id != '0000000000000000000000000000000000000000' and ref == request.GET.get('name'):
message = _parse_message(body, repo)
Thread(target=_dispatch, args=(deploy_id, ref, commit_id, message)).start()
return HttpResponse(status=202)
elif kind == 'tag' and _kind == 'tags':
Thread(target=_dispatch, args=(deploy_id, ref)).start()
return HttpResponse(status=202)
return HttpResponse(status=204)
except Exception as e:
return HttpResponseBadRequest(e)
def _parse_request(request):
api_key = AppSetting.get_default('api_key')
token, repo, body = None, None, None
token = request.headers.get('X-Gitlab-Token')
if 'X-Gitlab-Token' in request.headers:
token = request.headers['X-Gitlab-Token']
repo = 'Gitlab'
elif 'X-Gitee-Token' in request.headers:
token = request.headers['X-Gitee-Token']
repo = 'Gitee'
elif 'X-Codeup-Token' in request.headers:
token = request.headers['X-Codeup-Token']
repo = 'Codeup'
elif 'X-Gogs-Signature' in request.headers:
token = request.headers['X-Gogs-Signature']
repo = 'Gogs'
elif 'X-Hub-Signature-256' in request.headers:
token = request.headers['X-Hub-Signature-256'].replace('sha256=', '')
repo = 'Github'
elif 'X-Coding-Signature' in request.headers:
token = request.headers['X-Coding-Signature'].replace('sha1=', '')
repo = 'Coding'
elif 'token' in request.GET: # Compatible the old version of gitlab
token = request.GET.get('token')
repo = 'Gitlab'
if repo in ['Gitlab', 'Gitee', 'Codeup']:
if token != api_key:
return None, None
elif repo in ['Github', 'Gogs']:
en_api_key = hmac.new(api_key.encode(), request.body, hashlib.sha256).hexdigest()
if token != en_api_key:
return None, None
elif repo in ['Coding']:
en_api_key = hmac.new(api_key.encode(), request.body, hashlib.sha1).hexdigest()
if token != en_api_key:
return None, None
else:
return None, None
body = json.loads(request.body)
if repo == 'Gogs' and not body['ref'].startswith('refs/'):
body['ref'] = 'refs/tags/' + body['ref']
return repo, body
def _parse_message(body, repo):
message = None
if repo in ['Gitee', 'Github', 'Coding']:
message = body.get('head_commit', {}).get('message', '')
elif repo in ['Gitlab', 'Codeup', 'Gogs']:
if body.get('commits'):
message = body['commits'][0].get('message', '')
else:
raise ValueError(f'repo {repo} is not supported')
return message[:20].strip()
def _dispatch(deploy_id, ref, commit_id=None, message=None):
deploy = Deploy.objects.filter(pk=deploy_id).first()
if not deploy:
raise Exception(f'no such deploy id for {deploy_id}')
req = DeployRequest(
type='3',
status='0' if deploy.is_audit else '2',
deploy=deploy,
spug_version=Repository.make_spug_version(deploy.id),
host_ids=deploy.host_ids,
created_by=deploy.created_by
)
if commit_id: # branch
req.version = f'{ref}#{commit_id[:6]}'
req.name = message or req.version
if deploy.extend == '1':
req.extra = json.dumps(['branch', ref, commit_id])
else: # tag
req.version = ref
req.name = ref
if deploy.extend == '1':
req.extra = json.dumps(['tag', ref, None])
req.save()
if req.status == '2':
req.do_at = human_datetime()
req.do_by = deploy.created_by
req.save()
deploy_dispatch(req)
================================================
FILE: spug_api/apps/apis/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from apps.apis import config
from apps.apis import deploy
urlpatterns = [
path('config/', config.get_configs),
path('deploy///', deploy.auto_deploy)
]
================================================
FILE: spug_api/apps/app/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/app/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from django.conf import settings
from libs import ModelMixin, human_datetime
from apps.account.models import User
from apps.config.models import Environment
import subprocess
import json
import os
class App(models.Model, ModelMixin):
name = models.CharField(max_length=50)
key = models.CharField(max_length=50, unique=True)
desc = models.CharField(max_length=255, null=True)
rel_apps = models.TextField(null=True)
rel_services = models.TextField(null=True)
sort_id = models.IntegerField(default=0, db_index=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, on_delete=models.PROTECT)
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['rel_apps'] = json.loads(self.rel_apps) if self.rel_apps else []
tmp['rel_services'] = json.loads(self.rel_services) if self.rel_services else []
return tmp
def __repr__(self):
return f''
class Meta:
db_table = 'apps'
ordering = ('-sort_id',)
class Deploy(models.Model, ModelMixin):
EXTENDS = (
('1', '常规发布'),
('2', '自定义发布'),
)
app = models.ForeignKey(App, on_delete=models.PROTECT)
env = models.ForeignKey(Environment, on_delete=models.PROTECT)
host_ids = models.TextField()
extend = models.CharField(max_length=2, choices=EXTENDS)
is_audit = models.BooleanField()
is_parallel = models.BooleanField(default=True)
rst_notify = models.CharField(max_length=255, null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
updated_at = models.CharField(max_length=20, null=True)
updated_by = models.ForeignKey(User, models.PROTECT, related_name='+', null=True)
@property
def extend_obj(self):
cls = DeployExtend1 if self.extend == '1' else DeployExtend2
return cls.objects.filter(deploy=self).first()
def to_dict(self, *args, **kwargs):
deploy = super().to_dict(*args, **kwargs)
deploy['app_key'] = self.app_key if hasattr(self, 'app_key') else None
deploy['app_name'] = self.app_name if hasattr(self, 'app_name') else None
deploy['host_ids'] = json.loads(self.host_ids)
deploy['rst_notify'] = json.loads(self.rst_notify)
deploy.update(self.extend_obj.to_dict())
return deploy
def delete(self, using=None, keep_parents=False):
deploy_id = self.id
super().delete(using, keep_parents)
repo_dir = os.path.join(settings.REPOS_DIR, str(deploy_id))
build_dir = os.path.join(settings.BUILD_DIR, f'{deploy_id}_*')
subprocess.Popen(f'rm -rf {repo_dir} {repo_dir + "_*"} {build_dir}', shell=True)
def __repr__(self):
return '' % (self.app_id, self.env_id)
class Meta:
db_table = 'deploys'
ordering = ('-id',)
class DeployExtend1(models.Model, ModelMixin):
deploy = models.OneToOneField(Deploy, primary_key=True, on_delete=models.CASCADE)
git_repo = models.CharField(max_length=255)
dst_dir = models.CharField(max_length=255)
dst_repo = models.CharField(max_length=255)
versions = models.IntegerField()
filter_rule = models.TextField()
hook_pre_server = models.TextField(null=True)
hook_post_server = models.TextField(null=True)
hook_pre_host = models.TextField(null=True)
hook_post_host = models.TextField(null=True)
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['filter_rule'] = json.loads(self.filter_rule)
return tmp
def __repr__(self):
return '' % self.deploy_id
class Meta:
db_table = 'deploy_extend1'
class DeployExtend2(models.Model, ModelMixin):
deploy = models.OneToOneField(Deploy, primary_key=True, on_delete=models.CASCADE)
server_actions = models.TextField()
host_actions = models.TextField()
require_upload = models.BooleanField(default=False)
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['server_actions'] = json.loads(self.server_actions)
tmp['host_actions'] = json.loads(self.host_actions)
return tmp
def __repr__(self):
return '' % self.deploy_id
class Meta:
db_table = 'deploy_extend2'
================================================
FILE: spug_api/apps/app/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', AppView.as_view()),
path('kit/key/', kit_key),
path('deploy/', DeployView.as_view()),
path('deploy//versions/', get_versions),
]
================================================
FILE: spug_api/apps/app/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.conf import settings
from apps.app.models import Deploy
from apps.setting.utils import AppSetting
from libs.gitlib import Git
import shutil
import os
def parse_envs(text):
data = {}
if text:
for line in text.split('\n'):
fields = line.split('=', 1)
if len(fields) != 2 or fields[0].strip() == '':
raise Exception(f'解析自定义全局变量{line!r}失败，确认其遵循 key = value 格式')
data[fields[0].strip()] = fields[1].strip()
return data
def fetch_versions(deploy: Deploy):
git_repo = deploy.extend_obj.git_repo
repo_dir = os.path.join(settings.REPOS_DIR, str(deploy.id))
pkey = AppSetting.get_default('private_key')
with Git(git_repo, repo_dir, pkey) as git:
return git.fetch_branches_tags()
def fetch_repo(deploy_id, git_repo):
repo_dir = os.path.join(settings.REPOS_DIR, str(deploy_id))
pkey = AppSetting.get_default('private_key')
with Git(git_repo, repo_dir, pkey) as git:
return git.fetch_branches_tags()
def remove_repo(deploy_id):
shutil.rmtree(os.path.join(settings.REPOS_DIR, str(deploy_id)), True)
================================================
FILE: spug_api/apps/app/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.db.models import F
from libs import JsonParser, Argument, json_response, auth
from apps.app.models import App, Deploy, DeployExtend1, DeployExtend2
from apps.config.models import Config, ConfigHistory, Service
from apps.app.utils import fetch_versions, remove_repo
from apps.setting.utils import AppSetting
import json
import re
class AppView(View):
def get(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False)
).parse(request.GET)
if error is None:
if request.user.is_supper:
apps = App.objects.all()
else:
ids = request.user.deploy_perms['apps']
apps = App.objects.filter(id__in=ids)
if form.id:
app = apps.filter(pk=form.id).first()
return json_response(app)
return json_response(apps)
return json_response(error=error)
@auth('deploy.app.add|deploy.app.edit|config.app.add|config.app.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入服务名称'),
Argument('key', help='请输入唯一标识符'),
Argument('desc', required=False)
).parse(request.body)
if error is None:
if not re.fullmatch(r'\w+', form.key, re.ASCII):
return json_response(error='标识符必须为字母、数字和下划线的组合')
app = App.objects.filter(key=form.key).first()
if app and app.id != form.id:
return json_response(error='该识符已存在，请更改后重试')
service = Service.objects.filter(key=form.key).first()
if service:
return json_response(error=f'该标识符已被服务 {service.name} 使用，请更改后重试')
if form.id:
App.objects.filter(pk=form.id).update(**form)
else:
app = App.objects.create(created_by=request.user, **form)
app.sort_id = app.id
app.save()
return json_response(error=error)
@auth('deploy.app.edit|config.app.edit_config')
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('rel_apps', type=list, required=False),
Argument('rel_services', type=list, required=False),
Argument('sort', filter=lambda x: x in ('up', 'down'), required=False)
).parse(request.body)
if error is None:
app = App.objects.filter(pk=form.id).first()
if not app:
return json_response(error='未找到指定应用')
if form.rel_apps is not None:
app.rel_apps = json.dumps(form.rel_apps)
if form.rel_services is not None:
app.rel_services = json.dumps(form.rel_services)
if form.sort:
if form.sort == 'up':
tmp = App.objects.filter(sort_id__gt=app.sort_id).last()
else:
tmp = App.objects.filter(sort_id__lt=app.sort_id).first()
if tmp:
tmp.sort_id, app.sort_id = app.sort_id, tmp.sort_id
tmp.save()
app.save()
return json_response(error=error)
@auth('deploy.app.del|config.app.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
if Deploy.objects.filter(app_id=form.id).exists():
return json_response(error='该应用在应用发布中已存在关联的发布配置，请删除相关发布配置后再尝试删除')
# auto delete configs
Config.objects.filter(type='app', o_id=form.id).delete()
ConfigHistory.objects.filter(type='app', o_id=form.id).delete()
for app in App.objects.filter(rel_apps__isnull=False):
rel_apps = json.loads(app.rel_apps)
if form.id in rel_apps:
rel_apps.remove(form.id)
app.rel_apps = json.dumps(rel_apps)
app.save()
App.objects.filter(pk=form.id).delete()
return json_response(error=error)
class DeployView(View):
@auth('deploy.app.view|deploy.request.view')
def get(self, request):
form, error = JsonParser(
Argument('app_id', type=int, required=False)
).parse(request.GET, True)
if not request.user.is_supper:
perms = request.user.deploy_perms
form.app_id__in = perms['apps']
form.env_id__in = perms['envs']
deploys = Deploy.objects.filter(**form) \
.annotate(app_name=F('app__name'), app_key=F('app__key')) \
.order_by('-app__sort_id')
return json_response(deploys)
@auth('deploy.app.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('app_id', type=int, help='请选择应用'),
Argument('env_id', type=int, help='请选择环境'),
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择要部署的主机'),
Argument('rst_notify', type=dict, help='请选择发布结果通知方式'),
Argument('extend', filter=lambda x: x in dict(Deploy.EXTENDS), help='请选择发布类型'),
Argument('is_parallel', type=bool, default=True),
Argument('is_audit', type=bool, default=False)
).parse(request.body)
if error is None:
deploy = Deploy.objects.filter(app_id=form.app_id, env_id=form.env_id).first()
if deploy and deploy.id != form.id:
return json_response(error='应用在该环境下已经存在发布配置')
form.host_ids = json.dumps(form.host_ids)
form.rst_notify = json.dumps(form.rst_notify)
if form.extend == '1':
extend_form, error = JsonParser(
Argument('git_repo', handler=str.strip, help='请输入git仓库地址'),
Argument('dst_dir', handler=str.strip, help='请输入发布部署路径'),
Argument('dst_repo', handler=str.strip, help='请输入发布存储路径'),
Argument('versions', type=int, filter=lambda x: x > 0, help='请输入发布保留版本数量'),
Argument('filter_rule', type=dict, help='参数错误'),
Argument('hook_pre_server', handler=str.strip, default=''),
Argument('hook_post_server', handler=str.strip, default=''),
Argument('hook_pre_host', handler=str.strip, default=''),
Argument('hook_post_host', handler=str.strip, default='')
).parse(request.body)
if error:
return json_response(error=error)
extend_form.dst_dir = extend_form.dst_dir.rstrip('/')
extend_form.filter_rule = json.dumps(extend_form.filter_rule)
if form.id:
extend = DeployExtend1.objects.filter(deploy_id=form.id).first()
if extend.git_repo != extend_form.git_repo:
remove_repo(form.id)
Deploy.objects.filter(pk=form.id).update(**form)
DeployExtend1.objects.filter(deploy_id=form.id).update(**extend_form)
else:
deploy = Deploy.objects.create(created_by=request.user, **form)
DeployExtend1.objects.create(deploy=deploy, **extend_form)
elif form.extend == '2':
extend_form, error = JsonParser(
Argument('server_actions', type=list, help='请输入执行动作'),
Argument('host_actions', type=list, help='请输入执行动作')
).parse(request.body)
if error:
return json_response(error=error)
if len(extend_form.server_actions) + len(extend_form.host_actions) == 0:
return json_response(error='请至少设置一个执行的动作')
extend_form.require_upload = any(x.get('src_mode') == '1' for x in extend_form.host_actions)
extend_form.server_actions = json.dumps(extend_form.server_actions)
extend_form.host_actions = json.dumps(extend_form.host_actions)
if form.id:
Deploy.objects.filter(pk=form.id).update(**form)
DeployExtend2.objects.filter(deploy_id=form.id).update(**extend_form)
else:
deploy = Deploy.objects.create(created_by=request.user, **form)
DeployExtend2.objects.create(deploy=deploy, **extend_form)
return json_response(error=error)
@auth('deploy.app.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
deploy = Deploy.objects.get(pk=form.id)
if deploy.deployrequest_set.exists():
return json_response(error='已存在关联的发布记录，请删除关联的发布记录后再尝试删除发布配置')
for item in deploy.repository_set.all():
item.delete()
deploy.delete()
return json_response(error=error)
@auth('deploy.app.config|deploy.repository.add|deploy.request.add|deploy.request.edit')
def get_versions(request, d_id):
deploy = Deploy.objects.filter(pk=d_id).first()
if not deploy:
return json_response(error='未找到指定应用')
if deploy.extend == '2':
return json_response(error='该应用不支持此操作')
branches, tags = fetch_versions(deploy)
return json_response({'branches': branches, 'tags': tags})
@auth('deploy.app.config|deploy.app.edit')
def kit_key(request):
form, error = JsonParser(
Argument('key', filter=lambda x: x in ('api_key', 'public_key'), help='参数错误')
).parse(request.body)
if error is None:
api_key = AppSetting.get_default(form.key)
return json_response(api_key)
return json_response(error=error)
================================================
FILE: spug_api/apps/config/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/config/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs import ModelMixin, human_datetime
from apps.account.models import User
class Environment(models.Model, ModelMixin):
name = models.CharField(max_length=50)
key = models.CharField(max_length=50)
desc = models.CharField(max_length=255, null=True)
sort_id = models.IntegerField(default=0, db_index=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, on_delete=models.PROTECT)
def __repr__(self):
return f''
class Meta:
db_table = 'environments'
ordering = ('-sort_id',)
class Service(models.Model, ModelMixin):
name = models.CharField(max_length=50)
key = models.CharField(max_length=50, unique=True)
desc = models.CharField(max_length=255, null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, on_delete=models.PROTECT)
def __repr__(self):
return f''
class Meta:
db_table = 'services'
ordering = ('-id',)
class Config(models.Model, ModelMixin):
TYPES = (
('app', 'App'),
('src', 'Service')
)
type = models.CharField(max_length=5, choices=TYPES)
o_id = models.IntegerField()
key = models.CharField(max_length=50)
env = models.ForeignKey(Environment, on_delete=models.PROTECT)
value = models.TextField(null=True)
desc = models.CharField(max_length=255, null=True)
is_public = models.BooleanField(default=False)
updated_at = models.CharField(max_length=20)
updated_by = models.ForeignKey(User, on_delete=models.PROTECT)
def __repr__(self):
return f''
class Meta:
db_table = 'configs'
ordering = ('-key',)
class ConfigHistory(models.Model, ModelMixin):
ACTIONS = (
('1', '新增'),
('2', '更新'),
('3', '删除')
)
type = models.CharField(max_length=5)
o_id = models.IntegerField()
key = models.CharField(max_length=50)
env_id = models.IntegerField()
value = models.TextField(null=True)
desc = models.CharField(max_length=255, null=True)
is_public = models.BooleanField()
old_value = models.TextField(null=True)
action = models.CharField(max_length=2, choices=ACTIONS)
updated_at = models.CharField(max_length=20)
updated_by = models.ForeignKey(User, on_delete=models.PROTECT)
def __repr__(self):
return f''
class Meta:
db_table = 'config_histories'
ordering = ('key',)
================================================
FILE: spug_api/apps/config/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', ConfigView.as_view()),
path('parse/json/', parse_json),
path('parse/text/', parse_text),
path('diff/', post_diff),
path('environment/', EnvironmentView.as_view()),
path('service/', ServiceView.as_view()),
path('history/', HistoryView.as_view()),
]
================================================
FILE: spug_api/apps/config/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apps.config.models import Config, Service
from apps.app.models import App
import json
def compose_configs(app, env_id, no_prefix=False):
configs = dict()
# app own configs
for item in Config.objects.filter(type='app', o_id=app.id, env_id=env_id).only('key', 'value'):
key = item.key if no_prefix else f'{app.key}_{item.key}'
configs[key] = item.value
# relation app public configs
if app.rel_apps:
app_ids = json.loads(app.rel_apps)
if app_ids:
id_key_map = {x.id: x.key for x in App.objects.filter(id__in=app_ids)}
for item in Config.objects.filter(type='app', o_id__in=app_ids, env_id=env_id, is_public=True) \
.only('key', 'value'):
key = item.key if no_prefix else f'{id_key_map[item.o_id]}_{item.key}'
configs[key] = item.value
# relation service configs
if app.rel_services:
src_ids = json.loads(app.rel_services)
if src_ids:
id_key_map = {x.id: x.key for x in Service.objects.filter(id__in=src_ids)}
for item in Config.objects.filter(type='src', o_id__in=src_ids, env_id=env_id).only('key', 'value'):
key = item.key if no_prefix else f'{id_key_map[item.o_id]}_{item.key}'
configs[key] = item.value
return configs
================================================
FILE: spug_api/apps/config/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.db.models import F
from libs import json_response, JsonParser, Argument, auth
from apps.app.models import Deploy, App
from apps.repository.models import Repository
from apps.config.models import *
import json
import re
class EnvironmentView(View):
def get(self, request):
query = {}
if not request.user.is_supper:
query['id__in'] = request.user.deploy_perms['envs']
envs = Environment.objects.filter(**query)
return json_response(envs)
@auth('config.env.add|config.env.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入环境名称'),
Argument('key', help='请输入唯一标识符'),
Argument('desc', required=False)
).parse(request.body)
if error is None:
if not re.fullmatch(r'\w+', form.key, re.ASCII):
return json_response(error='标识符必须为字母、数字和下划线的组合')
env = Environment.objects.filter(key=form.key).first()
if env and env.id != form.id:
return json_response(error=f'唯一标识符 {form.key} 已存在，请更改后重试')
if form.id:
Environment.objects.filter(pk=form.id).update(**form)
else:
env = Environment.objects.create(created_by=request.user, **form)
env.sort_id = env.id
env.save()
return json_response(error=error)
@auth('config.env.edit')
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('sort', filter=lambda x: x in ('up', 'down'), required=False)
).parse(request.body)
if error is None:
env = Environment.objects.filter(pk=form.id).first()
if not env:
return json_response(error='未找到指定环境')
if form.sort:
if form.sort == 'up':
tmp = Environment.objects.filter(sort_id__gt=env.sort_id).last()
else:
tmp = Environment.objects.filter(sort_id__lt=env.sort_id).first()
if tmp:
tmp.sort_id, env.sort_id = env.sort_id, tmp.sort_id
tmp.save()
env.save()
return json_response(error=error)
@auth('config.env.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
if Deploy.objects.filter(env_id=form.id).exists():
return json_response(error='该环境已关联了发布配置，请删除相关发布配置后再尝试删除')
if Repository.objects.filter(env_id=form.id).exists():
return json_response(error='该环境关联了构建记录，请在删除应用发布/构建仓库中相关记录后再尝试')
# auto delete configs
Config.objects.filter(env_id=form.id).delete()
ConfigHistory.objects.filter(env_id=form.id).delete()
Environment.objects.filter(pk=form.id).delete()
return json_response(error=error)
class ServiceView(View):
@auth('config.src.view')
def get(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False)
).parse(request.GET)
if error is None:
if form.id:
service = Service.objects.get(pk=form.id)
return json_response(service)
services = Service.objects.all()
return json_response(services)
return json_response(error=error)
@auth('config.src.add|config.src.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入服务名称'),
Argument('key', help='请输入唯一标识符'),
Argument('desc', required=False)
).parse(request.body)
if error is None:
if not re.fullmatch(r'\w+', form.key, re.ASCII):
return json_response(error='标识符必须为字母、数字和下划线的组合')
service = Service.objects.filter(key=form.key).first()
if service and service.id != form.id:
return json_response(error='该标识符已存在，请更改后重试')
app = App.objects.filter(key=form.key).first()
if app:
return json_response(error=f'该标识符已被应用 {app.name} 使用，请更改后重试')
if form.id:
Service.objects.filter(pk=form.id).update(**form)
else:
Service.objects.create(created_by=request.user, **form)
return json_response(error=error)
@auth('config.src.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
rel_apps = []
for app in App.objects.filter(rel_services__isnull=False):
rel_services = json.loads(app.rel_services)
if form.id in rel_services:
rel_apps.append(app.name)
if rel_apps:
return json_response(
error=f'该服务在配置中心已被 "{", ".join(rel_apps)}" 依赖，请解除依赖关系后再尝试删除。')
# auto delete configs
Config.objects.filter(type='src', o_id=form.id).delete()
ConfigHistory.objects.filter(type='src', o_id=form.id).delete()
Service.objects.filter(pk=form.id).delete()
return json_response(error=error)
class ConfigView(View):
@auth('config.src.view_config|config.app.view_config')
def get(self, request):
form, error = JsonParser(
Argument('id', type=int, help='未指定操作对象'),
Argument('type', filter=lambda x: x in dict(Config.TYPES), help='缺少必要参数'),
Argument('env_id', type=int, help='缺少必要参数'),
).parse(request.GET)
if error is None:
form.o_id, data = form.pop('id'), []
for item in Config.objects.filter(**form).annotate(update_user=F('updated_by__nickname')):
tmp = item.to_dict()
tmp['update_user'] = item.update_user
data.append(tmp)
return json_response(data)
return json_response(error=error)
@auth('config.src.edit_config|config.app.edit_config')
def post(self, request):
form, error = JsonParser(
Argument('o_id', type=int, help='缺少必要参数'),
Argument('type', filter=lambda x: x in dict(Config.TYPES), help='缺少必要参数'),
Argument('envs', type=list, filter=lambda x: len(x), help='请选择环境'),
Argument('key', help='请输入Key'),
Argument('is_public', type=bool, help='缺少必要参数'),
Argument('value', type=str, default=''),
Argument('desc', required=False)
).parse(request.body)
if error is None:
form.value = form.value.strip()
form.updated_at = human_datetime()
form.updated_by = request.user
envs = form.pop('envs')
for env_id in envs:
cf = Config.objects.filter(o_id=form.o_id, type=form.type, env_id=env_id, key=form.key).first()
if cf:
raise Exception(f'{cf.env.name} 中已存在该Key')
Config.objects.create(env_id=env_id, **form)
ConfigHistory.objects.create(action='1', env_id=env_id, **form)
return json_response(error=error)
@auth('config.src.edit_config|config.app.edit_config')
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='缺少必要参数'),
Argument('value', type=str, default=''),
Argument('is_public', type=bool, help='缺少必要参数'),
Argument('desc', required=False)
).parse(request.body)
if error is None:
form.value = form.value.strip()
config = Config.objects.filter(pk=form.id).first()
if not config:
return json_response(error='未找到指定对象')
config.desc = form.desc
config.is_public = form.is_public
if config.value != form.value:
old_value = config.value
config.value = form.value
config.updated_at = human_datetime()
config.updated_by = request.user
ConfigHistory.objects.create(
action='2',
old_value=old_value,
**config.to_dict(excludes=('id',)))
config.save()
return json_response(error=error)
@auth('config.src.edit_config|config.app.edit_config')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='未指定操作对象')
).parse(request.GET)
if error is None:
config = Config.objects.filter(pk=form.id).first()
if config:
ConfigHistory.objects.create(
action='3',
old_value=config.value,
value='',
updated_at=human_datetime(),
updated_by=request.user,
**config.to_dict(excludes=('id', 'value', 'updated_at', 'updated_by_id'))
)
config.delete()
return json_response(error=error)
class HistoryView(View):
@auth('config.src.view_config|config.app.view_config')
def post(self, request):
form, error = JsonParser(
Argument('o_id', type=int, help='缺少必要参数'),
Argument('env_id', type=int, help='缺少必要参数'),
Argument('type', filter=lambda x: x in dict(Config.TYPES), help='缺少必要参数')
).parse(request.body)
if error is None:
data = []
for item in ConfigHistory.objects.filter(**form).annotate(update_user=F('updated_by__nickname')):
tmp = item.to_dict()
tmp['action_alias'] = item.get_action_display()
tmp['update_user'] = item.update_user
data.append(tmp)
return json_response(data)
return json_response(error=error)
@auth('config.src.view_config|config.app.view_config')
def post_diff(request):
form, error = JsonParser(
Argument('o_id', type=int, help='缺少必要参数'),
Argument('type', filter=lambda x: x in dict(Config.TYPES), help='缺少必要参数'),
Argument('envs', type=list, filter=lambda x: len(x), help='缺少必要参数'),
).parse(request.body)
if error is None:
data, form.env_id__in = {}, form.pop('envs')
for item in Config.objects.filter(**form).order_by('key'):
if item.key in data:
data[item.key][item.env_id] = item.value
else:
data[item.key] = {'key': item.key, item.env_id: item.value}
return json_response(list(data.values()))
return json_response(error=error)
@auth('config.src.edit_config|config.app.edit_config')
def parse_json(request):
form, error = JsonParser(
Argument('o_id', type=int, help='缺少必要参数'),
Argument('type', filter=lambda x: x in dict(Config.TYPES), help='缺少必要参数'),
Argument('env_id', type=int, help='缺少必要参数'),
Argument('data', type=dict, help='缺少必要参数')
).parse(request.body)
if error is None:
data = form.pop('data')
_parse(request, form, data)
return json_response(error=error)
@auth('config.src.edit_config|config.app.edit_config')
def parse_text(request):
form, error = JsonParser(
Argument('o_id', type=int, help='缺少必要参数'),
Argument('type', filter=lambda x: x in dict(Config.TYPES), help='缺少必要参数'),
Argument('env_id', type=int, help='缺少必要参数'),
Argument('data', handler=str.strip, help='缺少必要参数')
).parse(request.body)
if error is None:
data = {}
for line in form.pop('data').split('\n'):
line = line.strip()
if not line or line[0] in ('#', ';'):
continue
fields = line.split('=', 1)
if len(fields) != 2 or fields[0].strip() == '':
return json_response(error=f'解析配置{line!r}失败，确认其遵循 key = value 格式')
data[fields[0].strip()] = fields[1].strip()
_parse(request, form, data)
return json_response(error=error)
def _parse(request, query, data):
for item in Config.objects.filter(**query):
if item.key in data:
value = _filter_value(data.pop(item.key))
if item.value != value:
old_value = item.value
item.value = value
item.updated_at = human_datetime()
item.updated_by = request.user
item.save()
ConfigHistory.objects.create(
action='2',
old_value=old_value,
**item.to_dict(excludes=('id',)))
else:
ConfigHistory.objects.create(
action='3',
old_value=item.value,
value='',
updated_at=human_datetime(),
updated_by=request.user,
**item.to_dict(excludes=('id', 'value', 'updated_at', 'updated_by_id'))
)
item.delete()
for key, value in data.items():
query.key = key
query.is_public = False
query.value = _filter_value(value)
query.updated_at = human_datetime()
query.updated_by = request.user
Config.objects.create(**query)
ConfigHistory.objects.create(action='1', **query)
def _filter_value(value):
if isinstance(value, (str, int)):
value = str(value).strip()
else:
value = json.dumps(value)
return value
================================================
FILE: spug_api/apps/deploy/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/deploy/helper.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.template.defaultfilters import filesizeformat
from libs.utils import human_datetime, render_str, str_decode
from libs.spug import Notification
from apps.host.models import Host
from functools import partial
import subprocess
import json
import os
class SpugError(Exception):
pass
class Helper:
def __init__(self, rds, key):
self.rds = rds
self.key = key
self.callback = []
@classmethod
def make(cls, rds, key, host_ids=None):
if host_ids:
counter, tmp_key = 0, f'{key}_tmp'
data = rds.lrange(key, counter, counter + 9)
while data:
for item in data:
counter += 1
print(item)
tmp = json.loads(item.decode())
if tmp['key'] not in host_ids:
rds.rpush(tmp_key, item)
data = rds.lrange(key, counter, counter + 9)
rds.delete(key)
if rds.exists(tmp_key):
rds.rename(tmp_key, key)
else:
rds.delete(key)
return cls(rds, key)
@classmethod
def _make_dd_notify(cls, url, action, req, version, host_str):
texts = [
f'**申请标题：** {req.name}',
f'**应用名称：** {req.deploy.app.name}',
f'**应用版本：** {version}',
f'**发布环境：** {req.deploy.env.name}',
f'**发布主机：** {host_str}',
]
if action == 'approve_req':
texts.insert(0, '## %s ## ' % '发布审核申请')
texts.extend([
f'**申请人员：** {req.created_by.nickname}',
f'**申请时间：** {human_datetime()}',
'> 来自 Spug运维平台'
])
elif action == 'approve_rst':
color, text = ('#008000', '通过') if req.status == '1' else ('#f90202', '驳回')
texts.insert(0, '## %s ## ' % '发布审核结果')
texts.extend([
f'**审核人员：** {req.approve_by.nickname}',
f'**审核结果：** {text}',
f'**审核意见：** {req.reason or ""}',
f'**审核时间：** {human_datetime()}',
'> 来自 Spug运维平台'
])
else:
color, text = ('#008000', '成功') if req.status == '3' else ('#f90202', '失败')
texts.insert(0, '## %s ## ' % '发布结果通知')
if req.approve_at:
texts.append(f'**审核人员：** {req.approve_by.nickname}')
do_user = req.do_by.nickname if req.type != '3' else 'Webhook'
texts.extend([
f'**执行人员：** {do_user}',
f'**发布结果：** {text}',
f'**发布时间：** {human_datetime()}',
'> 来自 Spug运维平台'
])
data = {
'msgtype': 'markdown',
'markdown': {
'title': 'Spug 发布消息通知',
'text': '\n\n'.join(texts)
},
'at': {
'isAtAll': True
}
}
Notification.handle_request(url, data, 'dd')
@classmethod
def _make_wx_notify(cls, url, action, req, version, host_str):
texts = [
f'申请标题： {req.name}',
f'应用名称： {req.deploy.app.name}',
f'应用版本： {version}',
f'发布环境： {req.deploy.env.name}',
f'发布主机： {host_str}',
]
if action == 'approve_req':
texts.insert(0, '## %s' % '发布审核申请')
texts.extend([
f'申请人员： {req.created_by.nickname}',
f'申请时间： {human_datetime()}',
'> 来自 Spug运维平台'
])
elif action == 'approve_rst':
color, text = ('info', '通过') if req.status == '1' else ('warning', '驳回')
texts.insert(0, '## %s' % '发布审核结果')
texts.extend([
f'审核人员： {req.approve_by.nickname}',
f'审核结果： {text}',
f'审核意见： {req.reason or ""}',
f'审核时间： {human_datetime()}',
'> 来自 Spug运维平台'
])
else:
color, text = ('info', '成功') if req.status == '3' else ('warning', '失败')
texts.insert(0, '## %s' % '发布结果通知')
if req.approve_at:
texts.append(f'审核人员： {req.approve_by.nickname}')
do_user = req.do_by.nickname if req.type != '3' else 'Webhook'
texts.extend([
f'执行人员： {do_user}',
f'发布结果： {text}',
f'发布时间： {human_datetime()}',
'> 来自 Spug运维平台'
])
data = {
'msgtype': 'markdown',
'markdown': {
'content': '\n'.join(texts)
}
}
Notification.handle_request(url, data, 'wx')
@classmethod
def _make_fs_notify(cls, url, action, req, version, host_str):
texts = [
f'申请标题： {req.name}',
f'应用名称： {req.deploy.app.name}',
f'应用版本： {version}',
f'发布环境： {req.deploy.env.name}',
f'发布主机： {host_str}',
]
if action == 'approve_req':
title = '发布审核申请'
texts.extend([
f'申请人员： {req.created_by.nickname}',
f'申请时间： {human_datetime()}',
])
elif action == 'approve_rst':
title = '发布审核结果'
text = '通过' if req.status == '1' else '驳回'
texts.extend([
f'审核人员： {req.approve_by.nickname}',
f'审核结果： {text}',
f'审核意见： {req.reason or ""}',
f'审核时间： {human_datetime()}',
])
else:
title = '发布结果通知'
text = '成功 ✅' if req.status == '3' else '失败 ❗'
if req.approve_at:
texts.append(f'审核人员： {req.approve_by.nickname}')
do_user = req.do_by.nickname if req.type != '3' else 'Webhook'
texts.extend([
f'执行人员： {do_user}',
f'发布结果： {text}',
f'发布时间： {human_datetime()}',
])
data = {
'msg_type': 'post',
'content': {
'post': {
'zh_cn': {
'title': title,
'content': [[{'tag': 'text', 'text': x}] for x in texts] + [[{'tag': 'at', 'user_id': 'all'}]]
}
}
}
}
Notification.handle_request(url, data, 'fs')
@classmethod
def send_deploy_notify(cls, req, action=None):
rst_notify = json.loads(req.deploy.rst_notify)
host_ids = json.loads(req.host_ids) if isinstance(req.host_ids, str) else req.host_ids
if rst_notify['mode'] != '0' and rst_notify.get('value'):
url = rst_notify['value']
version = req.version
hosts = [{'id': x.id, 'name': x.name} for x in Host.objects.filter(id__in=host_ids)]
host_str = ', '.join(x['name'] for x in hosts[:2])
if len(hosts) > 2:
host_str += f'等{len(hosts)}台主机'
if rst_notify['mode'] == '1':
cls._make_dd_notify(url, action, req, version, host_str)
elif rst_notify['mode'] == '2':
data = {
'action': action,
'req_id': req.id,
'req_name': req.name,
'app_id': req.deploy.app_id,
'app_name': req.deploy.app.name,
'env_id': req.deploy.env_id,
'env_name': req.deploy.env.name,
'status': req.status,
'reason': req.reason,
'version': version,
'targets': hosts,
'is_success': req.status == '3',
'created_at': human_datetime()
}
Notification.handle_request(url, data)
elif rst_notify['mode'] == '3':
cls._make_wx_notify(url, action, req, version, host_str)
elif rst_notify['mode'] == '4':
cls._make_fs_notify(url, action, req, version, host_str)
else:
raise NotImplementedError
def add_callback(self, func):
self.callback.append(func)
def parse_filter_rule(self, data: str, sep='\n', env=None):
data, files = data.strip(), []
if data:
for line in data.split(sep):
line = line.strip()
if line and not line.startswith('#'):
files.append(render_str(line, env))
return files
def _send(self, message):
self.rds.rpush(self.key, json.dumps(message))
def send_info(self, key, message):
if message:
self._send({'key': key, 'data': message})
def send_error(self, key, message, with_break=True):
message = f'\r\n\033[31m{message}\033[0m'
self._send({'key': key, 'status': 'error', 'data': message})
if with_break:
raise SpugError
def send_step(self, key, step, data):
self._send({'key': key, 'step': step, 'data': data})
def clear(self):
self.rds.delete(f'{self.key}_tmp')
# save logs for two weeks
self.rds.expire(self.key, 14 * 24 * 60 * 60)
self.rds.close()
# callback
for func in self.callback:
func()
def progress_callback(self, key):
def func(k, n, t):
message = f'\r {filesizeformat(n):<8}/{filesizeformat(t):>8} '
self.send_info(k, message)
self.send_info(key, '\r\n')
return partial(func, key)
def local(self, command, env=None):
if env:
env = dict(env.items())
env.update(os.environ)
task = subprocess.Popen(command, env=env, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
message = b''
while True:
output = task.stdout.read(1)
if not output:
break
if output in (b'\r', b'\n'):
message += b'\r\n' if output == b'\n' else b'\r'
message = str_decode(message)
self.send_info('local', message)
message = b''
else:
message += output
if task.wait() != 0:
self.send_error('local', f'exit code: {task.returncode}')
def remote(self, key, ssh, command, env=None):
code = -1
for code, out in ssh.exec_command_with_stream(command, environment=env):
self.send_info(key, out)
if code != 0:
self.send_error(key, f'exit code: {code}')
def remote_raw(self, key, ssh, command):
code, out = ssh.exec_command_raw(command)
if code != 0:
self.send_error(key, f'exit code: {code}, {out}')
================================================
FILE: spug_api/apps/deploy/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from django.conf import settings
from libs import ModelMixin, human_datetime
from apps.account.models import User
from apps.app.models import Deploy
from apps.repository.models import Repository
import json
import os
class DeployRequest(models.Model, ModelMixin):
STATUS = (
('-3', '发布异常'),
('-1', '已驳回'),
('0', '待审核'),
('1', '待发布'),
('2', '发布中'),
('3', '发布成功'),
)
TYPES = (
('1', '正常发布'),
('2', '回滚'),
('3', '自动发布'),
)
deploy = models.ForeignKey(Deploy, on_delete=models.CASCADE)
repository = models.ForeignKey(Repository, null=True, on_delete=models.SET_NULL)
name = models.CharField(max_length=100)
type = models.CharField(max_length=2, choices=TYPES, default='1')
extra = models.TextField()
host_ids = models.TextField()
desc = models.CharField(max_length=255, null=True)
status = models.CharField(max_length=2, choices=STATUS)
reason = models.CharField(max_length=255, null=True)
version = models.CharField(max_length=100, null=True)
spug_version = models.CharField(max_length=50, null=True)
plan = models.DateTimeField(null=True)
fail_host_ids = models.TextField(default='[]')
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
approve_at = models.CharField(max_length=20, null=True)
approve_by = models.ForeignKey(User, models.PROTECT, related_name='+', null=True)
do_at = models.CharField(max_length=20, null=True)
do_by = models.ForeignKey(User, models.PROTECT, related_name='+', null=True)
@property
def is_quick_deploy(self):
if self.type in ('1', '3') and self.deploy.extend == '1' and self.extra:
extra = json.loads(self.extra)
return extra[0] in ('branch', 'tag')
return False
def delete(self, using=None, keep_parents=False):
super().delete(using, keep_parents)
if self.repository_id:
if not DeployRequest.objects.filter(repository=self.repository).exists():
self.repository.delete()
if self.deploy.extend == '2':
try:
os.remove(os.path.join(settings.REPOS_DIR, str(self.deploy_id), self.spug_version))
except FileNotFoundError:
pass
def __repr__(self):
return f''
class Meta:
db_table = 'deploy_requests'
ordering = ('-id',)
================================================
FILE: spug_api/apps/deploy/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('request/', RequestView.as_view()),
path('request/info/', get_request_info),
path('request/ext1/', post_request_ext1),
path('request/ext1/rollback/', post_request_ext1_rollback),
path('request/ext2/', post_request_ext2),
path('request/upload/', do_upload),
path('request//', RequestDetailView.as_view()),
]
================================================
FILE: spug_api/apps/deploy/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django_redis import get_redis_connection
from django.conf import settings
from django.db import close_old_connections
from libs.utils import AttrDict, human_time, render_str
from apps.host.models import Host
from apps.config.utils import compose_configs
from apps.repository.models import Repository
from apps.repository.utils import dispatch as build_repository
from apps.deploy.models import DeployRequest
from apps.deploy.helper import Helper, SpugError
from concurrent import futures
from functools import partial
import json
import uuid
import os
REPOS_DIR = settings.REPOS_DIR
BUILD_DIR = settings.BUILD_DIR
def dispatch(req, fail_mode=False):
rds = get_redis_connection()
rds_key = f'{settings.REQUEST_KEY}:{req.id}'
if fail_mode:
req.host_ids = req.fail_host_ids
req.fail_mode = fail_mode
req.host_ids = json.loads(req.host_ids)
req.fail_host_ids = req.host_ids[:]
helper = Helper.make(rds, rds_key, req.host_ids if fail_mode else None)
try:
api_token = uuid.uuid4().hex
rds.setex(api_token, 60 * 60, f'{req.deploy.app_id},{req.deploy.env_id}')
env = AttrDict(
SPUG_APP_NAME=req.deploy.app.name,
SPUG_APP_KEY=req.deploy.app.key,
SPUG_APP_ID=str(req.deploy.app_id),
SPUG_REQUEST_ID=str(req.id),
SPUG_REQUEST_NAME=req.name,
SPUG_DEPLOY_ID=str(req.deploy.id),
SPUG_ENV_ID=str(req.deploy.env_id),
SPUG_ENV_KEY=req.deploy.env.key,
SPUG_VERSION=req.version,
SPUG_BUILD_VERSION=req.spug_version,
SPUG_DEPLOY_TYPE=req.type,
SPUG_API_TOKEN=api_token,
SPUG_REPOS_DIR=REPOS_DIR,
)
# append configs
configs = compose_configs(req.deploy.app, req.deploy.env_id)
configs_env = {f'_SPUG_{k.upper()}': v for k, v in configs.items()}
env.update(configs_env)
if req.deploy.extend == '1':
_ext1_deploy(req, helper, env)
else:
_ext2_deploy(req, helper, env)
req.status = '3'
except Exception as e:
req.status = '-3'
raise e
finally:
close_old_connections()
DeployRequest.objects.filter(pk=req.id).update(
status=req.status,
repository=req.repository,
fail_host_ids=json.dumps(req.fail_host_ids),
)
helper.clear()
Helper.send_deploy_notify(req)
def _ext1_deploy(req, helper, env):
if not req.repository_id:
rep = Repository(
app_id=req.deploy.app_id,
env_id=req.deploy.env_id,
deploy_id=req.deploy_id,
version=req.version,
spug_version=req.spug_version,
extra=req.extra,
remarks='SPUG AUTO MAKE',
created_by_id=req.created_by_id
)
build_repository(rep, helper)
req.repository = rep
extras = json.loads(req.extra)
if extras[0] == 'repository':
extras = extras[1:]
if extras[0] == 'branch':
env.update(SPUG_GIT_BRANCH=extras[1], SPUG_GIT_COMMIT_ID=extras[2])
else:
env.update(SPUG_GIT_TAG=extras[1])
if req.deploy.is_parallel:
threads, latest_exception = [], None
max_workers = max(10, os.cpu_count() * 5)
with futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
for h_id in req.host_ids:
new_env = AttrDict(env.items())
t = executor.submit(_deploy_ext1_host, req, helper, h_id, new_env)
t.h_id = h_id
threads.append(t)
for t in futures.as_completed(threads):
exception = t.exception()
if exception:
latest_exception = exception
if not isinstance(exception, SpugError):
helper.send_error(t.h_id, f'Exception: {exception}', False)
else:
req.fail_host_ids.remove(t.h_id)
if latest_exception:
raise latest_exception
else:
host_ids = sorted(req.host_ids, reverse=True)
while host_ids:
h_id = host_ids.pop()
new_env = AttrDict(env.items())
try:
_deploy_ext1_host(req, helper, h_id, new_env)
req.fail_host_ids.remove(h_id)
except Exception as e:
helper.send_error(h_id, f'Exception: {e}', False)
for h_id in host_ids:
helper.send_error(h_id, '终止发布', False)
raise e
def _ext2_deploy(req, helper, env):
extend, step = req.deploy.extend_obj, 1
host_actions = json.loads(extend.host_actions)
server_actions = json.loads(extend.server_actions)
env.update({'SPUG_RELEASE': req.version})
if req.version:
for index, value in enumerate(req.version.split()):
env.update({f'SPUG_RELEASE_{index + 1}': value})
if not req.fail_mode:
helper.send_info('local', f'\033[32m完成√\033[0m\r\n')
for action in server_actions:
helper.send_step('local', step, f'{human_time()} {action["title"]}...\r\n')
helper.local(f'cd /tmp && {action["data"]}', env)
step += 1
for action in host_actions:
if action.get('type') == 'transfer':
action['src'] = render_str(action.get('src', '').strip().rstrip('/'), env)
action['dst'] = render_str(action['dst'].strip().rstrip('/'), env)
if action.get('src_mode') == '1': # upload when publish
extra = json.loads(req.extra)
if 'name' in extra:
action['name'] = extra['name']
break
helper.send_step('local', step, f'{human_time()} 检测到来源为本地路径的数据传输动作，执行打包... \r\n')
action['src'] = action['src'].rstrip('/ ')
action['dst'] = action['dst'].rstrip('/ ')
if not action['src'] or not action['dst']:
helper.send_error('local', f'Invalid path for transfer, src: {action["src"]} dst: {action["dst"]}')
if not os.path.exists(action['src']):
helper.send_error('local', f'No such file or directory: {action["src"]}')
is_dir, exclude = os.path.isdir(action['src']), ''
sp_dir, sd_dst = os.path.split(action['src'])
contain = sd_dst
if action['mode'] != '0' and is_dir:
files = helper.parse_filter_rule(action['rule'], ',', env)
if files:
if action['mode'] == '1':
contain = ' '.join(f'{sd_dst}/{x}' for x in files)
else:
excludes = []
for x in files:
if x.startswith('/'):
excludes.append(f'--exclude={sd_dst}{x}')
else:
excludes.append(f'--exclude={x}')
exclude = ' '.join(excludes)
tar_gz_file = f'{req.spug_version}.tar.gz'
helper.local(f'cd {sp_dir} && tar -zcf {tar_gz_file} {exclude} {contain}')
helper.send_info('local', f'{human_time()} \033[32m完成√\033[0m\r\n')
helper.add_callback(partial(os.remove, os.path.join(sp_dir, tar_gz_file)))
break
helper.send_step('local', 100, '')
if host_actions:
if req.deploy.is_parallel:
threads, latest_exception = [], None
max_workers = max(10, os.cpu_count() * 5)
with futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
for h_id in req.host_ids:
new_env = AttrDict(env.items())
t = executor.submit(_deploy_ext2_host, helper, h_id, host_actions, new_env, req.spug_version)
t.h_id = h_id
threads.append(t)
for t in futures.as_completed(threads):
exception = t.exception()
if exception:
latest_exception = exception
if not isinstance(exception, SpugError):
helper.send_error(t.h_id, f'Exception: {exception}', False)
else:
req.fail_host_ids.remove(t.h_id)
if latest_exception:
raise latest_exception
else:
host_ids = sorted(req.host_ids, reverse=True)
while host_ids:
h_id = host_ids.pop()
new_env = AttrDict(env.items())
try:
_deploy_ext2_host(helper, h_id, host_actions, new_env, req.spug_version)
req.fail_host_ids.remove(h_id)
except Exception as e:
helper.send_error(h_id, f'Exception: {e}', False)
for h_id in host_ids:
helper.send_error(h_id, '终止发布', False)
raise e
else:
req.fail_host_ids = []
helper.send_step('local', 100, f'\r\n{human_time()} ** 发布成功 **')
def _deploy_ext1_host(req, helper, h_id, env):
helper.send_step(h_id, 1, f'\033[32m就绪√\033[0m\r\n{human_time()} 数据准备... ')
host = Host.objects.filter(pk=h_id).first()
if not host:
helper.send_error(h_id, 'no such host')
env.update({'SPUG_HOST_ID': h_id, 'SPUG_HOST_NAME': host.hostname})
extend = req.deploy.extend_obj
extend.dst_dir = render_str(extend.dst_dir, env)
extend.dst_repo = render_str(extend.dst_repo, env)
env.update(SPUG_DST_DIR=extend.dst_dir)
with host.get_ssh(default_env=env) as ssh:
base_dst_dir = os.path.dirname(extend.dst_dir)
code, _ = ssh.exec_command_raw(
f'mkdir -p {extend.dst_repo} {base_dst_dir} && [ -e {extend.dst_dir} ] && [ ! -L {extend.dst_dir} ]')
if code == 0:
helper.send_error(host.id, f'检测到该主机的发布目录 {extend.dst_dir!r} 已存在，为了数据安全请自行备份后删除该目录，Spug 将会创建并接管该目录。')
if req.type == '2':
helper.send_step(h_id, 1, '\033[33m跳过√\033[0m\r\n')
else:
# clean
clean_command = f'ls -d {extend.deploy_id}_* 2> /dev/null | sort -t _ -rnk2 | tail -n +{extend.versions + 1} | xargs rm -rf'
helper.remote_raw(host.id, ssh, f'cd {extend.dst_repo} && {clean_command}')
# transfer files
tar_gz_file = f'{req.spug_version}.tar.gz'
try:
callback = helper.progress_callback(host.id)
ssh.put_file(
os.path.join(BUILD_DIR, tar_gz_file),
os.path.join(extend.dst_repo, tar_gz_file),
callback
)
except Exception as e:
helper.send_error(host.id, f'Exception: {e}')
command = f'cd {extend.dst_repo} && rm -rf {req.spug_version} && tar xf {tar_gz_file} && rm -f {req.deploy_id}_*.tar.gz'
helper.remote_raw(host.id, ssh, command)
helper.send_step(h_id, 1, '\033[32m完成√\033[0m\r\n')
# pre host
repo_dir = os.path.join(extend.dst_repo, req.spug_version)
if extend.hook_pre_host:
helper.send_step(h_id, 2, f'{human_time()} 发布前任务... \r\n')
command = f'cd {repo_dir} && {extend.hook_pre_host}'
helper.remote(host.id, ssh, command)
# do deploy
helper.send_step(h_id, 3, f'{human_time()} 执行发布... ')
helper.remote_raw(host.id, ssh, f'rm -f {extend.dst_dir} && ln -sfn {repo_dir} {extend.dst_dir}')
helper.send_step(h_id, 3, '\033[32m完成√\033[0m\r\n')
# post host
if extend.hook_post_host:
helper.send_step(h_id, 4, f'{human_time()} 发布后任务... \r\n')
command = f'cd {extend.dst_dir} && {extend.hook_post_host}'
helper.remote(host.id, ssh, command)
helper.send_step(h_id, 100, f'\r\n{human_time()} ** \033[32m发布成功\033[0m **')
def _deploy_ext2_host(helper, h_id, actions, env, spug_version):
helper.send_info(h_id, '\033[32m就绪√\033[0m\r\n')
host = Host.objects.filter(pk=h_id).first()
if not host:
helper.send_error(h_id, 'no such host')
env.update({'SPUG_HOST_ID': h_id, 'SPUG_HOST_NAME': host.hostname})
with host.get_ssh(default_env=env) as ssh:
for index, action in enumerate(actions):
helper.send_step(h_id, 1 + index, f'{human_time()} {action["title"]}...\r\n')
if action.get('type') == 'transfer':
if action.get('src_mode') == '1':
try:
dst = action['dst']
command = f'[ -e {dst} ] || mkdir -p $(dirname {dst}); [ -d {dst} ]'
code, _ = ssh.exec_command_raw(command)
if code == 0: # is dir
if not action.get('name'):
raise RuntimeError('internal error 1002')
dst = dst.rstrip('/') + '/' + action['name']
callback = helper.progress_callback(host.id)
ssh.put_file(os.path.join(REPOS_DIR, env.SPUG_DEPLOY_ID, spug_version), dst, callback)
except Exception as e:
helper.send_error(host.id, f'Exception: {e}')
helper.send_info(host.id, 'transfer completed\r\n')
continue
else:
sp_dir, sd_dst = os.path.split(action['src'])
tar_gz_file = f'{spug_version}.tar.gz'
try:
callback = helper.progress_callback(host.id)
ssh.put_file(os.path.join(sp_dir, tar_gz_file), f'/tmp/{tar_gz_file}', callback)
except Exception as e:
helper.send_error(host.id, f'Exception: {e}')
command = f'mkdir -p /tmp/{spug_version} && tar xf /tmp/{tar_gz_file} -C /tmp/{spug_version}/ '
command += f'&& rm -rf {action["dst"]} && mv /tmp/{spug_version}/{sd_dst} {action["dst"]} '
command += f'&& rm -rf /tmp/{spug_version}* && echo "transfer completed"'
else:
command = f'cd /tmp && {action["data"]}'
helper.remote(host.id, ssh, command)
helper.send_step(h_id, 100, f'\r\n{human_time()} ** \033[32m发布成功\033[0m **')
================================================
FILE: spug_api/apps/deploy/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.db.models import F
from django.conf import settings
from django.http.response import HttpResponseBadRequest
from django_redis import get_redis_connection
from libs import json_response, JsonParser, Argument, human_datetime, human_time, auth
from apps.deploy.models import DeployRequest
from apps.app.models import Deploy, DeployExtend2
from apps.repository.models import Repository
from apps.deploy.utils import dispatch, Helper
from apps.host.models import Host
from collections import defaultdict
from threading import Thread
from datetime import datetime
import subprocess
import json
import os
class RequestView(View):
@auth('deploy.request.view')
def get(self, request):
data, query, counter = [], {}, {}
if not request.user.is_supper:
perms = request.user.deploy_perms
query['deploy__app_id__in'] = perms['apps']
query['deploy__env_id__in'] = perms['envs']
for item in DeployRequest.objects.filter(**query).annotate(
env_id=F('deploy__env_id'),
env_name=F('deploy__env__name'),
app_id=F('deploy__app_id'),
app_name=F('deploy__app__name'),
app_host_ids=F('deploy__host_ids'),
app_extend=F('deploy__extend'),
rep_extra=F('repository__extra'),
do_by_user=F('do_by__nickname'),
approve_by_user=F('approve_by__nickname'),
created_by_user=F('created_by__nickname')):
tmp = item.to_dict()
tmp['env_id'] = item.env_id
tmp['env_name'] = item.env_name
tmp['app_id'] = item.app_id
tmp['app_name'] = item.app_name
tmp['app_extend'] = item.app_extend
tmp['host_ids'] = json.loads(item.host_ids)
tmp['fail_host_ids'] = json.loads(item.fail_host_ids)
tmp['extra'] = json.loads(item.extra) if item.extra else None
tmp['rep_extra'] = json.loads(item.rep_extra) if item.rep_extra else None
tmp['app_host_ids'] = json.loads(item.app_host_ids)
tmp['status_alias'] = item.get_status_display()
tmp['created_by_user'] = item.created_by_user
tmp['approve_by_user'] = item.approve_by_user
tmp['do_by_user'] = item.do_by_user
if item.app_extend == '1':
tmp['visible_rollback'] = item.deploy_id not in counter
counter[item.deploy_id] = True
data.append(tmp)
return json_response(data)
@auth('deploy.request.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('mode', filter=lambda x: x in ('count', 'expire', 'deploy'), required=False, help='参数错误'),
Argument('value', required=False),
).parse(request.GET)
if error is None:
if form.id:
deploy = DeployRequest.objects.filter(pk=form.id).first()
if not deploy or deploy.status not in ('0', '1', '-1'):
return json_response(error='未找到指定发布申请或当前状态不允许删除')
deploy.delete()
return json_response()
count = 0
if form.mode == 'count':
if not str(form.value).isdigit() or int(form.value) < 1:
return json_response(error='请输入正确的保留数量')
counter, form.value = defaultdict(int), int(form.value)
for item in DeployRequest.objects.all():
counter[item.deploy_id] += 1
if counter[item.deploy_id] > form.value:
count += 1
item.delete()
elif form.mode == 'expire':
for item in DeployRequest.objects.filter(created_at__lt=form.value):
count += 1
item.delete()
elif form.mode == 'deploy':
app_id, env_id = str(form.value).split(',')
for item in DeployRequest.objects.filter(deploy__app_id=app_id, deploy__env_id=env_id):
count += 1
item.delete()
return json_response(count)
return json_response(error=error)
class RequestDetailView(View):
@auth('deploy.request.view')
def get(self, request, r_id):
req = DeployRequest.objects.filter(pk=r_id).first()
if not req:
return json_response(error='未找到指定发布申请')
hosts = Host.objects.filter(id__in=json.loads(req.host_ids))
outputs = {x.id: {'id': x.id, 'title': x.name, 'data': f'{human_time()} 读取数据... '} for x in hosts}
response = {'outputs': outputs, 'status': req.status}
if req.is_quick_deploy:
outputs['local'] = {'id': 'local', 'data': ''}
if req.deploy.extend == '2':
outputs['local'] = {'id': 'local', 'data': f'{human_time()} 读取数据... '}
response['s_actions'] = json.loads(req.deploy.extend_obj.server_actions)
response['h_actions'] = json.loads(req.deploy.extend_obj.host_actions)
if not response['h_actions']:
response['outputs'] = {'local': outputs['local']}
rds, key, counter = get_redis_connection(), f'{settings.REQUEST_KEY}:{r_id}', 0
data = rds.lrange(key, counter, counter + 9)
while data:
for item in data:
counter += 1
item = json.loads(item.decode())
if item['key'] in outputs:
if 'data' in item:
outputs[item['key']]['data'] += item['data']
if 'step' in item:
outputs[item['key']]['step'] = item['step']
if 'status' in item:
outputs[item['key']]['status'] = item['status']
data = rds.lrange(key, counter, counter + 9)
response['index'] = counter
if counter == 0:
for item in outputs:
outputs[item]['data'] += '\r\n\r\n未读取到数据，Spug 仅保存最近2周的日志信息。'
if req.is_quick_deploy:
if outputs['local']['data']:
outputs['local']['data'] = f'{human_time()} 读取数据... ' + outputs['local']['data']
else:
outputs['local'].update(step=100, data=f'{human_time()} 已构建完成忽略执行。')
return json_response(response)
@auth('deploy.request.do')
def post(self, request, r_id):
form, _ = JsonParser(Argument('mode', default='all')).parse(request.body)
query = {'pk': r_id}
if not request.user.is_supper:
perms = request.user.deploy_perms
query['deploy__app_id__in'] = perms['apps']
query['deploy__env_id__in'] = perms['envs']
req = DeployRequest.objects.filter(**query).first()
if not req:
return json_response(error='未找到指定发布申请')
if req.status not in ('1', '-3'):
return json_response(error='该申请单当前状态还不能执行发布')
host_ids = req.fail_host_ids if form.mode == 'fail' else req.host_ids
hosts = Host.objects.filter(id__in=json.loads(host_ids))
message = f'{human_time()} 等待调度... '
outputs = {x.id: {'id': x.id, 'title': x.name, 'step': 0, 'data': message} for x in hosts}
req.status = '2'
req.do_at = human_datetime()
req.do_by = request.user
req.save()
Thread(target=dispatch, args=(req, form.mode == 'fail')).start()
if req.is_quick_deploy:
if req.repository_id:
outputs['local'] = {'id': 'local', 'step': 100, 'data': f'{human_time()} 已构建完成忽略执行。'}
else:
outputs['local'] = {'id': 'local', 'step': 0, 'data': f'{human_time()} 建立连接... '}
if req.deploy.extend == '2':
outputs['local'] = {'id': 'local', 'step': 0, 'data': f'{human_time()} 建立连接... '}
s_actions = json.loads(req.deploy.extend_obj.server_actions)
h_actions = json.loads(req.deploy.extend_obj.host_actions)
for item in h_actions:
if item.get('type') == 'transfer' and item.get('src_mode') == '0':
s_actions.append({'title': '执行打包'})
if not h_actions:
outputs = {'local': outputs['local']}
return json_response({'s_actions': s_actions, 'h_actions': h_actions, 'outputs': outputs})
return json_response({'outputs': outputs})
@auth('deploy.request.approve')
def patch(self, request, r_id):
form, error = JsonParser(
Argument('reason', required=False),
Argument('is_pass', type=bool, help='参数错误')
).parse(request.body)
if error is None:
req = DeployRequest.objects.filter(pk=r_id).first()
if not req:
return json_response(error='未找到指定申请')
if not form.is_pass and not form.reason:
return json_response(error='请输入驳回原因')
if req.status != '0':
return json_response(error='该申请当前状态不允许审核')
req.approve_at = human_datetime()
req.approve_by = request.user
req.status = '1' if form.is_pass else '-1'
req.reason = form.reason
req.save()
Thread(target=Helper.send_deploy_notify, args=(req, 'approve_rst')).start()
return json_response(error=error)
@auth('deploy.request.add|deploy.request.edit')
def post_request_ext1(request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('deploy_id', type=int, help='参数错误'),
Argument('name', help='请输入申请标题'),
Argument('extra', type=list, help='请选择发布版本'),
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择要部署的主机'),
Argument('type', default='1'),
Argument('plan', required=False),
Argument('desc', required=False),
).parse(request.body)
if error is None:
deploy = Deploy.objects.get(pk=form.deploy_id)
form.spug_version = Repository.make_spug_version(deploy.id)
if form.extra[0] == 'tag':
if not form.extra[1]:
return json_response(error='请选择要发布的版本')
form.version = form.extra[1]
elif form.extra[0] == 'branch':
if not form.extra[2]:
return json_response(error='请选择要发布的分支及Commit ID')
form.version = f'{form.extra[1]}#{form.extra[2][:6]}'
elif form.extra[0] == 'repository':
if not form.extra[1]:
return json_response(error='请选择要发布的版本')
repository = Repository.objects.get(pk=form.extra[1])
form.repository_id = repository.id
form.version = repository.version
form.spug_version = repository.spug_version
form.extra = ['repository'] + json.loads(repository.extra)
else:
return json_response(error='参数错误')
form.extra = json.dumps(form.extra)
form.status = '0' if deploy.is_audit else '1'
form.host_ids = json.dumps(sorted(form.host_ids))
if form.id:
req = DeployRequest.objects.get(pk=form.id)
is_required_notify = deploy.is_audit and req.status == '-1'
DeployRequest.objects.filter(pk=form.id).update(created_by=request.user, reason=None, **form)
else:
req = DeployRequest.objects.create(created_by=request.user, **form)
is_required_notify = deploy.is_audit
if is_required_notify:
Thread(target=Helper.send_deploy_notify, args=(req, 'approve_req')).start()
return json_response(error=error)
@auth('deploy.request.do')
def post_request_ext1_rollback(request):
form, error = JsonParser(
Argument('request_id', type=int, help='请选择要回滚的版本'),
Argument('name', help='请输入申请标题'),
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择要部署的主机'),
Argument('desc', required=False),
).parse(request.body)
if error is None:
req = DeployRequest.objects.get(pk=form.pop('request_id'))
requests = DeployRequest.objects.filter(deploy=req.deploy, status__in=('3', '-3'))
versions = list({x.spug_version: 1 for x in requests}.keys())
if req.spug_version not in versions[:req.deploy.extend_obj.versions + 1]:
return json_response(error='选择的版本超出了发布配置中设置的版本数量，无法快速回滚，可通过新建发布申请选择构建仓库里的该版本再次发布。')
form.status = '0' if req.deploy.is_audit else '1'
form.host_ids = json.dumps(sorted(form.host_ids))
new_req = DeployRequest.objects.create(
deploy_id=req.deploy_id,
repository_id=req.repository_id,
type='2',
extra=req.extra,
version=req.version,
spug_version=req.spug_version,
created_by=request.user,
**form
)
if req.deploy.is_audit:
Thread(target=Helper.send_deploy_notify, args=(new_req, 'approve_req')).start()
return json_response(error=error)
@auth('deploy.request.add|deploy.request.edit')
def post_request_ext2(request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('deploy_id', type=int, help='缺少必要参数'),
Argument('name', help='请输申请标题'),
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择要部署的主机'),
Argument('extra', type=dict, required=False),
Argument('version', default=''),
Argument('type', default='1'),
Argument('plan', required=False),
Argument('desc', required=False),
).parse(request.body)
if error is None:
deploy = Deploy.objects.filter(pk=form.deploy_id).first()
if not deploy:
return json_response(error='未找到该发布配置')
extra = form.pop('extra')
if DeployExtend2.objects.filter(deploy=deploy, host_actions__contains='"src_mode": "1"').exists():
if not extra:
return json_response(error='该应用的发布配置中使用了数据传输动作且设置为发布时上传，请上传要传输的数据')
form.spug_version = extra['path']
form.extra = json.dumps(extra)
else:
form.spug_version = Repository.make_spug_version(deploy.id)
form.name = form.name.replace("'", '')
form.status = '0' if deploy.is_audit else '1'
form.host_ids = json.dumps(form.host_ids)
if form.id:
req = DeployRequest.objects.get(pk=form.id)
is_required_notify = deploy.is_audit and req.status == '-1'
form.update(created_by=request.user, reason=None)
req.update_by_dict(form)
else:
req = DeployRequest.objects.create(created_by=request.user, **form)
is_required_notify = deploy.is_audit
if is_required_notify:
Thread(target=Helper.send_deploy_notify, args=(req, 'approve_req')).start()
return json_response(error=error)
@auth('deploy.request.view')
def get_request_info(request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
req = DeployRequest.objects.get(pk=form.id)
response = req.to_dict(selects=('status', 'reason'))
response['fail_host_ids'] = json.loads(req.fail_host_ids)
response['status_alias'] = req.get_status_display()
return json_response(response)
return json_response(error=error)
@auth('deploy.request.add')
def do_upload(request):
repos_dir = settings.REPOS_DIR
file = request.FILES['file']
deploy_id = request.POST.get('deploy_id')
if file and deploy_id:
dir_name = os.path.join(repos_dir, deploy_id)
file_name = datetime.now().strftime("%Y%m%d%H%M%S")
command = f'mkdir -p {dir_name} && cd {dir_name} && ls | sort -rn | tail -n +11 | xargs rm -rf'
code, outputs = subprocess.getstatusoutput(command)
if code != 0:
return json_response(error=outputs)
with open(os.path.join(dir_name, file_name), 'wb') as f:
for chunk in file.chunks():
f.write(chunk)
return json_response(file_name)
else:
return HttpResponseBadRequest()
================================================
FILE: spug_api/apps/exec/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/exec/executors.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django_redis import get_redis_connection
from libs.utils import human_seconds_time
from libs.ssh import SSH
import threading
import socket
import json
import time
def exec_worker_handler(job):
job = Job(**json.loads(job))
threading.Thread(target=job.run).start()
class Job:
def __init__(self, key, name, hostname, port, username, pkey, command, interpreter, params=None, token=None,
term=None):
self.ssh = SSH(hostname, port, username, pkey, term=term)
self.key = key
self.command = self._handle_command(command, interpreter)
self.token = token
self.rds = get_redis_connection()
self.env = dict(
SPUG_HOST_ID=str(self.key),
SPUG_HOST_NAME=name,
SPUG_HOST_HOSTNAME=hostname,
SPUG_SSH_PORT=str(port),
SPUG_SSH_USERNAME=username,
SPUG_INTERPRETER=interpreter
)
if isinstance(params, dict):
self.env.update({f'_SPUG_{k}': str(v) for k, v in params.items()})
def _send(self, message):
self.rds.publish(self.token, json.dumps(message))
def _handle_command(self, command, interpreter):
if interpreter == 'python':
attach = 'INTERPRETER=python\ncommand -v python3 &> /dev/null && INTERPRETER=python3'
return f'{attach}\n$INTERPRETER << EOF\n# -*- coding: UTF-8 -*-\n{command}\nEOF'
return command
def send(self, data):
self._send({'key': self.key, 'data': data})
def send_status(self, code):
self._send({'key': self.key, 'status': code})
def run(self):
if not self.token:
with self.ssh:
return self.ssh.exec_command(self.command, self.env)
flag = time.time()
self.send('\r\n\x1b[36m### Executing ...\x1b[0m\r\n')
code = -1
try:
with self.ssh:
for code, out in self.ssh.exec_command_with_stream(self.command, self.env):
self.send(out)
human_time = human_seconds_time(time.time() - flag)
self.send(f'\r\n\x1b[36m** 执行结束，总耗时：{human_time} **\x1b[0m')
except socket.timeout:
code = 130
self.send('\r\n\x1b[31m### Time out\x1b[0m')
except Exception as e:
code = 131
self.send(f'\r\n\x1b[31m### Exception {e}\x1b[0m')
raise e
finally:
self.send_status(code)
================================================
FILE: spug_api/apps/exec/management/commands/runworker.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from django.conf import settings
from django.db import connections
from django_redis import get_redis_connection
from concurrent.futures import ThreadPoolExecutor
from apps.schedule.executors import schedule_worker_handler
from apps.monitor.executors import monitor_worker_handler
from apps.exec.executors import exec_worker_handler
from apps.notify.models import Notify
from threading import Thread
import logging
import time
import os
EXEC_WORKER_KEY = settings.EXEC_WORKER_KEY
MONITOR_WORKER_KEY = settings.MONITOR_WORKER_KEY
SCHEDULE_WORKER_KEY = settings.SCHEDULE_WORKER_KEY
logging.basicConfig(level=logging.WARNING, format='%(asctime)s %(message)s')
class Worker:
def __init__(self):
self.rds = get_redis_connection()
self._executor = ThreadPoolExecutor(max_workers=max(100, os.cpu_count() * 50))
def job_done(self, future):
connections.close_all()
def queue_monitor(self):
counter = 0
while True:
time.sleep((counter or 1) ** 3 * 10)
qsize = self._executor._work_queue.qsize()
if qsize > 0:
if counter > 0:
content = '请检查监控、任务计划或批量执行等避免长耗时任务，必要时可重启服务清空队列。'
try:
Notify.make_system_notify(f'执行队列堆积（{qsize}）', content)
except Exception as e:
logging.warning(e)
finally:
connections.close_all()
logging.warning(f'!!! 执行队列堆积（{qsize}）')
counter += 1
else:
counter = 0
def run(self):
logging.warning('Running worker')
Thread(target=self.queue_monitor, daemon=True).start()
self.rds.delete(EXEC_WORKER_KEY, MONITOR_WORKER_KEY, SCHEDULE_WORKER_KEY)
while True:
key, job = self.rds.blpop([EXEC_WORKER_KEY, SCHEDULE_WORKER_KEY, MONITOR_WORKER_KEY])
key = key.decode()
if key == SCHEDULE_WORKER_KEY:
future = self._executor.submit(schedule_worker_handler, job)
elif key == MONITOR_WORKER_KEY:
future = self._executor.submit(monitor_worker_handler, job)
elif key == EXEC_WORKER_KEY:
future = self._executor.submit(exec_worker_handler, job)
else:
continue
future.add_done_callback(self.job_done)
class Command(BaseCommand):
help = 'Start worker process'
def handle(self, *args, **options):
w = Worker()
w.run()
================================================
FILE: spug_api/apps/exec/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs import ModelMixin, human_datetime
from apps.account.models import User
import json
class ExecTemplate(models.Model, ModelMixin):
name = models.CharField(max_length=50)
type = models.CharField(max_length=50)
body = models.TextField()
interpreter = models.CharField(max_length=20, default='sh')
host_ids = models.TextField(default='[]')
desc = models.CharField(max_length=255, null=True)
parameters = models.TextField(default='[]')
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
updated_at = models.CharField(max_length=20, null=True)
updated_by = models.ForeignKey(User, models.PROTECT, related_name='+', null=True)
def __repr__(self):
return '' % self.name
def to_view(self):
tmp = self.to_dict()
tmp['host_ids'] = json.loads(self.host_ids)
tmp['parameters'] = json.loads(self.parameters)
return tmp
class Meta:
db_table = 'exec_templates'
ordering = ('-id',)
class ExecHistory(models.Model, ModelMixin):
user = models.ForeignKey(User, on_delete=models.CASCADE)
template = models.ForeignKey(ExecTemplate, on_delete=models.SET_NULL, null=True)
digest = models.CharField(max_length=32, db_index=True)
interpreter = models.CharField(max_length=20)
command = models.TextField()
params = models.TextField(default='{}')
host_ids = models.TextField()
updated_at = models.CharField(max_length=20, default=human_datetime)
def to_view(self):
tmp = self.to_dict()
tmp['host_ids'] = json.loads(self.host_ids)
if self.template:
tmp['template_name'] = self.template.name
tmp['interpreter'] = self.template.interpreter
tmp['parameters'] = json.loads(self.template.parameters)
tmp['command'] = self.template.body
return tmp
class Meta:
db_table = 'exec_histories'
ordering = ('-updated_at',)
class Transfer(models.Model, ModelMixin):
user = models.ForeignKey(User, on_delete=models.CASCADE)
digest = models.CharField(max_length=32, db_index=True)
host_id = models.IntegerField(null=True)
src_dir = models.CharField(max_length=255)
dst_dir = models.CharField(max_length=255)
host_ids = models.TextField()
updated_at = models.CharField(max_length=20, default=human_datetime)
def to_view(self):
tmp = self.to_dict()
tmp['host_ids'] = json.loads(self.host_ids)
return tmp
class Meta:
db_table = 'exec_transfer'
ordering = ('-id',)
================================================
FILE: spug_api/apps/exec/transfer.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.conf import settings
from django.db import close_old_connections
from django_redis import get_redis_connection
from apps.exec.models import Transfer
from apps.account.utils import has_host_perm
from apps.host.models import Host
from apps.setting.utils import AppSetting
from libs import json_response, JsonParser, Argument, auth
from libs.utils import str_decode, human_seconds_time
from concurrent import futures
from threading import Thread
import subprocess
import tempfile
import uuid
import json
import time
import os
class TransferView(View):
@auth('exec.transfer.do')
def get(self, request):
records = Transfer.objects.filter(user=request.user)
return json_response([x.to_view() for x in records])
@auth('exec.transfer.do')
def post(self, request):
data = request.POST.get('data')
form, error = JsonParser(
Argument('host', required=False),
Argument('dst_dir', help='请输入目标路径'),
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择目标主机'),
).parse(data)
if error is None:
if not has_host_perm(request.user, form.host_ids):
return json_response(error='无权访问主机，请联系管理员')
host_id = None
token = uuid.uuid4().hex
base_dir = os.path.join(settings.TRANSFER_DIR, token)
if form.host:
host_id, path = json.loads(form.host)
if not path.strip('/'):
return json_response(error='请输入正确的数据源路径')
host = Host.objects.get(pk=host_id)
with host.get_ssh() as ssh:
code, _ = ssh.exec_command_raw(f'[ -d {path} ]')
if code != 0:
return json_response(error='数据源路径必须为该主机上已存在的目录')
os.makedirs(base_dir)
with tempfile.NamedTemporaryFile(mode='w') as fp:
fp.write(host.pkey or AppSetting.get('private_key'))
fp.flush()
target = f'{host.username}@{host.hostname}:{path}'
command = f'sshfs -o ro -o ssh_command="ssh -p {host.port} -i {fp.name}" {target} {base_dir}'
task = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
if task.returncode != 0:
os.system(f'umount -f {base_dir} &> /dev/null ; rm -rf {base_dir}')
return json_response(error=task.stdout.decode())
else:
os.makedirs(base_dir)
index = 0
while True:
file = request.FILES.get(f'file{index}')
if not file:
break
with open(os.path.join(base_dir, file.name), 'wb') as f:
for chunk in file.chunks():
f.write(chunk)
index += 1
Transfer.objects.create(
user=request.user,
digest=token,
host_id=host_id,
src_dir=base_dir,
dst_dir=form.dst_dir,
host_ids=json.dumps(form.host_ids),
)
return json_response(token)
return json_response(error=error)
@auth('exec.transfer.do')
def patch(self, request):
form, error = JsonParser(
Argument('token', help='参数错误')
).parse(request.body)
if error is None:
task = Transfer.objects.get(digest=form.token)
Thread(target=_dispatch_sync, args=(task,)).start()
return json_response(error=error)
def _dispatch_sync(task):
rds = get_redis_connection()
threads = []
max_workers = max(10, os.cpu_count() * 5)
with futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
for host in Host.objects.filter(id__in=json.loads(task.host_ids)):
t = executor.submit(_do_sync, rds, task, host)
t.token = task.digest
t.key = host.id
threads.append(t)
for t in futures.as_completed(threads):
exc = t.exception()
if exc:
rds.publish(
t.token,
json.dumps({'key': t.key, 'status': -1, 'data': f'\x1b[31mException: {exc}\x1b[0m'})
)
if task.host_id:
command = f'umount -f {task.src_dir} && rm -rf {task.src_dir}'
else:
command = f'rm -rf {task.src_dir}'
subprocess.run(command, shell=True)
close_old_connections()
def _do_sync(rds, task, host):
token = task.digest
rds.publish(token, json.dumps({'key': host.id, 'data': '\r\n\x1b[36m### Executing ...\x1b[0m\r\n'}))
with tempfile.NamedTemporaryFile(mode='w') as fp:
fp.write(host.pkey or AppSetting.get('private_key'))
fp.write('\n')
fp.flush()
flag = time.time()
options = '-azv --progress' if task.host_id else '-rzv --progress'
argument = f'{task.src_dir}/ {host.username}@{host.hostname}:{task.dst_dir}'
command = f'rsync {options} -h -e "ssh -p {host.port} -o StrictHostKeyChecking=no -i {fp.name}" {argument}'
task = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
message = b''
while True:
output = task.stdout.read(1)
if not output:
break
if output in (b'\r', b'\n'):
message += b'\r\n' if output == b'\n' else b'\r'
message = str_decode(message)
if 'rsync: command not found' in message:
data = '\r\n\x1b[31m检测到该主机未安装rsync，可通过批量执行/执行任务模块进行以下命令批量安装\x1b[0m'
data += '\r\nCentos/Redhat: yum install -y rsync'
data += '\r\nUbuntu/Debian: apt install -y rsync'
rds.publish(token, json.dumps({'key': host.id, 'data': data}))
break
rds.publish(token, json.dumps({'key': host.id, 'data': message}))
message = b''
else:
message += output
status = task.wait()
if status == 0:
human_time = human_seconds_time(time.time() - flag)
rds.publish(token, json.dumps({'key': host.id, 'data': f'\r\n\x1b[32m** 分发完成，总耗时：{human_time} **\x1b[0m'}))
rds.publish(token, json.dumps({'key': host.id, 'status': task.wait()}))
================================================
FILE: spug_api/apps/exec/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.conf.urls import url
from apps.exec.views import *
from apps.exec.transfer import TransferView
urlpatterns = [
url(r'template/$', TemplateView.as_view()),
url(r'do/$', TaskView.as_view()),
url(r'transfer/$', TransferView.as_view()),
]
================================================
FILE: spug_api/apps/exec/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django_redis import get_redis_connection
from django.conf import settings
from libs import json_response, JsonParser, Argument, human_datetime, auth
from apps.exec.models import ExecTemplate, ExecHistory
from apps.host.models import Host
from apps.account.utils import has_host_perm
import uuid
import json
class TemplateView(View):
@auth('exec.template.view|exec.task.do|schedule.schedule.add|schedule.schedule.edit|\
monitor.monitor.add|monitor.monitor.edit')
def get(self, request):
templates = ExecTemplate.objects.all()
types = [x['type'] for x in templates.order_by('type').values('type').distinct()]
return json_response({'types': types, 'templates': [x.to_view() for x in templates]})
@auth('exec.template.add|exec.template.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入模版名称'),
Argument('type', help='请选择模版类型'),
Argument('body', help='请输入模版内容'),
Argument('interpreter', default='sh'),
Argument('host_ids', type=list, handler=json.dumps, default=[]),
Argument('parameters', type=list, handler=json.dumps, default=[]),
Argument('desc', required=False)
).parse(request.body)
if error is None:
if form.id:
form.updated_at = human_datetime()
form.updated_by = request.user
ExecTemplate.objects.filter(pk=form.pop('id')).update(**form)
else:
form.created_by = request.user
ExecTemplate.objects.create(**form)
return json_response(error=error)
@auth('exec.template.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
ExecTemplate.objects.filter(pk=form.id).delete()
return json_response(error=error)
class TaskView(View):
@auth('exec.task.do')
def get(self, request):
records = ExecHistory.objects.filter(user=request.user).select_related('template')
return json_response([x.to_view() for x in records])
@auth('exec.task.do')
def post(self, request):
form, error = JsonParser(
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择执行主机'),
Argument('command', help='请输入执行命令内容'),
Argument('interpreter', default='sh'),
Argument('template_id', type=int, required=False),
Argument('params', type=dict, handler=json.dumps, default={})
).parse(request.body)
if error is None:
if not has_host_perm(request.user, form.host_ids):
return json_response(error='无权访问主机，请联系管理员')
token, rds = uuid.uuid4().hex, get_redis_connection()
form.host_ids.sort()
if form.template_id:
template = ExecTemplate.objects.filter(pk=form.template_id).first()
if not template or template.body != form.command:
form.template_id = None
ExecHistory.objects.create(
user=request.user,
digest=token,
interpreter=form.interpreter,
template_id=form.template_id,
command=form.command,
host_ids=json.dumps(form.host_ids),
params=form.params
)
return json_response(token)
return json_response(error=error)
@auth('exec.task.do')
def patch(self, request):
form, error = JsonParser(
Argument('token', help='参数错误'),
Argument('cols', type=int, required=False),
Argument('rows', type=int, required=False)
).parse(request.body)
if error is None:
term = None
if form.cols and form.rows:
term = {'width': form.cols, 'height': form.rows}
rds = get_redis_connection()
task = ExecHistory.objects.get(digest=form.token)
for host in Host.objects.filter(id__in=json.loads(task.host_ids)):
data = dict(
key=host.id,
name=host.name,
token=task.digest,
interpreter=task.interpreter,
hostname=host.hostname,
port=host.port,
username=host.username,
command=task.command,
pkey=host.private_key,
params=json.loads(task.params),
term=term
)
rds.rpush(settings.EXEC_WORKER_KEY, json.dumps(data))
return json_response(error=error)
================================================
FILE: spug_api/apps/file/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/file/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', FileView.as_view()),
path('object/', ObjectView.as_view()),
]
================================================
FILE: spug_api/apps/file/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.http import FileResponse
import stat
import time
import os
KB = 1024
MB = 1024 * 1024
GB = 1024 * 1024 * 1024
TB = 1024 * 1024 * 1024 * 1024
class FileResponseAfter(FileResponse):
def __init__(self, callback, *args, **kwargs):
super().__init__(*args, **kwargs)
self.callback = callback
def close(self):
super().close()
self.callback()
def parse_mode(obj):
if obj.st_mode:
mt = stat.S_IFMT(obj.st_mode)
if mt == stat.S_IFIFO:
kind = 'p'
elif mt == stat.S_IFCHR:
kind = 'c'
elif mt == stat.S_IFDIR:
kind = 'd'
elif mt == stat.S_IFBLK:
kind = 'b'
elif mt == stat.S_IFREG:
kind = '-'
elif mt == stat.S_IFLNK:
kind = 'l'
elif mt == stat.S_IFSOCK:
kind = 's'
else:
kind = '?'
code = obj._rwx(
(obj.st_mode & 448) >> 6, obj.st_mode & stat.S_ISUID
)
code += obj._rwx(
(obj.st_mode & 56) >> 3, obj.st_mode & stat.S_ISGID
)
code += obj._rwx(
obj.st_mode & 7, obj.st_mode & stat.S_ISVTX, True
)
return kind + code
else:
return '?---------'
def format_size(size):
if size:
if size < KB:
return f'{size}B'
if size < MB:
return f'{size / KB:.1f}K'
if size < GB:
return f'{size / MB:.1f}M'
if size < TB:
return f'{size / GB:.1f}G'
return f'{size / TB:.1f}T'
else:
return ''
def fetch_dir_list(host, path):
with host.get_ssh() as ssh:
objects = []
for item in ssh.list_dir_attr(path):
code = parse_mode(item)
kind, is_link, name = '?', False, getattr(item, 'filename', '?')
if stat.S_ISLNK(item.st_mode):
is_link = True
try:
item = ssh.sftp_stat(os.path.join(path, name))
except FileNotFoundError:
pass
if stat.S_ISREG(item.st_mode):
kind = '-'
elif stat.S_ISDIR(item.st_mode):
kind = 'd'
if (item.st_mtime is None) or (item.st_mtime == int(0xffffffff)):
date = '(unknown date)'
else:
date = time.strftime('%Y/%m/%d %H:%M:%S', time.localtime(item.st_mtime))
objects.append({
'name': name,
'size': '' if kind == 'd' else format_size(item.st_size or ''),
'date': date,
'kind': kind,
'code': code,
'is_link': is_link
})
return objects
================================================
FILE: spug_api/apps/file/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django_redis import get_redis_connection
from apps.host.models import Host
from apps.account.utils import has_host_perm
from apps.file.utils import FileResponseAfter, fetch_dir_list
from libs import json_response, JsonParser, Argument, auth
from functools import partial
import os
class FileView(View):
@auth('host.console.list')
def get(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('path', help='参数错误')
).parse(request.GET)
if error is None:
if not has_host_perm(request.user, form.id):
return json_response(error='无权访问主机，请联系管理员')
host = Host.objects.get(pk=form.id)
if not host:
return json_response(error='未找到指定主机')
objects = fetch_dir_list(host, form.path)
return json_response(objects)
return json_response(error=error)
class ObjectView(View):
@auth('host.console.list')
def get(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('file', help='请输入文件路径')
).parse(request.GET)
if error is None:
if not has_host_perm(request.user, form.id):
return json_response(error='无权访问主机，请联系管理员')
host = Host.objects.filter(pk=form.id).first()
if not host:
return json_response(error='未找到指定主机')
filename = os.path.basename(form.file)
ssh_cli = host.get_ssh().get_client()
sftp = ssh_cli.open_sftp()
f = sftp.open(form.file)
return FileResponseAfter(ssh_cli.close, f, as_attachment=True, filename=filename)
return json_response(error=error)
@auth('host.console.upload')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('token', help='参数错误'),
Argument('path', help='参数错误'),
).parse(request.POST)
if error is None:
if not has_host_perm(request.user, form.id):
return json_response(error='无权访问主机，请联系管理员')
file = request.FILES.get('file')
if not file:
return json_response(error='请选择要上传的文件')
host = Host.objects.get(pk=form.id)
if not host:
return json_response(error='未找到指定主机')
rds_cli = get_redis_connection()
callback = partial(self._compute_progress, rds_cli, form.token, file.size)
with host.get_ssh() as ssh:
ssh.put_file_by_fl(file, f'{form.path}/{file.name}', callback=callback)
return json_response(error=error)
@auth('host.console.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('file', help='请输入文件路径')
).parse(request.GET)
if error is None:
if not has_host_perm(request.user, form.id):
return json_response(error='无权访问主机，请联系管理员')
host = Host.objects.get(pk=form.id)
if not host:
return json_response(error='未找到指定主机')
with host.get_ssh() as ssh:
ssh.remove_file(form.file)
return json_response(error=error)
def _compute_progress(self, rds_cli, token, total, value, *args):
percent = '%.1f' % (value / total * 100)
rds_cli.publish(token, percent)
================================================
FILE: spug_api/apps/home/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/home/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs.mixins import ModelMixin
import json
class Notice(models.Model, ModelMixin):
title = models.CharField(max_length=100)
content = models.TextField()
is_stress = models.BooleanField(default=False)
read_ids = models.TextField(default='[]')
sort_id = models.IntegerField(default=0, db_index=True)
created_at = models.DateTimeField(auto_now_add=True)
def to_view(self):
tmp = self.to_dict()
tmp['read_ids'] = json.loads(self.read_ids)
return tmp
class Meta:
db_table = 'notices'
ordering = ('-sort_id',)
class Navigation(models.Model, ModelMixin):
title = models.CharField(max_length=64)
desc = models.CharField(max_length=128)
logo = models.TextField()
links = models.TextField()
sort_id = models.IntegerField(default=0, db_index=True)
created_at = models.DateTimeField(auto_now_add=True)
def to_view(self):
tmp = self.to_dict()
tmp['links'] = json.loads(self.links)
return tmp
class Meta:
db_table = 'navigations'
ordering = ('-sort_id',)
================================================
FILE: spug_api/apps/home/navigation.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from libs import json_response, JsonParser, Argument
from apps.home.models import Navigation
import json
class NavView(View):
def get(self, request):
navs = Navigation.objects.all()
return json_response([x.to_view() for x in navs])
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('title', help='请输入导航标题'),
Argument('desc', help='请输入导航描述'),
Argument('logo', help='请上传导航logo'),
Argument('links', type=list, filter=lambda x: len(x), help='请设置导航链接'),
).parse(request.body)
if error is None:
form.links = json.dumps(form.links)
if form.id:
Navigation.objects.filter(pk=form.id).update(**form)
else:
nav = Navigation.objects.create(**form)
nav.sort_id = nav.id
nav.save()
return json_response(error=error)
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('sort', filter=lambda x: x in ('up', 'down'), required=False),
).parse(request.body)
if error is None:
nav = Navigation.objects.filter(pk=form.id).first()
if not nav:
return json_response(error='未找到指定记录')
if form.sort:
if form.sort == 'up':
tmp = Navigation.objects.filter(sort_id__gt=nav.sort_id).last()
else:
tmp = Navigation.objects.filter(sort_id__lt=nav.sort_id).first()
if tmp:
tmp.sort_id, nav.sort_id = nav.sort_id, tmp.sort_id
tmp.save()
nav.save()
return json_response(error=error)
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
Navigation.objects.filter(pk=form.id).delete()
return json_response(error=error)
================================================
FILE: spug_api/apps/home/notice.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from libs import json_response, JsonParser, Argument
from apps.home.models import Notice
import json
class NoticeView(View):
def get(self, request):
notices = Notice.objects.all()
return json_response([x.to_view() for x in notices])
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('title', help='请输入标题'),
Argument('content', help='请输入内容'),
Argument('is_stress', type=bool, default=False),
).parse(request.body)
if error is None:
if form.is_stress:
Notice.objects.update(is_stress=False)
if form.id:
Notice.objects.filter(pk=form.id).update(**form)
else:
notice = Notice.objects.create(**form)
notice.sort_id = notice.id
notice.save()
return json_response(error=error)
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('sort', filter=lambda x: x in ('up', 'down'), required=False),
Argument('read', required=False)
).parse(request.body)
if error is None:
notice = Notice.objects.filter(pk=form.id).first()
if not notice:
return json_response(error='未找到指定记录')
if form.sort:
if form.sort == 'up':
tmp = Notice.objects.filter(sort_id__gt=notice.sort_id).last()
else:
tmp = Notice.objects.filter(sort_id__lt=notice.sort_id).first()
if tmp:
tmp.sort_id, notice.sort_id = notice.sort_id, tmp.sort_id
tmp.save()
if form.read:
read_ids = json.loads(notice.read_ids)
read_ids.append(str(request.user.id))
notice.read_ids = json.dumps(read_ids)
notice.save()
return json_response(error=error)
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
Notice.objects.filter(pk=form.id).delete()
return json_response(error=error)
================================================
FILE: spug_api/apps/home/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
from apps.home.notice import NoticeView
from apps.home.navigation import NavView
urlpatterns = [
path('statistic/', get_statistic),
path('alarm/', get_alarm),
path('deploy/', get_deploy),
path('request/', get_request),
path('notice/', NoticeView.as_view()),
path('navigation/', NavView.as_view()),
]
================================================
FILE: spug_api/apps/home/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apps.app.models import App
from apps.host.models import Host
from apps.schedule.models import Task
from apps.monitor.models import Detection
from apps.alarm.models import Alarm
from apps.deploy.models import Deploy, DeployRequest
from apps.account.utils import get_host_perms
from libs.utils import json_response, human_date, parse_time
from libs.parser import JsonParser, Argument
from libs.decorators import auth
from datetime import datetime, timedelta
import json
@auth('dashboard.dashboard.view')
def get_statistic(request):
if request.user.is_supper:
app = App.objects.count()
host = Host.objects.count()
else:
deploy_perms, host_perms = request.user.deploy_perms, get_host_perms(request.user)
app = App.objects.filter(id__in=deploy_perms['apps']).count()
host = len(host_perms)
data = {
'app': app,
'host': host,
'task': Task.objects.count(),
'detection': Detection.objects.count()
}
return json_response(data)
@auth('dashboard.dashboard.view')
def get_alarm(request):
form, error = JsonParser(
Argument('type', required=False),
Argument('name', required=False)
).parse(request.GET, True)
if error is None:
now = datetime.now()
data = {human_date(now - timedelta(days=x + 1)): 0 for x in range(14)}
for alarm in Alarm.objects.filter(status='1', created_at__gt=human_date(now - timedelta(days=14)), **form):
date = alarm.created_at[:10]
if date in data:
data[date] += 1
data = [{'date': k, 'value': v} for k, v in data.items()]
return json_response(data)
return json_response(error=error)
@auth('dashboard.dashboard.view')
def get_request(request):
form, error = JsonParser(
Argument('duration', type=list, help='参数错误')
).parse(request.body)
if error is None:
s_date = form.duration[0]
e_date = (parse_time(form.duration[1]) + timedelta(days=1)).strftime('%Y-%m-%d')
data = {x.id: {'name': x.name, 'count': 0} for x in App.objects.all()}
for req in DeployRequest.objects.filter(created_at__gt=s_date, created_at__lt=e_date):
data[req.deploy.app_id]['count'] += 1
data = sorted(data.values(), key=lambda x: x['count'], reverse=True)[:20]
return json_response(data)
return json_response(error=error)
@auth('dashboard.dashboard.view')
def get_deploy(request):
host = Host.objects.filter(deleted_at__isnull=True).count()
data = {x.id: {'name': x.name, 'count': 0} for x in App.objects.all()}
for dep in Deploy.objects.all():
data[dep.app_id]['count'] += len(json.loads(dep.host_ids))
data = filter(lambda x: x['count'], data.values())
return json_response({'host': host, 'res': list(data)})
================================================
FILE: spug_api/apps/host/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/host/add.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from libs import json_response, JsonParser, Argument, auth
from apps.host.models import Host, HostExtend, Group
from apps.host import utils
import json
@auth('host.host.add')
def get_regions(request):
form, error = JsonParser(
Argument('type', filter=lambda x: x in ('ali', 'tencent'), help='参数错误'),
Argument('ak', help='请输入AccessKey ID'),
Argument('ac', help='请输入AccessKey Secret'),
).parse(request.GET)
if error is None:
response = []
if form.type == 'ali':
for item in utils.fetch_ali_regions(form.ak, form.ac):
response.append({'id': item['RegionId'], 'name': item['LocalName']})
else:
for item in utils.fetch_tencent_regions(form.ak, form.ac):
response.append({'id': item['Region'], 'name': item['RegionName']})
return json_response(response)
return json_response(error=error)
@auth('host.host.add')
def cloud_import(request):
form, error = JsonParser(
Argument('type', filter=lambda x: x in ('ali', 'tencent'), help='参数错误'),
Argument('ak', help='请输入AccessKey ID'),
Argument('ac', help='请输入AccessKey Secret'),
Argument('region_id', help='请选择区域'),
Argument('group_id', type=int, help='请选择分组'),
Argument('username', help='请输入默认SSH用户名'),
Argument('port', type=int, help='请输入默认SSH端口号'),
Argument('host_type', filter=lambda x: x in ('public', 'private'), help='请选择连接地址'),
).parse(request.body)
if error is None:
group = Group.objects.filter(pk=form.group_id).first()
if not group:
return json_response(error='未找到指定分组')
if form.type == 'ali':
instances = utils.fetch_ali_instances(form.ak, form.ac, form.region_id)
else:
instances = utils.fetch_tencent_instances(form.ak, form.ac, form.region_id)
host_add_ids = []
for item in instances:
instance_id = item['instance_id']
host_name = item.pop('instance_name')
public_ips = item['public_ip_address'] or []
private_ips = item['private_ip_address'] or []
item['public_ip_address'] = json.dumps(public_ips)
item['private_ip_address'] = json.dumps(private_ips)
if HostExtend.objects.filter(instance_id=instance_id).exists():
HostExtend.objects.filter(instance_id=instance_id).update(**item)
else:
if form.host_type == 'public':
hostname = public_ips[0] if public_ips else ''
else:
hostname = private_ips[0] if private_ips else ''
host = Host.objects.create(
name=host_name,
hostname=hostname,
port=form.port,
username=form.username,
created_by=request.user)
HostExtend.objects.create(host=host, **item)
host_add_ids.append(host.id)
if host_add_ids:
group.hosts.add(*host_add_ids)
return json_response(len(instances))
return json_response(error=error)
================================================
FILE: spug_api/apps/host/extend.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from libs import json_response, JsonParser, Argument, human_datetime, auth
from apps.host.models import Host, HostExtend
from apps.host.utils import check_os_type, fetch_host_extend
import json
class ExtendView(View):
@auth('host.host.add|host.host.edit')
def get(self, request):
form, error = JsonParser(
Argument('host_id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
host = Host.objects.filter(pk=form.host_id).first()
if not host:
return json_response(error='未找到指定主机')
if not host.is_verified:
return json_response(error='该主机还未验证')
with host.get_ssh() as ssh:
response = fetch_host_extend(ssh)
return json_response(response)
return json_response(error=error)
@auth('host.host.add|host.host.edit')
def post(self, request):
form, error = JsonParser(
Argument('host_id', type=int, help='参数错误'),
Argument('instance_id', required=False),
Argument('os_name', help='请输入操作系统'),
Argument('cpu', type=int, help='请输入CPU核心数'),
Argument('memory', type=float, help='请输入内存大小'),
Argument('disk', type=list, filter=lambda x: len(x), help='请添加磁盘'),
Argument('private_ip_address', type=list, filter=lambda x: len(x), help='请添加内网IP'),
Argument('public_ip_address', type=list, required=False),
Argument('instance_charge_type', default='Other'),
Argument('internet_charge_type', default='Other'),
Argument('created_time', required=False),
Argument('expired_time', required=False)
).parse(request.body)
if error is None:
host = Host.objects.filter(pk=form.host_id).first()
form.disk = json.dumps(form.disk)
form.public_ip_address = json.dumps(form.public_ip_address) if form.public_ip_address else '[]'
form.private_ip_address = json.dumps(form.private_ip_address)
form.updated_at = human_datetime()
form.os_type = check_os_type(form.os_name)
if hasattr(host, 'hostextend'):
extend = host.hostextend
extend.update_by_dict(form)
else:
extend = HostExtend.objects.create(host=host, **form)
return json_response(extend.to_view())
return json_response(error=error)
================================================
FILE: spug_api/apps/host/group.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.db.models import F
from libs import json_response, JsonParser, Argument, auth
from apps.host.models import Group
from apps.account.models import Role
def fetch_children(data, with_hosts):
if data:
sub_data = dict()
for item in Group.objects.filter(parent_id__in=data.keys()):
tmp = item.to_view(with_hosts)
sub_data[item.id] = tmp
data[item.parent_id]['children'].append(tmp)
return fetch_children(sub_data, with_hosts)
def merge_children(data, prefix, childes):
prefix = f'{prefix}/' if prefix else ''
for item in childes:
name = f'{prefix}{item["title"]}'
item['name'] = name
if item.get('children'):
merge_children(data, name, item['children'])
else:
data[item['key']] = name
def filter_by_perm(data, result, ids):
for item in data:
if 'children' in item:
if item['key'] in ids:
result.append(item)
elif item['children']:
filter_by_perm(item['children'], result, ids)
class GroupView(View):
def get(self, request):
with_hosts = request.GET.get('with_hosts')
data, data2 = dict(), dict()
for item in Group.objects.filter(parent_id=0):
data[item.id] = item.to_view(with_hosts)
fetch_children(data, with_hosts)
if not data:
grp = Group.objects.create(name='Default', sort_id=1)
data[grp.id] = grp.to_view()
if request.user.is_supper:
tree_data = list(data.values())
else:
tree_data, ids = [], request.user.group_perms
filter_by_perm(data.values(), tree_data, ids)
merge_children(data2, '', tree_data)
return json_response({'treeData': tree_data, 'groups': data2})
@auth('admin')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('parent_id', type=int, default=0),
Argument('name', help='请输入分组名称')
).parse(request.body)
if error is None:
if form.id:
Group.objects.filter(pk=form.id).update(name=form.name)
else:
group = Group.objects.create(**form)
group.sort_id = group.id
group.save()
return json_response(error=error)
@auth('admin')
def patch(self, request):
form, error = JsonParser(
Argument('s_id', type=int, help='参数错误'),
Argument('d_id', type=int, help='参数错误'),
Argument('action', type=int, help='参数错误')
).parse(request.body)
if error is None:
src = Group.objects.get(pk=form.s_id)
dst = Group.objects.get(pk=form.d_id)
if form.action == 0:
src.parent_id = dst.id
dst = Group.objects.filter(parent_id=dst.id).first()
if not dst:
src.save()
return json_response()
form.action = -1
src.parent_id = dst.parent_id
if src.sort_id > dst.sort_id:
if form.action == -1:
dst = Group.objects.filter(sort_id__gt=dst.sort_id).last()
Group.objects.filter(sort_id__lt=src.sort_id, sort_id__gte=dst.sort_id).update(sort_id=F('sort_id') + 1)
else:
if form.action == 1:
dst = Group.objects.filter(sort_id__lt=dst.sort_id).first()
Group.objects.filter(sort_id__lte=dst.sort_id, sort_id__gt=src.sort_id).update(sort_id=F('sort_id') - 1)
src.sort_id = dst.sort_id
src.save()
return json_response(error=error)
@auth('admin')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
group = Group.objects.filter(pk=form.id).first()
if not group:
return json_response(error='未找到指定分组')
if Group.objects.filter(parent_id=group.id).exists():
return json_response(error='请移除子分组后再尝试删除')
if group.hosts.exists():
return json_response(error='请移除分组下的主机后再尝试删除')
if not Group.objects.exclude(pk=form.id).exists():
return json_response(error='请至少保留一个分组')
role = Role.objects.filter(group_perms__regex=fr'[^0-9]{form.id}[^0-9]').first()
if role:
return json_response(error=f'账户角色【{role.name}】的主机权限关联该分组，请解除关联后再尝试删除')
group.delete()
return json_response(error=error)
================================================
FILE: spug_api/apps/host/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs import ModelMixin, human_datetime
from apps.account.models import User
from apps.setting.utils import AppSetting
from libs.ssh import SSH
import json
class Host(models.Model, ModelMixin):
name = models.CharField(max_length=100)
hostname = models.CharField(max_length=50)
port = models.IntegerField(null=True)
username = models.CharField(max_length=50)
pkey = models.TextField(null=True)
desc = models.CharField(max_length=255, null=True)
is_verified = models.BooleanField(default=False)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
@property
def private_key(self):
return self.pkey or AppSetting.get('private_key')
def get_ssh(self, pkey=None, default_env=None):
pkey = pkey or self.private_key
return SSH(self.hostname, self.port, self.username, pkey, default_env=default_env)
def to_view(self):
tmp = self.to_dict()
if hasattr(self, 'hostextend'):
tmp.update(self.hostextend.to_view())
tmp['group_ids'] = []
return tmp
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'hosts'
ordering = ('-id',)
class HostExtend(models.Model, ModelMixin):
INSTANCE_CHARGE_TYPES = (
('PrePaid', '包年包月'),
('PostPaid', '按量计费'),
('Other', '其他')
)
INTERNET_CHARGE_TYPES = (
('PayByTraffic', '按流量计费'),
('PayByBandwidth', '按带宽计费'),
('Other', '其他')
)
host = models.OneToOneField(Host, on_delete=models.CASCADE)
instance_id = models.CharField(max_length=64, null=True)
zone_id = models.CharField(max_length=30, null=True)
cpu = models.IntegerField()
memory = models.FloatField()
disk = models.CharField(max_length=255, default='[]')
os_name = models.CharField(max_length=50)
os_type = models.CharField(max_length=20)
private_ip_address = models.CharField(max_length=255)
public_ip_address = models.CharField(max_length=255)
instance_charge_type = models.CharField(max_length=20, choices=INSTANCE_CHARGE_TYPES)
internet_charge_type = models.CharField(max_length=20, choices=INTERNET_CHARGE_TYPES)
created_time = models.CharField(max_length=20, null=True)
expired_time = models.CharField(max_length=20, null=True)
updated_at = models.CharField(max_length=20, default=human_datetime)
def to_view(self):
tmp = self.to_dict(excludes=('id',))
tmp['disk'] = json.loads(self.disk)
tmp['private_ip_address'] = json.loads(self.private_ip_address)
tmp['public_ip_address'] = json.loads(self.public_ip_address)
tmp['instance_charge_type_alias'] = self.get_instance_charge_type_display()
tmp['internet_charge_type_alisa'] = self.get_internet_charge_type_display()
return tmp
class Meta:
db_table = 'host_extend'
class Group(models.Model, ModelMixin):
name = models.CharField(max_length=50)
parent_id = models.IntegerField(default=0)
sort_id = models.IntegerField(default=0)
hosts = models.ManyToManyField(Host, related_name='groups')
def to_view(self, with_hosts=False):
response = dict(key=self.id, value=self.id, title=self.name, children=[])
if with_hosts:
def make_item(x):
return dict(title=x.name, hostname=x.hostname, key=f'{self.id}_{x.id}', id=x.id, isLeaf=True)
response['children'] = [make_item(x) for x in self.hosts.all()]
return response
class Meta:
db_table = 'host_groups'
ordering = ('-sort_id',)
================================================
FILE: spug_api/apps/host/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from apps.host.views import *
from apps.host.group import GroupView
from apps.host.extend import ExtendView
from apps.host.add import get_regions, cloud_import
urlpatterns = [
path('', HostView.as_view()),
path('extend/', ExtendView.as_view()),
path('group/', GroupView.as_view()),
path('import/', post_import),
path('import/cloud/', cloud_import),
path('import/region/', get_regions),
path('parse/', post_parse),
path('valid/', batch_valid),
]
================================================
FILE: spug_api/apps/host/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django_redis import get_redis_connection
from libs.helper import make_ali_request, make_tencent_request
from libs.ssh import SSH, AuthenticationException
from libs.utils import AttrDict, human_datetime
from libs.validators import ip_validator
from apps.host.models import HostExtend
from apps.setting.utils import AppSetting
from collections import defaultdict
from datetime import datetime, timezone
from concurrent import futures
import ipaddress
import json
import math
import os
def check_os_type(os_name):
os_name = os_name.lower()
types = ('centos', 'coreos', 'debian', 'suse', 'ubuntu', 'windows', 'freebsd', 'tencent', 'alibaba', 'fedora')
for t in types:
if t in os_name:
return t
return 'unknown'
def check_instance_charge_type(value, supplier):
if supplier == 'ali':
if value in ('PrePaid', 'PostPaid'):
return value
else:
return 'Other'
if supplier == 'tencent':
if value == 'PREPAID':
return 'PrePaid'
if value == 'POSTPAID_BY_HOUR':
return 'PostPaid'
return 'Other'
def check_internet_charge_type(value, supplier):
if supplier == 'ali':
if value in ('PayByTraffic', 'PayByBandwidth'):
return value
else:
return 'Other'
if supplier == 'tencent':
if value == 'TRAFFIC_POSTPAID_BY_HOUR':
return 'PayByTraffic'
if value in ('BANDWIDTH_PREPAID', 'BANDWIDTH_POSTPAID_BY_HOUR'):
return 'PayByBandwidth'
return 'Other'
def parse_utc_date(value):
if not value:
return None
s_format = '%Y-%m-%dT%H:%M:%SZ'
if len(value) == 17:
s_format = '%Y-%m-%dT%H:%MZ'
date = datetime.strptime(value, s_format).replace(tzinfo=timezone.utc)
return date.astimezone().strftime('%Y-%m-%d %H:%M:%S')
def fetch_ali_regions(ak, ac):
params = dict(Action='DescribeRegions')
res = make_ali_request(ak, ac, 'http://ecs.aliyuncs.com', params)
if 'Regions' in res:
return res['Regions']['Region']
else:
raise Exception(res)
def fetch_ali_disks(ak, ac, region_id, page_number=1):
data, page_size = defaultdict(list), 20
params = dict(
Action='DescribeDisks',
RegionId=region_id,
PageNumber=page_number,
PageSize=page_size
)
res = make_ali_request(ak, ac, 'http://ecs.aliyuncs.com', params)
if 'Disks' in res:
for item in res['Disks']['Disk']:
data[item['InstanceId']].append(item['Size'])
if len(res['Disks']['Disk']) == page_size:
page_number += 1
new_data = fetch_ali_disks(ak, ac, region_id, page_number)
data.update(new_data)
return data
else:
raise Exception(res)
def fetch_ali_instances(ak, ac, region_id, page_number=1):
data, page_size = {}, 20
params = dict(
Action='DescribeInstances',
RegionId=region_id,
PageNumber=page_number,
PageSize=page_size
)
res = make_ali_request(ak, ac, 'http://ecs.aliyuncs.com', params)
if 'Instances' not in res:
raise Exception(res)
for item in res['Instances']['Instance']:
if 'NetworkInterfaces' in item:
network_interface = item['NetworkInterfaces']['NetworkInterface']
else:
network_interface = []
data[item['InstanceId']] = dict(
instance_id=item['InstanceId'],
instance_name=item['InstanceName'],
os_name=item['OSName'],
os_type=check_os_type(item['OSName']),
cpu=item['Cpu'],
memory=item['Memory'] / 1024,
created_time=parse_utc_date(item['CreationTime']),
expired_time=parse_utc_date(item['ExpiredTime']),
instance_charge_type=check_instance_charge_type(item['InstanceChargeType'], 'ali'),
internet_charge_type=check_internet_charge_type(item['InternetChargeType'], 'ali'),
public_ip_address=item['PublicIpAddress']['IpAddress'],
private_ip_address=[x['PrimaryIpAddress'] for x in network_interface if x.get('PrimaryIpAddress')],
zone_id=item['ZoneId']
)
if len(res['Instances']['Instance']) == page_size:
new_data = fetch_ali_instances(ak, ac, region_id, page_number + 1)
data.update(new_data)
if page_number != 1:
return data
for instance_id, disk in fetch_ali_disks(ak, ac, region_id).items():
if instance_id in data:
data[instance_id]['disk'] = disk
return list(data.values())
def fetch_tencent_regions(ak, ac):
params = dict(Action='DescribeRegions')
res = make_tencent_request(ak, ac, 'cvm.tencentcloudapi.com', params)
if 'RegionSet' in res['Response']:
return res['Response']['RegionSet']
else:
raise Exception(res)
def fetch_tencent_instances(ak, ac, region_id, page_number=1):
data, page_size = [], 20
params = dict(
Action='DescribeInstances',
Region=region_id,
Offset=(page_number - 1) * page_size,
Limit=page_size
)
res = make_tencent_request(ak, ac, 'cvm.tencentcloudapi.com', params)
if 'InstanceSet' not in res['Response']:
raise Exception(res)
for item in res['Response']['InstanceSet']:
data_disks = list(map(lambda x: x['DiskSize'], item['DataDisks']))
internet_charge_type = item['InternetAccessible']['InternetChargeType']
data.append(dict(
instance_id=item['InstanceId'],
instance_name=item['InstanceName'],
os_name=item['OsName'],
os_type=check_os_type(item['OsName']),
cpu=item['CPU'],
memory=item['Memory'],
disk=[item['SystemDisk']['DiskSize']] + data_disks,
created_time=parse_utc_date(item['CreatedTime']),
expired_time=parse_utc_date(item['ExpiredTime']),
instance_charge_type=check_instance_charge_type(item['InstanceChargeType'], 'tencent'),
internet_charge_type=check_internet_charge_type(internet_charge_type, 'tencent'),
public_ip_address=item['PublicIpAddresses'],
private_ip_address=item['PrivateIpAddresses'],
zone_id=item['Placement']['Zone']
))
if len(res['Response']['InstanceSet']) == page_size:
page_number += 1
new_data = fetch_tencent_instances(ak, ac, region_id, page_number)
data.extend(new_data)
return data
def fetch_host_extend(ssh):
public_ip_address = set()
private_ip_address = set()
response = {'disk': []}
code, out = ssh.exec_command_raw('nproc')
if code != 0:
code, out = ssh.exec_command_raw("grep -c '^processor' /proc/cpuinfo")
if code == 0:
response['cpu'] = int(out.strip())
code, out = ssh.exec_command_raw("cat /etc/os-release | grep PRETTY_NAME | awk -F \\\" '{print $2}'")
if '/etc/os-release' in out:
code, out = ssh.exec_command_raw("cat /etc/issue | head -1 | awk '{print $1,$2,$3}'")
if code == 0:
response['os_name'] = out.strip()[:50]
code, out = ssh.exec_command_raw('hostname -I')
if code == 0:
for ip in out.strip().split():
if len(ip) > 15: # ignore ipv6
continue
if ipaddress.ip_address(ip).is_global:
if len(public_ip_address) < 10:
public_ip_address.add(ip)
elif len(private_ip_address) < 10:
private_ip_address.add(ip)
ssh_hostname = ssh.arguments.get('hostname')
if ip_validator(ssh_hostname):
if ipaddress.ip_address(ssh_hostname).is_global:
if ssh_hostname in public_ip_address:
public_ip_address.remove(ssh_hostname)
public_ip_address = [ssh_hostname] + list(public_ip_address)
else:
if ssh_hostname in private_ip_address:
private_ip_address.remove(ssh_hostname)
private_ip_address = [ssh_hostname] + list(private_ip_address)
code, out = ssh.exec_command_raw('lsblk -dbn -o SIZE -e 11 2> /dev/null')
if code == 0:
disks = []
for item in out.strip().splitlines():
item = item.strip()
size = math.ceil(int(item) / 1024 / 1024 / 1024)
if size > 10:
disks.append(size)
response['disk'] = disks[:10]
code, out = ssh.exec_command_raw("dmidecode -t 17 | grep -E 'Size: [0-9]+' | awk '{s+=$2} END {print s,$3}'")
if code == 0:
fields = out.strip().split()
if len(fields) == 2 and fields[1] in ('GB', 'MB'):
size, unit = out.strip().split()
if unit == 'GB':
response['memory'] = size
else:
response['memory'] = round(int(size) / 1024, 0)
if 'memory' not in response:
code, out = ssh.exec_command_raw("cat /proc/meminfo | grep 'MemTotal' | awk '{print $2}'")
if code == 0:
response['memory'] = math.ceil(int(out) / 1024 / 1024)
response['public_ip_address'] = list(public_ip_address)
response['private_ip_address'] = list(private_ip_address)
return response
def batch_sync_host(token, hosts, password=None):
private_key, public_key = AppSetting.get_ssh_key()
threads, latest_exception, rds = [], None, get_redis_connection()
max_workers = max(10, os.cpu_count() * 5)
with futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
for host in hosts:
if hasattr(host, 'password'):
password = host.password
t = executor.submit(_sync_host_extend, host, private_key, public_key, password)
t.host = host
threads.append(t)
for t in futures.as_completed(threads):
exception = t.exception()
if exception:
rds.rpush(token, json.dumps({'key': t.host.id, 'status': 'fail', 'message': f'{exception}'}))
else:
rds.rpush(token, json.dumps({'key': t.host.id, 'status': 'ok'}))
t.host.is_verified = True
t.host.save()
rds.expire(token, 60)
def _sync_host_extend(host, private_key=None, public_key=None, password=None, ssh=None):
if not ssh:
kwargs = host.to_dict(selects=('hostname', 'port', 'username'))
with _get_ssh(kwargs, host.pkey, private_key, public_key, password) as ssh:
return _sync_host_extend(host, ssh=ssh)
form = AttrDict(fetch_host_extend(ssh))
form.disk = json.dumps(form.disk)
form.public_ip_address = json.dumps(form.public_ip_address)
form.private_ip_address = json.dumps(form.private_ip_address)
form.updated_at = human_datetime()
form.os_type = check_os_type(form.os_name)
if hasattr(host, 'hostextend'):
extend = host.hostextend
extend.update_by_dict(form)
else:
extend = HostExtend.objects.create(host=host, **form)
return extend
def _get_ssh(kwargs, pkey=None, private_key=None, public_key=None, password=None):
try:
ssh = SSH(pkey=pkey or private_key, **kwargs)
ssh.get_client()
return ssh
except AuthenticationException as e:
if password:
with SSH(password=str(password), **kwargs) as ssh:
ssh.add_public_key(public_key)
return _get_ssh(kwargs, private_key)
raise e
================================================
FILE: spug_api/apps/host/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.db.models import F
from django.http.response import HttpResponseBadRequest
from libs import json_response, JsonParser, Argument, AttrDict, auth
from apps.setting.utils import AppSetting
from apps.account.utils import get_host_perms
from apps.host.models import Host, Group
from apps.host.utils import batch_sync_host, _sync_host_extend
from apps.exec.models import ExecTemplate
from apps.app.models import Deploy
from apps.schedule.models import Task
from apps.monitor.models import Detection
from libs.ssh import SSH, AuthenticationException
from paramiko.ssh_exception import BadAuthenticationType
from openpyxl import load_workbook
from threading import Thread
import socket
import uuid
class HostView(View):
def get(self, request):
hosts = Host.objects.select_related('hostextend')
if not request.user.is_supper:
hosts = hosts.filter(id__in=get_host_perms(request.user))
hosts = {x.id: x.to_view() for x in hosts}
for rel in Group.hosts.through.objects.filter(host_id__in=hosts.keys()):
hosts[rel.host_id]['group_ids'].append(rel.group_id)
return json_response(list(hosts.values()))
@auth('host.host.add|host.host.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('group_ids', type=list, filter=lambda x: len(x), help='请选择主机分组'),
Argument('name', help='请输主机名称'),
Argument('username', handler=str.strip, help='请输入登录用户名'),
Argument('hostname', handler=str.strip, help='请输入主机名或IP'),
Argument('port', type=int, help='请输入SSH端口'),
Argument('pkey', required=False),
Argument('desc', required=False),
Argument('password', required=False),
).parse(request.body)
if error is None:
if not _do_host_verify(form):
return json_response('auth fail')
group_ids = form.pop('group_ids')
other = Host.objects.filter(name=form.name).first()
if other and (not form.id or other.id != form.id):
return json_response(error=f'已存在的主机名称【{form.name}】')
if form.id:
Host.objects.filter(pk=form.id).update(is_verified=True, **form)
host = Host.objects.get(pk=form.id)
else:
host = Host.objects.create(created_by=request.user, is_verified=True, **form)
host.groups.set(group_ids)
response = host.to_view()
response['group_ids'] = group_ids
return json_response(response)
return json_response(error=error)
@auth('host.host.add|host.host.edit')
def put(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误')
).parse(request.body)
if error is None:
host = Host.objects.get(pk=form.id)
with host.get_ssh() as ssh:
_sync_host_extend(host, ssh=ssh)
return json_response(error=error)
@auth('admin')
def patch(self, request):
form, error = JsonParser(
Argument('host_ids', type=list, filter=lambda x: len(x), help='请选择主机'),
Argument('s_group_id', type=int, help='参数错误'),
Argument('t_group_id', type=int, help='参数错误'),
Argument('is_copy', type=bool, help='参数错误'),
).parse(request.body)
if error is None:
if form.t_group_id == form.s_group_id:
return json_response(error='不能选择本分组的主机')
s_group = Group.objects.get(pk=form.s_group_id)
t_group = Group.objects.get(pk=form.t_group_id)
t_group.hosts.add(*form.host_ids)
if not form.is_copy:
s_group.hosts.remove(*form.host_ids)
return json_response(error=error)
@auth('host.host.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('group_id', type=int, required=False),
).parse(request.GET)
if error is None:
if form.id:
host_ids = [form.id]
elif form.group_id:
group = Group.objects.get(pk=form.group_id)
host_ids = [x.id for x in group.hosts.all()]
else:
return json_response(error='参数错误')
for host_id in host_ids:
regex = fr'[^0-9]{host_id}[^0-9]'
deploy = Deploy.objects.filter(host_ids__regex=regex) \
.annotate(app_name=F('app__name'), env_name=F('env__name')).first()
if deploy:
return json_response(error=f'应用【{deploy.app_name}】在【{deploy.env_name}】的发布配置关联了该主机，请解除关联后再尝试删除该主机')
task = Task.objects.filter(targets__regex=regex).first()
if task:
return json_response(error=f'任务计划中的任务【{task.name}】关联了该主机，请解除关联后再尝试删除该主机')
detection = Detection.objects.filter(type__in=('3', '4'), targets__regex=regex).first()
if detection:
return json_response(error=f'监控中心的任务【{detection.name}】关联了该主机，请解除关联后再尝试删除该主机')
tpl = ExecTemplate.objects.filter(host_ids__regex=regex).first()
if tpl:
return json_response(error=f'执行模板【{tpl.name}】关联了该主机，请解除关联后再尝试删除该主机')
Host.objects.filter(id__in=host_ids).delete()
return json_response(error=error)
@auth('host.host.add')
def post_import(request):
group_id = request.POST.get('group_id')
file = request.FILES['file']
hosts = []
ws = load_workbook(file, read_only=True)['Sheet1']
summary = {'fail': 0, 'success': 0, 'invalid': [], 'skip': [], 'repeat': []}
for i, row in enumerate(ws.rows, start=1):
if i == 1: # 第1行是表头 略过
continue
if not all([row[x].value for x in range(4)]):
summary['invalid'].append(i)
summary['fail'] += 1
continue
data = AttrDict(
name=row[0].value,
hostname=row[1].value,
port=row[2].value,
username=row[3].value,
desc=row[5].value
)
if Host.objects.filter(hostname=data.hostname, port=data.port, username=data.username).exists():
summary['skip'].append(i)
summary['fail'] += 1
continue
if Host.objects.filter(name=data.name).exists():
summary['repeat'].append(i)
summary['fail'] += 1
continue
host = Host.objects.create(created_by=request.user, **data)
host.groups.add(group_id)
summary['success'] += 1
host.password = row[4].value
hosts.append(host)
token = uuid.uuid4().hex
if hosts:
Thread(target=batch_sync_host, args=(token, hosts)).start()
return json_response({'summary': summary, 'token': token, 'hosts': {x.id: {'name': x.name} for x in hosts}})
@auth('host.host.add')
def post_parse(request):
file = request.FILES['file']
if file:
data = file.read()
return json_response(data.decode())
else:
return HttpResponseBadRequest()
@auth('host.host.add')
def batch_valid(request):
form, error = JsonParser(
Argument('password', required=False),
Argument('range', filter=lambda x: x in ('1', '2'), help='参数错误')
).parse(request.body)
if error is None:
if form.range == '1': # all hosts
hosts = Host.objects.all()
else:
hosts = Host.objects.filter(is_verified=False).all()
token = uuid.uuid4().hex
Thread(target=batch_sync_host, args=(token, hosts, form.password)).start()
return json_response({'token': token, 'hosts': {x.id: {'name': x.name} for x in hosts}})
return json_response(error=error)
def _do_host_verify(form):
password = form.pop('password')
if form.pkey:
try:
with SSH(form.hostname, form.port, form.username, form.pkey) as ssh:
ssh.ping()
return True
except BadAuthenticationType:
raise Exception('该主机不支持密钥认证，请参考官方文档，错误代码：E01')
except AuthenticationException:
raise Exception('上传的独立密钥认证失败，请检查该密钥是否能正常连接主机（推荐使用全局密钥）')
except socket.timeout:
raise Exception('连接主机超时，请检查网络')
private_key, public_key = AppSetting.get_ssh_key()
if password:
try:
with SSH(form.hostname, form.port, form.username, password=password) as ssh:
ssh.add_public_key(public_key)
except BadAuthenticationType:
raise Exception('该主机不支持密码认证，请参考官方文档，错误代码：E00')
except AuthenticationException:
raise Exception('密码连接认证失败，请检查密码是否正确')
except socket.timeout:
raise Exception('连接主机超时，请检查网络')
try:
with SSH(form.hostname, form.port, form.username, private_key) as ssh:
ssh.ping()
except BadAuthenticationType:
raise Exception('该主机不支持密钥认证，请参考官方文档，错误代码：E01')
except AuthenticationException:
if password:
raise Exception('密钥认证失败，请参考官方文档，错误代码：E02')
return False
except socket.timeout:
raise Exception('连接主机超时，请检查网络')
return True
================================================
FILE: spug_api/apps/monitor/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/monitor/executors.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django_redis import get_redis_connection
from apps.host.models import Host
from apps.monitor.utils import handle_notify
from socket import socket
import subprocess
import platform
import requests
import logging
import json
import time
import re
logging.captureWarnings(True)
regex = re.compile(r'Failed to establish a new connection: (.*)\'\)+')
def site_check(url, limit):
try:
res = requests.get(url, timeout=30)
if limit:
duration = int(res.elapsed.total_seconds() * 1000)
if duration > int(limit):
return False, f'响应时间 {duration}ms 大于 {limit}ms'
return 200 <= res.status_code < 400, f'返回HTTP状态码 {res.status_code}'
except Exception as e:
error = e.__str__()
exps = re.findall(regex, error)
if exps:
error = exps[0]
return False, error
def port_check(addr, port):
try:
sock = socket()
sock.settimeout(5)
sock.connect((addr, int(port)))
sock.close()
return True, '端口状态检测正常'
except Exception as e:
return False, f'异常信息：{e}'
def ping_check(addr):
try:
if platform.system().lower() == 'windows':
command = f'ping -n 1 -w 3000 {addr}'
else:
command = f'ping -c 1 -W 3 {addr}'
task = subprocess.run(command, shell=True, stdout=subprocess.PIPE)
if task.returncode == 0:
return True, 'Ping检测正常'
else:
return False, 'Ping检测失败'
except Exception as e:
return False, f'异常信息：{e}'
def host_executor(host, command):
try:
with host.get_ssh() as ssh:
exit_code, out = ssh.exec_command_raw(command)
if exit_code == 0:
return True, out or '检测状态正常'
else:
return False, out or f'退出状态码：{exit_code}'
except Exception as e:
return False, f'异常信息：{e}'
def monitor_worker_handler(job):
task_id, tp, addr, extra, threshold, quiet = json.loads(job)
target = addr
if tp == '1':
is_ok, message = site_check(addr, extra)
elif tp == '2':
is_ok, message = port_check(addr, extra)
elif tp == '5':
is_ok, message = ping_check(addr)
elif tp not in ('3', '4'):
is_ok, message = False, f'invalid monitor type for {tp!r}'
else:
command = f'ps -ef|grep -v grep|grep {extra!r}' if tp == '3' else extra
host = Host.objects.filter(pk=addr).first()
if not host:
is_ok, message = False, f'unknown host id for {addr!r}'
else:
is_ok, message = host_executor(host, command)
target = f'{host.name}({host.hostname})'
rds, key, f_count, f_time = get_redis_connection(), f'spug:det:{task_id}', f'c_{addr}', f't_{addr}'
v_count, v_time = rds.hmget(key, f_count, f_time)
if is_ok:
if v_count:
rds.hdel(key, f_count, f_time)
if v_time:
logging.warning('send recovery notification')
handle_notify(task_id, target, is_ok, message, int(v_count) + 1)
return
v_count = rds.hincrby(key, f_count)
if v_count >= threshold:
if not v_time or int(time.time()) - int(v_time) >= quiet * 60:
rds.hset(key, f_time, int(time.time()))
logging.warning('send fault alarm notification')
handle_notify(task_id, target, is_ok, message, v_count)
def dispatch(tp, addr, extra):
if tp == '1':
return site_check(addr, extra)
elif tp == '2':
return port_check(addr, extra)
elif tp == '5':
return ping_check(addr)
elif tp == '3':
command = f'ps -ef|grep -v grep|grep {extra!r}'
elif tp == '4':
command = extra
else:
raise TypeError(f'invalid monitor type: {tp!r}')
host = Host.objects.filter(pk=addr).first()
return host_executor(host, command)
================================================
FILE: spug_api/apps/monitor/management/commands/runmonitor.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from apps.monitor.scheduler import Scheduler
import logging
logging.basicConfig(level=logging.WARNING, format='%(asctime)s %(message)s')
class Command(BaseCommand):
help = 'Start monitor process'
def handle(self, *args, **options):
s = Scheduler()
s.run()
================================================
FILE: spug_api/apps/monitor/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs import ModelMixin, human_datetime
from apps.account.models import User
import json
class Detection(models.Model, ModelMixin):
TYPES = (
('1', '站点检测'),
('2', '端口检测'),
('3', '进程检测'),
('4', '自定义脚本'),
('5', 'Ping检测'),
)
STATUS = (
(0, '正常'),
(1, '异常'),
)
name = models.CharField(max_length=50)
type = models.CharField(max_length=2, choices=TYPES)
group = models.CharField(max_length=255, null=True)
targets = models.TextField()
extra = models.TextField(null=True)
desc = models.CharField(max_length=255, null=True)
is_active = models.BooleanField(default=True)
rate = models.IntegerField(default=5)
threshold = models.IntegerField(default=3)
quiet = models.IntegerField(default=24 * 60)
fault_times = models.SmallIntegerField(default=0)
notify_mode = models.CharField(max_length=255)
notify_grp = models.CharField(max_length=255)
latest_run_time = models.CharField(max_length=20, null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
updated_at = models.CharField(max_length=20, null=True)
updated_by = models.ForeignKey(User, models.PROTECT, related_name='+', null=True)
def to_view(self):
tmp = self.to_dict()
tmp['type_alias'] = self.get_type_display()
tmp['notify_mode'] = json.loads(self.notify_mode)
tmp['notify_grp'] = json.loads(self.notify_grp)
tmp['targets'] = json.loads(self.targets)
return tmp
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'detections'
ordering = ('-id',)
================================================
FILE: spug_api/apps/monitor/scheduler.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.executors.pool import ThreadPoolExecutor
from apscheduler.triggers.interval import IntervalTrigger
from django_redis import get_redis_connection
from django.conf import settings
from django.db import connections
from django.db.utils import DatabaseError
from apps.monitor.models import Detection
from libs import AttrDict, human_datetime
from datetime import datetime, timedelta
from random import randint
import logging
import json
MONITOR_WORKER_KEY = settings.MONITOR_WORKER_KEY
class Scheduler:
timezone = settings.TIME_ZONE
def __init__(self):
self.scheduler = BackgroundScheduler(timezone=self.timezone, executors={'default': ThreadPoolExecutor(30)})
def _dispatch(self, task_id, tp, targets, extra, threshold, quiet):
Detection.objects.filter(pk=task_id).update(latest_run_time=human_datetime())
rds_cli = get_redis_connection()
for t in json.loads(targets):
rds_cli.rpush(MONITOR_WORKER_KEY, json.dumps([task_id, tp, t, extra, threshold, quiet]))
connections.close_all()
def _init(self):
self.scheduler.start()
try:
for item in Detection.objects.filter(is_active=True):
now = datetime.now()
trigger = IntervalTrigger(minutes=int(item.rate), timezone=self.timezone)
self.scheduler.add_job(
self._dispatch,
trigger,
id=str(item.id),
args=(item.id, item.type, item.targets, item.extra, item.threshold, item.quiet),
next_run_time=now + timedelta(seconds=randint(0, 60))
)
connections.close_all()
except DatabaseError:
pass
def run(self):
rds_cli = get_redis_connection()
self._init()
rds_cli.delete(settings.MONITOR_KEY)
logging.warning('Running monitor')
while True:
_, data = rds_cli.brpop(settings.MONITOR_KEY)
task = AttrDict(json.loads(data))
if task.action in ('add', 'modify'):
trigger = IntervalTrigger(minutes=int(task.rate), timezone=self.timezone)
self.scheduler.add_job(
self._dispatch,
trigger,
id=str(task.id),
args=(task.id, task.type, task.targets, task.extra, task.threshold, task.quiet),
replace_existing=True
)
elif task.action == 'remove':
job = self.scheduler.get_job(str(task.id))
if job:
job.remove()
================================================
FILE: spug_api/apps/monitor/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', DetectionView.as_view()),
path('overview/', get_overview),
path('test/', run_test),
]
================================================
FILE: spug_api/apps/monitor/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import close_old_connections
from apps.alarm.models import Alarm
from apps.monitor.models import Detection
from libs.spug import Notification
import json
def seconds_to_human(seconds):
text = ''
if seconds > 3600:
text = f'{int(seconds / 3600)}小时'
seconds = seconds % 3600
if seconds > 60:
text += f'{int(seconds / 60)}分钟'
seconds = seconds % 60
if seconds:
text += f'{seconds}秒'
return text
def _record_alarm(det, target, duration, status):
Alarm.objects.create(
name=det.name,
type=det.get_type_display(),
target=target,
status=status,
duration=duration,
notify_grp=det.notify_grp,
notify_mode=det.notify_mode)
def handle_notify(task_id, target, is_ok, out, fault_times):
close_old_connections()
det = Detection.objects.get(pk=task_id)
duration = seconds_to_human(det.rate * fault_times * 60)
event = '2' if is_ok else '1'
_record_alarm(det, target, duration, event)
grp = json.loads(det.notify_grp)
notify = Notification(grp, event, target, det.name, out, duration)
notify.dispatch_monitor(json.loads(det.notify_mode))
================================================
FILE: spug_api/apps/monitor/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.conf import settings
from django_redis import get_redis_connection
from libs import json_response, JsonParser, Argument, human_datetime, auth
from apps.monitor.models import Detection
from apps.monitor.executors import dispatch
from apps.setting.utils import AppSetting
from datetime import datetime
import json
class DetectionView(View):
@auth('dashboard.dashboard.view|monitor.monitor.view')
def get(self, request):
detections = Detection.objects.all()
groups = [x['group'] for x in detections.order_by('group').values('group').distinct()]
return json_response({'groups': groups, 'detections': [x.to_view() for x in detections]})
@auth('monitor.monitor.add|monitor.monitor.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('name', help='请输入任务名称'),
Argument('group', help='请选择任务分组'),
Argument('targets', type=list, filter=lambda x: len(x), help='请输入监控地址'),
Argument('type', filter=lambda x: x in dict(Detection.TYPES), help='请选择监控类型'),
Argument('extra', required=False),
Argument('desc', required=False),
Argument('rate', type=int, default=5),
Argument('threshold', type=int, default=3),
Argument('quiet', type=int, default=24 * 60),
Argument('notify_grp', type=list, help='请选择报警联系组'),
Argument('notify_mode', type=list, help='请选择报警方式'),
).parse(request.body)
if error is None:
if set(form.notify_mode).intersection(['1', '2', '6']):
if not AppSetting.get_default('spug_push_key'):
return json_response(error='报警方式微信、短信、电话需要配置推送服务（系统设置/推送服务设置），请配置后再启用该报警方式。')
form.targets = json.dumps(form.targets)
form.notify_grp = json.dumps(form.notify_grp)
form.notify_mode = json.dumps(form.notify_mode)
if form.id:
Detection.objects.filter(pk=form.id).update(
updated_at=human_datetime(),
updated_by=request.user,
**form)
task = Detection.objects.filter(pk=form.id).first()
if task and task.is_active:
form.action = 'modify'
rds_cli = get_redis_connection()
rds_cli.lpush(settings.MONITOR_KEY, json.dumps(form))
else:
dtt = Detection.objects.create(created_by=request.user, **form)
form.action = 'add'
form.id = dtt.id
rds_cli = get_redis_connection()
rds_cli.lpush(settings.MONITOR_KEY, json.dumps(form))
return json_response(error=error)
@auth('monitor.monitor.edit')
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象'),
Argument('is_active', type=bool, required=False)
).parse(request.body, True)
if error is None:
Detection.objects.filter(pk=form.id).update(**form)
if form.get('is_active') is not None:
if form.is_active:
task = Detection.objects.filter(pk=form.id).first()
message = {'id': form.id, 'action': 'add'}
message.update(task.to_dict(selects=('targets', 'extra', 'rate', 'type', 'threshold', 'quiet')))
else:
message = {'id': form.id, 'action': 'remove'}
rds_cli = get_redis_connection()
rds_cli.lpush(settings.MONITOR_KEY, json.dumps(message))
return json_response(error=error)
@auth('monitor.monitor.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
task = Detection.objects.filter(pk=form.id).first()
if task:
if task.is_active:
return json_response(error='该监控项正在运行中，请先停止后再尝试删除')
task.delete()
return json_response(error=error)
@auth('monitor.monitor.add|monitor.monitor.edit')
def run_test(request):
form, error = JsonParser(
Argument('type', help='请选择监控类型'),
Argument('targets', type=list, filter=lambda x: len(x), help='请输入监控地址'),
Argument('extra', required=False)
).parse(request.body)
if error is None:
is_success, message = dispatch(form.type, form.targets[0], form.extra)
return json_response({'is_success': is_success, 'message': message})
return json_response(error=error)
@auth('monitor.monitor.view')
def get_overview(request):
response = []
rds = get_redis_connection()
for item in Detection.objects.all():
data = {}
for key in json.loads(item.targets):
key = str(key)
data[key] = {
'id': f'{item.id}_{key}',
'group': item.group,
'name': item.name,
'type': item.get_type_display(),
'target': key,
'desc': item.desc,
'status': '0',
'latest_run_time': item.latest_run_time,
}
if item.is_active:
if item.latest_run_time:
data[key]['status'] = '1'
else:
data[key]['status'] = '10'
if item.is_active:
for key, val in rds.hgetall(f'spug:det:{item.id}').items():
prefix, key = key.decode().split('_', 1)
if key in data:
val = int(val)
if prefix == 'c':
if data[key]['status'] == '1':
data[key]['status'] = '2'
data[key]['count'] = val
elif prefix == 't':
date = datetime.fromtimestamp(val).strftime('%Y-%m-%d %H:%M:%S')
data[key].update(status='3', notified_at=date)
response.extend(list(data.values()))
return json_response(response)
================================================
FILE: spug_api/apps/notify/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/notify/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from django.core.cache import cache
from libs import ModelMixin, human_datetime
from libs.channel import Channel
import hashlib
class Notify(models.Model, ModelMixin):
TYPES = (
('1', '通知'),
('2', '待办'),
)
SOURCES = (
('monitor', '监控中心'),
('schedule', '任务计划'),
('flag', '应用发布'),
('alert', '系统警告'),
)
title = models.CharField(max_length=255)
source = models.CharField(max_length=10, choices=SOURCES)
type = models.CharField(max_length=2, choices=TYPES)
content = models.TextField(null=True)
unread = models.BooleanField(default=True)
link = models.CharField(max_length=255, null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
@classmethod
def make_system_notify(cls, title, content):
cls._make_notify('alert', '1', title, content)
@classmethod
def make_monitor_notify(cls, title, content):
cls._make_notify('monitor', '1', title, content)
@classmethod
def make_schedule_notify(cls, title, content):
cls._make_notify('schedule', '1', title, content)
@classmethod
def make_deploy_notify(cls, title, content):
cls._make_notify('flag', '1', title, content)
@classmethod
def _make_notify(cls, source, type, title, content):
tmp_str = f'{source},{type},{title},{content}'
digest = hashlib.md5(tmp_str.encode()).hexdigest()
unique_key = f'spug:notify:{digest}'
if not cache.get(unique_key): # 限制相同内容的发送频率
cache.set(unique_key, 1, 3600)
cls.objects.create(source=source, title=title, type=type, content=content)
Channel.send_notify(title, content)
def __repr__(self):
return '' % self.title
class Meta:
db_table = 'notifies'
ordering = ('-id',)
================================================
FILE: spug_api/apps/notify/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', NotifyView.as_view()),
]
================================================
FILE: spug_api/apps/notify/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from apps.notify.models import Notify
from libs import json_response, JsonParser, Argument
class NotifyView(View):
def get(self, request):
notifies = Notify.objects.all()
return json_response(notifies)
def patch(self, request):
form, error = JsonParser(
Argument('ids', type=list, help='参数错误')
).parse(request.body)
if error is None:
Notify.objects.filter(id__in=form.ids).update(unread=False)
return json_response(error=error)
================================================
FILE: spug_api/apps/repository/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/repository/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from django.conf import settings
from libs.mixins import ModelMixin
from apps.app.models import App, Environment, Deploy
from apps.account.models import User
from datetime import datetime
import json
import os
class Repository(models.Model, ModelMixin):
STATUS = (
('0', '未开始'),
('1', '构建中'),
('2', '失败'),
('5', '成功'),
)
app = models.ForeignKey(App, on_delete=models.PROTECT)
env = models.ForeignKey(Environment, on_delete=models.PROTECT)
deploy = models.ForeignKey(Deploy, on_delete=models.PROTECT)
version = models.CharField(max_length=100)
spug_version = models.CharField(max_length=50)
remarks = models.CharField(max_length=255, null=True)
extra = models.TextField()
status = models.CharField(max_length=2, choices=STATUS, default='0')
created_at = models.DateTimeField(auto_now_add=True)
created_by = models.ForeignKey(User, on_delete=models.PROTECT)
@staticmethod
def make_spug_version(deploy_id):
return f'{deploy_id}_{datetime.now().strftime("%Y%m%d%H%M%S")}'
def to_view(self):
tmp = self.to_dict()
tmp['extra'] = json.loads(self.extra)
tmp['status_alias'] = self.get_status_display()
if hasattr(self, 'app_name'):
tmp['app_name'] = self.app_name
if hasattr(self, 'env_name'):
tmp['env_name'] = self.env_name
if hasattr(self, 'created_by_user'):
tmp['created_by_user'] = self.created_by_user
return tmp
def delete(self, using=None, keep_parents=False):
super().delete(using, keep_parents)
try:
build_file = f'{self.spug_version}.tar.gz'
os.remove(os.path.join(settings.BUILD_DIR, build_file))
except FileNotFoundError:
pass
class Meta:
db_table = 'repositories'
ordering = ('-id',)
================================================
FILE: spug_api/apps/repository/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', RepositoryView.as_view()),
path('/', get_detail),
path('request/', get_requests),
]
================================================
FILE: spug_api/apps/repository/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django_redis import get_redis_connection
from django.conf import settings
from django.db import close_old_connections
from libs.utils import AttrDict, human_time, render_str
from apps.repository.models import Repository
from apps.app.utils import fetch_repo
from apps.config.utils import compose_configs
from apps.deploy.helper import Helper
import json
import uuid
import os
REPOS_DIR = settings.REPOS_DIR
BUILD_DIR = settings.BUILD_DIR
def dispatch(rep: Repository, helper=None):
rep.status = '1'
alone_build = helper is None
if not helper:
rds = get_redis_connection()
rds_key = f'{settings.BUILD_KEY}:{rep.spug_version}'
helper = Helper.make(rds, rds_key)
rep.save()
try:
api_token = uuid.uuid4().hex
helper.rds.setex(api_token, 60 * 60, f'{rep.app_id},{rep.env_id}')
helper.send_info('local', f'\033[32m完成√\033[0m\r\n{human_time()} 构建准备... ')
env = AttrDict(
SPUG_APP_NAME=rep.app.name,
SPUG_APP_KEY=rep.app.key,
SPUG_APP_ID=str(rep.app_id),
SPUG_DEPLOY_ID=str(rep.deploy_id),
SPUG_BUILD_ID=str(rep.id),
SPUG_ENV_ID=str(rep.env_id),
SPUG_ENV_KEY=rep.env.key,
SPUG_VERSION=rep.version,
SPUG_BUILD_VERSION=rep.spug_version,
SPUG_API_TOKEN=api_token,
SPUG_REPOS_DIR=REPOS_DIR,
)
# append configs
configs = compose_configs(rep.app, rep.env_id)
configs_env = {f'_SPUG_{k.upper()}': v for k, v in configs.items()}
env.update(configs_env)
_build(rep, helper, env)
rep.status = '5'
except Exception as e:
rep.status = '2'
raise e
finally:
helper.local(f'cd {REPOS_DIR} && rm -rf {rep.spug_version}')
close_old_connections()
if alone_build:
helper.clear()
rep.save()
return rep
elif rep.status == '5':
rep.save()
def _build(rep: Repository, helper, env):
extend = rep.deploy.extend_obj
extras = json.loads(rep.extra)
git_dir = os.path.join(REPOS_DIR, str(rep.deploy_id))
build_dir = os.path.join(REPOS_DIR, rep.spug_version)
tar_file = os.path.join(BUILD_DIR, f'{rep.spug_version}.tar.gz')
if extras[0] == 'branch':
tree_ish = extras[2]
env.update(SPUG_GIT_BRANCH=extras[1], SPUG_GIT_COMMIT_ID=extras[2])
else:
tree_ish = extras[1]
env.update(SPUG_GIT_TAG=extras[1])
env.update(SPUG_DST_DIR=render_str(extend.dst_dir, env))
fetch_repo(rep.deploy_id, extend.git_repo)
helper.send_info('local', '\033[32m完成√\033[0m\r\n')
if extend.hook_pre_server:
helper.send_step('local', 1, f'{human_time()} 检出前任务...\r\n')
helper.local(f'cd {git_dir} && {extend.hook_pre_server}', env)
helper.send_step('local', 2, f'{human_time()} 执行检出... ')
command = f'cd {git_dir} && git archive --prefix={rep.spug_version}/ {tree_ish} | (cd .. && tar xf -)'
helper.local(command)
helper.send_info('local', '\033[32m完成√\033[0m\r\n')
if extend.hook_post_server:
helper.send_step('local', 3, f'{human_time()} 检出后任务...\r\n')
helper.local(f'cd {build_dir} && {extend.hook_post_server}', env)
helper.send_step('local', 4, f'\r\n{human_time()} 执行打包... ')
filter_rule, exclude, contain = json.loads(extend.filter_rule), '', rep.spug_version
files = helper.parse_filter_rule(filter_rule['data'], env=env)
if files:
if filter_rule['type'] == 'exclude':
excludes = []
for x in files:
if x.startswith('/'):
excludes.append(f'--exclude={rep.spug_version}{x}')
else:
excludes.append(f'--exclude={x}')
exclude = ' '.join(excludes)
else:
contain = ' '.join(f'{rep.spug_version}/{x}' for x in files)
helper.local(f'mkdir -p {BUILD_DIR} && cd {REPOS_DIR} && tar zcf {tar_file} {exclude} {contain}')
helper.send_step('local', 5, f'\033[32m完成√\033[0m')
helper.send_step('local', 100, f'\r\n\r\n{human_time()} ** \033[32m构建成功\033[0m **')
================================================
FILE: spug_api/apps/repository/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django.db.models import F
from django.conf import settings
from django_redis import get_redis_connection
from libs import json_response, JsonParser, Argument, human_time, AttrDict, auth
from apps.repository.models import Repository
from apps.deploy.models import DeployRequest
from apps.repository.utils import dispatch
from apps.app.models import Deploy
from threading import Thread
import json
class RepositoryView(View):
@auth('deploy.repository.view|deploy.request.add|deploy.request.edit')
def get(self, request):
apps = request.user.deploy_perms['apps']
deploy_id = request.GET.get('deploy_id')
data = Repository.objects.filter(app_id__in=apps).annotate(
app_name=F('app__name'),
env_name=F('env__name'),
created_by_user=F('created_by__nickname'))
if deploy_id:
data = data.filter(deploy_id=deploy_id, status='5')
return json_response([x.to_view() for x in data])
response = dict()
for item in data:
if item.app_id in response:
response[item.app_id]['child'].append(item.to_view())
else:
tmp = item.to_view()
tmp['child'] = [item.to_view()]
response[item.app_id] = tmp
return json_response(list(response.values()))
@auth('deploy.repository.add')
def post(self, request):
form, error = JsonParser(
Argument('deploy_id', type=int, help='参数错误'),
Argument('version', help='请输入构建版本'),
Argument('extra', type=list, help='参数错误'),
Argument('remarks', required=False)
).parse(request.body)
if error is None:
deploy = Deploy.objects.filter(pk=form.deploy_id).first()
if not deploy:
return json_response(error='未找到指定发布配置')
form.extra = json.dumps(form.extra)
form.spug_version = Repository.make_spug_version(deploy.id)
rep = Repository.objects.create(
app_id=deploy.app_id,
env_id=deploy.env_id,
created_by=request.user,
**form)
Thread(target=dispatch, args=(rep,)).start()
return json_response(rep.to_view())
return json_response(error=error)
@auth('deploy.repository.add|deploy.repository.build')
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='参数错误'),
Argument('action', help='参数错误')
).parse(request.body)
if error is None:
rep = Repository.objects.filter(pk=form.id).first()
if not rep:
return json_response(error='未找到指定构建记录')
if form.action == 'rebuild':
Thread(target=dispatch, args=(rep,)).start()
return json_response(rep.to_view())
return json_response(error=error)
@auth('deploy.repository.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
repository = Repository.objects.filter(pk=form.id).first()
if not repository:
return json_response(error='未找到指定构建记录')
if repository.deployrequest_set.exists():
return json_response(error='已关联发布申请无法删除')
repository.delete()
return json_response(error=error)
@auth('deploy.repository.view')
def get_requests(request):
form, error = JsonParser(
Argument('repository_id', type=int, help='参数错误')
).parse(request.GET)
if error is None:
requests = []
for item in DeployRequest.objects.filter(repository_id=form.repository_id):
data = item.to_dict(selects=('id', 'name', 'created_at'))
data['host_ids'] = json.loads(item.host_ids)
data['status_alias'] = item.get_status_display()
requests.append(data)
return json_response(requests)
@auth('deploy.repository.view')
def get_detail(request, r_id):
repository = Repository.objects.filter(pk=r_id).first()
if not repository:
return json_response(error='未找到指定构建记录')
rds, counter = get_redis_connection(), 0
if repository.remarks == 'SPUG AUTO MAKE':
req = repository.deployrequest_set.last()
key = f'{settings.REQUEST_KEY}:{req.id}'
else:
key = f'{settings.BUILD_KEY}:{repository.spug_version}'
data = rds.lrange(key, counter, counter + 9)
response = AttrDict(data='', step=0, s_status='process', status=repository.status)
while data:
for item in data:
counter += 1
item = json.loads(item.decode())
if item['key'] == 'local':
if 'data' in item:
response.data += item['data']
if 'step' in item:
response.step = item['step']
if 'status' in item:
response.status = item['status']
data = rds.lrange(key, counter, counter + 9)
response.index = counter
if repository.status in ('0', '1'):
response.data = f'{human_time()} 建立连接... ' + response.data
elif not response.data:
response.data = f'{human_time()} 读取数据... \r\n\r\n未读取到数据，Spug 仅保存最近2周的构建日志。'
else:
response.data = f'{human_time()} 读取数据... ' + response.data
return json_response(response)
================================================
FILE: spug_api/apps/schedule/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/schedule/builtin.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import connections
from django.conf import settings
from apps.account.models import History, User
from apps.alarm.models import Alarm
from apps.schedule.models import Task, History as TaskHistory
from apps.deploy.models import DeployRequest
from apps.app.models import DeployExtend1
from apps.exec.models import ExecHistory, Transfer
from apps.notify.models import Notify
from apps.deploy.utils import dispatch
from apps.repository.models import Repository
from libs.utils import parse_time, human_datetime, human_date
from datetime import datetime, timedelta
from threading import Thread
from collections import defaultdict
from pathlib import Path
import time
import os
def auto_run_by_day():
try:
date_7 = human_date(datetime.now() - timedelta(days=7))
date_30 = human_date(datetime.now() - timedelta(days=30))
History.objects.filter(created_at__lt=date_30).delete()
Notify.objects.filter(created_at__lt=date_7, unread=False).delete()
Alarm.objects.filter(created_at__lt=date_30).delete()
for item in DeployExtend1.objects.all():
index = 0
for req in DeployRequest.objects.filter(deploy_id=item.deploy_id, repository_id__isnull=False):
if index > item.versions and req.repository_id:
req.repository.delete()
index += 1
timer = defaultdict(int)
for item in ExecHistory.objects.all():
if timer[item.user_id] >= 10:
item.delete()
else:
timer[item.user_id] += 1
timer = defaultdict(int)
for item in Transfer.objects.all():
if timer[item.user_id] >= 10:
item.delete()
else:
timer[item.user_id] += 1
for task in Task.objects.all():
try:
record = TaskHistory.objects.filter(task_id=task.id)[50]
TaskHistory.objects.filter(task_id=task.id, id__lt=record.id).delete()
except IndexError:
pass
timestamp = time.time() - 2 * 3600
for item in Path(settings.TRANSFER_DIR).iterdir():
if item.name != '.gitkeep':
if item.stat().st_atime < timestamp:
transfer_dir = item.absolute()
os.system(f'umount -f {transfer_dir} &> /dev/null ; rm -rf {transfer_dir}')
finally:
connections.close_all()
def auto_run_by_minute():
try:
now = datetime.now()
for req in DeployRequest.objects.filter(status='2'):
if (now - parse_time(req.do_at)).seconds > 3600:
req.status = '-3'
req.save()
for rep in Repository.objects.filter(status='1'):
if (now - parse_time(rep.created_at)).seconds > 3600:
rep.status = '2'
rep.save()
for req in DeployRequest.objects.filter(status='1', plan__lte=now):
req.status = '2'
req.do_at = human_datetime()
req.do_by = req.created_by
req.save()
Thread(target=dispatch, args=(req,)).start()
finally:
connections.close_all()
================================================
FILE: spug_api/apps/schedule/executors.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from libs.ssh import AuthenticationException
from django.db import close_old_connections, transaction
from apps.host.models import Host
from apps.schedule.models import History, Task
from apps.schedule.utils import send_fail_notify
import subprocess
import socket
import time
import json
def local_executor(command):
code, out, now = 1, None, time.time()
task = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
code = task.wait(3600)
out = task.stdout.read() + task.stderr.read()
out = out.decode()
except subprocess.TimeoutExpired:
# task.kill()
out = 'timeout, wait more than 1 hour'
return code, round(time.time() - now, 3), out
def host_executor(host, command):
code, out, now = 1, None, time.time()
try:
with host.get_ssh() as ssh:
code, out = ssh.exec_command_raw(command)
except AuthenticationException:
out = 'ssh authentication fail'
except socket.error as e:
out = f'network error {e}'
return code, round(time.time() - now, 3), out
def dispatch_job(host_id, interpreter, command):
if interpreter == 'python':
attach = 'INTERPRETER=python\ncommand -v python3 &> /dev/null && INTERPRETER=python3'
command = f'{attach}\n$INTERPRETER << EOF\n# -*- coding: UTF-8 -*-\n{command}\nEOF'
if host_id == 'local':
code, duration, out = local_executor(command)
else:
host = Host.objects.filter(pk=host_id).first()
if not host:
code, duration, out = 1, 0, f'unknown host id for {host_id!r}'
else:
code, duration, out = host_executor(host, command)
return code, duration, out
def schedule_worker_handler(job):
history_id, host_id, interpreter, command = json.loads(job)
code, duration, out = dispatch_job(host_id, interpreter, command)
close_old_connections()
with transaction.atomic():
history = History.objects.select_for_update().get(pk=history_id)
output = json.loads(history.output)
output[str(host_id)] = [code, duration, out]
history.output = json.dumps(output)
if all(output.values()):
history.status = '1' if sum(x[0] for x in output.values()) == 0 else '2'
history.save()
if history.status == '2':
task = Task.objects.get(pk=history.task_id)
send_fail_notify(task)
================================================
FILE: spug_api/apps/schedule/management/commands/runscheduler.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.core.management.base import BaseCommand
from apps.schedule.scheduler import Scheduler
import logging
logging.basicConfig(level=logging.WARNING, format='%(asctime)s %(message)s')
class Command(BaseCommand):
help = 'Start schedule process'
def handle(self, *args, **options):
s = Scheduler()
s.run()
================================================
FILE: spug_api/apps/schedule/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from libs import ModelMixin, human_datetime
from apps.account.models import User
import json
class History(models.Model, ModelMixin):
STATUS = (
(0, '执行中'),
(1, '成功'),
(2, '失败'),
)
task_id = models.IntegerField()
status = models.SmallIntegerField(choices=STATUS)
run_time = models.CharField(max_length=20)
output = models.TextField()
def to_list(self):
tmp = super().to_dict(selects=('id', 'status', 'run_time'))
tmp['status_alias'] = self.get_status_display()
return tmp
class Meta:
db_table = 'task_histories'
ordering = ('-id',)
class Task(models.Model, ModelMixin):
TRIGGERS = (
('date', '一次性'),
('calendarinterval', '日历间隔'),
('cron', 'UNIX cron'),
('interval', '普通间隔')
)
name = models.CharField(max_length=50)
type = models.CharField(max_length=50)
interpreter = models.CharField(max_length=20, default='sh')
command = models.TextField()
targets = models.TextField()
trigger = models.CharField(max_length=20, choices=TRIGGERS)
trigger_args = models.CharField(max_length=255)
is_active = models.BooleanField(default=False)
desc = models.CharField(max_length=255, null=True)
latest = models.ForeignKey(History, on_delete=models.PROTECT, null=True)
rst_notify = models.CharField(max_length=255, null=True)
created_at = models.CharField(max_length=20, default=human_datetime)
created_by = models.ForeignKey(User, models.PROTECT, related_name='+')
updated_at = models.CharField(max_length=20, null=True)
updated_by = models.ForeignKey(User, models.PROTECT, related_name='+', null=True)
def to_dict(self, *args, **kwargs):
tmp = super().to_dict(*args, **kwargs)
tmp['targets'] = json.loads(self.targets)
tmp['latest_status'] = self.latest.status if self.latest else None
tmp['latest_run_time'] = self.latest.run_time if self.latest else None
tmp['latest_status_alias'] = self.latest.get_status_display() if self.latest else None
tmp['rst_notify'] = json.loads(self.rst_notify) if self.rst_notify else {'mode': '0'}
if self.trigger == 'cron':
tmp['trigger_args'] = json.loads(self.trigger_args)
return tmp
def __repr__(self):
return '' % self.name
class Meta:
db_table = 'tasks'
ordering = ('-id',)
================================================
FILE: spug_api/apps/schedule/scheduler.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.executors.pool import ThreadPoolExecutor
from apscheduler.triggers.interval import IntervalTrigger
from apscheduler.triggers.date import DateTrigger
from apscheduler.triggers.cron import CronTrigger
from django_redis import get_redis_connection
from django.db import connections
from django.db.utils import DatabaseError
from apps.schedule.models import Task, History
from apps.schedule.builtin import auto_run_by_day, auto_run_by_minute
from django.conf import settings
from libs import AttrDict, human_datetime
import logging
import json
SCHEDULE_WORKER_KEY = settings.SCHEDULE_WORKER_KEY
class Scheduler:
timezone = settings.TIME_ZONE
week_map = {
'/': '/',
'-': '-',
',': ',',
'*': '*',
'7': '6',
'0': '6',
'1': '0',
'2': '1',
'3': '2',
'4': '3',
'5': '4',
'6': '5',
}
def __init__(self):
self.scheduler = BackgroundScheduler(timezone=self.timezone, executors={'default': ThreadPoolExecutor(30)})
@classmethod
def covert_week(cls, week_str):
return ''.join(map(lambda x: cls.week_map[x], week_str))
@classmethod
def parse_trigger(cls, trigger, trigger_args):
if trigger == 'interval':
return IntervalTrigger(seconds=int(trigger_args), timezone=cls.timezone)
elif trigger == 'date':
return DateTrigger(run_date=trigger_args, timezone=cls.timezone)
elif trigger == 'cron':
args = json.loads(trigger_args) if not isinstance(trigger_args, dict) else trigger_args
minute, hour, day, month, week = args['rule'].split()
week = cls.covert_week(week)
return CronTrigger(minute=minute, hour=hour, day=day, month=month, day_of_week=week,
start_date=args['start'], end_date=args['stop'])
else:
raise TypeError(f'unknown schedule policy: {trigger!r}')
def _init_builtin_jobs(self):
self.scheduler.add_job(auto_run_by_day, 'cron', hour=1, minute=20)
self.scheduler.add_job(auto_run_by_minute, 'interval', minutes=1)
def _dispatch(self, task_id, interpreter, command, targets):
output = {x: None for x in targets}
history = History.objects.create(
task_id=task_id,
status='0',
run_time=human_datetime(),
output=json.dumps(output)
)
Task.objects.filter(pk=task_id).update(latest_id=history.id)
rds_cli = get_redis_connection()
for t in targets:
rds_cli.rpush(SCHEDULE_WORKER_KEY, json.dumps([history.id, t, interpreter, command]))
connections.close_all()
def _init(self):
self.scheduler.start()
self._init_builtin_jobs()
try:
for task in Task.objects.filter(is_active=True):
trigger = self.parse_trigger(task.trigger, task.trigger_args)
self.scheduler.add_job(
self._dispatch,
trigger,
id=str(task.id),
args=(task.id, task.interpreter, task.command, json.loads(task.targets)),
)
connections.close_all()
except DatabaseError:
pass
def run(self):
rds_cli = get_redis_connection()
self._init()
rds_cli.delete(settings.SCHEDULE_KEY)
logging.warning('Running scheduler')
while True:
_, data = rds_cli.brpop(settings.SCHEDULE_KEY)
task = AttrDict(json.loads(data))
if task.action in ('add', 'modify'):
trigger = self.parse_trigger(task.trigger, task.trigger_args)
self.scheduler.add_job(
self._dispatch,
trigger,
id=str(task.id),
args=(task.id, task.interpreter, task.command, task.targets),
replace_existing=True
)
elif task.action == 'remove':
job = self.scheduler.get_job(str(task.id))
if job:
job.remove()
================================================
FILE: spug_api/apps/schedule/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from .views import *
urlpatterns = [
path('', Schedule.as_view()),
path('/', HistoryView.as_view()),
path('run_time/', next_run_time),
]
================================================
FILE: spug_api/apps/schedule/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from libs.utils import human_datetime
from libs.spug import Notification
from libs.push import push_server
from apps.setting.utils import AppSetting
import json
def send_fail_notify(task, msg=None):
rst_notify = json.loads(task.rst_notify)
mode = rst_notify.get('mode')
url = rst_notify.get('value')
if mode != '0' and url:
_do_notify(task, mode, url, msg)
def _do_notify(task, mode, url, msg):
if mode == '1':
texts = [
'## 任务执行失败通知 ## ',
f'**任务名称：** {task.name} ',
f'**任务类型：** {task.type} ',
f'**描述信息：** {msg or "请在任务计划执行历史中查看详情"} ',
f'**发生时间：** {human_datetime()} ',
'> 来自 Spug运维平台'
]
data = {
'msgtype': 'markdown',
'markdown': {
'title': '任务执行失败通知',
'text': '\n\n'.join(texts)
},
'at': {
'isAtAll': True
}
}
Notification.handle_request(url, data, 'dd')
elif mode == '2':
data = {
'task_id': task.id,
'task_name': task.name,
'task_type': task.type,
'message': msg or '请在任务计划执行历史中查看详情',
'created_at': human_datetime()
}
Notification.handle_request(url, data)
elif mode == '3':
texts = [
'## 任务执行失败通知',
f'任务名称： {task.name}',
f'任务类型： {task.type}',
f'描述信息： {msg or "请在任务计划执行历史中查看详情"}',
f'发生时间： {human_datetime()}',
'> 来自 Spug运维平台'
]
data = {
'msgtype': 'markdown',
'markdown': {
'content': '\n'.join(texts)
}
}
Notification.handle_request(url, data, 'wx')
elif mode == '4':
data = {
'msg_type': 'post',
'content': {
'post': {
'zh_cn': {
'title': '任务执行失败通知',
'content': [
[{'tag': 'text', 'text': f'任务名称： {task.name}'}],
[{'tag': 'text', 'text': f'任务类型： {task.type}'}],
[{'tag': 'text', 'text': f'描述信息： {msg or "请在任务计划执行历史中查看详情"}'}],
[{'tag': 'text', 'text': f'发生时间： {human_datetime()}'}],
[{'tag': 'at', 'user_id': 'all'}],
]
}
}
}
}
Notification.handle_request(url, data, 'fs')
elif mode == '5':
spug_push_key = AppSetting.get_default('spug_push_key')
if not spug_push_key:
return
data = {
'source': 'schedule',
'token': spug_push_key,
'targets': url,
'dataset': {
'name': task.name,
'type': task.type,
'message': msg or '请在任务计划执行历史中查看详情',
}
}
Notification.handle_request(f'{push_server}/spug/message/', data, 'spug')
================================================
FILE: spug_api/apps/schedule/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from django_redis import get_redis_connection
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from apps.schedule.scheduler import Scheduler
from apps.schedule.models import Task, History
from apps.schedule.executors import dispatch_job
from apps.host.models import Host
from django.conf import settings
from libs import json_response, JsonParser, Argument, human_datetime, auth
import json
class Schedule(View):
@auth('schedule.schedule.view')
def get(self, request):
tasks = Task.objects.all()
types = [x['type'] for x in tasks.order_by('type').values('type').distinct()]
return json_response({'types': types, 'tasks': [x.to_dict() for x in tasks]})
@auth('schedule.schedule.add|schedule.schedule.edit')
def post(self, request):
form, error = JsonParser(
Argument('id', type=int, required=False),
Argument('type', help='请输入任务类型'),
Argument('name', help='请输入任务名称'),
Argument('interpreter', help='请选择执行解释器'),
Argument('command', help='请输入任务内容'),
Argument('rst_notify', type=dict, help='请选择执行失败通知方式'),
Argument('targets', type=list, filter=lambda x: len(x), help='请选择执行对象'),
Argument('trigger', filter=lambda x: x in dict(Task.TRIGGERS), help='请选择触发器类型'),
Argument('trigger_args', help='请输入触发器参数'),
Argument('desc', required=False),
).parse(request.body)
if error is None:
form.targets = json.dumps(form.targets)
form.rst_notify = json.dumps(form.rst_notify)
if form.trigger == 'cron':
args = json.loads(form.trigger_args)['rule'].split()
if len(args) != 5:
return json_response(error='无效的执行规则，请更正后再试')
minute, hour, day, month, week = args
week = '0' if week == '7' else week
try:
CronTrigger(minute=minute, hour=hour, day=day, month=month, day_of_week=week)
except ValueError:
return json_response(error='无效的执行规则，请更正后再试')
if form.id:
Task.objects.filter(pk=form.id).update(
updated_at=human_datetime(),
updated_by=request.user,
**form
)
task = Task.objects.filter(pk=form.id).first()
if task and task.is_active:
form.action = 'modify'
form.targets = json.loads(form.targets)
rds_cli = get_redis_connection()
rds_cli.lpush(settings.SCHEDULE_KEY, json.dumps(form))
else:
Task.objects.create(created_by=request.user, **form)
return json_response(error=error)
@auth('schedule.schedule.edit')
def patch(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象'),
Argument('is_active', type=bool, required=False)
).parse(request.body, True)
if error is None:
task = Task.objects.get(pk=form.id)
if form.get('is_active') is not None:
task.is_active = form.is_active
task.latest_id = None
if form.is_active:
message = {'id': form.id, 'action': 'add'}
message.update(task.to_dict(selects=('interpreter', 'trigger', 'trigger_args', 'command', 'targets')))
else:
message = {'id': form.id, 'action': 'remove'}
rds_cli = get_redis_connection()
rds_cli.lpush(settings.SCHEDULE_KEY, json.dumps(message))
task.save()
return json_response(error=error)
@auth('schedule.schedule.del')
def delete(self, request):
form, error = JsonParser(
Argument('id', type=int, help='请指定操作对象')
).parse(request.GET)
if error is None:
task = Task.objects.filter(pk=form.id).first()
if task:
if task.is_active:
return json_response(error='该任务在运行中，请先停止任务再尝试删除')
task.delete()
History.objects.filter(task_id=task.id).delete()
return json_response(error=error)
class HistoryView(View):
@auth('schedule.schedule.view')
def get(self, request, t_id):
task = Task.objects.filter(pk=t_id).first()
if not task:
return json_response(error='未找到指定任务')
h_id = request.GET.get('id')
if h_id:
h_id = task.latest_id if h_id == 'latest' else h_id
return json_response(self._fetch_detail(h_id))
histories = History.objects.filter(task_id=t_id)
return json_response([x.to_list() for x in histories])
@auth('schedule.schedule.edit')
def post(self, request, t_id):
task = Task.objects.filter(pk=t_id).first()
if not task:
return json_response(error='未找到指定任务')
outputs, status = {}, 1
for host_id in json.loads(task.targets):
code, duration, out = dispatch_job(host_id, task.interpreter, task.command)
if code != 0:
status = 2
outputs[host_id] = [code, duration, out]
history = History.objects.create(
task_id=task.id,
status=status,
run_time=human_datetime(),
output=json.dumps(outputs)
)
return json_response(history.id)
def _fetch_detail(self, h_id):
record = History.objects.filter(pk=h_id).first()
outputs = json.loads(record.output)
host_ids = (x for x in outputs.keys() if x != 'local')
hosts_info = {str(x.id): x.name for x in Host.objects.filter(id__in=host_ids)}
data = {'run_time': record.run_time, 'success': 0, 'failure': 0, 'duration': 0, 'outputs': []}
for host_id, value in outputs.items():
if not value:
continue
code, duration, out = value
key = 'success' if code == 0 else 'failure'
data[key] += 1
data['duration'] += duration
data['outputs'].append({
'name': hosts_info.get(host_id, '本机'),
'code': code,
'duration': duration,
'output': out})
data['duration'] = f"{data['duration'] / len(outputs):.3f}"
return data
@auth('schedule.schedule.view|schedule.schedule.add|schedule.schedule.edit')
def next_run_time(request):
form, error = JsonParser(
Argument('rule', help='参数错误'),
Argument('start', required=False),
Argument('stop', required=False)
).parse(request.body)
if error is None:
try:
minute, hour, day, month, week = form.rule.split()
week = Scheduler.covert_week(week)
trigger = CronTrigger(minute=minute, hour=hour, day=day, month=month, day_of_week=week,
start_date=form.start, end_date=form.stop)
except (ValueError, KeyError):
return json_response({'success': False, 'msg': '无效的执行规则'})
scheduler = BackgroundScheduler(timezone=settings.TIME_ZONE)
scheduler.start()
job = scheduler.add_job(lambda: None, trigger)
run_time = job.next_run_time
scheduler.shutdown()
if run_time:
return json_response({'success': True, 'msg': run_time.strftime('%Y-%m-%d %H:%M:%S')})
else:
return json_response({'success': False, 'msg': '无法被触发'})
return json_response(error=error)
================================================
FILE: spug_api/apps/setting/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/apps/setting/models.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import models
from apps.account.models import User
from libs import ModelMixin
import json
KEYS_DEFAULT = {
'MFA': {'enable': False},
'verify_ip': True,
'bind_ip': True,
'ldap_service': {},
'spug_key': None,
'api_key': None,
'mail_service': {},
'private_key': None,
'public_key': None,
'spug_push_key': None,
}
class Setting(models.Model, ModelMixin):
key = models.CharField(max_length=50, unique=True)
value = models.TextField()
desc = models.CharField(max_length=255, null=True)
def to_view(self):
tmp = self.to_dict(selects=('key',))
tmp['value'] = self.real_val
return tmp
@property
def real_val(self):
if self.value:
return json.loads(self.value)
else:
return KEYS_DEFAULT.get(self.key)
def __repr__(self):
return '' % self.key
class Meta:
db_table = 'settings'
class UserSetting(models.Model, ModelMixin):
user = models.ForeignKey(User, on_delete=models.CASCADE)
key = models.CharField(max_length=32)
value = models.TextField()
class Meta:
db_table = 'user_settings'
unique_together = ('user', 'key')
================================================
FILE: spug_api/apps/setting/urls.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
# from django.urls import path
from django.conf.urls import url
from apps.setting.views import *
from apps.setting.user import UserSettingView
urlpatterns = [
url(r'^$', SettingView.as_view()),
url(r'^user/$', UserSettingView.as_view()),
url(r'^ldap_test/$', ldap_test),
url(r'^email_test/$', email_test),
url(r'^mfa/$', MFAView.as_view()),
url(r'^about/$', get_about),
url(r'^push/bind/$', handle_push_bind),
url(r'^push/balance/$', handle_push_balance),
]
================================================
FILE: spug_api/apps/setting/user.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from libs import JsonParser, Argument, json_response
from apps.setting.models import UserSetting
class UserSettingView(View):
def get(self, request):
response = {}
for item in UserSetting.objects.filter(user=request.user):
response[item.key] = item.value
return json_response(response)
def post(self, request):
form, error = JsonParser(
Argument('key', help='参数错误'),
Argument('value', help='参数错误'),
).parse(request.body)
if error is None:
UserSetting.objects.update_or_create(
user=request.user,
key=form.key,
defaults={'value': form.value}
)
return self.get(request)
return json_response(error=error)
================================================
FILE: spug_api/apps/setting/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from functools import lru_cache
from apps.setting.models import Setting, KEYS_DEFAULT
from libs.ssh import SSH
import json
class AppSetting:
@classmethod
@lru_cache(maxsize=64)
def get(cls, key):
info = Setting.objects.filter(key=key).first()
if not info:
raise KeyError(f'no such key for {key!r}')
return info.real_val
@classmethod
def get_default(cls, key, default=None):
info = Setting.objects.filter(key=key).first()
if not info:
return default
return info.real_val
@classmethod
def set(cls, key, value, desc=None):
if key in KEYS_DEFAULT:
value = json.dumps(value)
Setting.objects.update_or_create(key=key, defaults={'value': value, 'desc': desc})
else:
raise KeyError('invalid key')
@classmethod
def delete(cls, key):
Setting.objects.filter(key=key).delete()
@classmethod
def get_ssh_key(cls):
public_key = cls.get_default('public_key')
private_key = cls.get_default('private_key')
if not private_key or not public_key:
private_key, public_key = SSH.generate_key()
cls.set('private_key', private_key)
cls.set('public_key', public_key)
return private_key, public_key
================================================
FILE: spug_api/apps/setting/views.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
import django
from django.core.cache import cache
from django.conf import settings
from libs import JsonParser, Argument, json_response, auth
from libs.utils import generate_random_str
from libs.mail import Mail
from libs.push import get_balance, send_login_code
from libs.mixins import AdminView
from apps.setting.utils import AppSetting
from apps.setting.models import Setting, KEYS_DEFAULT
from copy import deepcopy
import platform
import ldap
class SettingView(AdminView):
def get(self, request):
response = deepcopy(KEYS_DEFAULT)
for item in Setting.objects.all():
if item.key == 'spug_push_key':
response[item.key] = f'{item.real_val[:8]}********{item.real_val[-8:]}'
else:
response[item.key] = item.real_val
return json_response(response)
def post(self, request):
form, error = JsonParser(
Argument('data', type=list, help='缺少必要的参数')
).parse(request.body)
if error is None:
for item in form.data:
AppSetting.set(**item)
return json_response(error=error)
class MFAView(AdminView):
def get(self, request):
if not request.user.wx_token:
return json_response(
error='检测到当前账户未配置推送标识（账户管理/编辑），请配置后再尝试启用MFA认证，否则可能造成系统无法正常登录。')
spug_push_key = AppSetting.get_default('spug_push_key')
if not spug_push_key:
return json_response(error='检测到当前账户未绑定推送服务，请在系统设置/推送服务设置中绑定推送助手账户。')
code = generate_random_str(6)
send_login_code(spug_push_key, request.user.wx_token, code)
cache.set(f'{request.user.username}:code', code, 300)
return json_response()
def post(self, request):
form, error = JsonParser(
Argument('enable', type=bool, help='参数错误'),
Argument('code', required=False)
).parse(request.body)
if error is None:
if form.enable:
if not form.code:
return json_response(error='请输入验证码')
key = f'{request.user.username}:code'
code = cache.get(key)
if not code:
return json_response(error='验证码已失效，请重新获取')
if code != form.code:
ttl = cache.ttl(key)
cache.expire(key, ttl - 100)
return json_response(error='验证码错误')
cache.delete(key)
AppSetting.set('MFA', {'enable': form.enable})
return json_response(error=error)
@auth('admin')
def ldap_test(request):
form, error = JsonParser(
Argument('server'),
Argument('port', type=int),
Argument('admin_dn'),
Argument('password'),
).parse(request.body)
if error is None:
try:
con = ldap.initialize("ldap://{0}:{1}".format(form.server, form.port), bytes_mode=False)
con.simple_bind_s(form.admin_dn, form.password)
return json_response()
except Exception as e:
error = eval(str(e))
return json_response(error=error['desc'])
return json_response(error=error)
@auth('admin')
def email_test(request):
form, error = JsonParser(
Argument('server', help='请输入邮件服务地址'),
Argument('port', type=int, help='请输入邮件服务端口号'),
Argument('username', help='请输入邮箱账号'),
Argument('password', help='请输入密码/授权码'),
).parse(request.body)
if error is None:
try:
mail = Mail(**form)
server = mail.get_server()
server.quit()
return json_response()
except Exception as e:
error = f'{e}'
return json_response(error=error)
@auth('admin')
def get_about(request):
return json_response({
'python_version': platform.python_version(),
'system_version': platform.platform(),
'spug_version': settings.SPUG_VERSION,
'django_version': django.get_version()
})
@auth('admin')
def handle_push_bind(request):
form, error = JsonParser(
Argument('spug_push_key', required=False),
).parse(request.body)
if error is None:
if not form.spug_push_key:
AppSetting.delete('spug_push_key')
return json_response()
try:
res = get_balance(form.spug_push_key)
except Exception as e:
return json_response(error=f'绑定失败：{e}')
AppSetting.set('spug_push_key', form.spug_push_key)
return json_response(res)
return json_response(error=error)
@auth('admin')
def handle_push_balance(request):
token = AppSetting.get_default('spug_push_key')
if not token:
return json_response(error='请先配置推送服务绑定账户')
res = get_balance(token)
return json_response(res)
================================================
FILE: spug_api/consumer/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/consumer/consumers.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.conf import settings
from django_redis import get_redis_connection
from asgiref.sync import async_to_sync
from apps.host.models import Host
from consumer.utils import BaseConsumer
from apps.account.utils import has_host_perm
from libs.utils import str_decode
from threading import Thread
import time
import json
class ComConsumer(BaseConsumer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
token = self.scope['url_route']['kwargs']['token']
module = self.scope['url_route']['kwargs']['module']
if module == 'build':
self.key = f'{settings.BUILD_KEY}:{token}'
elif module == 'request':
self.key = f'{settings.REQUEST_KEY}:{token}'
elif module == 'host':
self.key = token
else:
raise TypeError(f'unknown module for {module}')
self.rds = get_redis_connection()
def disconnect(self, code):
self.rds.close()
def get_response(self, index):
counter = 0
while counter < 30:
response = self.rds.lindex(self.key, index)
if response:
return response.decode()
counter += 1
time.sleep(0.2)
def receive(self, text_data='', **kwargs):
if text_data.isdigit():
index = int(text_data)
response = self.get_response(index)
while response:
index += 1
self.send(text_data=response)
response = self.get_response(index)
self.send(text_data='pong')
class SSHConsumer(BaseConsumer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.id = self.scope['url_route']['kwargs']['id']
self.chan = None
self.ssh = None
def loop_read(self):
is_ready, data = False, b''
while True:
out = self.chan.recv(32 * 1024)
if not out:
self.close(3333)
break
data += out
try:
text = data.decode()
except UnicodeDecodeError:
try:
text = data.decode(encoding='GBK')
except UnicodeDecodeError:
time.sleep(0.01)
if self.chan.recv_ready():
continue
text = data.decode(errors='ignore')
if not is_ready:
self.send(text_data='\033[2J\033[3J\033[1;1H')
is_ready = True
self.send(text_data=text)
data = b''
def receive(self, text_data=None, bytes_data=None):
data = text_data or bytes_data
if data and self.chan:
data = json.loads(data)
# print('write: {!r}'.format(data))
resize = data.get('resize')
if resize and len(resize) == 2:
self.chan.resize_pty(*resize)
else:
self.chan.send(data['data'])
def disconnect(self, code):
if self.chan:
self.chan.close()
if self.ssh:
self.ssh.close()
def init(self):
if has_host_perm(self.user, self.id):
self.send(text_data='\r\n正在连接至主机 ...')
host = Host.objects.filter(pk=self.id).first()
if not host:
return self.close_with_message('未找到指定主机，请刷新页面重试。')
try:
self.ssh = host.get_ssh().get_client()
except Exception as e:
return self.close_with_message(f'连接主机失败: {e}')
self.chan = self.ssh.invoke_shell(term='xterm')
self.chan.transport.set_keepalive(30)
Thread(target=self.loop_read).start()
else:
self.close_with_message('你当前无权限操作该主机，请联系管理员授权。')
class NotifyConsumer(BaseConsumer):
def init(self):
async_to_sync(self.channel_layer.group_add)('notify', self.channel_name)
def disconnect(self, code):
async_to_sync(self.channel_layer.group_discard)('notify', self.channel_name)
def receive(self, **kwargs):
self.send(text_data='pong')
def notify_message(self, event):
self.send(text_data=json.dumps(event))
class PubSubConsumer(BaseConsumer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.token = self.scope['url_route']['kwargs']['token']
self.rds = get_redis_connection()
self.p = self.rds.pubsub(ignore_subscribe_messages=True)
self.p.subscribe(self.token)
def disconnect(self, code):
self.p.close()
self.rds.close()
def receive(self, **kwargs):
response = self.p.get_message(timeout=10)
while response:
data = str_decode(response['data'])
self.send(text_data=data)
response = self.p.get_message(timeout=10)
self.send(text_data='pong')
================================================
FILE: spug_api/consumer/routing.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.urls import path
from channels.routing import URLRouter
from consumer.consumers import *
ws_router = URLRouter([
path('ws/ssh//', SSHConsumer),
path('ws/subscribe//', PubSubConsumer),
path('ws///', ComConsumer),
path('ws/notify/', NotifyConsumer),
])
================================================
FILE: spug_api/consumer/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.db import close_old_connections
from channels.generic.websocket import WebsocketConsumer
from apps.account.models import User
from apps.setting.utils import AppSetting
from libs.utils import get_request_real_ip
from urllib.parse import parse_qs
import time
def get_real_ip(headers):
decode_headers = {k.decode(): v.decode() for k, v in headers}
return get_request_real_ip(decode_headers)
class BaseConsumer(WebsocketConsumer):
def __init__(self, *args, **kwargs):
super(BaseConsumer, self).__init__(*args, **kwargs)
self.user = None
def close_with_message(self, content):
self.send(text_data=f'\r\n\x1b[31m{content}\x1b[0m\r\n')
self.close()
def connect(self):
self.accept()
close_old_connections()
query_string = self.scope['query_string'].decode()
x_real_ip = get_real_ip(self.scope['headers'])
token = parse_qs(query_string).get('x-token', [''])[0]
if token and len(token) == 32:
user = User.objects.filter(access_token=token).first()
if user and user.token_expired >= time.time() and user.is_active:
if x_real_ip == user.last_ip or AppSetting.get_default('bind_ip') is False:
self.user = user
if hasattr(self, 'init'):
self.init()
return None
self.close_with_message('触发登录IP绑定安全策略，请在系统设置/安全设置中查看配置。')
self.close_with_message('用户身份验证失败，请重新登录或刷新页面。')
================================================
FILE: spug_api/libs/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from .parser import JsonParser, Argument
from .decorators import *
from .validators import *
from .mixins import *
from .utils import *
================================================
FILE: spug_api/libs/channel.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from channels.layers import get_channel_layer
from asgiref.sync import async_to_sync
import uuid
layer = get_channel_layer()
class Channel:
@staticmethod
def get_token():
return uuid.uuid4().hex
@staticmethod
def send_notify(title, content):
message = {
'type': 'notify.message',
'title': title,
'content': content
}
async_to_sync(layer.group_send)('notify', message)
================================================
FILE: spug_api/libs/decorators.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from functools import wraps
from .utils import json_response
def auth(perm_list):
def decorate(view_func):
codes = perm_list.split('|')
@wraps(view_func)
def wrapper(*args, **kwargs):
user = None
for item in args[:2]:
if hasattr(item, 'user'):
user = item.user
break
if user and user.has_perms(codes):
return view_func(*args, **kwargs)
return json_response(error='权限拒绝')
return wrapper
return decorate
================================================
FILE: spug_api/libs/gitlib.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from git import Repo, RemoteReference, TagReference, InvalidGitRepositoryError, GitCommandError
from tempfile import NamedTemporaryFile
from datetime import datetime
import shutil
import os
class Git:
def __init__(self, git_repo, repo_dir, pkey=None):
self.git_repo = git_repo
self.repo_dir = repo_dir
self.repo = None
self.pkey = pkey
self.fd = None
self.env = {}
def archive(self, filepath, commit):
with open(filepath, 'wb') as f:
self.repo.archive(f, commit)
def fetch_branches_tags(self):
self.fetch()
branches, tags = {}, {}
for ref in self.repo.references:
if isinstance(ref, RemoteReference):
if ref.remote_head != 'HEAD':
branches[ref.remote_head] = self._get_commits(f'origin/{ref.remote_head}', 30)
elif isinstance(ref, TagReference):
tags[ref.name] = {
'id': ref.tag.hexsha,
'author': ref.tag.tagger.name,
'date': self._format_date(ref.tag.tagged_date),
'message': ref.tag.message.strip()
} if ref.tag else {
'id': ref.commit.binsha.hex(),
'author': ref.commit.author.name,
'date': self._format_date(ref.commit.authored_date),
'message': ref.commit.message.strip()
}
tags = sorted(tags.items(), key=lambda x: x[1]['date'], reverse=True)
return branches, dict(tags)
def fetch(self):
kwargs = dict(f=True, p=True)
if self.repo.git.version_info >= (2, 17, 0):
kwargs.update(P=True)
try:
self.repo.remotes.origin.fetch(**kwargs)
except GitCommandError as e:
if self.env:
self.repo.remotes.origin.fetch(env=self.env, **kwargs)
else:
raise e
def _get_repo(self):
if os.path.exists(self.repo_dir):
try:
return Repo(self.repo_dir)
except InvalidGitRepositoryError:
if os.path.isdir(self.repo_dir):
shutil.rmtree(self.repo_dir)
else:
os.remove(self.repo_dir)
try:
repo = Repo.clone_from(self.git_repo, self.repo_dir)
except GitCommandError as e:
if self.env:
repo = Repo.clone_from(self.git_repo, self.repo_dir, env=self.env)
else:
raise e
return repo
def _get_commits(self, branch, count=10):
commits = []
for commit in self.repo.iter_commits(branch):
if len(commits) == count:
break
commits.append({
'id': commit.hexsha,
'author': commit.author.name,
'date': self._format_date(commit.committed_date),
'message': commit.message.strip()
})
return commits
def _format_date(self, timestamp):
if isinstance(timestamp, int):
date = datetime.fromtimestamp(timestamp)
return date.strftime('%Y-%m-%d %H:%M')
return timestamp
def __enter__(self):
if self.pkey:
self.fd = NamedTemporaryFile()
self.fd.write(self.pkey.encode())
self.fd.flush()
self.env = {'GIT_SSH_COMMAND': f'ssh -o StrictHostKeyChecking=no -i {self.fd.name}'}
self.repo = self._get_repo()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if self.fd:
self.fd.close()
================================================
FILE: spug_api/libs/helper.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from urllib.parse import quote, urlencode
from datetime import datetime
from pytz import timezone
import requests
import hashlib
import base64
import random
import time
import hmac
import uuid
def _special_url_encode(value) -> str:
if isinstance(value, (str, bytes)):
rst = quote(value)
else:
rst = urlencode(value)
return rst.replace('+', '%20').replace('*', '%2A').replace('%7E', '~')
def _make_ali_signature(key: str, params: dict) -> bytes:
sorted_str = _special_url_encode(dict(sorted(params.items())))
sign_str = 'GET&%2F&' + _special_url_encode(sorted_str)
sign_digest = hmac.new(key.encode(), sign_str.encode(), hashlib.sha1).digest()
return base64.encodebytes(sign_digest).strip()
def _make_tencent_signature(endpoint: str, key: str, params: dict) -> bytes:
sorted_str = '&'.join(f'{k}={v}' for k, v in sorted(params.items()))
sign_str = f'POST{endpoint}/?{sorted_str}'
sign_digest = hmac.new(key.encode(), sign_str.encode(), hashlib.sha1).digest()
return base64.encodebytes(sign_digest).strip()
def make_ali_request(ak, ac, endpoint, params):
params.update(
AccessKeyId=ak,
Format='JSON',
SignatureMethod='HMAC-SHA1',
SignatureNonce=uuid.uuid4().hex,
SignatureVersion='1.0',
Timestamp=datetime.now(tz=timezone('UTC')).strftime('%Y-%m-%dT%H:%M:%SZ'),
Version='2014-05-26'
)
params['Signature'] = _make_ali_signature(ac + '&', params)
return requests.get(endpoint, params).json()
def make_tencent_request(ak, ac, endpoint, params):
params.update(
Nonce=int(random.random() * 10000),
SecretId=ak,
Timestamp=int(time.time()),
Version='2017-03-12'
)
params['Signature'] = _make_tencent_signature(endpoint, ac, params)
return requests.post(f'https://{endpoint}', data=params).json()
================================================
FILE: spug_api/libs/ldap.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
import ldap
class LDAP:
def __init__(self, server, port, rules, admin_dn, password, base_dn):
self.server = server
self.port = port
self.rules = rules
self.admin_dn = admin_dn
self.password = password
self.base_dn = base_dn
def valid_user(self, username, password):
try:
conn = ldap.initialize("ldap://{0}:{1}".format(self.server, self.port), bytes_mode=False)
conn.simple_bind_s(self.admin_dn, self.password)
search_filter = f'({self.rules}={username})'
ldap_result_id = conn.search(self.base_dn, ldap.SCOPE_SUBTREE, search_filter, None)
result_type, result_data = conn.result(ldap_result_id, 0)
if result_type == ldap.RES_SEARCH_ENTRY:
conn.simple_bind_s(result_data[0][0], password)
return True, None
else:
return False, None
except Exception as error:
args = error.args
return False, args[0].get('desc', '未知错误') if args else '%s' % error
================================================
FILE: spug_api/libs/mail.py
================================================
from email.header import Header
from email.mime.text import MIMEText
from email.utils import formataddr
import smtplib
class Mail:
def __init__(self, server, port, username, password, nickname=None):
self.host = server
self.port = int(port)
self.user = username
self.password = password
self.nickname = nickname
def get_server(self):
if self.port == 465:
server = smtplib.SMTP_SSL(self.host, self.port)
elif self.port == 587:
server = smtplib.SMTP(self.host, self.port)
server.ehlo()
server.starttls()
else:
server = smtplib.SMTP(self.host, self.port)
server.login(self.user, self.password)
return server
def send_text_mail(self, receivers, subject, body):
server = self.get_server()
msg = MIMEText(body, 'plain', 'utf-8')
msg['Subject'] = Header(subject, 'utf-8')
msg['From'] = formataddr((self.nickname, self.user)) if self.nickname else self.user
server.sendmail(self.user, receivers, msg.as_string())
server.quit()
================================================
FILE: spug_api/libs/middleware.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.utils.deprecation import MiddlewareMixin
from django.conf import settings
from .utils import json_response, get_request_real_ip
from apps.account.models import User
from apps.setting.utils import AppSetting
import traceback
import time
class HandleExceptionMiddleware(MiddlewareMixin):
"""
处理试图函数异常
"""
def process_exception(self, request, exception):
traceback.print_exc()
return json_response(error='Exception: %s' % exception)
class AuthenticationMiddleware(MiddlewareMixin):
"""
登录验证
"""
def process_request(self, request):
if request.path in settings.AUTHENTICATION_EXCLUDES:
return None
if any(x.match(request.path) for x in settings.AUTHENTICATION_EXCLUDES if hasattr(x, 'match')):
return None
access_token = request.headers.get('x-token') or request.GET.get('x-token')
if access_token and len(access_token) == 32:
x_real_ip = get_request_real_ip(request.headers)
user = User.objects.filter(access_token=access_token).first()
if user and user.token_expired >= time.time() and user.is_active:
if x_real_ip == user.last_ip or AppSetting.get_default('bind_ip') is False:
request.user = user
user.token_expired = time.time() + settings.TOKEN_TTL
user.save()
return None
response = json_response(error="验证失败，请重新登录")
response.status_code = 401
return response
================================================
FILE: spug_api/libs/mixins.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.views.generic import View
from .utils import json_response
# 混入类，提供Model实例to_dict方法
class ModelMixin(object):
__slots__ = ()
def to_dict(self, excludes: tuple = None, selects: tuple = None) -> dict:
if not hasattr(self, '_meta'):
raise TypeError('<%r> does not a django.db.models.Model object.' % self)
elif selects:
return {f: getattr(self, f) for f in selects}
elif excludes:
return {f.attname: getattr(self, f.attname) for f in self._meta.fields if f.attname not in excludes}
else:
return {f.attname: getattr(self, f.attname) for f in self._meta.fields}
def update_by_dict(self, data):
for key, value in data.items():
setattr(self, key, value)
self.save()
class AdminView(View):
def dispatch(self, request, *args, **kwargs):
if hasattr(request, 'user') and request.user.is_supper:
return super().dispatch(request, *args, **kwargs)
else:
return json_response(error='权限拒绝')
================================================
FILE: spug_api/libs/parser.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
import json
from .utils import AttrDict
# 自定义的解析异常
class ParseError(BaseException):
def __init__(self, message):
self.message = message
# 需要校验的参数对象
class Argument(object):
"""
:param name: name of option
:param default: default value if the argument if absent
:param bool required: is required
"""
def __init__(self, name, default=None, handler=None, required=True, type=str, filter=None, help=None):
self.name = name
self.default = default
self.type = type
self.required = required
self.filter = filter
self.help = help
self.handler = handler
if not isinstance(self.name, str):
raise TypeError('Argument name must be string')
if filter and not callable(self.filter):
raise TypeError('Argument filter is not callable')
def parse(self, has_key, value):
if not has_key:
if self.required and self.default is None:
raise ParseError(
self.help or 'Required Error: %s is required' % self.name)
else:
return self.default
elif value in [u'', '', None]:
if self.default is not None:
return self.default
elif self.required:
raise ParseError(self.help or 'Value Error: %s must not be null' % self.name)
elif self.help:
raise ParseError(self.help)
else:
return value
try:
if self.type:
if self.type in (list, dict) and isinstance(value, str):
value = json.loads(value)
assert isinstance(value, self.type)
elif self.type == bool and isinstance(value, str):
assert value.lower() in ['true', 'false']
value = value.lower() == 'true'
elif not isinstance(value, self.type):
value = self.type(value)
except (TypeError, ValueError, AssertionError):
raise ParseError(self.help or 'Type Error: %s type must be %s' % (
self.name, self.type))
if self.filter:
if not self.filter(value):
raise ParseError(
self.help or 'Value Error: %s filter check failed' % self.name)
if self.handler:
value = self.handler(value)
return value
# 解析器基类
class BaseParser(object):
def __init__(self, *args):
self.args = []
for e in args:
if isinstance(e, str):
e = Argument(e)
elif not isinstance(e, Argument):
raise TypeError('%r is not instance of Argument' % e)
self.args.append(e)
def _get(self, key):
raise NotImplementedError
def _init(self, data):
raise NotImplementedError
def add_argument(self, **kwargs):
self.args.append(Argument(**kwargs))
def parse(self, data=None, clear=False):
rst = AttrDict()
try:
self._init(data)
for e in self.args:
has_key, value = self._get(e.name)
if clear and has_key is False and e.required is False:
continue
rst[e.name] = e.parse(has_key, value)
except ParseError as err:
return None, err.message
return rst, None
# Json解析器
class JsonParser(BaseParser):
def __init__(self, *args):
self.__data = None
super(JsonParser, self).__init__(*args)
def _get(self, key):
return key in self.__data, self.__data.get(key)
def _init(self, data):
try:
if isinstance(data, (str, bytes)):
self.__data = json.loads(data) if data else {}
else:
assert hasattr(data, '__contains__')
assert hasattr(data, 'get')
assert callable(data.get)
self.__data = data
except (ValueError, AssertionError):
raise ParseError('Invalid data type for parse')
================================================
FILE: spug_api/libs/push.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apps.setting.utils import AppSetting
import requests
push_server = 'https://push.spug.cc'
def get_balance(token):
res = requests.get(f'{push_server}/spug/balance/', json={'token': token})
if res.status_code != 200:
raise Exception(f'status code: {res.status_code}')
res = res.json()
if res.get('error'):
raise Exception(res['error'])
return res['data']
def get_contacts(token):
try:
res = requests.post(f'{push_server}/spug/contacts/', json={'token': token})
res = res.json()
if res['data']:
return res['data']
except Exception:
return []
def send_login_code(token, user, code):
url = f'{push_server}/spug/message/'
data = {
'token': token,
'targets': [user],
'source': 'mfa',
'dataset': {
'code': code
}
}
res = requests.post(url, json=data, timeout=15)
if res.status_code != 200:
raise Exception(f'status code: {res.status_code}')
res = res.json()
if res.get('error'):
raise Exception(res['error'])
================================================
FILE: spug_api/libs/spug.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from apps.alarm.models import Group, Contact
from apps.setting.utils import AppSetting
from apps.notify.models import Notify
from libs.mail import Mail
from libs.utils import human_datetime
from libs.push import push_server
import requests
import json
import time
import hmac
import hashlib
import base64
from urllib.parse import urlencode
def _gen_dd_sign(secret):
timestamp = str(int(time.time() * 1000))
string_to_sign = f'{timestamp}\n{secret}'
hmac_code = hmac.new(secret.encode('utf-8'), string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()
sign = base64.b64encode(hmac_code).decode('utf-8')
return timestamp, sign
def _gen_fs_sign(secret):
timestamp = str(int(time.time()))
string_to_sign = f'{timestamp}\n{secret}'
hmac_code = hmac.new(string_to_sign.encode('utf-8'), b'', digestmod=hashlib.sha256).digest()
sign = base64.b64encode(hmac_code).decode('utf-8')
return timestamp, sign
class Notification:
def __init__(self, grp, event, target, title, message, duration):
self.grp = grp
self.event = event
self.title = title
self.target = target
self.message = message
self.duration = duration
self.spug_push_key = AppSetting.get_default('spug_push_key')
@staticmethod
def handle_request(url, data, mode=None):
try:
res = requests.post(url, json=data, timeout=15)
except Exception as e:
return Notify.make_system_notify('通知发送失败', f'接口调用异常: {e}')
if res.status_code != 200:
return Notify.make_system_notify('通知发送失败', f'返回状态码：{res.status_code}, 请求URL：{res.url}')
if mode in ['dd', 'wx']:
res = res.json()
if res.get('errcode') == 0:
return
elif mode == 'spug':
res = res.json()
if not res.get('error'):
return
elif mode == 'fs':
res = res.json()
if res.get('StatusCode') == 0:
return
else:
raise NotImplementedError
Notify.make_system_notify('通知发送失败', f'返回数据：{res}')
def monitor_by_email(self, users):
mail_service = AppSetting.get_default('mail_service', {})
body = [
f'告警名称：{self.title}',
f'告警对象：{self.target}',
f'{"告警" if self.event == "1" else "恢复"}时间：{human_datetime()}',
f'告警描述：{self.message}'
]
if self.event == '2':
body.append('故障持续：' + self.duration)
if mail_service.get('server'):
event_map = {'1': '监控告警通知', '2': '告警恢复通知'}
subject = f'{event_map[self.event]}-{self.title}'
mail = Mail(**mail_service)
mail.send_text_mail(users, subject, '\r\n'.join(body) + '\r\n\r\n自动发送，请勿回复。')
else:
Notify.make_monitor_notify(
'发送报警信息失败',
'未配置报警服务，请在系统管理/系统设置/报警服务设置中配置邮件服务。'
)
def monitor_by_dd(self, users):
texts = [
'## %s ## ' % ('监控告警通知' if self.event == '1' else '告警恢复通知'),
f'**告警名称：** {self.title} ',
f'**告警对象：** {self.target} ',
f'**{"告警" if self.event == "1" else "恢复"}时间：** {human_datetime()} ',
f'**告警描述：** {self.message} ',
]
if self.event == '2':
texts.append(f'**持续时间：** {self.duration} ')
data = {
'msgtype': 'markdown',
'markdown': {
'title': '监控告警通知',
'text': '\n\n'.join(texts) + '\n\n> ###### 来自 Spug运维平台'
},
'at': {
'isAtAll': True
}
}
for url, secret in users:
if secret:
timestamp, sign = _gen_dd_sign(secret)
url = f'{url}&{urlencode({"timestamp": timestamp, "sign": sign})}'
self.handle_request(url, data, 'dd')
def monitor_by_fs(self, users):
title = '监控告警通知' if self.event == '1' else '告警恢复通知'
content = [
[{'tag': 'text', 'text': f'告警名称：{self.title}'}],
[{'tag': 'text', 'text': f'告警对象：{self.target}'}],
[{'tag': 'text', 'text': f'{"告警" if self.event == "1" else "恢复"}时间：{human_datetime()}'}],
[{'tag': 'text', 'text': f'告警描述：{self.message}'}],
]
if self.event == '2':
content.append([{'tag': 'text', 'text': f'持续时间：{self.duration}'}])
content.append([{'tag': 'text', 'text': '来自 Spug运维平台'}])
for url, secret in users:
data = {
'msg_type': 'post',
'content': {
'post': {
'zh_cn': {
'title': title,
'content': content
}
}
}
}
if secret:
timestamp, sign = _gen_fs_sign(secret)
data['timestamp'] = timestamp
data['sign'] = sign
self.handle_request(url, data, 'fs')
def monitor_by_qy_wx(self, users):
color, title = ('warning', '监控告警通知') if self.event == '1' else ('info', '告警恢复通知')
texts = [
f'## {title}',
f'**告警名称：** {self.title} ',
f'**告警对象：** {self.target}',
f'**{"告警" if self.event == "1" else "恢复"}时间：** {human_datetime()} ',
f'**告警描述：** {self.message} ',
]
if self.event == '2':
texts.append(f'**持续时间：** {self.duration} ')
data = {
'msgtype': 'markdown',
'markdown': {
'content': '\n'.join(texts) + '\n> 来自 Spug运维平台'
}
}
for url in users:
self.handle_request(url, data, 'wx')
def monitor_by_spug_push(self, targets):
if not self.spug_push_key:
Notify.make_monitor_notify(
'发送报警信息失败',
'未绑定推送服务，请在系统管理/系统设置/推送服务设置中绑定推送助手账户。'
)
return
data = {
'source': 'monitor',
'token': self.spug_push_key,
'targets': list(targets),
'dataset': {
'title': self.title,
'target': self.target,
'message': self.message,
'duration': self.duration,
'event': self.event
}
}
self.handle_request(f'{push_server}/spug/message/', data, 'spug')
def dispatch_monitor(self, modes):
u_ids, push_ids = [], []
for item in Group.objects.filter(id__in=self.grp):
for x in json.loads(item.contacts):
if isinstance(x, str) and '_' in x:
push_ids.append(x)
else:
u_ids.append(x)
targets = set()
for mode in modes:
if mode == '1':
wx_mp_ids = set(x for x in push_ids if x.startswith('wx_mp_'))
targets.update(wx_mp_ids)
elif mode == '2':
sms_ids = set(x for x in push_ids if x.startswith('sms_'))
targets.update(sms_ids)
elif mode == '3':
contacts = Contact.objects.filter(id__in=u_ids, ding__isnull=False)
users = []
for c in contacts:
sec = None
if c.secret:
sec = json.loads(c.secret).get('ding')
users.append((c.ding, sec))
if not users:
Notify.make_monitor_notify(
'发送报警信息失败',
'未找到可用的通知对象，请确保设置了相关报警联系人的钉钉。'
)
continue
self.monitor_by_dd(users)
elif mode == '4':
mail_ids = set(x for x in push_ids if x.startswith('mail_'))
targets.update(mail_ids)
users = set(x.email for x in Contact.objects.filter(id__in=u_ids, email__isnull=False))
if not users:
if not mail_ids:
Notify.make_monitor_notify(
'发送报警信息失败',
'未找到可用的通知对象，请确保设置了相关报警联系人的邮件地址。'
)
continue
self.monitor_by_email(users)
elif mode == '5':
users = set(x.qy_wx for x in Contact.objects.filter(id__in=u_ids, qy_wx__isnull=False))
if not users:
Notify.make_monitor_notify(
'发送报警信息失败',
'未找到可用的通知对象，请确保设置了相关报警联系人的企业微信。'
)
continue
self.monitor_by_qy_wx(users)
elif mode == '6':
voice_ids = set(x for x in push_ids if x.startswith('voice_'))
targets.update(voice_ids)
elif mode == '7':
contacts = Contact.objects.filter(id__in=u_ids, feishu__isnull=False)
users = []
for c in contacts:
sec = None
if c.secret:
sec = json.loads(c.secret).get('feishu')
users.append((c.feishu, sec))
if not users:
Notify.make_monitor_notify(
'发送报警信息失败',
'未找到可用的通知对象，请确保设置了相关报警联系人的飞书。'
)
continue
self.monitor_by_fs(users)
if targets:
self.monitor_by_spug_push(targets)
================================================
FILE: spug_api/libs/ssh.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from paramiko.client import SSHClient, AutoAddPolicy
from paramiko.rsakey import RSAKey
from paramiko.auth_handler import AuthHandler
from paramiko.ssh_exception import AuthenticationException, SSHException
from paramiko.py3compat import b, u
from io import StringIO
from uuid import uuid4
import time
import re
def _finalize_pubkey_algorithm(self, key_type):
if "rsa" not in key_type:
return key_type
if re.search(r"-OpenSSH_(?:[1-6]|7\.[0-7])", self.transport.remote_version):
pubkey_algo = "ssh-rsa"
if key_type.endswith("-cert-v01@openssh.com"):
pubkey_algo += "-cert-v01@openssh.com"
self.transport._agreed_pubkey_algorithm = pubkey_algo
return pubkey_algo
my_algos = [x for x in self.transport.preferred_pubkeys if "rsa" in x]
if not my_algos:
raise SSHException(
"An RSA key was specified, but no RSA pubkey algorithms are configured!" # noqa
)
server_algo_str = u(
self.transport.server_extensions.get("server-sig-algs", b(""))
)
if server_algo_str:
server_algos = server_algo_str.split(",")
agreement = list(filter(server_algos.__contains__, my_algos))
if agreement:
pubkey_algo = agreement[0]
else:
err = "Unable to agree on a pubkey algorithm for signing a {!r} key!" # noqa
raise AuthenticationException(err.format(key_type))
else:
pubkey_algo = "ssh-rsa"
if key_type.endswith("-cert-v01@openssh.com"):
pubkey_algo += "-cert-v01@openssh.com"
self.transport._agreed_pubkey_algorithm = pubkey_algo
return pubkey_algo
AuthHandler._finalize_pubkey_algorithm = _finalize_pubkey_algorithm
class SSH:
def __init__(self, hostname, port=22, username='root', pkey=None, password=None, default_env=None,
connect_timeout=10, term=None):
self.stdout = None
self.client = None
self.channel = None
self.sftp = None
self.exec_file = None
self.term = term or {}
self.eof = 'Spug EOF 2108111926'
self.default_env = default_env
self.regex = re.compile(r'Spug EOF 2108111926 (-?\d+)[\r\n]?')
self.arguments = {
'hostname': hostname,
'port': port,
'username': username,
'password': password,
'pkey': RSAKey.from_private_key(StringIO(pkey)) if isinstance(pkey, str) else pkey,
'timeout': connect_timeout,
'allow_agent': False,
'look_for_keys': False,
'banner_timeout': 30
}
@staticmethod
def generate_key():
key_obj = StringIO()
key = RSAKey.generate(2048)
key.write_private_key(key_obj)
return key_obj.getvalue(), 'ssh-rsa ' + key.get_base64()
def get_client(self):
if self.client is not None:
return self.client
self.client = SSHClient()
self.client.set_missing_host_key_policy(AutoAddPolicy)
self.client.connect(**self.arguments)
return self.client
def ping(self):
return True
def add_public_key(self, public_key):
command = f'mkdir -p -m 700 ~/.ssh && \
echo {public_key!r} >> ~/.ssh/authorized_keys && \
chmod 600 ~/.ssh/authorized_keys'
exit_code, out = self.exec_command_raw(command)
if exit_code != 0:
raise Exception(f'add public key error: {out}')
def exec_command_raw(self, command, environment=None):
channel = self.client.get_transport().open_session()
if environment:
channel.update_environment(environment)
channel.set_combine_stderr(True)
channel.exec_command(command)
code, output = channel.recv_exit_status(), channel.recv(-1)
return code, self._decode(output)
def exec_command(self, command, environment=None):
channel = self._get_channel()
command = self._handle_command(command, environment)
channel.sendall(command)
out, exit_code = '', -1
for line in self.stdout:
match = self.regex.search(line)
if match:
exit_code = int(match.group(1))
line = line[:match.start()]
out += line
break
out += line
return exit_code, out
def _win_exec_command_with_stream(self, command, environment=None):
channel = self.client.get_transport().open_session()
if environment:
channel.update_environment(environment)
channel.set_combine_stderr(True)
channel.get_pty(width=102)
channel.exec_command(command)
stdout = channel.makefile("rb", -1)
out = stdout.readline()
while out:
yield channel.exit_status, self._decode(out)
out = stdout.readline()
yield channel.recv_exit_status(), self._decode(out)
def exec_command_with_stream(self, command, environment=None):
channel = self._get_channel()
command = self._handle_command(command, environment)
channel.sendall(command)
exit_code, line = -1, ''
while True:
line = self._decode(channel.recv(8196))
if not line:
break
match = self.regex.search(line)
if match:
exit_code = int(match.group(1))
line = line[:match.start()]
break
yield exit_code, line
yield exit_code, line
def put_file(self, local_path, remote_path, callback=None):
sftp = self._get_sftp()
sftp.put(local_path, remote_path, callback=callback, confirm=False)
def put_file_by_fl(self, fl, remote_path, callback=None):
sftp = self._get_sftp()
sftp.putfo(fl, remote_path, callback=callback, confirm=False)
def list_dir_attr(self, path):
sftp = self._get_sftp()
return sftp.listdir_attr(path)
def sftp_stat(self, path):
sftp = self._get_sftp()
return sftp.stat(path)
def remove_file(self, path):
sftp = self._get_sftp()
sftp.remove(path)
def _get_channel(self):
if self.channel:
return self.channel
counter = 0
self.channel = self.client.invoke_shell(**self.term)
command = '[ -n "$BASH_VERSION" ] && set +o history\n'
command += '[ -n "$ZSH_VERSION" ] && set +o zle && set -o no_nomatch\n'
command += 'export PS1= && stty -echo\n'
command = self._handle_command(command, self.default_env)
self.channel.sendall(command)
out = ''
while True:
if self.channel.recv_ready():
out += self._decode(self.channel.recv(8196))
if self.regex.search(out):
self.stdout = self.channel.makefile('r')
break
elif counter >= 100:
self.client.close()
raise Exception('Wait spug response timeout')
else:
counter += 1
time.sleep(0.1)
return self.channel
def _get_sftp(self):
if self.sftp:
return self.sftp
self.sftp = self.client.open_sftp()
return self.sftp
def _make_env_command(self, environment):
if not environment:
return None
str_envs = []
for k, v in environment.items():
k = k.replace('-', '_')
if isinstance(v, str):
v = v.replace("'", "'\"'\"'")
str_envs.append(f"{k}='{v}'")
str_envs = ' '.join(str_envs)
return f'export {str_envs}'
def _handle_command(self, command, environment):
new_command = commands = ''
if not self.exec_file:
self.exec_file = f'/tmp/spug.{uuid4().hex}'
commands += f'trap \'rm -f {self.exec_file}\' EXIT\n'
env_command = self._make_env_command(environment)
if env_command:
new_command += f'{env_command}\n'
new_command += command
new_command += f'\necho {self.eof} $?\n'
self.put_file_by_fl(StringIO(new_command), self.exec_file)
commands += f'. {self.exec_file}\n'
return commands
def _decode(self, content):
try:
content = content.decode()
except UnicodeDecodeError:
content = content.decode(encoding='GBK', errors='ignore')
return content
def __enter__(self):
self.get_client()
transport = self.client.get_transport()
if 'windows' in transport.remote_version.lower():
self.exec_command = self.exec_command_raw
self.exec_command_with_stream = self._win_exec_command_with_stream
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.client.close()
self.client = None
================================================
FILE: spug_api/libs/utils.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from django.http.response import HttpResponse
from django.db.models import QuerySet
from datetime import datetime, date as datetime_date
from decimal import Decimal
from string import Template
import string
import random
import json
# 转换时间格式到字符串
def human_datetime(date=None):
if date:
assert isinstance(date, datetime)
else:
date = datetime.now()
return date.strftime('%Y-%m-%d %H:%M:%S')
# 转换时间格式到字符串(天)
def human_date(date=None):
if date:
assert isinstance(date, datetime)
else:
date = datetime.now()
return date.strftime('%Y-%m-%d')
def human_time(date=None):
if date:
assert isinstance(date, datetime)
else:
date = datetime.now()
return date.strftime('%H:%M:%S')
def str_decode(data):
try:
data = data.decode()
except UnicodeDecodeError:
try:
data = data.decode(encoding='GBK')
except UnicodeDecodeError:
data = data.decode(errors='ignore')
return data
# 解析时间类型的数据
def parse_time(value):
if isinstance(value, datetime):
return value
if isinstance(value, str):
if len(value) == 10:
return datetime.strptime(value, '%Y-%m-%d')
elif len(value) == 19:
return datetime.strptime(value, '%Y-%m-%d %H:%M:%S')
raise TypeError('Expect a datetime.datetime value')
# 传两个时间得到一个时间差
def human_seconds_time(seconds):
text = ''
if seconds >= 3600:
text += '%d小时' % (seconds / 3600)
seconds = seconds % 3600
if seconds >= 60:
text += '%d分' % (seconds / 60)
seconds = seconds % 60
if seconds > 0:
if text or isinstance(seconds, int):
text += '%.d秒' % seconds
else:
text += '%.1f秒' % seconds
return text
# 字符串模版渲染
def render_str(template, datasheet):
return Template(template).safe_substitute(datasheet)
def json_response(data='', error=''):
content = AttrDict(data=data, error=error)
if error:
content.data = ''
elif hasattr(data, 'to_dict'):
content.data = data.to_dict()
elif isinstance(data, (list, QuerySet)) and all([hasattr(item, 'to_dict') for item in data]):
content.data = [item.to_dict() for item in data]
return HttpResponse(json.dumps(content, cls=DateTimeEncoder), content_type='application/json')
# 继承自dict，实现可以通过.来操作元素
class AttrDict(dict):
def __setattr__(self, key, value):
self.__setitem__(key, value)
def __getattr__(self, item):
try:
return self.__getitem__(item)
except KeyError:
raise AttributeError(item)
def __delattr__(self, item):
self.__delitem__(item)
# 日期json序列化
class DateTimeEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, datetime):
return o.strftime('%Y-%m-%d %H:%M:%S')
elif isinstance(o, datetime_date):
return o.strftime('%Y-%m-%d')
elif isinstance(o, Decimal):
return float(o)
return json.JSONEncoder.default(self, o)
# 生成指定长度的随机数
def generate_random_str(length: int = 4, is_digits: bool = True) -> str:
words = string.digits if is_digits else string.ascii_letters + string.digits
return ''.join(random.sample(words, length))
def get_request_real_ip(headers: dict):
x_real_ip = headers.get('x-forwarded-for')
if not x_real_ip:
x_real_ip = headers.get('x-real-ip', '')
return x_real_ip.split(',')[0]
================================================
FILE: spug_api/libs/validators.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
import ipaddress
from datetime import datetime
# 判断是否是ip地址
def ip_validator(value):
try:
ipaddress.ip_address(value)
return True
except ValueError:
return False
# 判断是否是日期字符串，支持 2018-04-11 或 2018-04-11 14:55:30
def date_validator(value: str) -> bool:
value = value.strip()
try:
if len(value) == 10:
datetime.strptime(value, '%Y-%m-%d')
return True
elif len(value) == 19:
datetime.strptime(value, '%Y-%m-%d %H:%M:%S')
return True
except ValueError:
pass
return False
================================================
FILE: spug_api/logs/.gitkeep
================================================
================================================
FILE: spug_api/manage.py
================================================
#!/usr/bin/env python
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
"""Django's command-line utility for administrative tasks."""
import os
import sys
def main():
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'spug.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
if __name__ == '__main__':
main()
================================================
FILE: spug_api/repos/.gitkeep
================================================
================================================
FILE: spug_api/repos/build/.gitkeep
================================================
================================================
FILE: spug_api/requirements.txt
================================================
apscheduler==3.7.0
Django==2.2.28
asgiref==3.2.10
channels==2.3.1
channels_redis==2.4.1
paramiko==2.11.0
django-redis==4.10.0
requests==2.32.0
GitPython==3.1.41
python-ldap==3.4.0
openpyxl==3.0.3
user_agents==2.2.0
================================================
FILE: spug_api/spug/__init__.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
================================================
FILE: spug_api/spug/asgi.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
"""
ASGI entrypoint. Configures Django and then runs the application
defined in the ASGI_APPLICATION setting.
"""
import os
import django
from channels.routing import get_default_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'spug.settings')
django.setup()
application = get_default_application()
================================================
FILE: spug_api/spug/routing.py
================================================
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
from channels.routing import ProtocolTypeRouter
from consumer import routing
application = ProtocolTypeRouter({
'websocket': routing.ws_router
})
================================================
FILE: spug_api/spug/settings.py
================================================
"""
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
Django settings for spug project.
Generated by 'django-admin startproject' using Django 2.2.7.
For more information on this file, see
https://docs.djangoproject.com/en/2.2/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/2.2/ref/settings/
"""
import os
import re
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.2/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'vk0do47)egwzz!uk49%(y3s(fpx4+ha@ugt-hcv&%&d@hwr&p7'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['127.0.0.1']
# Application definition
INSTALLED_APPS = [
'apps.account',
'apps.host',
'apps.setting',
'apps.exec',
'apps.schedule',
'apps.monitor',
'apps.alarm',
'apps.config',
'apps.app',
'apps.deploy',
'apps.notify',
'apps.repository',
'apps.home',
'channels',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.middleware.common.CommonMiddleware',
'libs.middleware.AuthenticationMiddleware',
'libs.middleware.HandleExceptionMiddleware',
]
ROOT_URLCONF = 'spug.urls'
WSGI_APPLICATION = 'spug.wsgi.application'
ASGI_APPLICATION = 'spug.routing.application'
# Database
# https://docs.djangoproject.com/en/2.2/ref/settings/#databases
DATABASES = {
'default': {
'ATOMIC_REQUESTS': True,
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": "redis://127.0.0.1:6379/1",
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient",
}
}
}
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("127.0.0.1", 6379)],
"capacity": 1000,
"expiry": 120,
},
},
}
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': False,
},
]
TOKEN_TTL = 8 * 3600
SCHEDULE_KEY = 'spug:schedule'
SCHEDULE_WORKER_KEY = 'spug:schedule:worker'
MONITOR_KEY = 'spug:monitor'
MONITOR_WORKER_KEY = 'spug:monitor:worker'
EXEC_WORKER_KEY = 'spug:exec:worker'
REQUEST_KEY = 'spug:request'
BUILD_KEY = 'spug:build'
REPOS_DIR = os.path.join(os.path.dirname(os.path.dirname(BASE_DIR)), 'repos')
BUILD_DIR = os.path.join(REPOS_DIR, 'build')
TRANSFER_DIR = os.path.join(BASE_DIR, 'storage', 'transfer')
# Internationalization
# https://docs.djangoproject.com/en/2.2/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_L10N = True
USE_TZ = False
AUTHENTICATION_EXCLUDES = (
'/account/login/',
'/setting/basic/',
re.compile('/apis/.*'),
)
SPUG_VERSION = 'v3.3.3'
# override default config
try:
from spug.overrides import *
except ImportError:
pass
================================================
FILE: spug_api/spug/urls.py
================================================
"""spug URL Configuration
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/2.2/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.urls import path, include
urlpatterns = [
path('account/', include('apps.account.urls')),
path('host/', include('apps.host.urls')),
path('exec/', include('apps.exec.urls')),
path('schedule/', include('apps.schedule.urls')),
path('monitor/', include('apps.monitor.urls')),
path('alarm/', include('apps.alarm.urls')),
path('setting/', include('apps.setting.urls')),
path('config/', include('apps.config.urls')),
path('app/', include('apps.app.urls')),
path('deploy/', include('apps.deploy.urls')),
path('repository/', include('apps.repository.urls')),
path('home/', include('apps.home.urls')),
path('notify/', include('apps.notify.urls')),
path('file/', include('apps.file.urls')),
path('apis/', include('apps.apis.urls')),
]
================================================
FILE: spug_api/spug/wsgi.py
================================================
"""
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
WSGI config for spug project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'spug.settings')
application = get_wsgi_application()
================================================
FILE: spug_api/storage/transfer/.gitkeep
================================================
================================================
FILE: spug_api/tools/migrate.py
================================================
import django
import sys
import os
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(BASE_DIR)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "spug.settings")
django.setup()
from django.conf import settings
import subprocess
import shutil
import sys
import os
import re
class Version:
def __init__(self, version):
self.version = re.sub('[^0-9.]', '', version).split('.')
def __gt__(self, other):
if not isinstance(other, Version):
raise TypeError('required type Version')
for v1, v2 in zip(self.version, other.version):
if int(v1) == int(v2):
continue
elif int(v1) > int(v2):
return True
else:
return False
return False
if __name__ == '__main__':
old_version = Version(sys.argv[1])
now_version = Version(settings.SPUG_VERSION)
if old_version < Version('v3.0.2'):
old_path = os.path.join(settings.BASE_DIR, 'repos')
new_path = os.path.join(settings.REPOS_DIR)
if not os.path.exists(new_path):
print('执行 v3.0.1-beta.8 repos目录迁移')
shutil.move(old_path, new_path)
task = subprocess.Popen(f'cd {settings.BASE_DIR} && git checkout -- repos', shell=True)
if task.wait() != 0:
print('repos目录迁移失败，请联系官方人员')
================================================
FILE: spug_api/tools/start-api.sh
================================================
#!/bin/bash
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
# start api service
cd $(dirname $(dirname $0))
if [ -f ./venv/bin/activate ]; then
source ./venv/bin/activate
fi
exec gunicorn -b 127.0.0.1:9001 -w 2 --threads 8 --access-logfile - spug.wsgi
================================================
FILE: spug_api/tools/start-monitor.sh
================================================
#!/bin/bash
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
# start monitor service
cd $(dirname $(dirname $0))
if [ -f ./venv/bin/activate ]; then
source ./venv/bin/activate
fi
if command -v python3 &> /dev/null; then
PYTHON=python3
else
PYTHON=python
fi
exec $PYTHON manage.py runmonitor
================================================
FILE: spug_api/tools/start-scheduler.sh
================================================
#!/bin/bash
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
# start schedule service
cd $(dirname $(dirname $0))
if [ -f ./venv/bin/activate ]; then
source ./venv/bin/activate
fi
if command -v python3 &> /dev/null; then
PYTHON=python3
else
PYTHON=python
fi
exec $PYTHON manage.py runscheduler
================================================
FILE: spug_api/tools/start-worker.sh
================================================
#!/bin/bash
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
# start worker service
cd $(dirname $(dirname $0))
if [ -f ./venv/bin/activate ]; then
source ./venv/bin/activate
fi
if command -v python3 &> /dev/null; then
PYTHON=python3
else
PYTHON=python
fi
exec $PYTHON manage.py runworker
================================================
FILE: spug_api/tools/start-ws.sh
================================================
#!/bin/bash
# Copyright: (c) OpenSpug Organization. https://github.com/openspug/spug
# Copyright: (c)
# Released under the AGPL-3.0 License.
# start websocket service
cd $(dirname $(dirname $0))
if [ -f ./venv/bin/activate ]; then
source ./venv/bin/activate
fi
exec daphne -p 9002 spug.asgi:application
================================================
FILE: spug_api/tools/supervisor-spug.ini
================================================
[program:spug-api]
command = bash /data/spug/spug_api/tools/start-api.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/api.log
redirect_stderr = true
[program:spug-ws]
command = bash /data/spug/spug_api/tools/start-ws.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/ws.log
redirect_stderr = true
[program:spug-worker]
command = bash /data/spug/spug_api/tools/start-worker.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/worker.log
redirect_stderr = true
[program:spug-monitor]
command = bash /data/spug/spug_api/tools/start-monitor.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/monitor.log
redirect_stderr = true
[program:spug-scheduler]
command = bash /data/spug/spug_api/tools/start-scheduler.sh
autostart = true
stdout_logfile = /data/spug/spug_api/logs/scheduler.log
redirect_stderr = true
================================================
FILE: spug_web/.gitignore
================================================
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
# dependencies
/node_modules
/.pnp
.pnp.js
/.idea/
# testing
/coverage
# production
/build
# misc
.DS_Store
.env.local
.env.development.local
.env.test.local
.env.production.local
npm-debug.log*
yarn-debug.log*
yarn-error.log*
================================================
FILE: spug_web/README.md
================================================
spug web
================================================
FILE: spug_web/config-overrides.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
const {override, addDecoratorsLegacy, addLessLoader} = require('customize-cra');
module.exports = override(
addDecoratorsLegacy(),
addLessLoader({
lessOptions: {
javascriptEnabled: true,
modifyVars: {
'@primary-color': '#2563fc'
}
}
}),
);
================================================
FILE: spug_web/jsconfig.json
================================================
{
"compilerOptions": {
"baseUrl": "src"
},
"include": [
"src"
]
}
================================================
FILE: spug_web/package.json
================================================
{
"name": "spug_web",
"version": "3.0.0",
"private": true,
"dependencies": {
"@ant-design/icons": "^4.3.0",
"ace-builds": "^1.4.13",
"antd": "4.21.5",
"axios": "^0.21.0",
"bizcharts": "^3.5.9",
"history": "^4.10.1",
"lodash": "^4.17.19",
"mobx": "^5.15.6",
"mobx-react": "^6.3.0",
"moment": "^2.24.0",
"react": "^16.13.1",
"react-ace": "^9.5.0",
"react-dom": "^16.13.1",
"react-router-dom": "^5.2.0",
"react-scripts": "3.4.3",
"xterm": "^4.6.0",
"xterm-addon-fit": "^0.5.0"
},
"scripts": {
"start": "react-app-rewired start",
"build": "GENERATE_SOURCEMAP=false react-app-rewired build",
"test": "react-app-rewired test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": "react-app"
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
},
"devDependencies": {
"@babel/plugin-proposal-decorators": "^7.10.5",
"customize-cra": "^1.0.0",
"http-proxy-middleware": "0.19.2",
"less": "^3.12.2",
"less-loader": "^7.1.0",
"react-app-rewired": "^2.1.6"
}
}
================================================
FILE: spug_web/public/index.html
================================================
Spug
================================================
FILE: spug_web/public/manifest.json
================================================
{
"short_name": "Spug",
"name": "Spug",
"icons": [
{
"src": "favicon.ico",
"sizes": "64x64 32x32 24x24 16x16",
"type": "image/x-icon"
},
{
"src": "logo.png",
"type": "image/png",
"sizes": "192x192"
},
{
"src": "logo.png",
"type": "image/png",
"sizes": "512x512"
}
],
"start_url": ".",
"display": "standalone",
"theme_color": "#000000",
"background_color": "#ffffff"
}
================================================
FILE: spug_web/public/robots.txt
================================================
User-agent: *
Disallow: /
================================================
FILE: spug_web/src/App.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React, { Component } from 'react';
import {Switch, Route} from 'react-router-dom';
import Login from './pages/login';
import WebSSH from './pages/ssh';
import Layout from './layout';
class App extends Component {
render() {
return (
);
}
}
export default App;
================================================
FILE: spug_web/src/components/ACEditor.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import Editor from 'react-ace';
import 'ace-builds/src-noconflict/mode-sh';
import 'ace-builds/src-noconflict/mode-text';
import 'ace-builds/src-noconflict/mode-json';
import 'ace-builds/src-noconflict/mode-space';
import 'ace-builds/src-noconflict/mode-python';
import 'ace-builds/src-noconflict/theme-tomorrow';
export default function (props) {
const style = {fontFamily: 'Source Code Pro, Courier New, Courier, Monaco, monospace, PingFang SC, Microsoft YaHei', ...props.style}
return (
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { Link as ALink } from 'react-router-dom';
import { Divider, Button as AButton } from 'antd';
import { hasPermission } from 'libs';
function canVisible(auth) {
return !auth || hasPermission(auth)
}
class Action extends React.Component {
static Link(props) {
return
* Released under the AGPL-3.0 License.
*/
import React, { useState, useEffect } from 'react';
import { Link } from 'react-router-dom';
import { observer } from 'mobx-react';
import { Modal, Menu, Spin, Input } from 'antd';
import { OrderedListOutlined, BuildOutlined, SearchOutlined } from '@ant-design/icons';
import { includes, http } from 'libs';
import styles from './index.module.less';
import envStore from 'pages/config/environment/store';
import lds from 'lodash';
export default observer(function AppSelector(props) {
const [fetching, setFetching] = useState(false);
const [env_id, setEnvId] = useState();
const [search, setSearch] = useState();
const [deploys, setDeploys] = useState([]);
useEffect(() => {
setFetching(true);
http.get('/api/app/deploy/')
.then(res => setDeploys(res))
.finally(() => setFetching(false))
if (!envStore.records.length) {
envStore.fetchRecords().then(_initEnv)
} else {
_initEnv()
}
}, [])
function _initEnv() {
if (envStore.records.length) {
setEnvId(envStore.records[0].id)
}
}
let records = deploys.filter(x => x.env_id === Number(env_id));
if (search) records = records.filter(x => includes(x['app_name'], search) || includes(x['app_key'], search));
if (props.filter) records = records.filter(x => props.filter(x));
return (
)
})
================================================
FILE: spug_web/src/components/AuthButton.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { Button } from 'antd';
import { hasPermission } from 'libs';
export default function AuthButton(props) {
let disabled = props.disabled;
if (props.auth && !hasPermission(props.auth)) {
disabled = true;
}
return disabled ? null :
}
================================================
FILE: spug_web/src/components/AuthCard.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import {Card} from 'antd';
import { hasPermission } from 'libs';
export default function AuthCard(props) {
let disabled = props.disabled === undefined ? false : props.disabled;
if (props.auth && !hasPermission(props.auth)) {
disabled = true;
}
return disabled ? null : {props.children}
}
================================================
FILE: spug_web/src/components/AuthDiv.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { hasPermission } from 'libs';
export default function AuthDiv(props) {
let disabled = props.disabled === undefined ? false : props.disabled;
if (props.auth && !hasPermission(props.auth)) {
disabled = true;
}
return disabled ? null :
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { hasPermission } from 'libs';
export default function AuthFragment(props) {
return hasPermission(props.auth) ? {props.children} : null
}
================================================
FILE: spug_web/src/components/Breadcrumb.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { Breadcrumb } from 'antd';
import styles from './index.module.less';
export default class extends React.Component {
static Item = Breadcrumb.Item
render() {
let title = this.props.title;
if (!title) {
const rawChildren = this.props.children;
if (Array.isArray(rawChildren)) {
title = rawChildren[rawChildren.length - 1].props.children
} else {
title = rawChildren.props.children
}
}
return (
{this.props.children}
{this.props.extra ? (
)
}
}
================================================
FILE: spug_web/src/components/Link.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react'
function Link(props) {
return (
{props.title}
)
}
export default Link
================================================
FILE: spug_web/src/components/LinkButton.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { Button } from 'antd';
import { hasPermission } from 'libs';
export default function LinkButton(props) {
let disabled = props.disabled;
if (props.auth && !hasPermission(props.auth)) {
disabled = true;
}
return
}
================================================
FILE: spug_web/src/components/NotFound.js
================================================
import React from 'react';
import styles from './index.module.less';
export default function NotFound() {
return (
)
}
================================================
FILE: spug_web/src/components/SearchForm.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { Row, Col, Form } from 'antd';
import styles from './index.module.less';
export default class extends React.Component {
static Item(props) {
return (
{props.children}
)
}
render() {
return (
)
}
}
================================================
FILE: spug_web/src/components/StatisticsCard.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React from 'react';
import { Card, Col, Row } from 'antd';
import lodash from 'lodash';
import styles from './index.module.less';
class StatisticsCard extends React.Component {
static Item = (props) => {
return (
{items.map((item, index) => (
)
}
}
export default StatisticsCard
================================================
FILE: spug_web/src/components/TableCard.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
* Released under the AGPL-3.0 License.
*/
import React, { useState, useEffect, useRef } from 'react';
import { Table, Space, Divider, Popover, Checkbox, Button, Input, Select } from 'antd';
import { ReloadOutlined, SettingOutlined, FullscreenOutlined, SearchOutlined } from '@ant-design/icons';
import styles from './index.module.less';
let TableFields = localStorage.getItem('TableFields')
TableFields = TableFields ? JSON.parse(TableFields) : {}
function Search(props) {
let keys = props.keys || [''];
keys = keys.map(x => x.split('/'));
const [key, setKey] = useState(keys[0][0]);
return (
}
onChange={e => props.onChange(key, e.target.value)}
addonBefore={(
)}/>
)
}
function Footer(props) {
const actions = props.actions || [];
const length = props.selected.length;
return length > 0 ? (
{actions.map((item, index) => (
{item}
))}
) : null
}
function Header(props) {
const columns = props.columns || [];
const actions = props.actions || [];
const fields = props.fields || [];
const onFieldsChange = props.onFieldsChange;
const Fields = () => {
return (
{columns.map((item, index) => (
{item.title}
))}
)
}
function handleCheckAll(e) {
if (e.target.checked) {
onFieldsChange(columns.map((_, index) => index))
} else {
onFieldsChange([])
}
}
function handleFullscreen() {
if (props.rootRef.current && document.fullscreenEnabled) {
if (document.fullscreenElement) {
document.exitFullscreen()
} else {
props.rootRef.current.requestFullscreen()
}
}
}
return (
{actions.map((item, index) => (
{item}
))}
{actions.length ?
列展示,
]}
overlayClassName={styles.tableFields}
trigger="click"
placement="bottomRight"
content={
)
}
function TableCard(props) {
const rootRef = useRef();
const batchActions = props.batchActions || [];
const selected = props.selected || [];
const [fields, setFields] = useState([]);
const [defaultFields, setDefaultFields] = useState([]);
const [columns, setColumns] = useState([]);
useEffect(() => {
let [_columns, _fields] = [props.columns, []];
if (props.children) {
if (Array.isArray(props.children)) {
_columns = props.children.filter(x => x.props).map(x => x.props)
} else {
_columns = [props.children.props]
}
}
let hideFields = _columns.filter(x => x.hide).map(x => x.title)
if (props.tKey) {
if (TableFields[props.tKey]) {
hideFields = TableFields[props.tKey]
} else {
TableFields[props.tKey] = hideFields
localStorage.setItem('TableFields', JSON.stringify(TableFields))
}
}
for (let [index, item] of _columns.entries()) {
if (!hideFields.includes(item.title)) _fields.push(index)
}
setFields(_fields);
setColumns(_columns);
setDefaultFields(_fields);
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [])
function handleFieldsChange(fields) {
setFields(fields)
if (props.tKey) {
TableFields[props.tKey] = columns.filter((_, index) => !fields.includes(index)).map(x => x.title)
localStorage.setItem('TableFields', JSON.stringify(TableFields))
}
}
return (
fields.includes(index))}
dataSource={props.dataSource}
rowSelection={props.rowSelection}
expandable={props.expandable}
pagination={props.pagination}/>
{selected.length ?
{lds.get(envStore.idMap, `${env_id}.name`)}
}
onChange={e => setSearch(e.target.value)}/>
{records.map(item => (
props.onSelect(item)}>
{item.extend === '1' ?
))}
{records.length === 0 &&
{item.app_name}
{item.app_key}
该环境下还没有可发布或构建的应用哦，快去应用管理创建应用发布配置吧。
}
{props.children}
}
================================================
FILE: spug_web/src/components/AuthFragment.js
================================================
/**
* Copyright (c) OpenSpug Organization. https://github.com/openspug/spug
* Copyright (c)
{title}
{this.props.extra}
) : null}
404
抱歉，你访问的页面不存在
{props.title}
)
};
render() {
let items = lodash.get(this.props, 'children', []);
if (!lodash.isArray(items)) items = [items];
const span = Math.ceil(24 / (items.length || 1));
return (
{props.value}
{props.bordered !== false && }已选择 {length} 项
{props.title}