Showing preview only (236K chars total). Download the full file or copy to clipboard to get everything.
Repository: SolarEdgeTech/pyctuator
Branch: master
Commit: 45c93757088c
Files: 86
Total size: 215.4 KB
Directory structure:
gitextract_1r5sy0a9/
├── .coveragerc
├── .github/
│ └── workflows/
│ ├── python_package_build.yml
│ └── python_package_publish.yml
├── .gitignore
├── .pylintrc
├── LICENSE
├── Makefile
├── README.md
├── examples/
│ ├── Advanced/
│ │ ├── README.md
│ │ ├── advanced_example_app.py
│ │ ├── docker-compose.yml
│ │ └── pyproject.toml
│ ├── FastAPI/
│ │ ├── README.md
│ │ ├── fastapi_example_app.py
│ │ ├── fastapi_with_authentication_example_app.py
│ │ └── pyproject.toml
│ ├── Flask/
│ │ ├── README.md
│ │ ├── flask_example_app.py
│ │ └── pyproject.toml
│ ├── __init__.py
│ ├── aiohttp/
│ │ ├── README.md
│ │ ├── aiohttp_example_app.py
│ │ └── pyproject.toml
│ └── tornado/
│ ├── README.md
│ ├── pyproject.toml
│ └── tornado_example_app.py
├── mypy.ini
├── pyctuator/
│ ├── __init__.py
│ ├── auth.py
│ ├── endpoints.py
│ ├── environment/
│ │ ├── __init__.py
│ │ ├── custom_environment_provider.py
│ │ ├── environment_provider.py
│ │ ├── os_env_variables_impl.py
│ │ └── scrubber.py
│ ├── health/
│ │ ├── __init__.py
│ │ ├── composite_health_provider.py
│ │ ├── db_health_provider.py
│ │ ├── diskspace_health_impl.py
│ │ ├── health_provider.py
│ │ └── redis_health_provider.py
│ ├── httptrace/
│ │ ├── __init__.py
│ │ ├── http_header_scrubber.py
│ │ └── http_tracer.py
│ ├── impl/
│ │ ├── __init__.py
│ │ ├── aiohttp_pyctuator.py
│ │ ├── fastapi_pyctuator.py
│ │ ├── flask_pyctuator.py
│ │ ├── pyctuator_impl.py
│ │ ├── pyctuator_router.py
│ │ ├── spring_boot_admin_registration.py
│ │ └── tornado_pyctuator.py
│ ├── logfile/
│ │ └── logfile.py
│ ├── logging/
│ │ ├── __init__.py
│ │ └── pyctuator_logging.py
│ ├── metrics/
│ │ ├── __init__.py
│ │ ├── memory_metrics_impl.py
│ │ ├── metrics_provider.py
│ │ └── thread_metrics_impl.py
│ ├── py.typed
│ ├── pyctuator.py
│ └── threads/
│ ├── __init__.py
│ └── thread_dump_provider.py
├── pyproject.toml
└── tests/
├── __init__.py
├── aiohttp_test_server.py
├── conftest.py
├── environment/
│ ├── __init__.py
│ ├── test_custom_environment_provider.py
│ └── test_scrubber.py
├── fast_api_test_server.py
├── flask_test_server.py
├── health/
│ ├── __init__.py
│ ├── test_composite_health_provider.py
│ ├── test_db_health_provider.py
│ ├── test_health_status.py
│ └── test_redis_health_provider.py
├── httptrace/
│ ├── __init__.py
│ ├── test_http_header_scrubber.py
│ └── test_tornado_pyctuator.py
├── logfile/
│ ├── __init__.py
│ └── test_logfile.py
├── test_disabled_endpoints.py
├── test_pyctuator_e2e.py
├── test_spring_boot_admin_registration.py
└── tornado_test_server.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .coveragerc
================================================
[run]
omit = .venv/*
================================================
FILE: .github/workflows/python_package_build.yml
================================================
# This workflow will install dependencies, build Pyctuator, run tests (+coverage) and lint
name: build
on:
push:
pull_request:
jobs:
run_image:
runs-on: [ubuntu-latest]
container:
image: matanrubin/python-poetry:3.9
env:
TEST_REDIS_SERVER: True
REDIS_HOST: redis
services:
# Use a redis container for testing the redis health-provider
redis:
image: redis:5.0.3
steps:
- uses: actions/checkout@v2
- run: make bootstrap
- run: poetry build -vvv
# Install all dependencies except for psutil and run the tests with coverage - this tests handling missing psutil
- run: poetry install --extras flask --extras fastapi --extras aiohttp --extras tornado --extras db --extras redis
- run: make coverage
# Run pylint+mypy after installing psutil so they don't complain on missing dependencies
- run: poetry install --extras psutil
- run: make check
# Run tests with coverage again - this adds tests that require psutil
- run: make coverage
# Upload coverage files to codecov
- uses: actions/upload-artifact@v2
with:
name: htmlcov.zip
path: htmlcov/
- uses: codecov/codecov-action@v1
================================================
FILE: .github/workflows/python_package_publish.yml
================================================
# This workflow will build Pyctuator, and publish it to pypi.org
name: publish
on:
release:
types: [published]
jobs:
run_image:
runs-on: [ubuntu-latest]
container:
image: matanrubin/python-poetry:3.9
steps:
- uses: actions/checkout@v2
- run: make bootstrap
- run: poetry update -vvv
- run: poetry build -vvv
- run: poetry publish --username __token__ --password ${{ secrets.PYPI_TOKEN }}
================================================
FILE: .gitignore
================================================
# Created by .ignore support plugin (hsz.mobi)
### macOS template
# General
.DS_Store
.AppleDouble
.LSOverride
# Icon must end with two \r
Icon
# Thumbnails
._*
# Files that might appear in the root of a volume
.DocumentRevisions-V100
.fseventsd
.Spotlight-V100
.TemporaryItems
.Trashes
.VolumeIcon.icns
.com.apple.timemachine.donotpresent
# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk
### Java template
# Compiled class file
*.class
# Log file
*.log
# BlueJ files
*.ctxt
# Mobile Tools for Java (J2ME)
.mtj.tmp/
# Package Files #
*.jar
*.war
*.nar
*.ear
*.zip
*.tar.gz
*.rar
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*
### Maven template
target/
pom.xml.tag
pom.xml.releaseBackup
pom.xml.versionsBackup
pom.xml.next
release.properties
dependency-reduced-pom.xml
buildNumber.properties
.mvn/timing.properties
.mvn/wrapper/maven-wrapper.jar
### Python template
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
### JetBrains template
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
### Eclipse template
.metadata
bin/
tmp/
*.tmp
*.bak
*.swp
*~.nib
local.properties
.settings/
.loadpath
.recommenders
# External tool builders
.externalToolBuilders/
# Locally stored "Eclipse launch configurations"
*.launch
# PyDev specific (Python IDE for Eclipse)
*.pydevproject
# CDT-specific (C/C++ Development Tooling)
.cproject
# CDT- autotools
.autotools
# Java annotation processor (APT)
.factorypath
# PDT-specific (PHP Development Tools)
.buildpath
# sbteclipse plugin
.target
# Tern plugin
.tern-project
# TeXlipse plugin
.texlipse
# STS (Spring Tool Suite)
.springBeans
# Code Recommenders
.recommenders/
# Annotation Processing
.apt_generated/
# Scala IDE specific (Scala & Java development for Eclipse)
.cache-main
.scala_dependencies
.worksheet
### VisualStudio template
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
##
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
# User-specific files
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# Visual Studio 2015/2017 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# Visual Studio 2017 auto generated files
Generated\ Files/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUNIT
*.VisualState.xml
TestResult.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# Benchmark Results
BenchmarkDotNet.Artifacts/
# .NET Core
project.lock.json
project.fragment.lock.json
artifacts/
# StyleCop
StyleCopReport.xml
# Files built by Visual Studio
*_i.c
*_p.c
*_i.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
*.pdb
*.ipdb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*.log
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# Visual Studio Trace Files
*.e2e
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# JustCode is a .NET coding add-in
.JustCode
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# AxoCover is a Code Coverage Tool
.axoCover/*
!.axoCover/settings.json
# Visual Studio code coverage results
*.coverage
*.coveragexml
# NCrunch
_NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# Note: Comment the next line if you want to checkin your web deploy settings,
# but database connection strings (with potential passwords) will be unencrypted
*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# The packages folder can be ignored because of Package Restore
**/[Pp]ackages/*
# except build/, which is used as an MSBuild target.
!**/[Pp]ackages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/[Pp]ackages/repositories.config
# NuGet v3's project.json files produces more ignorable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
*.appx
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
orleans.codegen.cs
# Including strong name files can present a security risk
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
#*.snk
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
*.rptproj.bak
# SQL Server files
*.mdf
*.ldf
*.ndf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
*.rptproj.rsuser
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
node_modules/
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
*.vbw
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
.paket/paket.exe
paket-files/
# FAKE - F# Make
.fake/
# JetBrains Rider
.idea/
*.sln.iml
# CodeRush
.cr/
# Python Tools for Visual Studio (PTVS)
__pycache__/
*.pyc
# Visual Studio Code
.vscode/
# Cake - Uncomment if you are using it
# tools/**
# !tools/packages.config
# Tabs Studio
*.tss
# Telerik's JustMock configuration file
*.jmconfig
# BizTalk build output
*.btp.cs
*.btm.cs
*.odx.cs
*.xsd.cs
# OpenCover UI analysis results
OpenCover/
# Azure Stream Analytics local run output
ASALocalRun/
# MSBuild Binary and Structured Log
*.binlog
# NVidia Nsight GPU debugger configuration file
*.nvuser
# MFractors (Xamarin productivity tool) working folder
.mfractor/
### Windows template
# Windows thumbnail cache files
Thumbs.db
ehthumbs.db
ehthumbs_vista.db
# Dump file
*.stackdump
# Folder config file
[Dd]esktop.ini
# Recycle Bin used on file shares
$RECYCLE.BIN/
# Windows Installer files
*.cab
*.msi
*.msix
*.msm
*.msp
# Windows shortcuts
*.lnk
### Kotlin template
# Compiled class file
*.class
# Log file
*.log
# BlueJ files
*.ctxt
# Mobile Tools for Java (J2ME)
.mtj.tmp/
# Package Files #
*.jar
*.war
*.nar
*.ear
*.zip
*.tar.gz
*.rar
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid*
### VirtualEnv template
# Virtualenv
# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/
.Python
[Bb]in
[Ii]nclude
[Ll]ib
[Ll]ib64
[Ll]ocal
[Ss]cripts
pyvenv.cfg
.venv
pip-selfcheck.json
================================================
FILE: .pylintrc
================================================
[MASTER]
# Add files or directories to the blacklist. They should be base names, not
# paths.
ignore=CVS
# Pickle collected data for later comparisons.
persistent=yes
[MESSAGES CONTROL]
# Only show warnings with the listed confidence levels. Leave empty to show
# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED.
confidence=
# Disable the message, report, category or checker with the given id(s). You
# can either give multiple identifiers separated by comma (,) or put this
# option multiple times (only on the command line, not in the configuration
# file where it should appear only once). You can also use "--disable=all" to
# disable everything first and then reenable specific checks. For example, if
# you want to run only the similarities checker, you can use "--disable=all
# --enable=similarities". If you want to run only the classes checker, but have
# no Warning level messages displayed, use "--disable=all --enable=classes
# --disable=W".
disable=
# Pylint Defaults
raw-checker-failed,
bad-inline-option,
locally-disabled,
file-ignored,
suppressed-message,
useless-suppression,
deprecated-pragma,
use-symbolic-message-instead,
missing-docstring,
logging-fstring-interpolation,
invalid-name,
no-member, # Pylint doesn't currently support subclassing Enum and issues this warning everywhere
duplicate-code, # in order to support the domain/db/api duplication such as with InverterSpec
too-few-public-methods,
too-many-arguments,
redefined-outer-name, # false positive on pytest fixtures
# Enable the message, report, category or checker with the given id(s). You can
# either give multiple identifier separated by comma (,) or put this option
# multiple time (only on the command line, not in the configuration file where
# it should appear only once). See also the "--disable" option for examples.
enable=c-extension-no-member
[REPORTS]
# Python expression which should return a note less than 10 (10 is the highest
# note). You have access to the variables errors warning, statement which
# respectively contain the number of errors / warnings messages and the total
# number of statements analyzed. This is used by the global evaluation report
# (RP0004).
evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
# Template used to display messages. This is a python new-style format string
# used to format the message information. See doc for all details.
#msg-template=
# Set the output format. Available formats are text, parseable, colorized, json
# and msvs (visual studio). You can also give a reporter class, e.g.
# mypackage.mymodule.MyReporterClass.
output-format=text
# Tells whether to display a full report or only the messages.
reports=no
# Activate the evaluation score.
score=yes
[REFACTORING]
# Maximum number of nested blocks for function / method body
max-nested-blocks=5
# Complete name of functions that never returns. When checking for
# inconsistent-return-statements if a never returning function is called then
# it will be considered as an explicit return statement and no message will be
# printed.
never-returning-functions=sys.exit
[LOGGING]
# Format style used to check logging format string. `old` means using %
# formatting, while `new` is for `{}` formatting.
logging-format-style=old
# Logging modules to check that the string format arguments are in logging
# function parameter format.
logging-modules=logging
[SPELLING]
# Limits count of emitted suggestions for spelling mistakes.
max-spelling-suggestions=4
# Spelling dictionary name. Available dictionaries: none. To make it working
# install python-enchant package..
spelling-dict=
# List of comma separated words that should not be checked.
spelling-ignore-words=
# A path to a file that contains private dictionary; one word per line.
spelling-private-dict-file=
# Tells whether to store unknown words to indicated private dictionary in
# --spelling-private-dict-file option instead of raising a message.
spelling-store-unknown-words=no
[MISCELLANEOUS]
# List of note tags to take in consideration, separated by a comma.
notes=FIXME,
XXX,
TODO
[TYPECHECK]
# List of decorators that produce context managers, such as
# contextlib.contextmanager. Add to this list to register other decorators that
# produce valid context managers.
contextmanager-decorators=contextlib.contextmanager
# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=
# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
ignore-mixin-members=yes
# Tells whether to warn about missing members when the owner of the attribute
# is inferred to be None.
ignore-none=yes
# This flag controls whether pylint should warn about no-member and similar
# checks whenever an opaque object is returned when inferring. The inference
# can return multiple potential results while evaluating a Python object, but
# some branches might not be evaluated, which results in partial inference. In
# that case, it might be useful to still emit no-member and other checks for
# the rest of the inferred objects.
ignore-on-opaque-inference=yes
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
# qualified names.
ignored-classes=optparse.Values,thread._local,_thread._local
# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=
# Show a hint with possible names when a member name was not found. The aspect
# of finding the hint is based on edit distance.
missing-member-hint=yes
# The minimum edit distance a name should have in order to be considered a
# similar match for a missing member name.
missing-member-hint-distance=1
# The total number of similar names that should be taken in consideration when
# showing a hint for a missing member.
missing-member-max-choices=1
[VARIABLES]
# List of additional names supposed to be defined in builtins. Remember that
# you should avoid defining new builtins when possible.
additional-builtins=
# Tells whether unused global variables should be treated as a violation.
allow-global-unused-variables=yes
# List of strings which can identify a callback function by name. A callback
# name must start or end with one of those strings.
callbacks=cb_,
_cb
# A regular expression matching the name of dummy variables (i.e. expected to
# not be used).
dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_
# Argument names that match this expression will be ignored. Default to name
# with leading underscore.
ignored-argument-names=_.*|^ignored_|^unused_
# Tells whether we should check for unused import in __init__ files.
init-import=no
# List of qualified module names which can have objects that can redefine
# builtins.
redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io
[FORMAT]
# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.
expected-line-ending-format=
# Regexp for a line that is allowed to be longer than the limit.
ignore-long-lines=^\s*(# )?<?https?://\S+>?$
# Number of spaces of indent required inside a hanging or continued line.
indent-after-paren=4
# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
# tab).
indent-string=' '
# Maximum number of characters on a single line.
max-line-length=120
# Maximum number of lines in a module.
max-module-lines=1000
# List of optional constructs for which whitespace checking is disabled. `dict-
# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}.
# `trailing-comma` allows a space between comma and closing bracket: (a, ).
# `empty-line` allows space-only lines.
# no-space-check=trailing-comma,
# dict-separator
# Allow the body of a class to be on the same line as the declaration if body
# contains single statement.
single-line-class-stmt=no
# Allow the body of an if to be on the same line as the test if there is no
# else.
single-line-if-stmt=no
[SIMILARITIES]
# Ignore comments when computing similarities.
ignore-comments=yes
# Ignore docstrings when computing similarities.
ignore-docstrings=yes
# Ignore imports when computing similarities.
ignore-imports=no
# Minimum lines number of a similarity.
min-similarity-lines=4
[BASIC]
# Naming style matching correct argument names.
argument-naming-style=snake_case
# Regular expression matching correct argument names. Overrides argument-
# naming-style.
#argument-rgx=
# Naming style matching correct attribute names.
attr-naming-style=snake_case
# Regular expression matching correct attribute names. Overrides attr-naming-
# style.
#attr-rgx=
# Bad variable names which should always be refused, separated by a comma.
bad-names=foo,
bar,
baz,
toto,
tutu,
tata
# Naming style matching correct class attribute names.
class-attribute-naming-style=any
# Regular expression matching correct class attribute names. Overrides class-
# attribute-naming-style.
#class-attribute-rgx=
# Naming style matching correct class names.
class-naming-style=PascalCase
# Regular expression matching correct class names. Overrides class-naming-
# style.
#class-rgx=
# Naming style matching correct constant names.
const-naming-style=UPPER_CASE
# Regular expression matching correct constant names. Overrides const-naming-
# style.
#const-rgx=
# Minimum line length for functions/classes that require docstrings, shorter
# ones are exempt.
docstring-min-length=-1
# Naming style matching correct function names.
function-naming-style=snake_case
# Regular expression matching correct function names. Overrides function-
# naming-style.
#function-rgx=
# Good variable names which should always be accepted, separated by a comma.
good-names=i,
j,
k,
ex,
Run,
_
# Include a hint for the correct naming format with invalid-name.
include-naming-hint=no
# Naming style matching correct inline iteration names.
inlinevar-naming-style=any
# Regular expression matching correct inline iteration names. Overrides
# inlinevar-naming-style.
#inlinevar-rgx=
# Naming style matching correct method names.
method-naming-style=snake_case
# Regular expression matching correct method names. Overrides method-naming-
# style.
#method-rgx=
# Naming style matching correct module names.
module-naming-style=snake_case
# Regular expression matching correct module names. Overrides module-naming-
# style.
#module-rgx=
# Colon-delimited sets of names that determine each other's naming style when
# the name regexes allow several styles.
name-group=
# Regular expression which should only match function or class names that do
# not require a docstring.
no-docstring-rgx=^_
# List of decorators that produce properties, such as abc.abstractproperty. Add
# to this list to register other decorators that produce valid properties.
# These decorators are taken in consideration only for invalid-name.
property-classes=abc.abstractproperty
# Naming style matching correct variable names.
variable-naming-style=snake_case
# Regular expression matching correct variable names. Overrides variable-
# naming-style.
#variable-rgx=
[STRING]
# This flag controls whether the implicit-str-concat-in-sequence should
# generate a warning on implicit string concatenation in sequences defined over
# several lines.
check-str-concat-over-line-jumps=no
[IMPORTS]
# Allow wildcard imports from modules that define __all__.
allow-wildcard-with-all=no
# Analyse import fallback blocks. This can be used to support both Python 2 and
# 3 compatible code, which means that the block might have code that exists
# only in one or another interpreter, leading to false positives when analysed.
analyse-fallback-blocks=no
# Deprecated modules which should not be used, separated by a comma.
deprecated-modules=optparse,tkinter.tix
# Create a graph of external dependencies in the given file (report RP0402 must
# not be disabled).
ext-import-graph=
# Create a graph of every (i.e. internal and external) dependencies in the
# given file (report RP0402 must not be disabled).
import-graph=
# Create a graph of internal dependencies in the given file (report RP0402 must
# not be disabled).
int-import-graph=
# Force import order to recognize a module as part of the standard
# compatibility libraries.
known-standard-library=
# Force import order to recognize a module as part of a third party library.
known-third-party=enchant
[CLASSES]
# List of method names used to declare (i.e. assign) instance attributes.
defining-attr-methods=__init__,
__new__,
setUp
# List of member names, which should be excluded from the protected access
# warning.
exclude-protected=_asdict,
_fields,
_replace,
_source,
_make
# List of valid names for the first argument in a class method.
valid-classmethod-first-arg=cls
# List of valid names for the first argument in a metaclass class method.
valid-metaclass-classmethod-first-arg=cls
[DESIGN]
# Maximum number of arguments for function / method.
max-args=5
# Maximum number of attributes for a class (see R0902).
max-attributes=7
# Maximum number of boolean expressions in an if statement.
max-bool-expr=5
# Maximum number of branch for function / method body.
max-branches=12
# Maximum number of locals for function / method body.
max-locals=15
# Maximum number of parents for a class (see R0901).
max-parents=7
# Maximum number of public methods for a class (see R0904).
max-public-methods=20
# Maximum number of return / yield for function / method body.
max-returns=6
# Maximum number of statements in function / method body.
max-statements=50
# Minimum number of public methods for a class (see R0903).
min-public-methods=2
[EXCEPTIONS]
# Exceptions that will emit a warning when being caught. Defaults to
# "BaseException, Exception".
overgeneral-exceptions=builtins.BaseException,
builtins.Exception
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2020 SolarEdge Technologies Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: Makefile
================================================
all: check
help:
@echo "Available targets:"
@echo "- help Show this help message"
@echo "- bootstrap Installs required dependencies"
@echo "- check Runs static code analyzers"
@echo "- test Run unit tests"
@echo "- coverage Check test coverage"
bootstrap:
poetry --version || curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python
poetry install
check: pylint mypy
test:
poetry run pytest --log-cli-level=4 -vv tests
coverage:
poetry run pytest --cov-append --cov-report xml:./coverage.xml --cov-report html --cov-report term --cov=pyctuator --log-cli-level=4 -vv tests
pylint:
poetry run pylint --exit-zero pyctuator tests
mypy:
poetry run pip install types-redis
poetry run pip install types-requests
poetry run mypy -p pyctuator -p tests
package:
poetry build
clean:
find . -type d -name __pycache__ -print | xargs rm -rf
find . -type d -name .pytest_cache -print | xargs rm -rf
rm -rf dist htmlcov .mypy_cache
.PHONY: all help bootstrap check test coverage pylint mypy package clean
================================================
FILE: README.md
================================================
[](https://pypi.org/project/pyctuator/)
[](https://github.com/SolarEdgeTech/pyctuator/)
[](https://codecov.io/gh/SolarEdgeTech/pyctuator)
# Pyctuator
Monitor Python web apps using
[Spring Boot Admin](https://github.com/codecentric/spring-boot-admin).
Pyctuator supports **[Flask](https://palletsprojects.com/p/flask/)**, **[FastAPI](https://fastapi.tiangolo.com/)**, **[aiohttp](https://docs.aiohttp.org/)** and **[Tornado](https://www.tornadoweb.org/)**. **Django** support is planned as well.
The following video shows a FastAPI web app being monitored and controled using Spring Boot Admin.
The complete example can be found in [Advanced example](examples/Advanced/README.md).
## Requirements
Python 3.9+
Pyctuator has zero hard dependencies.
## Installing
Install Pyctuator using pip: `pip3 install pyctuator`
## Why?
Many Java shops use Spring Boot as their main web framework for developing
microservices.
These organizations often use Spring Actuator together with Spring Boot Admin
to monitor their microservices' status, gain access to applications'
state and configuration, manipulate log levels, etc.
While Spring Boot is suitable for many use-cases, it is very common for organizations
to also have a couple of Python microservices, as Python is often more suitable for
some types of applications. The most common examples are Data Science and Machine Learning
applications.
Setting up a proper monitoring tool for these microservices is a complex task, and might
not be justified for just a few Python microservices in a sea of Java microservices.
This is where Pyctuator comes in. It allows you to easily integrate your Python
microservices into your existing Spring Boot Admin deployment.
## Main Features
Pyctuator is a partial Python implementation of the
[Spring Actuator API](https://docs.spring.io/spring-boot/docs/2.1.8.RELEASE/actuator-api/html/) .
It currently supports the following Actuator features:
* **Application details**
* **Metrics**
* Memory usage
* Disk usage
* Custom metrics
* **Health monitors**
* Built in MySQL health monitor
* Built in Redis health monitor
* Custom health monitors
* **Environment**
* **Loggers** - Easily change log levels during runtime
* **Log file** - Tail the application's log file
* **Thread dump** - See which threads are running
* **HTTP traces** - Tail recent HTTP requests, including status codes and latency
## Quickstart
The examples below show a minimal integration of **FastAPI**, **Flask** and **aiohttp** applications with **Pyctuator**.
After installing Flask/FastAPI/aiohttp and Pyctuator, start by launching a local Spring Boot Admin instance:
```sh
docker run --rm -p 8080:8080 --add-host=host.docker.internal:host-gateway michayaak/spring-boot-admin:2.2.3-1
```
Then go to `http://localhost:8080` to get to the web UI.
### Flask
The following example is complete and should run as is.
```python
from flask import Flask
from pyctuator.pyctuator import Pyctuator
app_name = "Flask App with Pyctuator"
app = Flask(app_name)
@app.route("/")
def hello():
return "Hello World!"
Pyctuator(
app,
app_name,
app_url="http://host.docker.internal:5000",
pyctuator_endpoint_url="http://host.docker.internal:5000/pyctuator",
registration_url="http://localhost:8080/instances"
)
app.run(debug=False, port=5000)
```
The application will automatically register with Spring Boot Admin upon start up.
Log in to the Spring Boot Admin UI at `http://localhost:8080` to interact with the application.
### FastAPI
The following example is complete and should run as is.
```python
from fastapi import FastAPI
from uvicorn import Server
from uvicorn.config import Config
from pyctuator.pyctuator import Pyctuator
app_name = "FastAPI App with Pyctuator"
app = FastAPI(title=app_name)
@app.get("/")
def hello():
return "Hello World!"
Pyctuator(
app,
"FastAPI Pyctuator",
app_url="http://host.docker.internal:8000",
pyctuator_endpoint_url="http://host.docker.internal:8000/pyctuator",
registration_url="http://localhost:8080/instances"
)
Server(config=(Config(app=app, loop="asyncio"))).run()
```
The application will automatically register with Spring Boot Admin upon start up.
Log in to the Spring Boot Admin UI at `http://localhost:8080` to interact with the application.
### aiohttp
The following example is complete and should run as is.
```python
from aiohttp import web
from pyctuator.pyctuator import Pyctuator
app = web.Application()
routes = web.RouteTableDef()
@routes.get("/")
def hello():
return web.Response(text="Hello World!")
Pyctuator(
app,
"aiohttp Pyctuator",
app_url="http://host.docker.internal:8888",
pyctuator_endpoint_url="http://host.docker.internal:8888/pyctuator",
registration_url="http://localhost:8080/instances"
)
app.add_routes(routes)
web.run_app(app, port=8888)
```
The application will automatically register with Spring Boot Admin upon start up.
Log in to the Spring Boot Admin UI at `http://localhost:8080` to interact with the application.
### Registration Notes
When registering a service in Spring Boot Admin, note that:
* **Docker** - If the Spring Boot Admin is running in a container while the managed service is running in the docker-host directly, the `app_url` and `pyctuator_endpoint_url` should use `host.docker.internal` as the url's host so Spring Boot Admin will be able to connect to the monitored service.
* **Http Traces** - In order for the "Http Traces" tab to be able to hide requests sent by Spring Boot Admin to the Pyctuator endpoint, `pyctuator_endpoint_url` must be using the same host and port as `app_url`.
* **HTTPS** - If Pyctuator is to be registered with Spring Boot Admin using HTTPS and the default SSL context is inappropriate, you can provide your own `ssl.SSLContext` using the `ssl_context` optional parameter of the `Pyctuator` constructor.
* **Insecure HTTPS** - If Spring Boot Admin is using HTTPS with self-signed certificate, set the `PYCTUATOR_REGISTRATION_NO_CERT` environment variable so Pyctuator will disable certificate validation when registering (and deregistering).
## Advanced Configuration
The following sections are intended for advanced users who want to configure advanced Pyctuator features.
### Application Info
While Pyctuator only needs to know the application's name, we recommend that applications monitored by Spring
Boot Admin will show additional build and git details.
This becomes handy when scaling out a service to multiple instances by showing the version of each instance.
To do so, you can provide additional build and git info using methods of the Pyctuator object:
```python
pyctuator = Pyctuator(...) # arguments removed for brevity
pyctuator.set_build_info(
name="app",
version="1.3.1",
time=datetime.fromisoformat("2019-12-21T10:09:54.876091"),
)
pyctuator.set_git_info(
commit="7d4fef3",
time=datetime.fromisoformat("2019-12-24T14:18:32.123432"),
branch="origin/master",
)
```
Once you configure build and git info, you should see them in the Details tab of Spring Boot Admin:
### Additional Application Info
In addition to adding build and git info, Pyctuator allows adding arbitrary application details to the "Info" section in SBA.
This is done by initializing the `additional_app_info` parameter with an arbitrary dictionary.
For example, you can provide links to your application's metrics:
```python
Pyctuator(
app,
"Flask Pyctuator",
app_url=f"http://172.18.0.1:5000",
pyctuator_endpoint_url=f"http://172.18.0.1:5000/pyctuator",
registration_url=f"http://localhost:8080/instances",
app_description="Demonstrate Spring Boot Admin Integration with Flask",
additional_app_info=dict(
serviceLinks=dict(
metrics="http://xyz/service/metrics"
),
podLinks=dict(
metrics=["http://xyz/pod/metrics/memory", "http://xyz/pod/metrics/cpu"]
)
)
)
```
This will result with the following Info page in SBA:
### DB Health
For services that use SQL database via SQLAlchemy, Pyctuator can easily monitor and expose the connection's health
using the DbHealthProvider class as demonstrated below:
```python
engine = create_engine("mysql+pymysql://root:root@localhost:3306")
pyctuator = Pyctuator(...) # arguments removed for brevity
pyctuator.register_health_provider(DbHealthProvider(engine))
```
Once you configure the health provider, you should see DB health info in the Details tab of Spring Boot Admin:
### Redis health
If your service is using Redis, Pyctuator can monitor the connection to Redis by simply initializing a `RedisHealthProvider`:
```python
r = redis.Redis()
pyctuator = Pyctuator(...) # arguments removed for brevity
pyctuator.register_health_provider(RedisHealthProvider(r))
```
### Custom Environment
Out of the box, Pyctuator exposes Python's environment variables to Spring Boot Admin.
In addition, an application may register an environment provider to provide additional configuration that should be exposed via Spring Boot Admin.
When the environment provider is called it should return a dictionary describing the environment. The returned dictionary is exposed to Spring Boot Admin.
Since Spring Boot Admin doesn't support hierarchical environment (only a flat key/value mapping), the provided environment is flattened as dot-delimited keys.
Pyctuator tries to hide secrets from being exposed to Spring Boot Admin by replacing the values of "suspicious" keys with ***.
Suspicious keys are keys that contain the words "secret", "password" and some forms of "key".
For example, if an application's configuration looks like this:
```python
config = {
"a": "s1",
"b": {
"secret": "ha ha",
"c": 625,
},
"d": {
"e": True,
"f": "hello",
"g": {
"h": 123,
"i": "abcde"
}
}
}
```
An environment provider can be registered like so:
```python
pyctuator.register_environment_provider("config", lambda: config)
```
### Filesystem and Memory Metrics
Pyctuator can provide filesystem and memory metrics.
To enable these metrics, install [psutil](https://github.com/giampaolo/psutil)
Note that the `psutil` dependency is **optional** and is only required if you want to enable filesystem and memory monitoring.
### Loggers
Pyctuator leverages Python's builtin `logging` framework and allows controlling log levels at runtime.
Note that in order to control uvicorn's log level, you need to provide a logger object when instantiating it. For example:
```python
myFastAPIServer = Server(
config=Config(
logger=logging.getLogger("uvi"),
app=app,
loop="asyncio"
)
)
```
### Spring Boot Admin Using Basic Authentication
Pyctuator supports registration with Spring Boot Admin that requires basic authentications. The credentials are provided when initializing the Pyctuator instance as follows:
```python
# NOTE: Never include secrets in your code !!!
auth = BasicAuth(os.getenv("sba-username"), os.getenv("sba-password"))
Pyctuator(
app,
"Flask Pyctuator",
app_url="http://localhost:5000",
pyctuator_endpoint_url=f"http://localhost:5000/pyctuator",
registration_url=f"http://spring-boot-admin:8080/instances",
registration_auth=auth,
)
```
### Protecting Pyctuator with authentication
Since there are numerous standard approaches to protect an API, Pyctuator doesn't explicitly support any of them. Instead, Pyctuator allows to customize its integration with the web-framework.
See the example in [fastapi_with_authentication_example_app.py](examples/FastAPI/fastapi_with_authentication_example_app.py).
## Full blown examples
The `examples` folder contains full blown Python projects that are built using [Poetry](https://python-poetry.org/).
To run these examples, you'll need to have Spring Boot Admin running in a local docker container. A Spring Boot Admin Docker image is available [here](https://hub.docker.com/r/michayaak/spring-boot-admin).
Unless the example includes a docker-compose file, you'll need to start Spring Boot Admin using docker directly:
```sh
docker run --rm -p 8080:8080 --add-host=host.docker.internal:host-gateway michayaak/spring-boot-admin:2.2.3-1
```
(the docker image's tag represents the version of Spring Boot Admin, so if you need to use version `2.0.0`, use `michayaak/spring-boot-admin:2.0.0` instead, note it accepts connections on port 8082).
The examples include
* [FastAPI Example](examples/FastAPI/README.md) - demonstrates integrating Pyctuator with the FastAPI web framework.
* [Flask Example](examples/Flask/README.md) - demonstrates integrating Pyctuator with the Flask web framework.
* [Advanced Example](examples/Advanced/README.md) - demonstrates configuring and using all the advanced features of Pyctuator.
## Contributing
To set up a development environment, make sure you have Python 3.9 or newer installed, and run `make bootstrap`.
Use `make check` to run static analysis tools.
Use `make test` to run tests.
================================================
FILE: examples/Advanced/README.md
================================================
# Advanced Example
This example demonstrates using the optional features and customizations Pyctuator is offering.
## Running the example
Before running this example, you'll need SBA (Spring Boot Admin), MySQL and Redis running on the same machine the example application will be running.
It is recommended to start these services using the `docker-compose.yml` part of this example, from the `examples/Advanced` directory perform:
```shell
docker-compose --project-name example --file docker-compose.yml up --detach --force-recreate
```
Next, from the `examples/Advanced` directory, run the example application using poetry as follows:
```shell
poetry install
poetry run python advanced_example_app.py
```
# Using the example
The example application, available from http://localhost:8000 exposes example APIs for accessing the DB and Redis:
* http://localhost/db/version - returns the DB's version
* http://localhost:8000/redis/a-key - returns the value of the `a-key` key in redis
Connect to Spring Boot Admin using http://localhost:8082.
## Insights Details
1. Monitor disk space (requires [psutil](https://pypi.org/project/psutil/)):
2. Monitor connection to the DB (requies [sqlalchemy](https://pypi.org/project/SQLAlchemy/) and drivers specific to the DB being used)
3. Monitor Redis client (requires [redis](https://pypi.org/project/redis/))
4. Show build details
5. Show Git details
## Insights Metrics
If [psutil](https://pypi.org/project/psutil/) is installed, Pyctuator provides various process metrics in the "Metrics" tab:
## Insights Environment
Pyctuator automatically exposes all environment variables, after scrubbing secrets, via the "Environment" tab under "systemEnvironment":
Additionally, Pyctuator can be configured to expose application-specific configuration via SBA (after scrubbing commonly identified secrets):
Note that SBA only support flattened configuration hierarchy, which is automatically handled by Pyctuator.
# Secret scrubbing
Pyctuator is using a "secret scrubber" for scrubbing/masking secrets from environment-variables and config-entries that are being reported to SBA.
The default secret scrubber is taking care fore masking values of keys that are expected to keep secrets.
Additionally, the default scrubber is masking credentials that are included in URLs.
It is possible to override the default scrubber by calling `set_secret_scrubber` providing it a mapping function that will hide/mask the desired keys.
Note that the pattern used by the built in `SecretScrubber` can be replaced.
For example:
```python
pyctuator = Pyctuator(...) # arguments removed for brevity
secret_scrubber = SecretScrubber(keys_to_scrub=re.compile("^ABC$|^xyz$", re.IGNORECASE)).scrub_secrets
pyctuator.set_secret_scrubber(secret_scrubber)
```
# Further customization
Using Pyctuator, it is possible to have SBA monitor application-specific health aspects using custom health-providers.
Health status may include multiple checks and may also include details on failures or the apps health.
To demonstrate this, the example application exposes additional API for setting the health, http://localhost:8000/health - posting a JSON formatted `pyctuator.health_provider.HealthDetails` to set the current health status.
For example, the call bellow will make the application report its down.
```shell
curl -X POST localhost:8000/health -d '{"status": "DOWN", "details": {"backend_connectivity": "Down", "available_resources": 41}}'
```
================================================
FILE: examples/Advanced/advanced_example_app.py
================================================
import datetime
import logging
import random
from dataclasses import dataclass
from typing import Any, Dict, List
from starlette.requests import Request
import redis
from fastapi import FastAPI
from sqlalchemy.engine import Engine, create_engine
from uvicorn import Server
from uvicorn.config import Config
from pyctuator.health.db_health_provider import DbHealthProvider
from pyctuator.health.health_provider import HealthProvider, HealthStatus, Status, HealthDetails
from pyctuator.health.redis_health_provider import RedisHealthProvider
from pyctuator.pyctuator import Pyctuator
# ----------------------------------------------------------------------------------------------------------------------
# The `app_config` variable below holds all the settings of the application which in addition for being used by the
# application, is also exposed to SBA (after scrubbing secrets) through Pyctuator.
# ----------------------------------------------------------------------------------------------------------------------
app_config = {
"app": {
"name": "Advanced Example Server",
"version": "1.3.1",
"build_time": datetime.datetime.fromisoformat("2019-12-21T10:09:54.876091"),
"description": "Demonstrate Spring Boot Admin Integration with FastAPI",
"git": {
"commit": "7d4fef3",
"time": datetime.datetime.fromisoformat("2019-12-24T14:18:32.123432"),
"branch": "master",
},
# the URL to use when accessing the application
"public_endpoint": f"http://host.docker.internal:8000",
},
"mysql": {
"host": "localhost:3306",
"user": "root",
# NOTE: don't put secrets in code, get them from env! (although Pyctuator will scrub this)
"password": "root",
},
"monitoring": {
# Because SBA runs in a container, this is the URL of the app/pyctuator as seen from the SBA container
"pyctuator_endpoint": f"http://host.docker.internal:8000/pyctuator",
# Spring Boot Admin registration URL
"sba_registration_endpoint": f"http://localhost:8080/instances",
}
}
def get_conf(key: str) -> Any:
def recursive_get(child_conf: Dict, key_parts: List[str]) -> Any:
if len(key_parts) == 1:
return child_conf[key_parts[0]]
return recursive_get(child_conf[key_parts[0]], key_parts[1:])
return recursive_get(app_config, key.split("."))
# ----------------------------------------------------------------------------------------------------------------------
# A FastAPI application is initialized providing some test API
# ----------------------------------------------------------------------------------------------------------------------
logger = logging.getLogger("ExampleApp")
# Initialize a connection to the DB which the app is using
db_engine: Engine = create_engine(
"mysql+pymysql://{user}:{password}@{host}".format(
user=get_conf("mysql.user"),
password=get_conf("mysql.password"),
host=get_conf("mysql.host"),
),
echo=True)
# Initialize a redis client for the app to use
redis_client = redis.Redis()
app = FastAPI(
title=get_conf("app.name"),
description=get_conf("app.description"),
docs_url="/api",
)
@dataclass
class AppSpecificHealthDetails(HealthDetails):
backend_connectivity: str
available_resources: int
app_specific_health = HealthStatus(
status=Status.UP,
details=AppSpecificHealthDetails(backend_connectivity="Connected", available_resources=35)
)
@app.get("/")
def hello():
logger.debug(f"{datetime.datetime.now()} - {str(random.randint(0, 100))}")
print("Printing to STDOUT")
return "Hello World!"
@app.get("/redis/{key}")
def redis_get(key: str) -> Any:
return redis_client.get(key)
@app.get("/db/version")
def db_version() -> Any:
return db_engine.execute("SELECT version()").next()[0]
@app.post("/health")
def health_up(request: Request, health: Dict) -> None: # health should be of type HealthStatus
global app_specific_health
app_specific_health = HealthStatus(Status[health["status"]], details=health["details"])
# ----------------------------------------------------------------------------------------------------------------------
# Initialize Pyctuator with the SBA endpoint and all the extensions
# ----------------------------------------------------------------------------------------------------------------------
pyctuator = Pyctuator(
app,
get_conf("app.name"),
get_conf("app.public_endpoint"),
get_conf("monitoring.pyctuator_endpoint"),
get_conf("monitoring.sba_registration_endpoint"),
app_description=app.description,
)
# Provide app's build info
pyctuator.set_build_info(
name=get_conf("app.name"),
version=get_conf("app.version"),
time=get_conf("app.build_time"),
)
# Provide git commit info
pyctuator.set_git_info(
commit=get_conf("app.git.commit"),
time=get_conf("app.git.time"),
branch=get_conf("app.git.branch"),
)
# Expose app's config via the Pyctuator API for SBA to show the scrubbed version in the UI
pyctuator.register_environment_provider("conf", lambda: app_config)
# Add health check for the DB connection
pyctuator.register_health_provider(DbHealthProvider(db_engine))
# Add health check for the Redis client
pyctuator.register_health_provider(RedisHealthProvider(redis_client))
# Register a custom health provider that reflects the contents of `healthy` and `health_details` to SBA
class CustomHealthProvider(HealthProvider):
def is_supported(self) -> bool:
return True
def get_name(self) -> str:
return "app-specific-health"
def get_health(self) -> HealthStatus:
return app_specific_health
pyctuator.register_health_provider(CustomHealthProvider())
# ----------------------------------------------------------------------------------------------------------------------
# The server is started after Pyctuator is created to allow Pyctuator to fully integrate with FastAPI
# ----------------------------------------------------------------------------------------------------------------------
server = Server(config=(Config(app=app, loop="asyncio", host="0.0.0.0")))
server.run()
================================================
FILE: examples/Advanced/docker-compose.yml
================================================
version: '3'
services:
mysql:
image: mysql:5.7.22
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=root
redis:
image: redis:alpine
ports:
- 6379:6379
sba:
image: michayaak/spring-boot-admin:2.2.2
ports:
- 8080:8080
================================================
FILE: examples/Advanced/pyproject.toml
================================================
[tool.poetry]
name = "fastapi-pyctuator-example"
version = "1.0.0"
description = "Example of using Pyctuator"
authors = [
"Luke Skywalker <Luke@starwars.com>",
]
[tool.poetry.dependencies]
python = "^3.9"
psutil = { version = "^5.6" }
fastapi = { version = "^0.65.2" }
uvicorn = { version = "^0.11.7" }
pyctuator = { version = "^1.2.0" }
sqlalchemy = { version = "^1.3" }
PyMySQL = { version = "^0.9.3" }
cryptography = { version = "^2.8" }
redis = { version = "^3.3" }
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
================================================
FILE: examples/FastAPI/README.md
================================================
# FastAPI example
This example demonstrates the integration with the [FastAPI](https://fastapi.tiangolo.com/) web-framework.
## Running the example
1. Start an instance of SBA (Spring Boot Admin):
```sh
docker run --rm -p 8080:8080 --add-host=host.docker.internal:host-gateway michayaak/spring-boot-admin:2.2.3-1
```
2. Once Spring Boot Admin is running, you can run the examples as follow:
```sh
cd examples/FastAPI
poetry install
poetry run python -m fastapi_example_app
```
## Running an example where pyctuator requires authentication
In order to protect the Pyctuator endpoint, a customizer is used to make the required configuration changes to the API router.
In addition, the credentials need to be included in the registration request sent to SBA in order for it it could authenticate when querying the pyctuator API.
================================================
FILE: examples/FastAPI/fastapi_example_app.py
================================================
import datetime
import logging
import random
from fastapi import FastAPI
from uvicorn import Server
from uvicorn.config import Config
from pyctuator.pyctuator import Pyctuator
my_logger = logging.getLogger("example")
app = FastAPI(
title="FastAPI Example Server",
description="Demonstrate Spring Boot Admin Integration with FastAPI",
docs_url="/api",
)
@app.get("/")
def read_root():
my_logger.debug(f"{datetime.datetime.now()} - {str(random.randint(0, 100))}")
print("Printing to STDOUT")
return "Hello World!"
example_app_address = "host.docker.internal"
example_sba_address = "localhost"
pyctuator = Pyctuator(
app,
"Example FastAPI",
app_url=f"http://{example_app_address}:8000",
pyctuator_endpoint_url=f"http://{example_app_address}:8000/pyctuator",
registration_url=f"http://{example_sba_address}:8080/instances",
app_description=app.description,
)
server = Server(config=(Config(
app=app,
loop="asyncio",
host="0.0.0.0",
log_level=logging.WARNING,
)))
server.run()
================================================
FILE: examples/FastAPI/fastapi_with_authentication_example_app.py
================================================
import datetime
import logging
import random
import secrets
from fastapi import FastAPI, Depends, APIRouter, HTTPException
from fastapi.security import HTTPBasicCredentials, HTTPBasic
from starlette import status
from uvicorn import Server
from uvicorn.config import Config
from pyctuator.pyctuator import Pyctuator
my_logger = logging.getLogger("example")
class SimplisticBasicAuth:
def __init__(self, username: str, password: str):
"""
Initializes a simplistic basic-auth FastAPI dependency with hardcoded username and password -
don't do this at home!
"""
self.username = username
self.password = password
def __call__(self, credentials: HTTPBasicCredentials = Depends(HTTPBasic(realm="pyctuator"))):
correct_username = secrets.compare_digest(credentials.username, self.username)
correct_password = secrets.compare_digest(credentials.password, self.password) if self.password else True
if not (correct_username and correct_password):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Incorrect username or password",
headers={"WWW-Authenticate": "Basic"},
)
username = "u1"
password = "p2"
security = SimplisticBasicAuth(username, password)
app = FastAPI(
title="FastAPI Example Server",
description="Demonstrate Spring Boot Admin Integration with FastAPI",
docs_url="/api",
)
def add_authentication_to_pyctuator(router: APIRouter) -> None:
router.dependencies = [Depends(security)]
@app.get("/")
def read_root(credentials: HTTPBasicCredentials = Depends(security)):
my_logger.debug(f"{datetime.datetime.now()} - {str(random.randint(0, 100))}")
return {"username": credentials.username, "password": credentials.password}
example_app_address = "172.18.0.1"
example_sba_address = "localhost"
pyctuator = Pyctuator(
app,
"Example FastAPI",
app_url=f"http://{example_app_address}:8000",
pyctuator_endpoint_url=f"http://{example_app_address}:8000/pyctuator",
registration_url=f"http://{example_sba_address}:8080/instances",
app_description=app.description,
customizer=add_authentication_to_pyctuator, # Customize Pyctuator's API router to require authentication
metadata={
"user.name": username, # Include the credentials in the registration request sent to SBA
"user.password": password,
}
)
# Keep the console clear - configure uvicorn (FastAPI's WSGI web app) not to log the detail of every incoming request
uvicorn_logger = logging.getLogger("uvicorn")
uvicorn_logger.setLevel(logging.WARNING)
server = Server(config=(Config(
app=app,
loop="asyncio",
host="0.0.0.0",
logger=uvicorn_logger,
)))
server.run()
================================================
FILE: examples/FastAPI/pyproject.toml
================================================
[tool.poetry]
name = "fastapi-pyctuator-example"
version = "1.0.0"
description = "Example of using Pyctuator"
authors = [
"Luke Skywalker <Luke@starwars.com>",
]
[tool.poetry.dependencies]
python = "^3.9"
psutil = { version = "^5.6" }
fastapi = { version = "^0.82.0" }
uvicorn = { version = "^0.18.2" }
pyctuator = { version = "^1.2.0" }
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
================================================
FILE: examples/Flask/README.md
================================================
# Flask example
This example demonstrates the integration with the [Flask](https://flask.palletsprojects.com/) web-framework.
## Running the example
1. Start an instance of SBA (Spring Boot Admin):
```sh
docker run --rm -p 8080:8080 --add-host=host.docker.internal:host-gateway michayaak/spring-boot-admin:2.2.3-1
```
2. Once Spring Boot Admin is running, you can run the examples as follow:
```sh
cd examples/Flask
poetry install
poetry run python -m flask_example_app
```
## Notes
* Note that when Flask debugging is enabled, Pyctuator and Flask are initialized twice because Flask reloads the script. This causes Pyctuator to register twice thus the `startup` time alternates between the time these instances started.
```Python
app.run(port=5000, host="0.0.0.0", debug=True)
```
================================================
FILE: examples/Flask/flask_example_app.py
================================================
import datetime
import logging
import random
from flask import Flask
from pyctuator.pyctuator import Pyctuator
# Keep the console clear - configure werkzeug (flask's WSGI web app) not to log the detail of every incoming request
logging.getLogger("werkzeug").setLevel(logging.WARNING)
my_logger = logging.getLogger("example")
app = Flask("Flask Example Server")
@app.route("/")
def hello():
my_logger.debug(f"{datetime.datetime.now()} - {str(random.randint(0, 100))}")
print("Printing to STDOUT")
return "Hello World!"
example_app_address = "host.docker.internal"
example_sba_address = "localhost"
Pyctuator(
app,
"Flask Pyctuator",
app_url=f"http://{example_app_address}:5000",
pyctuator_endpoint_url=f"http://{example_app_address}:5000/pyctuator",
registration_url=f"http://{example_sba_address}:8080/instances",
app_description="Demonstrate Spring Boot Admin Integration with Flask",
)
app.run(port=5000, host="0.0.0.0")
================================================
FILE: examples/Flask/pyproject.toml
================================================
[tool.poetry]
name = "flask-pyctuator-example"
version = "1.0.0"
description = "Example of using Pyctuator"
authors = [
"Luke Skywalker <Luke@starwars.com>",
]
[tool.poetry.dependencies]
python = "^3.9"
psutil = { version = "^5.6" }
flask = { version = "^2.2.2" }
pyctuator = { version = "^1.2.0" }
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
================================================
FILE: examples/__init__.py
================================================
================================================
FILE: examples/aiohttp/README.md
================================================
# aiohttp example
This example demonstrates the integration with the [aiohttp](https://docs.aiohttp.org).
## Running the example
1. Start an instance of SBA (Spring Boot Admin):
```sh
docker run --rm -p 8080:8080 --add-host=host.docker.internal:host-gateway michayaak/spring-boot-admin:2.2.3-1
```
2. Once Spring Boot Admin is running, you can run the examples as follow:
```sh
cd examples/aiohttp
poetry install
poetry run python -m aiohttp_example_app
```
================================================
FILE: examples/aiohttp/aiohttp_example_app.py
================================================
import datetime
import logging
import random
from aiohttp import web
from pyctuator.pyctuator import Pyctuator
my_logger = logging.getLogger("example")
app = web.Application()
routes = web.RouteTableDef()
@routes.get('/')
def home(request: web.Request) -> web.Response:
my_logger.debug(f"{datetime.datetime.now()} - {str(random.randint(0, 100))}")
print("Printing to STDOUT")
return web.Response(text="Hello World!")
example_app_address = "host.docker.internal"
example_sba_address = "localhost"
pyctuator = Pyctuator(
app,
"Example aiohttp",
app_url=f"http://{example_app_address}:8888",
pyctuator_endpoint_url=f"http://{example_app_address}:8888/pyctuator",
registration_url=f"http://{example_sba_address}:8080/instances",
app_description="Demonstrate Spring Boot Admin Integration with aiohttp",
)
app.add_routes(routes)
web.run_app(app, port=8888)
================================================
FILE: examples/aiohttp/pyproject.toml
================================================
[tool.poetry]
name = "aiohttp-pyctuator-example"
version = "1.0.0"
description = "Example of using Pyctuator"
authors = [
"Luke Skywalker <Luke@starwars.com>",
]
[tool.poetry.dependencies]
python = "^3.9"
psutil = { version = "^5.6" }
aiohttp = { version = "^3.6.2" }
pyctuator = { version = "^1.2.0" }
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
================================================
FILE: examples/tornado/README.md
================================================
# Tornado example
This example demonstrates the integration with the [Tornado](https://www.tornadoweb.org/).
## Running the example
1. Start an instance of SBA (Spring Boot Admin):
```sh
docker run --rm -p 8080:8080 --add-host=host.docker.internal:host-gateway michayaak/spring-boot-admin:2.2.3-1
```
2. Once Spring Boot Admin is running, you can run the examples as follow:
```sh
cd examples/tornado
poetry install
poetry run python -m tornado_example_app
```
================================================
FILE: examples/tornado/pyproject.toml
================================================
[tool.poetry]
name = "tornado-pyctuator-example"
version = "1.0.0"
description = "Example of using Pyctuator"
authors = [
"Desmond Stonie <aneasystone@gmail.com>",
]
[tool.poetry.dependencies]
python = "^3.9"
psutil = { version = "^5.6" }
tornado = { version = "^6.0.4" }
pyctuator = { version = "^1.2.0" }
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
================================================
FILE: examples/tornado/tornado_example_app.py
================================================
import datetime
import logging
import random
from tornado import ioloop
from tornado.httpserver import HTTPServer
from tornado.web import Application, RequestHandler
from pyctuator.pyctuator import Pyctuator
my_logger = logging.getLogger("example")
class HomeHandler(RequestHandler):
def get(self):
my_logger.debug(f"{datetime.datetime.now()} - {str(random.randint(0, 100))}")
self.write("Hello World!")
app = Application(
[
(r"/", HomeHandler)
],
debug=False
)
example_app_address = "host.docker.internal"
example_sba_address = "localhost"
Pyctuator(
app,
"Tornado Pyctuator",
app_url=f"http://{example_app_address}:5000",
pyctuator_endpoint_url=f"http://{example_app_address}:5000/pyctuator",
registration_url=f"http://{example_sba_address}:8080/instances",
app_description="Demonstrate Spring Boot Admin Integration with Tornado",
)
http_server = HTTPServer(app, decompress_request=True)
http_server.listen(5000)
ioloop.IOLoop.current().start()
================================================
FILE: mypy.ini
================================================
[mypy]
disallow_untyped_defs = True
warn_return_any = True
[mypy-pytest]
ignore_missing_imports = True
[mypy-psutil]
ignore_missing_imports = True
[mypy-starlette.*]
ignore_missing_imports = True
[mypy-uvicorn.*]
ignore_missing_imports = True
[mypy-_pytest.monkeypatch.*]
ignore_missing_imports = True
================================================
FILE: pyctuator/__init__.py
================================================
__version__ = '0.1.0'
================================================
FILE: pyctuator/auth.py
================================================
from dataclasses import dataclass
from typing import Optional
@dataclass
class Auth:
pass
@dataclass
class BasicAuth(Auth):
username: str
password: Optional[str]
================================================
FILE: pyctuator/endpoints.py
================================================
from enum import Flag, auto
class Endpoints(Flag):
NONE = 0
ENV = auto()
INFO = auto()
HEALTH = auto()
METRICS = auto()
LOGGERS = auto()
THREAD_DUMP = auto()
LOGFILE = auto()
HTTP_TRACE = auto()
================================================
FILE: pyctuator/environment/__init__.py
================================================
================================================
FILE: pyctuator/environment/custom_environment_provider.py
================================================
from typing import Callable, Dict
from pyctuator.environment.environment_provider import PropertiesSource, EnvironmentProvider, PropertyValue
def _flatten(prefix: str, dict_to_flatten: Dict) -> Dict:
"""
Recursively flattens a dictionary that may contain literal values (numbers and strings) and other dictionaries.
For example, given the following dictionary: {
"a": 1,
"b": {
"c": 2
"d": {
"e": 3
}
}
}
The flattened dictionary will be: {
"a": 1,
"b.c": 2,
"b.d.e": 3
}
:param prefix: when descending to a sub-dictionary, the prefix represents the keys higher in the hierarchy
:param dict_to_flatten: a dictionary, or a sub-dictionary to be flattened
:return: a dictionary from a dot-separated key to a literal value
"""
res: Dict = {}
for key, value in dict_to_flatten.items():
key_with_prefix = f"{prefix}{key}."
if isinstance(value, dict):
res = {**res, **_flatten(key_with_prefix, value)}
else:
res[key_with_prefix[:-1]] = value
return res
class CustomEnvironmentProvider(EnvironmentProvider):
def __init__(self, name: str, env_provider: Callable[[], Dict]) -> None:
self.name = name
self.env_provider = env_provider
def get_properties_source(self, secret_scrubber: Callable[[Dict], Dict]) -> PropertiesSource:
flattened_env = _flatten("", self.env_provider())
scrubbed_env = secret_scrubber(flattened_env)
properties_dict = {key: PropertyValue(value) for (key, value) in scrubbed_env.items()}
return PropertiesSource(self.name, properties_dict)
================================================
FILE: pyctuator/environment/environment_provider.py
================================================
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Mapping, Optional, List, Any, Callable, Dict
@dataclass
class PropertyValue:
value: Any
origin: Optional[str] = None
@dataclass
class PropertiesSource:
name: str
properties: Mapping[str, PropertyValue]
@dataclass
class EnvironmentData:
activeProfiles: List[str]
propertySources: List[PropertiesSource]
class EnvironmentProvider(ABC):
@abstractmethod
def get_properties_source(self, secret_scrubber: Callable[[Dict], Dict]) -> PropertiesSource:
pass
================================================
FILE: pyctuator/environment/os_env_variables_impl.py
================================================
import os
from typing import Dict, Callable
from pyctuator.environment.environment_provider import PropertiesSource, PropertyValue, EnvironmentProvider
class OsEnvironmentVariableProvider(EnvironmentProvider):
def get_properties_source(self, secret_scrubber: Callable[[Dict], Dict]) -> PropertiesSource:
scrubbed_env = secret_scrubber(os.environ) # type: ignore
properties_dict = {key: PropertyValue(value) for (key, value) in scrubbed_env.items()}
return PropertiesSource("systemEnvironment", properties_dict)
================================================
FILE: pyctuator/environment/scrubber.py
================================================
import re
from typing import Any, Dict, Pattern
default_keys_to_scrub = re.compile("^(.*[^A-Za-z])?key([^A-Za-z].*)?$|.*secret.*|.*password.*|.*token.*", re.IGNORECASE)
class SecretScrubber:
def __init__(self, keys_to_scrub: Pattern[str] = default_keys_to_scrub) -> None:
self.keys_to_scrub = keys_to_scrub
self.url_keys_to_scrub = re.compile(".*url.*", re.IGNORECASE)
def scrub_secrets(self, mapping: Dict) -> Dict:
"""Scrubs secrets from a dictionary replacing them with stars
:param mapping: a mapping with "primitive" values that may include secrets
:return: a copy of the input mapping having all secrets replaced with stars
"""
def scrub(key: Any, value: Any) -> Any:
if self.keys_to_scrub.match(key):
return "******"
if self.url_keys_to_scrub.match(key):
return re.sub(r"(.*//[^:]*:).*(@.*)", r"\1******\2", str(value))
return value
return {k: scrub(k, v) for (k, v) in mapping.items()}
================================================
FILE: pyctuator/health/__init__.py
================================================
================================================
FILE: pyctuator/health/composite_health_provider.py
================================================
from dataclasses import dataclass
from typing import Mapping
from pyctuator.health.health_provider import HealthProvider, HealthStatus, Status
@dataclass
class CompositeHealthStatus(HealthStatus):
status: Status
details: Mapping[str, HealthStatus] # type: ignore[assignment]
class CompositeHealthProvider(HealthProvider):
def __init__(self, name: str, *health_providers: HealthProvider) -> None:
super().__init__()
self.name = name
self.health_providers = health_providers
def is_supported(self) -> bool:
return True
def get_name(self) -> str:
return self.name
def get_health(self) -> CompositeHealthStatus:
health_statuses: Mapping[str, HealthStatus] = {
provider.get_name(): provider.get_health()
for provider in self.health_providers
if provider.is_supported()
}
# Health is UP if no provider is registered
if not health_statuses:
return CompositeHealthStatus(Status.UP, health_statuses)
# If there's at least one provider and any of the providers is DOWN, the service is DOWN
service_is_down = any(health_status.status == Status.DOWN for health_status in health_statuses.values())
if service_is_down:
return CompositeHealthStatus(Status.DOWN, health_statuses)
# If there's at least one provider and none of the providers is DOWN and at least one is UP, the service is UP
service_is_up = any(health_status.status == Status.UP for health_status in health_statuses.values())
if service_is_up:
return CompositeHealthStatus(Status.UP, health_statuses)
# else, all providers are unknown so the service is UNKNOWN
return CompositeHealthStatus(Status.UNKNOWN, health_statuses)
================================================
FILE: pyctuator/health/db_health_provider.py
================================================
import importlib.util
from dataclasses import dataclass
from typing import Optional
from sqlalchemy.engine import Engine
from pyctuator.health.health_provider import HealthProvider, HealthStatus, Status, HealthDetails
@dataclass
class DbHealthDetails(HealthDetails):
engine: str
failure: Optional[str] = None
@dataclass
class DbHealthStatus(HealthStatus):
status: Status
details: DbHealthDetails
class DbHealthProvider(HealthProvider):
def __init__(self, engine: Engine, name: str = "db") -> None:
super().__init__()
self.engine = engine
self.name = name
def is_supported(self) -> bool:
return importlib.util.find_spec("sqlalchemy") is not None
def get_name(self) -> str:
return self.name
def get_health(self) -> DbHealthStatus:
try:
with self.engine.connect() as conn:
if self.engine.dialect.do_ping(conn.connection): # type: ignore[arg-type]
return DbHealthStatus(
status=Status.UP,
details=DbHealthDetails(self.engine.name)
)
return DbHealthStatus(
status=Status.UNKNOWN,
details=DbHealthDetails(self.engine.name, "Pinging failed"))
except Exception as e: # pylint: disable=broad-except
return DbHealthStatus(status=Status.DOWN, details=DbHealthDetails(self.engine.name, str(e)))
================================================
FILE: pyctuator/health/diskspace_health_impl.py
================================================
# pylint: disable=import-outside-toplevel
import importlib.util
from dataclasses import dataclass
from pyctuator.health.health_provider import HealthProvider, HealthDetails, HealthStatus, Status
@dataclass
class DiskSpaceHealthDetails(HealthDetails):
total: int
free: int
threshold: int
@dataclass
class DiskSpaceHealth(HealthStatus):
status: Status
details: DiskSpaceHealthDetails
class DiskSpaceHealthProvider(HealthProvider):
def __init__(self, free_bytes_down_threshold: int) -> None:
self.free_bytes_down_threshold = free_bytes_down_threshold
if importlib.util.find_spec("psutil"):
# psutil is optional and must only be imported if it is installed
import psutil
self.psutil = psutil
else:
self.psutil = None
def is_supported(self) -> bool:
return self.psutil is not None
def get_name(self) -> str:
return "diskSpace"
def get_health(self) -> DiskSpaceHealth:
usage = self.psutil.disk_usage(".")
return DiskSpaceHealth(
Status.UP if usage.free > self.free_bytes_down_threshold else Status.DOWN,
DiskSpaceHealthDetails(usage.total, usage.free, self.free_bytes_down_threshold)
)
================================================
FILE: pyctuator/health/health_provider.py
================================================
import abc
from abc import ABC
from dataclasses import dataclass
from enum import Enum
from http import HTTPStatus
from typing import Mapping
class Status(str, Enum):
UP = "UP"
DOWN = "DOWN"
UNKNOWN = "UNKNOWN"
@dataclass
class HealthDetails:
pass
@dataclass
class HealthStatus:
status: Status
details: HealthDetails
@dataclass
class HealthSummary:
status: Status
details: Mapping[str, HealthStatus]
def http_status(self) -> int:
"""
:return: The HTTP according to the service's health. Done according to the documentation in
https://docs.spring.io/spring-boot/docs/2.7.0/reference/htmlsingle/#actuator.endpoints.health.writing-custom-health-indicators
The HTTP status code in the response reflects the overall health status. By default, OUT_OF_SERVICE
and DOWN map to 503. Any unmapped health statuses, including UP, map to 200.
"""
if self.status == Status.DOWN:
return HTTPStatus.SERVICE_UNAVAILABLE
return HTTPStatus.OK
class HealthProvider(ABC):
@abc.abstractmethod
def is_supported(self) -> bool:
pass
@abc.abstractmethod
def get_name(self) -> str:
pass
@abc.abstractmethod
def get_health(self) -> HealthStatus:
pass
================================================
FILE: pyctuator/health/redis_health_provider.py
================================================
import importlib.util
from dataclasses import dataclass
from typing import Optional
from redis import Redis
from pyctuator.health.health_provider import HealthProvider, HealthStatus, Status, HealthDetails
@dataclass
class RedisHealthDetails(HealthDetails):
version: Optional[str] = None
mode: Optional[str] = None
failure: Optional[str] = None
@dataclass
class RedisHealthStatus(HealthStatus):
status: Status
details: RedisHealthDetails
class RedisHealthProvider(HealthProvider):
def __init__(self, redis: Redis, name: str = "redis") -> None:
super().__init__()
self.redis = redis
self.name = name
def is_supported(self) -> bool:
return importlib.util.find_spec("redis") is not None
def get_name(self) -> str:
return self.name
def get_health(self) -> RedisHealthStatus:
try:
info = self.redis.info()
return RedisHealthStatus(
status=Status.UP,
details=RedisHealthDetails(
version=info["redis_version"],
mode=info["redis_mode"],
))
except Exception as e: # pylint: disable=broad-except
return RedisHealthStatus(
status=Status.DOWN,
details=RedisHealthDetails(
failure=str(e)
))
================================================
FILE: pyctuator/httptrace/__init__.py
================================================
from dataclasses import dataclass
from datetime import datetime
from typing import List, Mapping, Optional
@dataclass
class TraceResponse:
status: int
headers: Mapping[str, List[str]]
@dataclass
class TraceRequest:
method: str
uri: str
headers: Mapping[str, List[str]]
@dataclass
class Session:
id: str
@dataclass
class Principal:
name: str
@dataclass
class TraceRecord:
timestamp: datetime
principal: Optional[Principal]
session: Optional[Session]
request: TraceRequest
response: TraceResponse
timeTaken: int
@dataclass
class Traces:
traces: List[TraceRecord]
================================================
FILE: pyctuator/httptrace/http_header_scrubber.py
================================================
import re
_keys_to_scrub = re.compile(
"^(.*[^A-Za-z])?key([^A-Za-z].*)?$|"
".*secret.*|"
".*password.*|"
".*token.*|"
".*authorization.*|"
".*authentication.*|"
".*cookie.*",
re.IGNORECASE
)
def scrub_header_value(key: str, value: str) -> str:
if _keys_to_scrub.match(key):
return "******"
return value
================================================
FILE: pyctuator/httptrace/http_tracer.py
================================================
import collections
from typing import List, Mapping
from pyctuator.httptrace.http_header_scrubber import scrub_header_value
from pyctuator.httptrace import Traces, TraceRecord
class HttpTracer:
def __init__(self) -> None:
self.traces_list: collections.deque = collections.deque(maxlen=100)
def get_httptrace(self) -> Traces:
return Traces(list(self.traces_list))
def add_record(self, record: TraceRecord) -> None:
record.request.headers = self._scrub_and_normalize_headers(
record.request.headers)
record.response.headers = self._scrub_and_normalize_headers(
record.response.headers)
self.traces_list.append(record)
def _scrub_and_normalize_headers(self, headers: Mapping[str, List[str]]) -> Mapping[str, List[str]]:
return {header: [scrub_header_value(header, value) for value in values] for (header, values) in headers.items()}
================================================
FILE: pyctuator/impl/__init__.py
================================================
SBA_V2_CONTENT_TYPE = "application/vnd.spring-boot.actuator.v2+json;charset=UTF-8"
================================================
FILE: pyctuator/impl/aiohttp_pyctuator.py
================================================
import dataclasses
import json
from collections import defaultdict
from datetime import datetime
from functools import partial
from http import HTTPStatus
from typing import Any, Callable, List, Mapping
from aiohttp import web
from multidict import CIMultiDictProxy
from pyctuator.endpoints import Endpoints
from pyctuator.httptrace import TraceRecord, TraceRequest, TraceResponse
from pyctuator.impl import SBA_V2_CONTENT_TYPE
from pyctuator.impl.pyctuator_impl import PyctuatorImpl
from pyctuator.impl.pyctuator_router import PyctuatorRouter
# pylint: disable=too-many-locals,unused-argument
class AioHttpPyctuator(PyctuatorRouter):
def __init__(self, app: web.Application, pyctuator_impl: PyctuatorImpl, disabled_endpoints: Endpoints) -> None:
super().__init__(app, pyctuator_impl)
custom_dumps = partial(
json.dumps, default=self._custom_json_serializer
)
async def empty_handler(request: web.Request) -> web.Response:
return web.Response(text='')
async def get_endpoints(request: web.Request) -> web.Response:
return web.json_response(self.get_endpoints_data(), dumps=custom_dumps)
async def get_environment(request: web.Request) -> web.Response:
return web.json_response(pyctuator_impl.get_environment(), dumps=custom_dumps)
async def get_info(request: web.Request) -> web.Response:
return web.json_response(pyctuator_impl.get_app_info(), dumps=custom_dumps)
async def get_health(request: web.Request) -> web.Response:
health = pyctuator_impl.get_health()
return web.json_response(health, status=health.http_status(), dumps=custom_dumps)
async def get_metric_names(request: web.Request) -> web.Response:
return web.json_response(pyctuator_impl.get_metric_names(), dumps=custom_dumps)
async def get_loggers(request: web.Request) -> web.Response:
return web.json_response(pyctuator_impl.logging.get_loggers(), dumps=custom_dumps)
async def set_logger_level(request: web.Request) -> web.Response:
request_dict = await request.json()
pyctuator_impl.logging.set_logger_level(
request.match_info["logger_name"],
request_dict.get("configuredLevel", None),
)
return web.json_response({})
async def get_logger(request: web.Request) -> web.Response:
logger_name = request.match_info["logger_name"]
return web.json_response(pyctuator_impl.logging.get_logger(logger_name), dumps=custom_dumps)
async def get_thread_dump(request: web.Request) -> web.Response:
return web.json_response(pyctuator_impl.get_thread_dump(), dumps=custom_dumps)
async def get_httptrace(request: web.Request) -> web.Response:
raw_data = pyctuator_impl.http_tracer.get_httptrace()
return web.json_response(raw_data, dumps=custom_dumps)
async def get_metric_measurement(request: web.Request) -> web.Response:
return web.json_response(
pyctuator_impl.get_metric_measurement(request.match_info["metric_name"]),
dumps=custom_dumps)
async def get_logfile(request: web.Request) -> web.Response:
range_header = request.headers.get("range")
if not range_header:
return web.Response(
body=f"{pyctuator_impl.logfile.log_messages.get_range()}"
)
str_res, start, end = pyctuator_impl.logfile.get_logfile(range_header)
response = web.Response(
status=HTTPStatus.PARTIAL_CONTENT.value,
body=str_res,
headers={
"Content-Type": "text/html; charset=UTF-8",
"Accept-Ranges": "bytes",
"Content-Range": f"bytes {start}-{end}/{end}",
},
)
return response
@web.middleware
async def intercept_requests_and_responses(request: web.Request, handler: Callable) -> Any:
request_time = datetime.now()
response = await handler(request)
response_time = datetime.now()
# Set the SBA-V2 content type for responses from Pyctuator
if request.url.path.startswith(self.pyctuator_impl.pyctuator_endpoint_path_prefix):
response.headers["Content-Type"] = SBA_V2_CONTENT_TYPE
# Record the request and response
new_record = self._create_record(
request, response, request_time, response_time
)
self.pyctuator_impl.http_tracer.add_record(record=new_record)
return response
routes = [
web.get("/pyctuator", get_endpoints),
]
if Endpoints.ENV not in disabled_endpoints:
routes.append(web.options("/pyctuator/env", empty_handler))
routes.append(web.get("/pyctuator/env", get_environment))
if Endpoints.INFO not in disabled_endpoints:
routes.append(web.options("/pyctuator/info", empty_handler))
routes.append(web.get("/pyctuator/info", get_info))
if Endpoints.HEALTH not in disabled_endpoints:
routes.append(web.options("/pyctuator/health", empty_handler))
routes.append(web.get("/pyctuator/health", get_health))
if Endpoints.METRICS not in disabled_endpoints:
routes.append(web.options("/pyctuator/metrics", empty_handler))
routes.append(web.get("/pyctuator/metrics", get_metric_names))
routes.append(web.get("/pyctuator/metrics/{metric_name}", get_metric_measurement))
if Endpoints.LOGGERS not in disabled_endpoints:
routes.append(web.options("/pyctuator/loggers", empty_handler))
routes.append(web.get("/pyctuator/loggers", get_loggers))
routes.append(web.get("/pyctuator/loggers/{logger_name}", get_logger))
routes.append(web.post("/pyctuator/loggers/{logger_name}", set_logger_level))
if Endpoints.THREAD_DUMP not in disabled_endpoints:
routes.append(web.options("/pyctuator/dump", empty_handler))
routes.append(web.options("/pyctuator/threaddump", empty_handler))
routes.append(web.get("/pyctuator/dump", get_thread_dump))
routes.append(web.get("/pyctuator/threaddump", get_thread_dump))
if Endpoints.LOGFILE not in disabled_endpoints:
routes.append(web.options("/pyctuator/logfile", empty_handler))
routes.append(web.get("/pyctuator/logfile", get_logfile))
if Endpoints.HTTP_TRACE not in disabled_endpoints:
routes.append(web.options("/pyctuator/trace", empty_handler))
routes.append(web.options("/pyctuator/httptrace", empty_handler))
routes.append(web.get("/pyctuator/trace", get_httptrace))
routes.append(web.get("/pyctuator/httptrace", get_httptrace))
app.add_routes(routes)
app.middlewares.append(intercept_requests_and_responses)
def _custom_json_serializer(self, value: Any) -> Any:
if dataclasses.is_dataclass(value):
return dataclasses.asdict(value)
if isinstance(value, datetime):
return str(value)
return None
def _create_headers_dictionary(self, headers: CIMultiDictProxy[str]) -> Mapping[str, List[str]]:
headers_dict: Mapping[str, List[str]] = defaultdict(list)
for (key, value) in headers.items():
headers_dict[key].append(value)
return dict(headers_dict)
def _create_record(
self,
request: web.Request,
response: web.Response,
request_time: datetime,
response_time: datetime
) -> TraceRecord:
new_record: TraceRecord = TraceRecord(
request_time,
None,
None,
TraceRequest(
request.method,
str(request.url),
self._create_headers_dictionary(request.headers),
),
TraceResponse(
response.status,
self._create_headers_dictionary(CIMultiDictProxy(response.headers))
),
int((response_time.timestamp() - request_time.timestamp()) * 1000),
)
return new_record
================================================
FILE: pyctuator/impl/fastapi_pyctuator.py
================================================
from collections import defaultdict
from datetime import datetime
from http import HTTPStatus
from typing import Mapping, List, Callable
from typing import Optional, Dict, Awaitable
from fastapi import APIRouter, FastAPI, Header
from pydantic import BaseModel
from starlette.datastructures import Headers
from starlette.requests import Request
from starlette.responses import Response
from pyctuator.endpoints import Endpoints
from pyctuator.environment.environment_provider import EnvironmentData
from pyctuator.httptrace import TraceRecord, TraceRequest, TraceResponse
from pyctuator.httptrace.http_tracer import Traces
from pyctuator.impl import SBA_V2_CONTENT_TYPE
from pyctuator.impl.pyctuator_impl import PyctuatorImpl
from pyctuator.impl.pyctuator_router import PyctuatorRouter
from pyctuator.logging.pyctuator_logging import LoggersData, LoggerLevels
from pyctuator.metrics.metrics_provider import Metric, MetricNames
from pyctuator.threads.thread_dump_provider import ThreadDump
class FastApiLoggerItem(BaseModel):
configuredLevel: Optional[str]
# pylint: disable=too-many-locals
class FastApiPyctuator(PyctuatorRouter):
# pylint: disable=unused-variable
def __init__(
self,
app: FastAPI,
pyctuator_impl: PyctuatorImpl,
include_in_openapi_schema: bool,
customizer: Optional[Callable[[APIRouter], None]],
disabled_endpoints: Endpoints,
) -> None:
super().__init__(app, pyctuator_impl)
router = APIRouter()
if customizer:
customizer(router)
@router.get("/", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_endpoints() -> object:
return {"_links": self.get_endpoints_links()}
@router.options("/env", include_in_schema=include_in_openapi_schema)
@router.options("/info", include_in_schema=include_in_openapi_schema)
@router.options("/health", include_in_schema=include_in_openapi_schema)
@router.options("/metrics", include_in_schema=include_in_openapi_schema)
@router.options("/loggers", include_in_schema=include_in_openapi_schema)
@router.options("/dump", include_in_schema=include_in_openapi_schema)
@router.options("/threaddump", include_in_schema=include_in_openapi_schema)
@router.options("/logfile", include_in_schema=include_in_openapi_schema)
@router.options("/trace", include_in_schema=include_in_openapi_schema)
@router.options("/httptrace", include_in_schema=include_in_openapi_schema)
def options() -> None:
"""
Spring boot admin, after registration, issues multiple OPTIONS request to the monitored application in order
to determine the supported capabilities (endpoints).
Here we "acknowledge" that env, info and health are supported.
The "include_in_schema=False" is used to prevent from these OPTIONS endpoints to show up in the
documentation.
"""
if Endpoints.ENV not in disabled_endpoints:
@router.get("/env", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_environment() -> EnvironmentData:
return pyctuator_impl.get_environment()
if Endpoints.INFO not in disabled_endpoints:
@router.get("/info", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_info() -> Dict:
return pyctuator_impl.get_app_info()
if Endpoints.HEALTH not in disabled_endpoints:
@router.get("/health", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_health(response: Response) -> object:
health = pyctuator_impl.get_health()
response.status_code = health.http_status()
return health
if Endpoints.METRICS not in disabled_endpoints:
@router.get("/metrics", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_metric_names() -> MetricNames:
return pyctuator_impl.get_metric_names()
@router.get("/metrics/{metric_name}", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_metric_measurement(metric_name: str) -> Metric:
return pyctuator_impl.get_metric_measurement(metric_name)
# Retrieving All Loggers
if Endpoints.LOGGERS not in disabled_endpoints:
@router.get("/loggers", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_loggers() -> LoggersData:
return pyctuator_impl.logging.get_loggers()
@router.post("/loggers/{logger_name}", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def set_logger_level(item: FastApiLoggerItem, logger_name: str) -> Dict:
pyctuator_impl.logging.set_logger_level(logger_name, item.configuredLevel)
return {}
@router.get("/loggers/{logger_name}", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_logger(logger_name: str) -> LoggerLevels:
return pyctuator_impl.logging.get_logger(logger_name)
if Endpoints.THREAD_DUMP not in disabled_endpoints:
@router.get("/dump", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
@router.get("/threaddump", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_thread_dump() -> ThreadDump:
return pyctuator_impl.get_thread_dump()
if Endpoints.LOGFILE not in disabled_endpoints:
@router.get("/logfile", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_logfile(range_header: str = Header(default=None,
alias="range")) -> Response: # pylint: disable=redefined-builtin
if not range_header:
return Response(content=pyctuator_impl.logfile.log_messages.get_range())
str_res, start, end = pyctuator_impl.logfile.get_logfile(range_header)
my_res = Response(
status_code=HTTPStatus.PARTIAL_CONTENT.value,
content=str_res,
headers={
"Content-Type": "text/html; charset=UTF-8",
"Accept-Ranges": "bytes",
"Content-Range": f"bytes {start}-{end}/{end}",
})
return my_res
if Endpoints.HTTP_TRACE not in disabled_endpoints:
@router.get("/trace", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
@router.get("/httptrace", include_in_schema=include_in_openapi_schema, tags=["pyctuator"])
def get_httptrace() -> Traces:
return pyctuator_impl.http_tracer.get_httptrace()
@app.middleware("http")
async def intercept_requests_and_responses(
request: Request,
call_next: Callable[[Request], Awaitable[Response]]
) -> Response:
request_time = datetime.now()
response: Response = await call_next(request)
response_time = datetime.now()
# Set the SBA-V2 content type for responses from Pyctuator
if request.url.path.startswith(self.pyctuator_impl.pyctuator_endpoint_path_prefix):
response.headers["Content-Type"] = SBA_V2_CONTENT_TYPE
# Record the request and response
new_record = self._create_record(request, response, request_time, response_time)
self.pyctuator_impl.http_tracer.add_record(record=new_record)
return response
app.include_router(router, prefix=pyctuator_impl.pyctuator_endpoint_path_prefix)
def _create_headers_dictionary(self, headers: Headers) -> Mapping[str, List[str]]:
headers_dict: Mapping[str, List[str]] = defaultdict(list)
for (key, value) in headers.items():
headers_dict[key].append(value)
return headers_dict
def _create_record(
self,
request: Request,
response: Response,
request_time: datetime,
response_time: datetime,
) -> TraceRecord:
new_record: TraceRecord = TraceRecord(
request_time,
None,
None,
TraceRequest(request.method, str(request.url), self._create_headers_dictionary(request.headers)),
TraceResponse(response.status_code, self._create_headers_dictionary(response.headers)),
int((response_time.timestamp() - request_time.timestamp()) * 1000),
)
return new_record
================================================
FILE: pyctuator/impl/flask_pyctuator.py
================================================
import json
from collections import defaultdict
from datetime import datetime, date
from http import HTTPStatus
from typing import Dict, Tuple, Any, Mapping, List
from flask import Flask, Blueprint, request, jsonify, after_this_request
from flask import Response, make_response
from flask.json.provider import DefaultJSONProvider
# from flask.json import JSONEncoder
from werkzeug.datastructures import Headers
from pyctuator.endpoints import Endpoints
from pyctuator.httptrace import TraceRecord, TraceRequest, TraceResponse
from pyctuator.impl import SBA_V2_CONTENT_TYPE
from pyctuator.impl.pyctuator_impl import PyctuatorImpl
from pyctuator.impl.pyctuator_router import PyctuatorRouter
class IsoTimeJSONProvider(DefaultJSONProvider):
""" Override Flask's JSON encoding of datetime to assure ISO format is used.
By default, when Flask is rendering a response to JSON, it is formatting datetime, date and time according to
RFC-822 which is different from the ISO format used by SBA.
As of 2.2.*, changing the datetime and date JSON encoding is done globally,
see https://stackoverflow.com/a/74618781/2692895 (which is an updated reply to
https://stackoverflow.com/questions/43663552/keep-a-datetime-date-in-yyyy-mm-dd-format-when-using-flasks-jsonify)
"""
def default(self, o: Any) -> Any:
if isinstance(o, (date, datetime)):
return o.isoformat()
return super().default(o)
class FlaskPyctuator(PyctuatorRouter):
# pylint: disable=too-many-locals, unused-variable
def __init__(
self,
app: Flask,
pyctuator_impl: PyctuatorImpl,
disabled_endpoints: Endpoints,
) -> None:
super().__init__(app, pyctuator_impl)
path_prefix: str = pyctuator_impl.pyctuator_endpoint_path_prefix
flask_blueprint: Blueprint = Blueprint("flask_blueprint", "pyctuator", )
app.json = IsoTimeJSONProvider(app)
@app.before_request
def intercept_requests_and_responses() -> None:
request_time = datetime.now()
@after_this_request
def after_response(response: Response) -> Response:
response_time = datetime.now()
# Set the SBA-V2 content type for responses from Pyctuator
if request.path.startswith(self.pyctuator_impl.pyctuator_endpoint_path_prefix):
response.headers["Content-Type"] = SBA_V2_CONTENT_TYPE
# Record the request and response
self.record_request_and_response(response, request_time, response_time)
return response
@flask_blueprint.route("/")
def get_endpoints() -> Any:
return jsonify(self.get_endpoints_data())
if Endpoints.ENV not in disabled_endpoints:
@flask_blueprint.route("/env")
def get_environment() -> Any:
return jsonify(pyctuator_impl.get_environment())
if Endpoints.INFO not in disabled_endpoints:
@flask_blueprint.route("/info")
def get_info() -> Any:
return jsonify(pyctuator_impl.get_app_info())
if Endpoints.HEALTH not in disabled_endpoints:
@flask_blueprint.route("/health")
def get_health() -> Any:
health = pyctuator_impl.get_health()
return jsonify(health), health.http_status()
if Endpoints.METRICS not in disabled_endpoints:
@flask_blueprint.route("/metrics")
def get_metric_names() -> Any:
return jsonify(pyctuator_impl.get_metric_names())
@flask_blueprint.route("/metrics/<metric_name>")
def get_metric_measurement(metric_name: str) -> Any:
return jsonify(pyctuator_impl.get_metric_measurement(metric_name))
# Retrieving All Loggers
if Endpoints.LOGGERS not in disabled_endpoints:
@flask_blueprint.route("/loggers")
def get_loggers() -> Any:
return jsonify(pyctuator_impl.logging.get_loggers())
@flask_blueprint.route("/loggers/<logger_name>", methods=['POST'])
def set_logger_level(logger_name: str) -> Dict:
request_dict = json.loads(request.data)
pyctuator_impl.logging.set_logger_level(logger_name, request_dict.get("configuredLevel", None))
return {}
@flask_blueprint.route("/loggers/<logger_name>")
def get_logger(logger_name: str) -> Any:
return jsonify(pyctuator_impl.logging.get_logger(logger_name))
if Endpoints.THREAD_DUMP not in disabled_endpoints:
@flask_blueprint.route("/threaddump")
@flask_blueprint.route("/dump")
def get_thread_dump() -> Any:
return jsonify(pyctuator_impl.get_thread_dump())
if Endpoints.LOGFILE not in disabled_endpoints:
@flask_blueprint.route("/logfile")
def get_logfile() -> Tuple[Response, int]:
range_header = request.environ.get('HTTP_RANGE')
if not range_header:
response: Response = make_response(pyctuator_impl.logfile.log_messages.get_range())
return response, HTTPStatus.OK
str_res, start, end = pyctuator_impl.logfile.get_logfile(range_header)
resp: Response = make_response(str_res)
resp.headers["Content-Type"] = "text/html; charset=UTF-8"
resp.headers["Accept-Ranges"] = "bytes"
resp.headers["Content-Range"] = f"bytes {start}-{end}/{end}"
return resp, HTTPStatus.PARTIAL_CONTENT
if Endpoints.HTTP_TRACE not in disabled_endpoints:
@flask_blueprint.route("/trace")
@flask_blueprint.route("/httptrace")
def get_httptrace() -> Any:
return jsonify(pyctuator_impl.http_tracer.get_httptrace())
app.register_blueprint(flask_blueprint, url_prefix=path_prefix)
def _create_headers_dictionary_flask(self, headers: Headers) -> Mapping[str, List[str]]:
headers_dict: Mapping[str, List[str]] = defaultdict(list)
for (key, value) in headers.items():
headers_dict[key].append(value)
return dict(headers_dict)
def record_request_and_response(
self,
response: Response,
request_time: datetime,
response_time: datetime,
) -> None:
new_record = TraceRecord(
request_time,
None,
None,
TraceRequest(request.method, str(request.url), self._create_headers_dictionary_flask(request.headers)),
TraceResponse(response.status_code, self._create_headers_dictionary_flask(response.headers)),
int((response_time.timestamp() - request_time.timestamp()) * 1000),
)
self.pyctuator_impl.http_tracer.add_record(record=new_record)
================================================
FILE: pyctuator/impl/pyctuator_impl.py
================================================
import dataclasses
from dataclasses import dataclass
from datetime import datetime
from typing import List, Dict, Mapping, Optional, Callable
from urllib.parse import urlparse
from pyctuator.endpoints import Endpoints
from pyctuator.environment.environment_provider import EnvironmentData, EnvironmentProvider
from pyctuator.environment.scrubber import SecretScrubber
from pyctuator.health.health_provider import HealthStatus, HealthSummary, Status, HealthProvider
from pyctuator.httptrace.http_tracer import HttpTracer
from pyctuator.logfile.logfile import PyctuatorLogfile # type: ignore
from pyctuator.logging.pyctuator_logging import PyctuatorLogging
from pyctuator.metrics.metrics_provider import Metric, MetricNames, MetricsProvider
from pyctuator.threads.thread_dump_provider import ThreadDump, ThreadDumpProvider
@dataclass
class GitCommitInfo:
time: datetime
id: str
@dataclass
class GitInfo:
commit: GitCommitInfo
branch: Optional[str] = None
@dataclass
class BuildInfo:
name: Optional[str] = None
artifact: Optional[str] = None
group: Optional[str] = None
version: Optional[str] = None
time: Optional[datetime] = None
@dataclass
class AppDetails:
name: str
description: Optional[str] = None
@dataclass
class AppInfo:
app: AppDetails
build: Optional[BuildInfo] = None
git: Optional[GitInfo] = None
class PyctuatorImpl:
# pylint: disable=too-many-instance-attributes
def __init__(
self,
app_info: AppInfo,
pyctuator_endpoint_url: str,
logfile_max_size: int,
logfile_formatter: str,
additional_app_info: Optional[dict],
disabled_endpoints: Endpoints
):
self.app_info = app_info
self.pyctuator_endpoint_url = pyctuator_endpoint_url
self.additional_app_info = additional_app_info
self.disabled_endpoints = disabled_endpoints
self.metrics_providers: List[MetricsProvider] = []
self.health_providers: List[HealthProvider] = []
self.environment_providers: List[EnvironmentProvider] = []
self.logging = PyctuatorLogging()
self.thread_dump_provider = ThreadDumpProvider()
self.logfile = PyctuatorLogfile(max_size=logfile_max_size, formatter=logfile_formatter)
self.http_tracer = HttpTracer()
self.secret_scrubber: Callable[[Dict], Dict] = SecretScrubber().scrub_secrets
# Determine the endpoint's URL path prefix and make sure it doesn't end with a "/"
self.pyctuator_endpoint_path_prefix = urlparse(pyctuator_endpoint_url).path
if self.pyctuator_endpoint_path_prefix[-1:] == "/":
self.pyctuator_endpoint_path_prefix = self.pyctuator_endpoint_path_prefix[:-1]
def register_metrics_provider(self, provider: MetricsProvider) -> None:
self.metrics_providers.append(provider)
def register_health_providers(self, provider: HealthProvider) -> None:
self.health_providers.append(provider)
def register_environment_provider(self, provider: EnvironmentProvider) -> None:
self.environment_providers.append(provider)
def set_secret_scrubber(self, secret_scrubber: Callable[[Dict], Dict]) -> None:
self.secret_scrubber = secret_scrubber
def get_environment(self) -> EnvironmentData:
active_profiles: List[str] = []
env_data = EnvironmentData(
active_profiles,
[source.get_properties_source(self.secret_scrubber) for source in self.environment_providers]
)
return env_data
def set_git_info(self, git_info: GitInfo) -> None:
self.app_info.git = git_info
def set_build_info(self, build_info: BuildInfo) -> None:
self.app_info.build = build_info
def get_health(self) -> HealthSummary:
health_statuses: Mapping[str, HealthStatus] = {
provider.get_name(): provider.get_health()
for provider in self.health_providers
if provider.is_supported()
}
# Health is UP if no provider is registered
if not health_statuses:
return HealthSummary(Status.UP, health_statuses)
# If there's at least one provider and any of the providers is DOWN, the service is DOWN
service_is_down = any(health_status.status == Status.DOWN for health_status in health_statuses.values())
if service_is_down:
return HealthSummary(Status.DOWN, health_statuses)
# If there's at least one provider and none of the providers is DOWN and at least one is UP, the service is UP
service_is_up = any(health_status.status == Status.UP for health_status in health_statuses.values())
if service_is_up:
return HealthSummary(Status.UP, health_statuses)
# else, all providers are unknown so the service is UNKNOWN
return HealthSummary(Status.UNKNOWN, health_statuses)
def get_metric_names(self) -> MetricNames:
metric_names = []
for provider in self.metrics_providers:
for metric_name in provider.get_supported_metric_names():
metric_names.append(metric_name)
return MetricNames(metric_names)
def get_metric_measurement(self, metric_name: str) -> Metric:
for provider in self.metrics_providers:
if metric_name.startswith(provider.get_prefix()):
return provider.get_metric(metric_name)
raise KeyError(f"Unknown metric {metric_name}")
def get_thread_dump(self) -> ThreadDump:
return self.thread_dump_provider.get_thread_dump()
def get_app_info(self) -> Dict:
app_info_dict = {k: v for (k, v) in dataclasses.asdict(self.app_info).items() if v}
if self.additional_app_info:
app_info_dict = {**app_info_dict, **self.additional_app_info}
return app_info_dict
================================================
FILE: pyctuator/impl/pyctuator_router.py
================================================
from abc import ABC
from dataclasses import dataclass
from typing import Any, Optional, Mapping
from pyctuator.endpoints import Endpoints
from pyctuator.impl.pyctuator_impl import PyctuatorImpl
@dataclass
class LinkHref:
href: str
templated: bool
@dataclass
class EndpointsData:
_links: Mapping[str, LinkHref]
class PyctuatorRouter(ABC):
def __init__(
self,
app: Any,
pyctuator_impl: PyctuatorImpl,
):
self.app = app
self.pyctuator_impl = pyctuator_impl
def get_endpoints_data(self) -> EndpointsData:
return EndpointsData(self.get_endpoints_links())
def get_endpoints_links(self) -> Mapping[str, LinkHref]:
def link_href(endpoint: Endpoints, path: str) -> Optional[LinkHref]:
return None if endpoint in self.pyctuator_impl.disabled_endpoints \
else LinkHref(self.pyctuator_impl.pyctuator_endpoint_url + path, False)
endpoints = {
"self": LinkHref(self.pyctuator_impl.pyctuator_endpoint_url, False),
"env": link_href(Endpoints.ENV, "/env"),
"info": link_href(Endpoints.INFO, "/info"),
"health": link_href(Endpoints.HEALTH, "/health"),
"metrics": link_href(Endpoints.METRICS, "/metrics"),
"loggers": link_href(Endpoints.LOGGERS, "/loggers"),
"dump": link_href(Endpoints.THREAD_DUMP, "/dump"),
"threaddump": link_href(Endpoints.THREAD_DUMP, "/threaddump"),
"logfile": link_href(Endpoints.LOGFILE, "/logfile"),
"httptrace": link_href(Endpoints.HTTP_TRACE, "/httptrace"),
}
return {endpoint: link_href for (endpoint, link_href) in endpoints.items() if link_href is not None}
================================================
FILE: pyctuator/impl/spring_boot_admin_registration.py
================================================
import http.client
import json
import logging
import os
import ssl
import threading
import urllib.parse
from base64 import b64encode
from datetime import datetime
from http.client import HTTPConnection, HTTPResponse
from typing import Optional, Dict
from pyctuator.auth import Auth, BasicAuth
# pylint: disable=too-many-instance-attributes
class BootAdminRegistrationHandler:
def __init__(
self,
registration_url: str,
registration_auth: Optional[Auth],
application_name: str,
pyctuator_base_url: str,
start_time: datetime,
service_url: str,
registration_interval_sec: float,
application_metadata: Optional[dict] = None,
ssl_context: Optional[ssl.SSLContext] = None,
) -> None:
self.registration_url = registration_url
self.registration_auth = registration_auth
self.application_name = application_name
self.pyctuator_base_url = pyctuator_base_url
self.start_time = start_time
self.service_url = service_url if service_url.endswith("/") else service_url + "/"
self.registration_interval_sec = registration_interval_sec
self.instance_id = None
self.application_metadata = application_metadata if application_metadata else {}
self.ssl_context = ssl_context
self.should_continue_registration_schedule: bool = False
self.disable_certificate_validation_for_https_registration: bool = \
os.getenv("PYCTUATOR_REGISTRATION_NO_CERT") is not None
def _schedule_next_registration(self, registration_interval_sec: float) -> None:
timer = threading.Timer(
registration_interval_sec,
self._register_with_admin_server,
[]
)
timer.setDaemon(True)
timer.start()
def _register_with_admin_server(self) -> None:
# When waking up, make sure registration is still needed
if not self.should_continue_registration_schedule:
return
registration_data = {
"name": self.application_name,
"managementUrl": self.pyctuator_base_url,
"healthUrl": f"{self.pyctuator_base_url}/health",
"serviceUrl": self.service_url,
"metadata": {
"startup": self.start_time.isoformat(),
**self.application_metadata
}
}
logging.debug("Trying to post registration data to %s: %s", self.registration_url, registration_data)
conn: Optional[HTTPConnection] = None
try:
headers = {"Content-type": "application/json"}
self.authenticate(headers)
response = self._http_request(self.registration_url, "POST", headers, json.dumps(registration_data))
if response.status < 200 or response.status >= 300:
logging.warning("Failed registering with boot-admin, got %s - %s", response.status, response.read())
else:
self.instance_id = json.loads(response.read().decode('utf-8'))["id"]
except Exception as e: # pylint: disable=broad-except
logging.warning("Failed registering with boot-admin, %s (%s)", e, type(e))
finally:
if conn:
conn.close()
# Schedule the next registration unless asked to abort
if self.should_continue_registration_schedule:
self._schedule_next_registration(self.registration_interval_sec)
def deregister_from_admin_server(self) -> None:
if self.instance_id is None:
return
headers = {}
self.authenticate(headers)
deregistration_url = f"{self.registration_url}/{self.instance_id}"
logging.info("Deregistering from %s", deregistration_url)
conn: Optional[HTTPConnection] = None
try:
response = self._http_request(deregistration_url, "DELETE", headers)
if response.status < 200 or response.status >= 300:
logging.warning("Failed deregistering from boot-admin, got %s - %s", response.status, response.read())
except Exception as e: # pylint: disable=broad-except
logging.warning("Failed deregistering from boot-admin, %s (%s)", e, type(e))
finally:
if conn:
conn.close()
def authenticate(self, headers: Dict) -> None:
if isinstance(self.registration_auth, BasicAuth):
password = self.registration_auth.password if self.registration_auth.password else ""
authorization_string = self.registration_auth.username + ":" + password
encoded_authorization: str = b64encode(bytes(authorization_string, "utf-8")).decode("ascii")
headers["Authorization"] = f"Basic {encoded_authorization}"
def start(self, initial_delay_sec: Optional[float] = None) -> None:
logging.info("Starting recurring registration of %s with %s",
self.pyctuator_base_url, self.registration_url)
self.should_continue_registration_schedule = True
self._schedule_next_registration(initial_delay_sec or self.registration_interval_sec)
def stop(self) -> None:
logging.info("Stopping recurring registration")
self.should_continue_registration_schedule = False
def _http_request(self, url: str, method: str, headers: Dict[str, str], body: Optional[str] = None) -> HTTPResponse:
url_parts = urllib.parse.urlsplit(url)
if not url_parts.hostname:
raise ValueError(f"Unknown host in {url}")
hostname: str = url_parts.hostname
if url_parts.scheme == "http":
conn = http.client.HTTPConnection(host=hostname, port=url_parts.port)
elif url_parts.scheme == "https":
context = self.ssl_context
if not context and self.disable_certificate_validation_for_https_registration:
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_NONE
conn = http.client.HTTPSConnection(url_parts.hostname, url_parts.port, context=context)
else:
raise ValueError(f"Unknown scheme in {url}")
conn.request(
method,
url_parts.path,
body=body,
headers=headers,
)
return conn.getresponse()
================================================
FILE: pyctuator/impl/tornado_pyctuator.py
================================================
import dataclasses
import json
from datetime import datetime, timedelta
from functools import partial
from http import HTTPStatus
from typing import Any, Optional, Callable, Mapping, List
from tornado.httputil import HTTPHeaders
from tornado.web import Application, RequestHandler
from pyctuator.endpoints import Endpoints
from pyctuator.httptrace import TraceRecord, TraceRequest, TraceResponse
from pyctuator.impl import SBA_V2_CONTENT_TYPE
from pyctuator.impl.pyctuator_impl import PyctuatorImpl
from pyctuator.impl.pyctuator_router import PyctuatorRouter
# pylint: disable=abstract-method
class AbstractPyctuatorHandler(RequestHandler):
pyctuator_router: Optional[PyctuatorRouter] = None
dumps: Optional[Callable[[Any], str]] = None
def initialize(self) -> None:
self.pyctuator_router = self.application.settings.get("pyctuator_router")
self.dumps = self.application.settings.get("custom_dumps")
self.set_header("Content-Type", SBA_V2_CONTENT_TYPE)
def options(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write("")
class PyctuatorHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.get_endpoints_data()))
# GET /env
class EnvHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.get_environment()))
# GET /info
class InfoHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.get_app_info()))
# GET /health
class HealthHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
health = self.pyctuator_router.pyctuator_impl.get_health()
self.set_status(health.http_status())
self.write(self.dumps(health))
# GET /metrics
class MetricsHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.get_metric_names()))
# GET "/metrics/{metric_name}"
class MetricsNameHandler(AbstractPyctuatorHandler):
def get(self, metric_name: str) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.get_metric_measurement(metric_name)))
# GET /loggers
class LoggersHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.logging.get_loggers()))
# GET /loggers/{logger_name}
# POST /loggers/{logger_name}
class LoggersNameHandler(AbstractPyctuatorHandler):
def get(self, logger_name: str) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.logging.get_logger(logger_name)))
def post(self, logger_name: str) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
body_str = self.request.body.decode("utf-8")
body = json.loads(body_str)
self.pyctuator_router.pyctuator_impl.logging.set_logger_level(logger_name, body.get("configuredLevel", None))
self.write("")
# GET /threaddump
class ThreadDumpHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.get_thread_dump()))
# GET /logfile
class LogFileHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
range_header = self.request.headers.get("range")
if not range_header:
self.write(f"{self.pyctuator_router.pyctuator_impl.logfile.log_messages.get_range()}")
else:
str_res, start, end = self.pyctuator_router.pyctuator_impl.logfile.get_logfile(range_header)
self.set_status(HTTPStatus.PARTIAL_CONTENT.value)
self.add_header("Content-Type", "text/html; charset=UTF-8")
self.add_header("Accept-Ranges", "bytes")
self.add_header("Content-Range", f"bytes {start}-{end}/{end}")
self.write(str_res)
# GET /httptrace
class HttpTraceHandler(AbstractPyctuatorHandler):
def get(self) -> None:
assert self.pyctuator_router is not None
assert self.dumps is not None
self.write(self.dumps(self.pyctuator_router.pyctuator_impl.http_tracer.get_httptrace()))
# pylint: disable=too-many-locals,unused-argument
class TornadoHttpPyctuator(PyctuatorRouter):
def __init__(self, app: Application, pyctuator_impl: PyctuatorImpl, disabled_endpoints: Endpoints) -> None:
super().__init__(app, pyctuator_impl)
custom_dumps = partial(
json.dumps, default=self._custom_json_serializer
)
app.settings.setdefault("pyctuator_router", self)
app.settings.setdefault("custom_dumps", custom_dumps)
# Register a log-function that records request and response in traces and than delegates to the original func
self.delegate_log_function = app.settings.get("log_function")
app.settings.setdefault("log_function", self._intercept_request_and_response)
handlers: list = [(r"/pyctuator", PyctuatorHandler)]
if Endpoints.ENV not in disabled_endpoints:
handlers.append((r"/pyctuator/env", EnvHandler))
if Endpoints.INFO not in disabled_endpoints:
handlers.append((r"/pyctuator/info", InfoHandler))
if Endpoints.HEALTH not in disabled_endpoints:
handlers.append((r"/pyctuator/health", HealthHandler))
if Endpoints.METRICS not in disabled_endpoints:
handlers.append((r"/pyctuator/metrics", MetricsHandler))
handlers.append((r"/pyctuator/metrics/(?P<metric_name>.*$)", MetricsNameHandler))
if Endpoints.LOGGERS not in disabled_endpoints:
handlers.append((r"/pyctuator/loggers", LoggersHandler))
handlers.append((r"/pyctuator/loggers/(?P<logger_name>.*$)", LoggersNameHandler))
if Endpoints.THREAD_DUMP not in disabled_endpoints:
handlers.append((r"/pyctuator/dump", ThreadDumpHandler))
handlers.append((r"/pyctuator/threaddump", ThreadDumpHandler))
if Endpoints.LOGFILE not in disabled_endpoints:
handlers.append((r"/pyctuator/logfile", LogFileHandler))
if Endpoints.HTTP_TRACE not in disabled_endpoints:
handlers.append((r"/pyctuator/trace", HttpTraceHandler))
handlers.append((r"/pyctuator/httptrace", HttpTraceHandler))
app.add_handlers(".*$", handlers)
def _intercept_request_and_response(self, handler: RequestHandler) -> None:
# Record the request and response
record = TraceRecord(
timestamp=datetime.now() - timedelta(seconds=handler.request.request_time()),
principal=None,
session=None,
request=TraceRequest(
method=handler.request.method or "",
uri=handler.request.full_url(),
headers=get_headers(handler.request.headers)
),
response=TraceResponse(
status=handler.get_status(),
headers=get_headers(handler._headers) # pylint: disable=protected-access
),
timeTaken=int(handler.request.request_time() * 1000),
)
self.pyctuator_impl.http_tracer.add_record(record)
if self.delegate_log_function:
self.delegate_log_function(handler)
def _custom_json_serializer(self, value: Any) -> Any:
if dataclasses.is_dataclass(value):
return dataclasses.asdict(value)
if isinstance(value, datetime):
return str(value)
return None
def get_headers(headers: HTTPHeaders) -> Mapping[str, List[str]]:
""" Tornado's HTTPHeaders contains multiple entries of the same header name if multiple values were used, this
function groups headers by header name. See documentation of `tornado.httputil.HTTPHeaders` """
return {header.lower(): headers.get_list(header) for header in headers.keys()}
================================================
FILE: pyctuator/logfile/logfile.py
================================================
import logging
import re
from typing import Optional, Tuple
logfile_request_range_pattern = re.compile("bytes=(\\d*)-(\\d*)")
class LogMessageBuffer(logging.Handler):
def __init__(self, max_size: int, formatter: str) -> None:
super().__init__()
self.setFormatter(logging.Formatter(formatter))
self._max_size = max_size
self._buffer: str = ""
self._offset: int = 0
def emit(self, record: logging.LogRecord) -> None:
msg = self.format(record) + "\n"
msg_len = len(msg)
if len(self._buffer) + msg_len > self._max_size:
self._buffer = self._buffer[-(self._max_size - msg_len):]
self._offset += msg_len
self._buffer += msg
def get_range(self, start: Optional[int] = None, end: Optional[int] = None) -> str:
start = start - self._offset if start else 0
return self._buffer[start:end or len(self._buffer) - 1]
def get_offset(self) -> int:
return self._offset
def get_offset_tuple(self, start: Optional[int], end: Optional[int]) -> Tuple[int, int]:
res_start = self._offset + (start or 0)
res_end = self._offset + (end or len(self._buffer))
return res_start, res_end
class PyctuatorLogfile:
def __init__(self, max_size: int, formatter: str) -> None:
self.log_messages = LogMessageBuffer(max_size=max_size, formatter=formatter)
def get_logfile(self, range_substring: str) -> Tuple[str, int, int]:
logging.debug("Received logfile request with range header: %s", range_substring)
start = None
end = None
range_substring_match = logfile_request_range_pattern.match(range_substring)
if range_substring_match:
start_str, end_str = range_substring_match.groups()
start = int(start_str) if start_str.strip() else None
end = int(end_str) if end_str.strip() else None
str_res = self.log_messages.get_range(start, end)
end = len(str_res) if (start is None) and end else end # Handle 0-307200 initial range edge-case
res_start, res_end = self.log_messages.get_offset_tuple(start, end)
logging.debug(f"Returning logfile response with range header: bytes=%d-%d/%d", res_start, res_end, res_end)
return str_res, res_start, res_end
def get_log_buffer_offset(self) -> int:
return self.log_messages.get_offset()
================================================
FILE: pyctuator/logging/__init__.py
================================================
================================================
FILE: pyctuator/logging/pyctuator_logging.py
================================================
import logging
from dataclasses import dataclass
from typing import Dict, List, Optional
@dataclass
class LoggerLevels:
configuredLevel: str
effectiveLevel: str
@dataclass
class LoggersData:
levels: List[str]
loggers: Dict[str, LoggerLevels]
groups: Dict[str, LoggerLevels]
@dataclass
class LogLevelMapping:
boot_admin_log_level: str
python_log_level: int
python_from_log_level: int
_log_level_mapping: List[LogLevelMapping] = [
LogLevelMapping("DEBUG", logging.DEBUG, logging.NOTSET),
LogLevelMapping("INFO", logging.INFO, logging.DEBUG),
LogLevelMapping("WARN", logging.WARNING, logging.INFO),
LogLevelMapping("ERROR", logging.ERROR, logging.WARNING),
LogLevelMapping("OFF", logging.NOTSET, -1),
]
def _python_to_admin_log_level(log_level: int) -> str:
for mapping in _log_level_mapping:
if mapping.python_from_log_level < log_level <= mapping.python_log_level:
return mapping.boot_admin_log_level
# If log_level is unknown, simply return its string representation
return str(log_level)
def _admin_to_python_log_level(log_level: str) -> int:
log_level_mapping = next(mapping for mapping in _log_level_mapping if mapping.boot_admin_log_level == log_level)
return log_level_mapping.python_log_level
class PyctuatorLogging:
def set_logger_level(self, logger_name: str, logger_level: Optional[str]) -> None:
logger = logging.getLogger(logger_name)
level = logger_level or "OFF"
if level == "OFF":
logging.disable(logging.CRITICAL) # disable all logging calls of (CRITICAL) severity lvl and below
logger.setLevel(0)
else:
logger.setLevel(_admin_to_python_log_level(level))
logging.debug("Setting logger '%s' level to %s", logger_name, level)
def get_loggers(self) -> LoggersData:
level_names = [mapping.boot_admin_log_level for mapping
in _log_level_mapping
if mapping.boot_admin_log_level != "OFF"]
loggers = {}
for logger_dict_member in logging.root.manager.loggerDict: # type: ignore
logger_inst = logging.getLogger(logger_dict_member)
level = _python_to_admin_log_level(logger_inst.level)
loggers[logger_inst.name] = LoggerLevels(level, level)
return LoggersData(levels=level_names, loggers=loggers, groups={})
def get_logger(self, logger_name: str) -> LoggerLevels:
logger = logging.getLogger(logger_name)
level = _python_to_admin_log_level(logger.level)
return LoggerLevels(level, level)
================================================
FILE: pyctuator/metrics/__init__.py
================================================
================================================
FILE: pyctuator/metrics/memory_metrics_impl.py
================================================
# pylint: disable=import-outside-toplevel
import importlib.util
from typing import List
from pyctuator.metrics.metrics_provider import MetricsProvider, Metric, Measurement
PREFIX = "memory."
class MemoryMetricsProvider(MetricsProvider):
def __init__(self) -> None:
if importlib.util.find_spec("psutil"):
# psutil is optional and must only be imported if it is installed
import psutil
self.process = psutil.Process()
else:
self.process = None
def get_prefix(self) -> str:
return PREFIX
def get_supported_metric_names(self) -> List[str]:
if not self.process:
return []
keys: List[str] = list(self.process.memory_info()._asdict().keys())
return list(map(lambda metric: PREFIX + metric, list(keys)))
def get_metric(self, metric_name: str) -> Metric:
measurements: List[Measurement] = []
if self.process:
name = metric_name[len(PREFIX):]
measurements = [Measurement("VALUE", getattr(self.process.memory_info(), name))]
return Metric(metric_name, None, "bytes", measurements, [])
================================================
FILE: pyctuator/metrics/metrics_provider.py
================================================
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List, Optional
@dataclass
class MetricNames:
names: List[str]
@dataclass
class Measurement:
statistic: str # one of TOTAL, TOTAL_TIME, COUNT, MAX, VALUE, UNKNOWN, ACTIVE_TASKS, DURATION
value: float # can be an int as well
@dataclass
class MetricTag:
tag: str
values: List[str]
@dataclass
class Metric:
name: str
description: Optional[str]
baseUnit: str
measurements: List[Measurement]
availableTags: List[MetricTag]
class MetricsProvider(ABC):
@abstractmethod
def get_prefix(self) -> str:
pass
@abstractmethod
def get_supported_metric_names(self) -> List[str]:
pass
@abstractmethod
def get_metric(self, metric_name: str) -> Metric:
pass
================================================
FILE: pyctuator/metrics/thread_metrics_impl.py
================================================
# pylint: disable=import-outside-toplevel
import importlib.util
from typing import List
from pyctuator.metrics.metrics_provider import MetricsProvider, Metric, Measurement
PREFIX = "thread."
THREAD_COUNT = PREFIX + "count"
class ThreadMetricsProvider(MetricsProvider):
def __init__(self) -> None:
if importlib.util.find_spec("psutil"):
# psutil is optional and must only be imported if it is installed
import psutil
self.process = psutil.Process()
else:
self.process = None
def get_prefix(self) -> str:
return PREFIX
def get_supported_metric_names(self) -> List[str]:
return [THREAD_COUNT] if self.process else []
def get_metric(self, metric_name: str) -> Metric:
measurements = [Measurement("COUNT", self.process.num_threads())] if self.process else []
return Metric(metric_name, None, "Integer", measurements, [])
================================================
FILE: pyctuator/py.typed
================================================
================================================
FILE: pyctuator/pyctuator.py
================================================
# pylint: disable=import-outside-toplevel
import atexit
import importlib.util
import logging
import ssl
from datetime import datetime, timezone
from typing import Any, Optional, Dict, Callable
# A note about imports: this module ensure that only relevant modules are imported.
# For example, if the webapp is a Flask webapp, we do not want to import FastAPI, and vice versa.
# To do that, all imports are in conditional branches after detecting which frameworks are installed.
# DO NOT add any web-framework-dependent imports to the global scope.
from pyctuator.auth import Auth
from pyctuator.endpoints import Endpoints
from pyctuator.environment.custom_environment_provider import CustomEnvironmentProvider
from pyctuator.environment.os_env_variables_impl import OsEnvironmentVariableProvider
from pyctuator.health.diskspace_health_impl import DiskSpaceHealthProvider
from pyctuator.health.health_provider import HealthProvider
from pyctuator.metrics.memory_metrics_impl import MemoryMetricsProvider
from pyctuator.metrics.thread_metrics_impl import ThreadMetricsProvider
from pyctuator.impl.pyctuator_impl import PyctuatorImpl, AppInfo, BuildInfo, GitInfo, GitCommitInfo, AppDetails
from pyctuator.impl.spring_boot_admin_registration import BootAdminRegistrationHandler
default_logfile_format = '%(asctime)s %(levelname)-5s %(process)d -- [%(threadName)s] %(module)s: %(message)s'
class Pyctuator:
# pylint: disable=too-many-locals
def __init__(
self,
app: Any,
app_name: str,
app_url: str,
pyctuator_endpoint_url: str,
registration_url: Optional[str],
registration_auth: Optional[Auth] = None,
app_description: Optional[str] = None,
registration_interval_sec: float = 10,
free_disk_space_down_threshold_bytes: int = 1024 * 1024 * 100,
logfile_max_size: int = 10000,
logfile_formatter: str = default_logfile_format,
auto_deregister: bool = True,
metadata: Optional[dict] = None,
additional_app_info: Optional[dict] = None,
ssl_context: Optional[ssl.SSLContext] = None,
customizer: Optional[Callable] = None,
disabled_endpoints: Endpoints = Endpoints.NONE,
) -> None:
"""The entry point for integrating pyctuator with a web-frameworks such as FastAPI and Flask.
Given an application built on top of a supported web-framework, it'll add to the application the REST API
endpoints that required for Spring Boot Admin to monitor and manage the application.
Pyctuator currently supports application built on top of FastAPI and Flask. The type of first argument, app is
specific to the target web-framework:
* FastAPI - `app` is an instance of `fastapi.applications.FastAPI`
* Flask - `app` is an instance of `flask.app.Flask`
* aiohttp - `app` is an instance of `aiohttp.web.Application`
* Tornado - `app` is an instance of `tornado.web.Application`
:param app: an instance of a supported web-framework with which the pyctuator endpoints will be registered
:param app_name: the application's name that will be presented in the "Info" section in boot-admin
:param app_description: a description that will be presented in the "Info" section in boot-admin
:param app_url: the full URL of the application being monitored which will be displayed in spring-boot-admin, we
recommend this URL to be accessible by those who manage the application (i.e. don't use "http://localhost..."
as it is only accessible from within the application's host)
:param pyctuator_endpoint_url: the public URL from which Pyctuator REST API will be accessible, used for
registering the application with spring-boot-admin, must be accessible from spring-boot-admin server (i.e.
don't use http://localhost:8080/... unless spring-boot-admin is running on the same host as the monitored
application)
:param registration_url: the spring-boot-admin endpoint to which registration requests must be posted
:param registration_auth: optional authentication details to use when registering with spring-boot-admin
:param registration_interval_sec: how often pyctuator will renew its registration with spring-boot-admin
:param free_disk_space_down_threshold_bytes: amount of free space in bytes in "./" (the application's current
working directory) below which the built-in disk-space health-indicator will fail
:param auto_deregister: if true, pyctuator will automatically deregister from SBA during shutdown, needed for
example when running in k8s so every time a new pod is created it is assigned a different IP address, resulting
with SBA showing "offline" instances
:param metadata: optional metadata key-value pairs that are displayed in SBA main page of an instance
:param additional_app_info: additional arbitrary information to add to the application's "Info" section
:param ssl_context: optional SSL context to be used when registering with SBA
:param customizer: a function that can customize the integration with the web-framework which is therefore web-
framework specific. For FastAPI, the function receives pyctuator's APIRouter allowing to add "dependencies" and
anything else that's provided by the router. See fastapi_with_authentication_example_app.py
:param disabled_endpoints: optional set of endpoints (such as /pyctuator/health) that should be disabled
"""
self.auto_deregister = auto_deregister
start_time = datetime.now(timezone.utc)
# Instantiate an instance of PyctuatorImpl which abstracts the state and logic of the pyctuator
self.pyctuator_impl = PyctuatorImpl(
AppInfo(app=AppDetails(name=app_name, description=app_description)),
pyctuator_endpoint_url,
logfile_max_size,
logfile_formatter,
additional_app_info,
disabled_endpoints,
)
# Register default health/metrics/environment providers
self.pyctuator_impl.register_environment_provider(OsEnvironmentVariableProvider())
self.pyctuator_impl.register_health_providers(DiskSpaceHealthProvider(free_disk_space_down_threshold_bytes))
self.pyctuator_impl.register_metrics_provider(MemoryMetricsProvider())
self.pyctuator_impl.register_metrics_provider(ThreadMetricsProvider())
self.boot_admin_registration_handler: Optional[BootAdminRegistrationHandler] = None
self.metadata = metadata
self.ssl_context = ssl_context
root_logger = logging.getLogger()
# If application did not initiate logging module, add default handler to root logger
# logging.info implicitly calls logging.basicConfig(), see logging.basicConfig in Python's documentation.
if not root_logger.hasHandlers():
logging.info("Logging not configured, using logging.basicConfig()")
root_logger.addHandler(self.pyctuator_impl.logfile.log_messages)
# Find and initialize an integration layer between the web-framework adn pyctuator
framework_integrations: Dict[str, Callable[[Any, PyctuatorImpl, Optional[Callable], Endpoints], bool]] = {
"flask": self._integrate_flask,
"fastapi": self._integrate_fastapi,
"aiohttp": self._integrate_aiohttp,
"tornado": self._integrate_tornado
}
for framework_name, framework_integration_function in framework_integrations.items():
if self._is_framework_installed(framework_name):
logging.debug("Framework %s is installed, trying to integrate with it", framework_name)
success = framework_integration_function(app, self.pyctuator_impl, customizer, disabled_endpoints)
if success:
logging.debug("Integrated with framework %s", framework_name)
if registration_url is not None:
self.boot_admin_registration_handler = BootAdminRegistrationHandler(
registration_url,
registration_auth,
app_name,
self.pyctuator_impl.pyctuator_endpoint_url,
start_time,
app_url,
registration_interval_sec,
self.metadata,
self.ssl_context,
)
# Deregister from SBA on exit
if self.auto_deregister:
atexit.register(self.boot_admin_registration_handler.deregister_from_admin_server)
self.boot_admin_registration_handler.start()
return
# Fail in case no framework was found for the target app
raise EnvironmentError("No framework was found that is matching the target app "
"(is it properly installed and imported?)")
def stop(self) -> None:
if self.boot_admin_registration_handler:
self.boot_admin_registration_handler.stop()
self.boot_admin_registration_handler = None
def set_secret_scrubber(self, secret_scrubber: Callable[[Dict], Dict]) -> None:
"""Overrides the default secret scrubber with a custom one. See SecretScrubber for example scrubber."""
self.pyctuator_impl.set_secret_scrubber(secret_scrubber)
def register_environment_provider(self, name: str, env_provider: Callable[[], Dict]) -> None:
self.pyctuator_impl.register_environment_provider(CustomEnvironmentProvider(name, env_provider))
def register_health_provider(self, provider: HealthProvider) -> None:
self.pyctuator_impl.register_health_providers(provider)
def set_git_info(self, commit: str, time: datetime, branch: Optional[str] = None) -> None:
self.pyctuator_impl.set_git_info(GitInfo(GitCommitInfo(time, commit), branch))
def set_build_info(
self,
artifact: Optional[str] = None,
group: Optional[str] = None,
name: Optional[str] = None,
version: Optional[str] = None,
time: Optional[datetime] = None,
) -> None:
self.pyctuator_impl.set_build_info(BuildInfo(name, artifact, group, version, time))
def _is_framework_installed(self, framework_name: str) -> bool:
return importlib.util.find_spec(framework_name) is not None
def _integrate_fastapi(
self,
app: Any,
pyctuator_impl: PyctuatorImpl,
customizer: Optional[Callable],
disabled_endpoints: Endpoints,
) -> bool:
"""
This method should only be called if we detected that FastAPI is installed.
It will then check whether the given app is a FastAPI app, and if so - it will add the Pyctuator
endpoints to it.
"""
from fastapi import FastAPI
if isinstance(app, FastAPI):
from pyctuator.impl.fastapi_pyctuator import FastApiPyctuator
FastApiPyctuator(app, pyctuator_impl, False, customizer, disabled_endpoints)
return True
return False
# pylint: disable=unused-argument
def _integrate_flask(
self,
app: Any,
pyctuator_impl: PyctuatorImpl,
customizer: Optional[Callable],
disabled_endpoints: Endpoints,
) -> bool:
"""
This method should only be called if we detected that Flask is installed.
It will then check whether the given app is a Flask app, and if so - it will add the Pyctuator
endpoints to it.
"""
from flask import Flask
if isinstance(app, Flask):
from pyctuator.impl.flask_pyctuator import FlaskPyctuator
FlaskPyctuator(app, pyctuator_impl, disabled_endpoints)
return True
return False
# pylint: disable=unused-argument
def _integrate_aiohttp(
self,
app: Any,
pyctuator_impl: PyctuatorImpl,
customizer: Optional[Callable],
disabled_endpoints: Endpoints,
) -> bool:
"""
This method should only be called if we detected that aiohttp is installed.
It will then check whether the given app is a aiohttp app, and if so - it will add the Pyctuator
endpoints to it.
"""
from aiohttp.web import Application
if isinstance(app, Application):
from pyctuator.impl.aiohttp_pyctuator import AioHttpPyctuator
AioHttpPyctuator(app, pyctuator_impl, disabled_endpoints)
return True
return False
# pylint: disable=unused-argument
def _integrate_tornado(
self,
app: Any,
pyctuator_impl: PyctuatorImpl,
customizer: Optional[Callable],
disabled_endpoints: Endpoints,
) -> bool:
"""
This method should only be called if we detected that tornado is installed.
It will then check whether the given app is a tornado app, and if so - it will add the Pyctuator
endpoints to it.
"""
from tornado.web import Application
if isinstance(app, Application):
from pyctuator.impl.tornado_pyctuator import TornadoHttpPyctuator
TornadoHttpPyctuator(app, pyctuator_impl, disabled_endpoints)
return True
return False
================================================
FILE: pyctuator/threads/__init__.py
================================================
================================================
FILE: pyctuator/threads/thread_dump_provider.py
================================================
import sys
import threading
from threading import Thread
from dataclasses import dataclass
from pathlib import Path
from typing import List, Dict, Any, Optional
@dataclass
class StackFrame:
methodName: str
fileName: str
lineNumber: int
className: Optional[str]
nativeMethod: bool
@dataclass
class ThreadInfo:
threadName: str
threadId: Optional[int]
daemon: bool
suspended: bool
threadState: str
stackTrace: List[StackFrame]
@dataclass
class ThreadDump:
threads: List[ThreadInfo]
class ThreadDumpProvider:
# pylint: disable=protected-access
def get_thread_dump(self) -> ThreadDump:
frames: Dict[Any, Any] = sys._current_frames()
return ThreadDump([
self._extract_thread_info(frames, thread)
for thread in threading.enumerate()
])
def _extract_thread_info(self, frames: Dict[Any, Any], thread: Thread) -> ThreadInfo:
return ThreadInfo(
threadName=thread.name,
threadId=thread.ident,
daemon=thread.daemon,
suspended=not thread.is_alive(),
threadState=self._calc_thread_state(thread),
stackTrace=self._build_thread_stack_trace(thread, frames),
)
def _build_thread_stack_trace(self, thread: Thread, frames: Dict[Any, Any]) -> List[StackFrame]:
def guess_class_name() -> Optional[str]:
"""
Tries to find a class name if one exists.
Fails if the frame is not in a class, or if the method does not call itself "self"
Does not support static and class methods.
"""
try:
return str(frame.f_locals["self"].__class__.__name__)
except KeyError:
return None
stack_frames = []
frame = frames[thread.ident] if thread.ident in frames else None
while frame is not None:
stack_frames.append(StackFrame(
methodName=frame.f_code.co_name,
fileName=Path(frame.f_code.co_filename).name,
lineNumber=frame.f_lineno,
className=guess_class_name(),
nativeMethod=False
))
frame = frame.f_back # move one frame back
return stack_frames
def _calc_thread_state(self, thread: threading.Thread) -> str:
if thread.ident and thread.ident < 0:
return "NEW"
return "RUNNABLE"
================================================
FILE: pyproject.toml
================================================
[tool.poetry]
name = "pyctuator"
version = "1.2.0"
description = "A Python implementation of the Spring Actuator API for popular web frameworks"
authors = [
"Michael Yakobi <michael.yakobi@solaredge.com>",
"Inbal Levi <inbal.levi@solaredge.com>",
"Yanay Reingewertz <yanay.reingewertz@solaredge.com>",
"Matan Rubin <matan.rubin@solaredge.com>"
]
maintainers = [
"Matan Rubin <matan.rubin@solaredge.com>",
"Michael Yakobi <michael.yakobi@solaredge.com>"
]
readme = "README.md"
homepage = "https://github.com/SolarEdgeTech/pyctuator"
repository = "https://github.com/SolarEdgeTech/pyctuator"
keywords = ["spring boot admin", "actuator", "pyctuator", "fastapi", "flask", "aiohttp", "tornado"]
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Web Environment",
"Framework :: Flask",
"Framework :: FastAPI",
"Framework :: aiohttp",
"Intended Audience :: Developers",
"Programming Language :: Python :: 3.9",
"Topic :: Internet :: WWW/HTTP :: HTTP Servers",
"Topic :: Internet :: WWW/HTTP :: WSGI :: Application",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: System :: Monitoring",
"Typing :: Typed",
"License :: OSI Approved :: Apache Software License",
]
[tool.poetry.dependencies]
python = "^3.9"
psutil = { version = "^5.6", optional = true }
flask = { version = "^2.3.0", optional = true }
fastapi = { version = "^0.100.1", optional = true }
uvicorn = { version = "^0.23.0", optional = true }
sqlalchemy = {version = "^2.0.4", optional = true}
PyMySQL = {version = "^1.0.2", optional = true}
cryptography = {version = ">=39.0.1,<40.0.0", optional = true}
redis = {version = "^4.3.4", optional = true}
aiohttp = {version = "^3.6.2", optional = true}
tornado = {version = "^6.0.4", optional = true}
[tool.poetry.dev-dependencies]
requests = "^2.22"
pytest = "^7.1.3"
mypy = "^1.0.1"
pylint = "^2.15.0" # v2.5 does not properly run on docker image...
pytest-cov = "^4.0.0"
autopep8 = "^2.0.0"
[tool.poetry.extras]
psutil = ["psutil"]
fastapi = ["fastapi", "uvicorn"]
flask = ["flask"]
aiohttp = ["aiohttp"]
tornado = ["tornado"]
db = ["sqlalchemy", "PyMySQL", "cryptography"]
redis = ["redis"]
[build-system]
requires = ["poetry>=1.1"]
build-backend = "poetry.masonry.api"
================================================
FILE: tests/__init__.py
================================================
================================================
FILE: tests/aiohttp_test_server.py
================================================
import asyncio
import logging
import threading
import time
from aiohttp import web
from pyctuator.endpoints import Endpoints
from pyctuator.pyctuator import Pyctuator
from tests.conftest import PyctuatorServer
bind_port = 6000
# mypy: ignore_errors
# pylint: disable=unused-variable
class AiohttpPyctuatorServer(PyctuatorServer):
def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> None:
global bind_port
self.port = bind_port
bind_port += 1
self.app = web.Application()
self.routes = web.RouteTableDef()
self.pyctuator = Pyctuator(
self.app,
"AIOHTTP Pyctuator",
f"http://localhost:{self.port}",
f"http://localhost:{self.port}/pyctuator",
"http://localhost:8001/register",
registration_interval_sec=1,
metadata=self.metadata,
additional_app_info=self.additional_app_info,
disabled_endpoints=disabled_endpoints,
)
@self.routes.get("/logfile_test_repeater")
async def logfile_test_repeater(request: web.Request) -> web.Response:
repeated_string = request.query.get("repeated_string")
logging.error(repeated_string)
return web.Response(text=repeated_string)
@self.routes.get("/httptrace_test_url")
async def get_httptrace_test_url(request: web.Request) -> web.Response:
# Sleep if requested to sleep - used for asserting httptraces timing
sleep_sec = request.query.get("sleep_sec")
if sleep_sec:
logging.info("Sleeping %s seconds before replying", sleep_sec)
time.sleep(int(sleep_sec))
# Echo 'User-Data' header as 'resp-data' - used for asserting headers are captured properly
headers = {
"resp-data": str(request.headers.get("User-Data")),
"response-secret": "my password"
}
return web.Response(headers=headers, body="my content")
self.app.add_routes(self.routes)
self.thread = threading.Thread(target=self._start_in_thread)
self.should_stop_server = False
self.server_started = False
async def _run_server(self) -> None:
logging.info("Preparing to start aiohttp server")
runner = web.AppRunner(self.app)
await runner.setup()
logging.info("Starting aiohttp server")
site = web.TCPSite(runner, port=self.port)
await site.start()
self.server_started = True
logging.info("aiohttp server started")
while not self.should_stop_server:
await asyncio.sleep(1)
logging.info("Shutting down aiohttp server")
await runner.shutdown()
await runner.cleanup()
logging.info("aiohttp server is shutdown")
def _start_in_thread(self) -> None:
loop = asyncio.new_event_loop()
loop.run_until_complete(self._run_server())
loop.stop()
def start(self) -> None:
self.thread.start()
while not self.server_started:
time.sleep(0.01)
def stop(self) -> None:
logging.info("Stopping aiohttp server")
self.pyctuator.stop()
self.should_stop_server = True
self.thread.join()
logging.info("aiohttp server stopped")
def atexit(self) -> None:
if self.pyctuator.boot_admin_registration_handler:
self.pyctuator.boot_admin_registration_handler.deregister_from_admin_server()
================================================
FILE: tests/conftest.py
================================================
import logging
import random
import secrets
import threading
import time
from abc import ABC, abstractmethod
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Generator, Optional, Dict
import pytest
import requests
from fastapi import FastAPI, Depends, HTTPException
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from pydantic import BaseModel
from starlette import status
from uvicorn.config import Config
from uvicorn.main import Server
from pyctuator.endpoints import Endpoints
REQUEST_TIMEOUT = 10
class RegistrationRequest(BaseModel):
name: str
managementUrl: str
healthUrl: str
serviceUrl: str
metadata: dict
@dataclass
# pylint: disable=too-many-instance-attributes
class RegisteredEndpoints:
root: str
pyctuator: str
env: Optional[str]
info: Optional[str]
health: Optional[str]
metrics: Optional[str]
loggers: Optional[str]
threads: Optional[str]
logfile: Optional[str]
httptrace: Optional[str]
@dataclass
class RegistrationTrackerFixture:
registration: Optional[RegistrationRequest]
count: int
start_time: Optional[str]
deregistration_time: Optional[datetime]
test_start_time: datetime
endpoint_href_path = {
Endpoints.ENV: "env",
Endpoints.INFO: "info",
Endpoints.HEALTH: "health",
Endpoints.METRICS: "metrics",
Endpoints.LOGGERS: "loggers",
Endpoints.THREAD_DUMP: "threaddump",
Endpoints.LOGFILE: "logfile",
Endpoints.HTTP_TRACE: "httptrace",
}
class CustomServer(Server):
def install_signal_handlers(self) -> None:
pass
@pytest.fixture
def registration_tracker() -> RegistrationTrackerFixture:
return RegistrationTrackerFixture(None, 0, None, None, datetime.now(timezone.utc))
@pytest.fixture
def boot_admin_server(registration_tracker: RegistrationTrackerFixture) -> Generator:
boot_admin_app = FastAPI(
title="Boot Admin Mock Server",
description="Demonstrate Spring Boot Admin Integration with FastAPI",
docs_url="/api",
)
security = HTTPBasic()
def get_current_username(credentials: HTTPBasicCredentials = Depends(security)) -> str:
correct_username = secrets.compare_digest(credentials.username, "moo")
correct_password = secrets.compare_digest(credentials.password, "haha")
if not (correct_username and correct_password):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Moo haha",
)
return credentials.username
# pylint: disable=unused-variable
@boot_admin_app.post("/register", tags=["admin-server"])
def register(registration: RegistrationRequest) -> Dict[str, str]:
logging.debug("Got registration post %s, %d registrations since %s",
registration, registration_tracker.count, registration_tracker.start_time)
registration_tracker.registration = registration
registration_tracker.count += 1
if registration_tracker.start_time is None:
registration_tracker.start_time = registration.metadata["startup"]
return {"id": "JB007"}
# pylint: disable=unused-variable
@boot_admin_app.post("/register-with-basic-auth", tags=["admin-server"])
def register_with_basic_auth(
registration: RegistrationRequest,
username: str = Depends(get_current_username)) -> Dict[str, str]:
logging.debug("Got registration post %s from %s, %d registrations since %s",
registration, username, registration_tracker.count, registration_tracker.start_time)
registration_tracker.registration = registration
registration_tracker.count += 1
if registration_tracker.start_time is None:
registration_tracker.start_time = registration.metadata["startup"]
return {"id": "JB007"}
# pylint: disable=unused-argument,unused-variable
@boot_admin_app.delete("/register/{registration_id}", tags=["admin-server"])
def deregister(registration_id: str) -> None:
logging.debug("Got deregistration, delete %s (previous deregistration time is %s)",
registration_id, registration_tracker.deregistration_time)
registration_tracker.deregistration_time = datetime.now(timezone.utc)
# Start the mock boot-admin server that is needed to test pyctuator's registration
boot_admin_config = Config(app=boot_admin_app, port=8001, lifespan="off", log_level="info")
boot_admin_server = CustomServer(config=boot_admin_config)
boot_admin_thread = threading.Thread(target=boot_admin_server.run)
boot_admin_thread.start()
while not boot_admin_server.started:
time.sleep(0.01)
logging.info("Spring-boot-admin mock-server started")
# Yield back to pytest until the module is done
yield None
logging.info("Stopping spring-boot-admin mock-server")
boot_admin_server.should_exit = True
boot_admin_server.force_exit = True
boot_admin_thread.join()
@pytest.mark.usefixtures("boot_admin_server")
@pytest.fixture
def registered_endpoints(registration_tracker: RegistrationTrackerFixture) -> RegisteredEndpoints:
# time.sleep(600)
# Wait for pyctuator to register with the boot-admin at least once
while registration_tracker.registration is None:
time.sleep(0.01)
assert isinstance(registration_tracker.registration, RegistrationRequest)
response = requests.get(registration_tracker.registration.managementUrl, timeout=REQUEST_TIMEOUT)
assert response.status_code == 200
links = response.json()["_links"]
def link_href(endpoint: Endpoints) -> Optional[str]:
link = endpoint_href_path.get(endpoint)
return str(links[link]["href"]) if link in links else None
return RegisteredEndpoints(
root=registration_tracker.registration.serviceUrl,
pyctuator=links["self"]["href"],
env=link_href(Endpoints.ENV),
info=link_href(Endpoints.INFO),
health=link_href(Endpoints.HEALTH),
metrics=link_href(Endpoints.METRICS),
loggers=link_href(Endpoints.LOGGERS),
threads=link_href(Endpoints.THREAD_DUMP),
logfile=link_href(Endpoints.LOGFILE),
httptrace=link_href(Endpoints.HTTP_TRACE),
)
class PyctuatorServer(ABC):
metadata: Optional[dict] = {f"k{i}": f"v{i}" for i in range(random.randrange(10))}
additional_app_info = {
"serviceLinks": {
"metrics": "http://xyz/service/metrics",
},
"podLinks": {
"metrics": ["http://xyz/pod/metrics/memory", "http://xyz/pod/metrics/cpu"],
},
}
@abstractmethod
def start(self) -> None:
pass
@abstractmethod
def stop(self) -> None:
pass
@abstractmethod
def atexit(self) -> None:
pass
================================================
FILE: tests/environment/__init__.py
================================================
================================================
FILE: tests/environment/test_custom_environment_provider.py
================================================
from typing import Dict
from pyctuator.environment.custom_environment_provider import CustomEnvironmentProvider
from pyctuator.environment.environment_provider import PropertyValue
from pyctuator.environment.scrubber import SecretScrubber
def test_custom_environment_provider() -> None:
def produce_env() -> Dict:
return {
"a": "s1",
"b": {
"secret": "ha ha",
"c": 625,
},
"d": {
"e": True,
"f": "hello",
"g": {
"h": 123,
"i": "abcde"
}
}
}
provider = CustomEnvironmentProvider("custom", produce_env)
properties_source = provider.get_properties_source(SecretScrubber().scrub_secrets)
assert properties_source.name == "custom"
assert properties_source.properties == {
"a": PropertyValue(value="s1"),
"b.secret": PropertyValue(value="******"),
"b.c": PropertyValue(value=625),
"d.e": PropertyValue(value=True),
"d.f": PropertyValue(value="hello"),
"d.g.h": PropertyValue(value=123),
"d.g.i": PropertyValue(value="abcde"),
}
================================================
FILE: tests/environment/test_scrubber.py
================================================
import re
from pyctuator.environment.scrubber import SecretScrubber
def test_scrub_secrets() -> None:
with_secrets = {
"some.value": "Good",
"another.value": 10,
"another.value.and.another": 10.0,
"a.boolean": True,
"some.api_key": "Bad",
"a.key": "Bad",
"a.keyboard": "Good",
"db.url.1": "mysql+pymysql://user:Bad@host:3306/schema",
"db.url.2": "mysql+pymysql://joe:Bad@host/schema",
"db.url.3": "mysql+pymysql://joe:Bad@host",
"db.url.4": "mysql+pymysql://host",
}
expected_without_secrets = {
"some.value": "Good",
"another.value": 10,
"another.value.and.another": 10.0,
"a.boolean": True,
"some.api_key": "******",
"a.key": "******",
"a.keyboard": "Good",
"db.url.1": "mysql+pymysql://user:******@host:3306/schema",
"db.url.2": "mysql+pymysql://joe:******@host/schema",
"db.url.3": "mysql+pymysql://joe:******@host",
"db.url.4": "mysql+pymysql://host",
}
scrubbed = SecretScrubber().scrub_secrets(with_secrets)
assert scrubbed == expected_without_secrets
def test_custom_scrub_secrets() -> None:
with_secrets = {
"some.value": "Good",
"another.value": 10,
"another.value.and.another": 10.0,
"a.boolean": True,
"some.api_key": "Bad",
"a.key": "Bad",
"a.keyboard": "Good",
}
expected_without_secrets = {
"some.value": "******",
"another.value": 10,
"another.value.and.another": 10.0,
"a.boolean": "******",
"some.api_key": "Bad",
"a.key": "Bad",
"a.keyboard": "Good",
}
scrubbed = SecretScrubber(keys_to_scrub=re.compile("^SOME.VALUE$|^a.BOOlean$", re.IGNORECASE))\
.scrub_secrets(with_secrets)
assert scrubbed == expected_without_secrets
================================================
FILE: tests/fast_api_test_server.py
================================================
import logging
import threading
import time
from typing import Optional
from fastapi import FastAPI
from starlette.requests import Request
from starlette.responses import Response
from uvicorn.config import Config
from pyctuator.endpoints import Endpoints
from pyctuator.pyctuator import Pyctuator
from tests.conftest import PyctuatorServer, CustomServer
bind_port = 7000
class FastApiPyctuatorServer(PyctuatorServer):
def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> None:
global bind_port
self.port = bind_port
bind_port += 1
self.app = FastAPI(
title="FastAPI Example Server",
description="Demonstrate Spring Boot Admin Integration with FastAPI",
docs_url="/api",
)
self.pyctuator = Pyctuator(
self.app,
"FastAPI Pyctuator",
f"http://localhost:{self.port}",
f"http://localhost:{self.port}/pyctuator",
"http://localhost:8001/register",
registration_interval_sec=1,
metadata=self.metadata,
additional_app_info=self.additional_app_info,
disabled_endpoints=disabled_endpoints,
)
@self.app.get("/logfile_test_repeater", tags=["pyctuator"])
# pylint: disable=unused-variable
def logfile_test_repeater(repeated_string: str) -> str:
logging.error(repeated_string)
return repeated_string
self.server = CustomServer(config=Config(app=self.app, port=self.port, lifespan="off", log_level="info"))
self.thread = threading.Thread(target=self.server.run)
@self.app.get("/httptrace_test_url")
# pylint: disable=unused-variable
def get_httptrace_test_url(request: Request, sleep_sec: Optional[int] = None) -> Response:
# Sleep if requested to sleep - used for asserting httptraces timing
if sleep_sec:
logging.info("Sleeping %s seconds before replying", sleep_sec)
time.sleep(sleep_sec)
# Echo 'User-Data' header as 'resp-data' - used for asserting headers are captured properly
headers = {
"resp-data": str(request.headers.get("User-Data")),
"response-secret": "my password"
}
return Response(headers=headers, content="my content")
def start(self) -> None:
self.thread.start()
while not self.server.started:
time.sleep(0.01)
def stop(self) -> None:
logging.info("Stopping FastAPI server")
self.pyctuator.stop()
# Allow the recurring registration to complete any in-progress request before stopping FastAPI
time.sleep(1)
self.server.should_exit = True
self.server.force_exit = True
self.thread.join()
logging.info("FastAPI server stopped")
def atexit(self) -> None:
if self.pyctuator.boot_admin_registration_handler:
self.pyctuator.boot_admin_registration_handler.deregister_from_admin_server()
================================================
FILE: tests/flask_test_server.py
================================================
import logging
import threading
import time
from wsgiref.simple_server import make_server
import requests
from flask import Flask, request, Response
from pyctuator.endpoints import Endpoints
from pyctuator.pyctuator import Pyctuator
from tests.conftest import PyctuatorServer
REQUEST_TIMEOUT = 10
bind_port = 5000
class FlaskPyctuatorServer(PyctuatorServer):
def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> None:
global bind_port
self.port = bind_port
bind_port += 1
self.app = Flask("Flask Example Server")
self.server = make_server('127.0.0.1', self.port, self.app)
self.ctx = self.app.app_context()
self.ctx.push()
self.thread = threading.Thread(target=self.server.serve_forever)
self.pyctuator = Pyctuator(
self.app,
"Flask Pyctuator",
f"http://localhost:{self.port}",
f"http://localhost:{self.port}/pyctuator",
"http://localhost:8001/register",
registration_interval_sec=1,
metadata=self.metadata,
additional_app_info=self.additional_app_info,
disabled_endpoints=disabled_endpoints,
)
@self.app.route("/logfile_test_repeater")
# pylint: disable=unused-variable
def logfile_test_repeater() -> str:
repeated_string: str = str(request.args.get("repeated_string"))
logging.error(repeated_string)
return repeated_string
@self.app.route("/httptrace_test_url", methods=["GET"])
# pylint: disable=unused-variable
def get_httptrace_test_url() -> Response:
# Sleep if requested to sleep - used for asserting httptraces timing
sleep_sec = request.args.get("sleep_sec")
if sleep_sec:
logging.info("Sleeping %s seconds before replying", sleep_sec)
time.sleep(int(sleep_sec))
# Echo 'User-Data' header as 'resp-data' - used for asserting headers are captured properly
resp = Response()
resp.headers["resp-data"] = str(request.headers.get("User-Data"))
resp.headers["response-secret"] = str(
request.headers.get("my password"))
return resp
def start(self) -> None:
logging.info("Starting Flask server")
self.thread.start()
while True:
time.sleep(0.5)
try:
requests.get(f"http://localhost:{self.port}/pyctuator", timeout=REQUEST_TIMEOUT)
logging.info("Flask server started")
return
except requests.exceptions.RequestException: # Catches all exceptions that Requests raises!
pass
def stop(self) -> None:
logging.info("Stopping Flask server")
self.pyctuator.stop()
self.server.shutdown()
self.thread.join()
logging.info("Flask server stopped")
def atexit(self) -> None:
if self.pyctuator.boot_admin_registration_handler:
self.pyctuator.boot_admin_registration_handler.deregister_from_admin_server()
================================================
FILE: tests/health/__init__.py
================================================
================================================
FILE: tests/health/test_composite_health_provider.py
================================================
from dataclasses import dataclass
from pyctuator.health.composite_health_provider import CompositeHealthProvider, CompositeHealthStatus
from pyctuator.health.health_provider import HealthProvider, HealthStatus, Status, HealthDetails
@dataclass
class CustomHealthDetails(HealthDetails):
details: str
class CustomHealthProvider(HealthProvider):
def __init__(self, name: str, status: HealthStatus) -> None:
super().__init__()
self.name = name
self.status = status
def is_supported(self) -> bool:
return True
def get_name(self) -> str:
return self.name
def get_health(self) -> HealthStatus:
return self.status
def test_composite_health_provider_no_providers() -> None:
health_provider = CompositeHealthProvider(
"comp1",
)
assert health_provider.get_name() == "comp1"
assert health_provider.get_health() == CompositeHealthStatus(
status=Status.UP,
details={}
)
def test_composite_health_provider_all_up() -> None:
health_provider = CompositeHealthProvider(
"comp2",
CustomHealthProvider("hp1", HealthStatus(Status.UP, CustomHealthDetails("d1"))),
CustomHealthProvider("hp2", HealthStatus(Status.UP, CustomHealthDetails("d2"))),
)
assert health_provider.get_name() == "comp2"
assert health_provider.get_health() == CompositeHealthStatus(
status=Status.UP,
details={
"hp1": HealthStatus(Status.UP, CustomHealthDetails("d1")),
"hp2": HealthStatus(Status.UP, CustomHealthDetails("d2")),
}
)
def test_composite_health_provider_one_down() -> None:
health_provider = CompositeHealthProvider(
"comp3",
CustomHealthProvider("hp1", HealthStatus(Status.UP, CustomHealthDetails("d1"))),
CustomHealthProvider("hp2", HealthStatus(Status.DOWN, CustomHealthDetails("d2"))),
)
assert health_provider.get_name() == "comp3"
assert health_provider.get_health() == CompositeHealthStatus(
status=Status.DOWN,
details={
"hp1": HealthStatus(Status.UP, CustomHealthDetails("d1")),
"hp2": HealthStatus(Status.DOWN, CustomHealthDetails("d2")),
}
)
================================================
FILE: tests/health/test_db_health_provider.py
================================================
# pylint: disable=import-outside-toplevel
import importlib.util
import os
import pytest
@pytest.fixture
def require_sql_alchemy() -> None:
if not importlib.util.find_spec("sqlalchemy"):
pytest.skip("sqlalchemy is missing, skipping")
@pytest.fixture
def require_pymysql() -> None:
if not importlib.util.find_spec("pymysql"):
pytest.skip("PyMySQL is missing, skipping")
@pytest.fixture
def require_mysql_server() -> None:
should_test_with_mysql = os.getenv("TEST_MYSQL_SERVER", None)
if not should_test_with_mysql:
pytest.skip("No MySQL server (env TEST_MYSQL_SERVER isn't True), skipping")
@pytest.mark.usefixtures("require_sql_alchemy")
def test_sqlite_health() -> None:
from sqlalchemy import create_engine
from pyctuator.health.db_health_provider import DbHealthProvider, DbHealthDetails, DbHealthStatus
from pyctuator.health.health_provider import Status
engine = create_engine("sqlite:///:memory:", echo=True)
health_provider = DbHealthProvider(engine)
assert health_provider.get_health() == DbHealthStatus(status=Status.UP, details=DbHealthDetails("sqlite"))
assert health_provider.get_name() == "db"
assert DbHealthProvider(engine, "kuki").get_name() == "kuki"
@pytest.mark.usefixtures("require_sql_alchemy", "require_pymysql", "require_mysql_server")
def test_mysql_health() -> None:
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from pyctuator.health.db_health_provider import DbHealthProvider, DbHealthDetails, DbHealthStatus
from pyctuator.health.health_provider import Status
engine: Engine = create_engine("mysql+pymysql://root:root@localhost:3306", echo=True)
health_provider = DbHealthProvider(engine)
assert health_provider.get_health() == DbHealthStatus(status=Status.UP, details=DbHealthDetails("mysql"))
engine = create_engine("mysql+pymysql://kukipuki:blahblah@localhost:3306", echo=True)
health_provider = DbHealthProvider(engine)
health = health_provider.get_health()
assert health.status == Status.DOWN
details: DbHealthDetails = health.details
assert details.failure is not None
assert "Access denied for user" in details.failure
================================================
FILE: tests/health/test_health_status.py
================================================
import pytest
from pyctuator.endpoints import Endpoints
from pyctuator.health.health_provider import HealthStatus, Status, HealthDetails, HealthProvider
from pyctuator.impl.pyctuator_impl import PyctuatorImpl, AppInfo, AppDetails
from pyctuator.pyctuator import default_logfile_format
class MyHealthProvider(HealthProvider):
def __init__(self, name: str = "kuki") -> None:
self.name = name
self.status = Status.UNKNOWN
def down(self) -> None:
self.status = Status.DOWN
def up(self) -> None:
self.status = Status.UP
def is_supported(self) -> bool:
return True
def get_health(self) -> HealthStatus:
return HealthStatus(self.status, HealthDetails())
def get_name(self) -> str:
return self.name
@pytest.fixture
def pyctuator_impl() -> PyctuatorImpl:
return PyctuatorImpl(
AppInfo(app=AppDetails(name="appy")),
"http://appy/pyctuator",
10,
default_logfile_format,
None,
Endpoints.NONE,
)
def test_health_status_single_provider(pyctuator_impl: PyctuatorImpl) -> None:
health_provider = MyHealthProvider()
pyctuator_impl.register_health_providers(health_provider)
# Test's default status is UNKNOWN
assert pyctuator_impl.get_health().status == Status.UNKNOWN
health_provider.down()
assert pyctuator_impl.get_health().status == Status.DOWN
health_provider.up()
assert pyctuator_impl.get_health().status == Status.UP
def test_health_status_multiple_providers(pyctuator_impl: PyctuatorImpl) -> None:
health_providers = [MyHealthProvider("kuki"), MyHealthProvider("puki"), MyHealthProvider("ruki")]
for health_provider in health_providers:
pyctuator_impl.register_health_providers(health_provider)
# Test's default status is UNKNOWN - all 3 are UNKNOWN
assert pyctuator_impl.get_health().status == Status.UNKNOWN
health_providers[0].down()
assert pyctuator_impl.get_health().status == Status.DOWN
health_providers[0].up()
assert pyctuator_impl.get_health().status == Status.UP
# first provider is UP, but the second is DOWN
health_providers[1].down()
assert pyctuator_impl.get_health().status == Status.DOWN
# first and second providers are UP, 3rd is UNKNOWN
health_provide
gitextract_1r5sy0a9/
├── .coveragerc
├── .github/
│ └── workflows/
│ ├── python_package_build.yml
│ └── python_package_publish.yml
├── .gitignore
├── .pylintrc
├── LICENSE
├── Makefile
├── README.md
├── examples/
│ ├── Advanced/
│ │ ├── README.md
│ │ ├── advanced_example_app.py
│ │ ├── docker-compose.yml
│ │ └── pyproject.toml
│ ├── FastAPI/
│ │ ├── README.md
│ │ ├── fastapi_example_app.py
│ │ ├── fastapi_with_authentication_example_app.py
│ │ └── pyproject.toml
│ ├── Flask/
│ │ ├── README.md
│ │ ├── flask_example_app.py
│ │ └── pyproject.toml
│ ├── __init__.py
│ ├── aiohttp/
│ │ ├── README.md
│ │ ├── aiohttp_example_app.py
│ │ └── pyproject.toml
│ └── tornado/
│ ├── README.md
│ ├── pyproject.toml
│ └── tornado_example_app.py
├── mypy.ini
├── pyctuator/
│ ├── __init__.py
│ ├── auth.py
│ ├── endpoints.py
│ ├── environment/
│ │ ├── __init__.py
│ │ ├── custom_environment_provider.py
│ │ ├── environment_provider.py
│ │ ├── os_env_variables_impl.py
│ │ └── scrubber.py
│ ├── health/
│ │ ├── __init__.py
│ │ ├── composite_health_provider.py
│ │ ├── db_health_provider.py
│ │ ├── diskspace_health_impl.py
│ │ ├── health_provider.py
│ │ └── redis_health_provider.py
│ ├── httptrace/
│ │ ├── __init__.py
│ │ ├── http_header_scrubber.py
│ │ └── http_tracer.py
│ ├── impl/
│ │ ├── __init__.py
│ │ ├── aiohttp_pyctuator.py
│ │ ├── fastapi_pyctuator.py
│ │ ├── flask_pyctuator.py
│ │ ├── pyctuator_impl.py
│ │ ├── pyctuator_router.py
│ │ ├── spring_boot_admin_registration.py
│ │ └── tornado_pyctuator.py
│ ├── logfile/
│ │ └── logfile.py
│ ├── logging/
│ │ ├── __init__.py
│ │ └── pyctuator_logging.py
│ ├── metrics/
│ │ ├── __init__.py
│ │ ├── memory_metrics_impl.py
│ │ ├── metrics_provider.py
│ │ └── thread_metrics_impl.py
│ ├── py.typed
│ ├── pyctuator.py
│ └── threads/
│ ├── __init__.py
│ └── thread_dump_provider.py
├── pyproject.toml
└── tests/
├── __init__.py
├── aiohttp_test_server.py
├── conftest.py
├── environment/
│ ├── __init__.py
│ ├── test_custom_environment_provider.py
│ └── test_scrubber.py
├── fast_api_test_server.py
├── flask_test_server.py
├── health/
│ ├── __init__.py
│ ├── test_composite_health_provider.py
│ ├── test_db_health_provider.py
│ ├── test_health_status.py
│ └── test_redis_health_provider.py
├── httptrace/
│ ├── __init__.py
│ ├── test_http_header_scrubber.py
│ └── test_tornado_pyctuator.py
├── logfile/
│ ├── __init__.py
│ └── test_logfile.py
├── test_disabled_endpoints.py
├── test_pyctuator_e2e.py
├── test_spring_boot_admin_registration.py
└── tornado_test_server.py
SYMBOL INDEX (321 symbols across 51 files)
FILE: examples/Advanced/advanced_example_app.py
function get_conf (line 57) | def get_conf(key: str) -> Any:
class AppSpecificHealthDetails (line 92) | class AppSpecificHealthDetails(HealthDetails):
function hello (line 104) | def hello():
function redis_get (line 111) | def redis_get(key: str) -> Any:
function db_version (line 116) | def db_version() -> Any:
function health_up (line 121) | def health_up(request: Request, health: Dict) -> None: # health should ...
class CustomHealthProvider (line 164) | class CustomHealthProvider(HealthProvider):
method is_supported (line 166) | def is_supported(self) -> bool:
method get_name (line 169) | def get_name(self) -> str:
method get_health (line 172) | def get_health(self) -> HealthStatus:
FILE: examples/FastAPI/fastapi_example_app.py
function read_root (line 21) | def read_root():
FILE: examples/FastAPI/fastapi_with_authentication_example_app.py
class SimplisticBasicAuth (line 17) | class SimplisticBasicAuth:
method __init__ (line 18) | def __init__(self, username: str, password: str):
method __call__ (line 26) | def __call__(self, credentials: HTTPBasicCredentials = Depends(HTTPBas...
function add_authentication_to_pyctuator (line 50) | def add_authentication_to_pyctuator(router: APIRouter) -> None:
function read_root (line 55) | def read_root(credentials: HTTPBasicCredentials = Depends(security)):
FILE: examples/Flask/flask_example_app.py
function hello (line 18) | def hello():
FILE: examples/aiohttp/aiohttp_example_app.py
function home (line 15) | def home(request: web.Request) -> web.Response:
FILE: examples/tornado/tornado_example_app.py
class HomeHandler (line 14) | class HomeHandler(RequestHandler):
method get (line 15) | def get(self):
FILE: pyctuator/auth.py
class Auth (line 6) | class Auth:
class BasicAuth (line 11) | class BasicAuth(Auth):
FILE: pyctuator/endpoints.py
class Endpoints (line 4) | class Endpoints(Flag):
FILE: pyctuator/environment/custom_environment_provider.py
function _flatten (line 6) | def _flatten(prefix: str, dict_to_flatten: Dict) -> Dict:
class CustomEnvironmentProvider (line 38) | class CustomEnvironmentProvider(EnvironmentProvider):
method __init__ (line 40) | def __init__(self, name: str, env_provider: Callable[[], Dict]) -> None:
method get_properties_source (line 44) | def get_properties_source(self, secret_scrubber: Callable[[Dict], Dict...
FILE: pyctuator/environment/environment_provider.py
class PropertyValue (line 8) | class PropertyValue:
class PropertiesSource (line 14) | class PropertiesSource:
class EnvironmentData (line 20) | class EnvironmentData:
class EnvironmentProvider (line 25) | class EnvironmentProvider(ABC):
method get_properties_source (line 28) | def get_properties_source(self, secret_scrubber: Callable[[Dict], Dict...
FILE: pyctuator/environment/os_env_variables_impl.py
class OsEnvironmentVariableProvider (line 7) | class OsEnvironmentVariableProvider(EnvironmentProvider):
method get_properties_source (line 9) | def get_properties_source(self, secret_scrubber: Callable[[Dict], Dict...
FILE: pyctuator/environment/scrubber.py
class SecretScrubber (line 7) | class SecretScrubber:
method __init__ (line 9) | def __init__(self, keys_to_scrub: Pattern[str] = default_keys_to_scrub...
method scrub_secrets (line 13) | def scrub_secrets(self, mapping: Dict) -> Dict:
FILE: pyctuator/health/composite_health_provider.py
class CompositeHealthStatus (line 8) | class CompositeHealthStatus(HealthStatus):
class CompositeHealthProvider (line 13) | class CompositeHealthProvider(HealthProvider):
method __init__ (line 15) | def __init__(self, name: str, *health_providers: HealthProvider) -> None:
method is_supported (line 20) | def is_supported(self) -> bool:
method get_name (line 23) | def get_name(self) -> str:
method get_health (line 26) | def get_health(self) -> CompositeHealthStatus:
FILE: pyctuator/health/db_health_provider.py
class DbHealthDetails (line 11) | class DbHealthDetails(HealthDetails):
class DbHealthStatus (line 17) | class DbHealthStatus(HealthStatus):
class DbHealthProvider (line 22) | class DbHealthProvider(HealthProvider):
method __init__ (line 24) | def __init__(self, engine: Engine, name: str = "db") -> None:
method is_supported (line 29) | def is_supported(self) -> bool:
method get_name (line 32) | def get_name(self) -> str:
method get_health (line 35) | def get_health(self) -> DbHealthStatus:
FILE: pyctuator/health/diskspace_health_impl.py
class DiskSpaceHealthDetails (line 9) | class DiskSpaceHealthDetails(HealthDetails):
class DiskSpaceHealth (line 16) | class DiskSpaceHealth(HealthStatus):
class DiskSpaceHealthProvider (line 21) | class DiskSpaceHealthProvider(HealthProvider):
method __init__ (line 23) | def __init__(self, free_bytes_down_threshold: int) -> None:
method is_supported (line 33) | def is_supported(self) -> bool:
method get_name (line 36) | def get_name(self) -> str:
method get_health (line 39) | def get_health(self) -> DiskSpaceHealth:
FILE: pyctuator/health/health_provider.py
class Status (line 10) | class Status(str, Enum):
class HealthDetails (line 17) | class HealthDetails:
class HealthStatus (line 22) | class HealthStatus:
class HealthSummary (line 28) | class HealthSummary:
method http_status (line 32) | def http_status(self) -> int:
class HealthProvider (line 44) | class HealthProvider(ABC):
method is_supported (line 46) | def is_supported(self) -> bool:
method get_name (line 50) | def get_name(self) -> str:
method get_health (line 54) | def get_health(self) -> HealthStatus:
FILE: pyctuator/health/redis_health_provider.py
class RedisHealthDetails (line 11) | class RedisHealthDetails(HealthDetails):
class RedisHealthStatus (line 18) | class RedisHealthStatus(HealthStatus):
class RedisHealthProvider (line 23) | class RedisHealthProvider(HealthProvider):
method __init__ (line 25) | def __init__(self, redis: Redis, name: str = "redis") -> None:
method is_supported (line 30) | def is_supported(self) -> bool:
method get_name (line 33) | def get_name(self) -> str:
method get_health (line 36) | def get_health(self) -> RedisHealthStatus:
FILE: pyctuator/httptrace/__init__.py
class TraceResponse (line 7) | class TraceResponse:
class TraceRequest (line 13) | class TraceRequest:
class Session (line 20) | class Session:
class Principal (line 25) | class Principal:
class TraceRecord (line 30) | class TraceRecord:
class Traces (line 40) | class Traces:
FILE: pyctuator/httptrace/http_header_scrubber.py
function scrub_header_value (line 15) | def scrub_header_value(key: str, value: str) -> str:
FILE: pyctuator/httptrace/http_tracer.py
class HttpTracer (line 8) | class HttpTracer:
method __init__ (line 9) | def __init__(self) -> None:
method get_httptrace (line 12) | def get_httptrace(self) -> Traces:
method add_record (line 15) | def add_record(self, record: TraceRecord) -> None:
method _scrub_and_normalize_headers (line 24) | def _scrub_and_normalize_headers(self, headers: Mapping[str, List[str]...
FILE: pyctuator/impl/aiohttp_pyctuator.py
class AioHttpPyctuator (line 20) | class AioHttpPyctuator(PyctuatorRouter):
method __init__ (line 21) | def __init__(self, app: web.Application, pyctuator_impl: PyctuatorImpl...
method _custom_json_serializer (line 156) | def _custom_json_serializer(self, value: Any) -> Any:
method _create_headers_dictionary (line 164) | def _create_headers_dictionary(self, headers: CIMultiDictProxy[str]) -...
method _create_record (line 170) | def _create_record(
FILE: pyctuator/impl/fastapi_pyctuator.py
class FastApiLoggerItem (line 25) | class FastApiLoggerItem(BaseModel):
class FastApiPyctuator (line 30) | class FastApiPyctuator(PyctuatorRouter):
method __init__ (line 33) | def __init__(
method _create_headers_dictionary (line 163) | def _create_headers_dictionary(self, headers: Headers) -> Mapping[str,...
method _create_record (line 169) | def _create_record(
FILE: pyctuator/impl/flask_pyctuator.py
class IsoTimeJSONProvider (line 20) | class IsoTimeJSONProvider(DefaultJSONProvider):
method default (line 31) | def default(self, o: Any) -> Any:
class FlaskPyctuator (line 37) | class FlaskPyctuator(PyctuatorRouter):
method __init__ (line 40) | def __init__(
method _create_headers_dictionary_flask (line 144) | def _create_headers_dictionary_flask(self, headers: Headers) -> Mappin...
method record_request_and_response (line 150) | def record_request_and_response(
FILE: pyctuator/impl/pyctuator_impl.py
class GitCommitInfo (line 19) | class GitCommitInfo:
class GitInfo (line 25) | class GitInfo:
class BuildInfo (line 31) | class BuildInfo:
class AppDetails (line 40) | class AppDetails:
class AppInfo (line 46) | class AppInfo:
class PyctuatorImpl (line 52) | class PyctuatorImpl:
method __init__ (line 54) | def __init__(
method register_metrics_provider (line 83) | def register_metrics_provider(self, provider: MetricsProvider) -> None:
method register_health_providers (line 86) | def register_health_providers(self, provider: HealthProvider) -> None:
method register_environment_provider (line 89) | def register_environment_provider(self, provider: EnvironmentProvider)...
method set_secret_scrubber (line 92) | def set_secret_scrubber(self, secret_scrubber: Callable[[Dict], Dict])...
method get_environment (line 95) | def get_environment(self) -> EnvironmentData:
method set_git_info (line 103) | def set_git_info(self, git_info: GitInfo) -> None:
method set_build_info (line 106) | def set_build_info(self, build_info: BuildInfo) -> None:
method get_health (line 109) | def get_health(self) -> HealthSummary:
method get_metric_names (line 133) | def get_metric_names(self) -> MetricNames:
method get_metric_measurement (line 140) | def get_metric_measurement(self, metric_name: str) -> Metric:
method get_thread_dump (line 146) | def get_thread_dump(self) -> ThreadDump:
method get_app_info (line 149) | def get_app_info(self) -> Dict:
FILE: pyctuator/impl/pyctuator_router.py
class LinkHref (line 10) | class LinkHref:
class EndpointsData (line 16) | class EndpointsData:
class PyctuatorRouter (line 20) | class PyctuatorRouter(ABC):
method __init__ (line 22) | def __init__(
method get_endpoints_data (line 30) | def get_endpoints_data(self) -> EndpointsData:
method get_endpoints_links (line 33) | def get_endpoints_links(self) -> Mapping[str, LinkHref]:
FILE: pyctuator/impl/spring_boot_admin_registration.py
class BootAdminRegistrationHandler (line 17) | class BootAdminRegistrationHandler:
method __init__ (line 19) | def __init__(
method _schedule_next_registration (line 46) | def _schedule_next_registration(self, registration_interval_sec: float...
method _register_with_admin_server (line 55) | def _register_with_admin_server(self) -> None:
method deregister_from_admin_server (line 96) | def deregister_from_admin_server(self) -> None:
method authenticate (line 120) | def authenticate(self, headers: Dict) -> None:
method start (line 127) | def start(self, initial_delay_sec: Optional[float] = None) -> None:
method stop (line 133) | def stop(self) -> None:
method _http_request (line 137) | def _http_request(self, url: str, method: str, headers: Dict[str, str]...
FILE: pyctuator/impl/tornado_pyctuator.py
class AbstractPyctuatorHandler (line 19) | class AbstractPyctuatorHandler(RequestHandler):
method initialize (line 23) | def initialize(self) -> None:
method options (line 28) | def options(self) -> None:
class PyctuatorHandler (line 34) | class PyctuatorHandler(AbstractPyctuatorHandler):
method get (line 35) | def get(self) -> None:
class EnvHandler (line 42) | class EnvHandler(AbstractPyctuatorHandler):
method get (line 43) | def get(self) -> None:
class InfoHandler (line 50) | class InfoHandler(AbstractPyctuatorHandler):
method get (line 51) | def get(self) -> None:
class HealthHandler (line 58) | class HealthHandler(AbstractPyctuatorHandler):
method get (line 59) | def get(self) -> None:
class MetricsHandler (line 68) | class MetricsHandler(AbstractPyctuatorHandler):
method get (line 69) | def get(self) -> None:
class MetricsNameHandler (line 76) | class MetricsNameHandler(AbstractPyctuatorHandler):
method get (line 77) | def get(self, metric_name: str) -> None:
class LoggersHandler (line 84) | class LoggersHandler(AbstractPyctuatorHandler):
method get (line 85) | def get(self) -> None:
class LoggersNameHandler (line 93) | class LoggersNameHandler(AbstractPyctuatorHandler):
method get (line 94) | def get(self, logger_name: str) -> None:
method post (line 99) | def post(self, logger_name: str) -> None:
class ThreadDumpHandler (line 109) | class ThreadDumpHandler(AbstractPyctuatorHandler):
method get (line 110) | def get(self) -> None:
class LogFileHandler (line 117) | class LogFileHandler(AbstractPyctuatorHandler):
method get (line 118) | def get(self) -> None:
class HttpTraceHandler (line 136) | class HttpTraceHandler(AbstractPyctuatorHandler):
method get (line 137) | def get(self) -> None:
class TornadoHttpPyctuator (line 144) | class TornadoHttpPyctuator(PyctuatorRouter):
method __init__ (line 145) | def __init__(self, app: Application, pyctuator_impl: PyctuatorImpl, di...
method _intercept_request_and_response (line 191) | def _intercept_request_and_response(self, handler: RequestHandler) -> ...
method _custom_json_serializer (line 213) | def _custom_json_serializer(self, value: Any) -> Any:
function get_headers (line 222) | def get_headers(headers: HTTPHeaders) -> Mapping[str, List[str]]:
FILE: pyctuator/logfile/logfile.py
class LogMessageBuffer (line 8) | class LogMessageBuffer(logging.Handler):
method __init__ (line 9) | def __init__(self, max_size: int, formatter: str) -> None:
method emit (line 16) | def emit(self, record: logging.LogRecord) -> None:
method get_range (line 24) | def get_range(self, start: Optional[int] = None, end: Optional[int] = ...
method get_offset (line 28) | def get_offset(self) -> int:
method get_offset_tuple (line 31) | def get_offset_tuple(self, start: Optional[int], end: Optional[int]) -...
class PyctuatorLogfile (line 37) | class PyctuatorLogfile:
method __init__ (line 38) | def __init__(self, max_size: int, formatter: str) -> None:
method get_logfile (line 41) | def get_logfile(self, range_substring: str) -> Tuple[str, int, int]:
method get_log_buffer_offset (line 61) | def get_log_buffer_offset(self) -> int:
FILE: pyctuator/logging/pyctuator_logging.py
class LoggerLevels (line 7) | class LoggerLevels:
class LoggersData (line 13) | class LoggersData:
class LogLevelMapping (line 20) | class LogLevelMapping:
function _python_to_admin_log_level (line 35) | def _python_to_admin_log_level(log_level: int) -> str:
function _admin_to_python_log_level (line 44) | def _admin_to_python_log_level(log_level: str) -> int:
class PyctuatorLogging (line 49) | class PyctuatorLogging:
method set_logger_level (line 50) | def set_logger_level(self, logger_name: str, logger_level: Optional[st...
method get_loggers (line 61) | def get_loggers(self) -> LoggersData:
method get_logger (line 74) | def get_logger(self, logger_name: str) -> LoggerLevels:
FILE: pyctuator/metrics/memory_metrics_impl.py
class MemoryMetricsProvider (line 10) | class MemoryMetricsProvider(MetricsProvider):
method __init__ (line 11) | def __init__(self) -> None:
method get_prefix (line 19) | def get_prefix(self) -> str:
method get_supported_metric_names (line 22) | def get_supported_metric_names(self) -> List[str]:
method get_metric (line 28) | def get_metric(self, metric_name: str) -> Metric:
FILE: pyctuator/metrics/metrics_provider.py
class MetricNames (line 8) | class MetricNames:
class Measurement (line 13) | class Measurement:
class MetricTag (line 19) | class MetricTag:
class Metric (line 25) | class Metric:
class MetricsProvider (line 33) | class MetricsProvider(ABC):
method get_prefix (line 36) | def get_prefix(self) -> str:
method get_supported_metric_names (line 40) | def get_supported_metric_names(self) -> List[str]:
method get_metric (line 44) | def get_metric(self, metric_name: str) -> Metric:
FILE: pyctuator/metrics/thread_metrics_impl.py
class ThreadMetricsProvider (line 11) | class ThreadMetricsProvider(MetricsProvider):
method __init__ (line 12) | def __init__(self) -> None:
method get_prefix (line 20) | def get_prefix(self) -> str:
method get_supported_metric_names (line 23) | def get_supported_metric_names(self) -> List[str]:
method get_metric (line 26) | def get_metric(self, metric_name: str) -> Metric:
FILE: pyctuator/pyctuator.py
class Pyctuator (line 27) | class Pyctuator:
method __init__ (line 29) | def __init__(
method stop (line 161) | def stop(self) -> None:
method set_secret_scrubber (line 166) | def set_secret_scrubber(self, secret_scrubber: Callable[[Dict], Dict])...
method register_environment_provider (line 170) | def register_environment_provider(self, name: str, env_provider: Calla...
method register_health_provider (line 173) | def register_health_provider(self, provider: HealthProvider) -> None:
method set_git_info (line 176) | def set_git_info(self, commit: str, time: datetime, branch: Optional[s...
method set_build_info (line 179) | def set_build_info(
method _is_framework_installed (line 189) | def _is_framework_installed(self, framework_name: str) -> bool:
method _integrate_fastapi (line 192) | def _integrate_fastapi(
method _integrate_flask (line 212) | def _integrate_flask(
method _integrate_aiohttp (line 232) | def _integrate_aiohttp(
method _integrate_tornado (line 252) | def _integrate_tornado(
FILE: pyctuator/threads/thread_dump_provider.py
class StackFrame (line 10) | class StackFrame:
class ThreadInfo (line 19) | class ThreadInfo:
class ThreadDump (line 29) | class ThreadDump:
class ThreadDumpProvider (line 33) | class ThreadDumpProvider:
method get_thread_dump (line 36) | def get_thread_dump(self) -> ThreadDump:
method _extract_thread_info (line 43) | def _extract_thread_info(self, frames: Dict[Any, Any], thread: Thread)...
method _build_thread_stack_trace (line 53) | def _build_thread_stack_trace(self, thread: Thread, frames: Dict[Any, ...
method _calc_thread_state (line 79) | def _calc_thread_state(self, thread: threading.Thread) -> str:
FILE: tests/aiohttp_test_server.py
class AiohttpPyctuatorServer (line 17) | class AiohttpPyctuatorServer(PyctuatorServer):
method __init__ (line 19) | def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> ...
method _run_server (line 66) | async def _run_server(self) -> None:
method _start_in_thread (line 85) | def _start_in_thread(self) -> None:
method start (line 90) | def start(self) -> None:
method stop (line 95) | def stop(self) -> None:
method atexit (line 102) | def atexit(self) -> None:
FILE: tests/conftest.py
class RegistrationRequest (line 25) | class RegistrationRequest(BaseModel):
class RegisteredEndpoints (line 35) | class RegisteredEndpoints:
class RegistrationTrackerFixture (line 49) | class RegistrationTrackerFixture:
class CustomServer (line 69) | class CustomServer(Server):
method install_signal_handlers (line 70) | def install_signal_handlers(self) -> None:
function registration_tracker (line 75) | def registration_tracker() -> RegistrationTrackerFixture:
function boot_admin_server (line 80) | def boot_admin_server(registration_tracker: RegistrationTrackerFixture) ...
function registered_endpoints (line 150) | def registered_endpoints(registration_tracker: RegistrationTrackerFixtur...
class PyctuatorServer (line 181) | class PyctuatorServer(ABC):
method start (line 193) | def start(self) -> None:
method stop (line 197) | def stop(self) -> None:
method atexit (line 201) | def atexit(self) -> None:
FILE: tests/environment/test_custom_environment_provider.py
function test_custom_environment_provider (line 8) | def test_custom_environment_provider() -> None:
FILE: tests/environment/test_scrubber.py
function test_scrub_secrets (line 6) | def test_scrub_secrets() -> None:
function test_custom_scrub_secrets (line 39) | def test_custom_scrub_secrets() -> None:
FILE: tests/fast_api_test_server.py
class FastApiPyctuatorServer (line 18) | class FastApiPyctuatorServer(PyctuatorServer):
method __init__ (line 19) | def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> ...
method start (line 66) | def start(self) -> None:
method stop (line 71) | def stop(self) -> None:
method atexit (line 83) | def atexit(self) -> None:
FILE: tests/flask_test_server.py
class FlaskPyctuatorServer (line 18) | class FlaskPyctuatorServer(PyctuatorServer):
method __init__ (line 19) | def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> ...
method start (line 66) | def start(self) -> None:
method stop (line 78) | def stop(self) -> None:
method atexit (line 85) | def atexit(self) -> None:
FILE: tests/health/test_composite_health_provider.py
class CustomHealthDetails (line 8) | class CustomHealthDetails(HealthDetails):
class CustomHealthProvider (line 12) | class CustomHealthProvider(HealthProvider):
method __init__ (line 14) | def __init__(self, name: str, status: HealthStatus) -> None:
method is_supported (line 19) | def is_supported(self) -> bool:
method get_name (line 22) | def get_name(self) -> str:
method get_health (line 25) | def get_health(self) -> HealthStatus:
function test_composite_health_provider_no_providers (line 29) | def test_composite_health_provider_no_providers() -> None:
function test_composite_health_provider_all_up (line 42) | def test_composite_health_provider_all_up() -> None:
function test_composite_health_provider_one_down (line 60) | def test_composite_health_provider_one_down() -> None:
FILE: tests/health/test_db_health_provider.py
function require_sql_alchemy (line 9) | def require_sql_alchemy() -> None:
function require_pymysql (line 15) | def require_pymysql() -> None:
function require_mysql_server (line 21) | def require_mysql_server() -> None:
function test_sqlite_health (line 28) | def test_sqlite_health() -> None:
function test_mysql_health (line 41) | def test_mysql_health() -> None:
FILE: tests/health/test_health_status.py
class MyHealthProvider (line 9) | class MyHealthProvider(HealthProvider):
method __init__ (line 10) | def __init__(self, name: str = "kuki") -> None:
method down (line 14) | def down(self) -> None:
method up (line 17) | def up(self) -> None:
method is_supported (line 20) | def is_supported(self) -> bool:
method get_health (line 23) | def get_health(self) -> HealthStatus:
method get_name (line 26) | def get_name(self) -> str:
function pyctuator_impl (line 31) | def pyctuator_impl() -> PyctuatorImpl:
function test_health_status_single_provider (line 42) | def test_health_status_single_provider(pyctuator_impl: PyctuatorImpl) ->...
function test_health_status_multiple_providers (line 56) | def test_health_status_multiple_providers(pyctuator_impl: PyctuatorImpl)...
FILE: tests/health/test_redis_health_provider.py
function require_redis (line 10) | def require_redis() -> None:
function require_redis_server (line 17) | def require_redis_server() -> None:
function redis_host (line 23) | def redis_host() -> str:
function test_redis_health (line 27) | def test_redis_health(redis_host: str) -> None:
function test_redis_bad_password (line 37) | def test_redis_bad_password(redis_host: str) -> None:
FILE: tests/httptrace/test_http_header_scrubber.py
function test_scrubbing (line 12) | def test_scrubbing(key: str, value: str) -> None:
function test_non_scrubbing (line 17) | def test_non_scrubbing(key: str, value: str) -> None:
FILE: tests/httptrace/test_tornado_pyctuator.py
function test_get_headers (line 6) | def test_get_headers() -> None:
FILE: tests/logfile/test_logfile.py
function test_empty_response (line 10) | def test_empty_response() -> None:
function test_buffer_not_full (line 18) | def test_buffer_not_full() -> None:
function test_buffer_overflow (line 30) | def test_buffer_overflow() -> None:
function test_forgotten_records (line 47) | def test_forgotten_records() -> None:
FILE: tests/test_disabled_endpoints.py
function disabled_endpoints (line 24) | def disabled_endpoints(request) -> Generator: # type: ignore
function pyctuator_server (line 32) | def pyctuator_server(disabled_endpoints: Endpoints, request) -> Generato...
function test_disabled_endpoints_not_shown (line 45) | def test_disabled_endpoints_not_shown(
function test_disabled_endpoints_not_found (line 70) | def test_disabled_endpoints_not_found(
FILE: tests/test_pyctuator_e2e.py
function pyctuator_server (line 32) | def pyctuator_server(request) -> Generator: # type: ignore
function test_response_content_type (line 45) | def test_response_content_type(
function test_self_endpoint (line 83) | def test_self_endpoint(registered_endpoints: RegisteredEndpoints) -> None:
function test_env_endpoint (line 90) | def test_env_endpoint(registered_endpoints: RegisteredEndpoints) -> None:
function test_info_endpoint (line 106) | def test_info_endpoint(registered_endpoints: RegisteredEndpoints, pyctua...
function test_health_endpoint_with_psutil (line 114) | def test_health_endpoint_with_psutil(registered_endpoints: RegisteredEnd...
function test_diskspace_no_psutil (line 147) | def test_diskspace_no_psutil(registered_endpoints: RegisteredEndpoints) ...
function test_metrics_endpoint (line 160) | def test_metrics_endpoint(registered_endpoints: RegisteredEndpoints) -> ...
function test_recurring_registration_and_deregistration (line 186) | def test_recurring_registration_and_deregistration(
function test_threads_endpoint (line 216) | def test_threads_endpoint(registered_endpoints: RegisteredEndpoints) -> ...
function test_loggers_endpoint (line 233) | def test_loggers_endpoint(registered_endpoints: RegisteredEndpoints) -> ...
function test_logfile_endpoint (line 275) | def test_logfile_endpoint(registered_endpoints: RegisteredEndpoints) -> ...
function test_traces_endpoint (line 294) | def test_traces_endpoint(registered_endpoints: RegisteredEndpoints) -> N...
FILE: tests/test_spring_boot_admin_registration.py
function test_registration_no_auth (line 13) | def test_registration_no_auth(registration_tracker: RegistrationTrackerF...
function test_registration_basic_auth_no_creds (line 25) | def test_registration_basic_auth_no_creds(registration_tracker: Registra...
function test_registration_basic_auth_bad_creds (line 43) | def test_registration_basic_auth_bad_creds(registration_tracker: Registr...
function test_registration_basic_auth (line 64) | def test_registration_basic_auth(registration_tracker: RegistrationTrack...
function get_registration_handler (line 78) | def get_registration_handler(registration_url: str, registration_auth: O...
function _start_registration (line 90) | def _start_registration(registration_handler: BootAdminRegistrationHandl...
FILE: tests/tornado_test_server.py
class TornadoPyctuatorServer (line 17) | class TornadoPyctuatorServer(PyctuatorServer):
method __init__ (line 18) | def __init__(self, disabled_endpoints: Endpoints = Endpoints.NONE) -> ...
method _start_in_thread (line 72) | def _start_in_thread(self) -> None:
method start (line 77) | def start(self) -> None:
method stop (line 82) | def stop(self) -> None:
method atexit (line 96) | def atexit(self) -> None:
Condensed preview — 86 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (235K chars).
[
{
"path": ".coveragerc",
"chars": 20,
"preview": "[run]\nomit = .venv/*"
},
{
"path": ".github/workflows/python_package_build.yml",
"chars": 1258,
"preview": "# This workflow will install dependencies, build Pyctuator, run tests (+coverage) and lint\n\nname: build\n\non:\n push:\n p"
},
{
"path": ".github/workflows/python_package_publish.yml",
"chars": 448,
"preview": "# This workflow will build Pyctuator, and publish it to pypi.org\n\nname: publish\n\non:\n release:\n types: [published]\n\n"
},
{
"path": ".gitignore",
"chars": 10682,
"preview": "# Created by .ignore support plugin (hsz.mobi)\n### macOS template\n# General\n.DS_Store\n.AppleDouble\n.LSOverride\n\n# Icon m"
},
{
"path": ".pylintrc",
"chars": 14725,
"preview": "[MASTER]\n\n# Add files or directories to the blacklist. They should be base names, not\n# paths.\nignore=CVS\n\n# Pickle coll"
},
{
"path": "LICENSE",
"chars": 11357,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "Makefile",
"chars": 1135,
"preview": "\nall: check\n\nhelp:\n\t@echo \"Available targets:\"\n\t@echo \"- help Show this help message\"\n\t@echo \"- bootst"
},
{
"path": "README.md",
"chars": 13525,
"preview": "[](https://pypi.org/project/pyctuator/)\n[![bui"
},
{
"path": "examples/Advanced/README.md",
"chars": 3982,
"preview": "# Advanced Example\nThis example demonstrates using the optional features and customizations Pyctuator is offering.\n\n## R"
},
{
"path": "examples/Advanced/advanced_example_app.py",
"chars": 6261,
"preview": "import datetime\nimport logging\nimport random\nfrom dataclasses import dataclass\nfrom typing import Any, Dict, List\nfrom s"
},
{
"path": "examples/Advanced/docker-compose.yml",
"chars": 277,
"preview": "version: '3'\nservices:\n mysql:\n image: mysql:5.7.22\n ports:\n - 3306:3306\n environment:\n - MYSQL_ROOT"
},
{
"path": "examples/Advanced/pyproject.toml",
"chars": 557,
"preview": "[tool.poetry]\nname = \"fastapi-pyctuator-example\"\nversion = \"1.0.0\"\ndescription = \"Example of using Pyctuator\"\nauthors = "
},
{
"path": "examples/FastAPI/README.md",
"chars": 904,
"preview": "# FastAPI example\nThis example demonstrates the integration with the [FastAPI](https://fastapi.tiangolo.com/) web-framew"
},
{
"path": "examples/FastAPI/fastapi_example_app.py",
"chars": 1046,
"preview": "import datetime\nimport logging\nimport random\n\nfrom fastapi import FastAPI\nfrom uvicorn import Server\nfrom uvicorn.config"
},
{
"path": "examples/FastAPI/fastapi_with_authentication_example_app.py",
"chars": 2796,
"preview": "import datetime\nimport logging\nimport random\nimport secrets\n\nfrom fastapi import FastAPI, Depends, APIRouter, HTTPExcept"
},
{
"path": "examples/FastAPI/pyproject.toml",
"chars": 425,
"preview": "[tool.poetry]\nname = \"fastapi-pyctuator-example\"\nversion = \"1.0.0\"\ndescription = \"Example of using Pyctuator\"\nauthors = "
},
{
"path": "examples/Flask/README.md",
"chars": 868,
"preview": "# Flask example\nThis example demonstrates the integration with the [Flask](https://flask.palletsprojects.com/) web-frame"
},
{
"path": "examples/Flask/flask_example_app.py",
"chars": 970,
"preview": "import datetime\nimport logging\nimport random\n\nfrom flask import Flask\n\nfrom pyctuator.pyctuator import Pyctuator\n\n# Keep"
},
{
"path": "examples/Flask/pyproject.toml",
"chars": 386,
"preview": "[tool.poetry]\nname = \"flask-pyctuator-example\"\nversion = \"1.0.0\"\ndescription = \"Example of using Pyctuator\"\nauthors = [\n"
},
{
"path": "examples/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "examples/aiohttp/README.md",
"chars": 536,
"preview": "# aiohttp example\nThis example demonstrates the integration with the [aiohttp](https://docs.aiohttp.org).\n\n## Running th"
},
{
"path": "examples/aiohttp/aiohttp_example_app.py",
"chars": 897,
"preview": "import datetime\nimport logging\nimport random\n\nfrom aiohttp import web\n\nfrom pyctuator.pyctuator import Pyctuator\n\nmy_log"
},
{
"path": "examples/aiohttp/pyproject.toml",
"chars": 390,
"preview": "[tool.poetry]\nname = \"aiohttp-pyctuator-example\"\nversion = \"1.0.0\"\ndescription = \"Example of using Pyctuator\"\nauthors = "
},
{
"path": "examples/tornado/README.md",
"chars": 539,
"preview": "# Tornado example\nThis example demonstrates the integration with the [Tornado](https://www.tornadoweb.org/).\n\n## Running"
},
{
"path": "examples/tornado/pyproject.toml",
"chars": 394,
"preview": "[tool.poetry]\nname = \"tornado-pyctuator-example\"\nversion = \"1.0.0\"\ndescription = \"Example of using Pyctuator\"\nauthors = "
},
{
"path": "examples/tornado/tornado_example_app.py",
"chars": 1063,
"preview": "import datetime\r\nimport logging\r\nimport random\r\n\r\nfrom tornado import ioloop\r\nfrom tornado.httpserver import HTTPServer\r"
},
{
"path": "mypy.ini",
"chars": 307,
"preview": "[mypy]\ndisallow_untyped_defs = True\nwarn_return_any = True\n\n[mypy-pytest]\nignore_missing_imports = True\n\n[mypy-psutil]\ni"
},
{
"path": "pyctuator/__init__.py",
"chars": 22,
"preview": "__version__ = '0.1.0'\n"
},
{
"path": "pyctuator/auth.py",
"chars": 178,
"preview": "from dataclasses import dataclass\nfrom typing import Optional\n\n\n@dataclass\nclass Auth:\n pass\n\n\n@dataclass\nclass Basic"
},
{
"path": "pyctuator/endpoints.py",
"chars": 233,
"preview": "from enum import Flag, auto\n\n\nclass Endpoints(Flag):\n NONE = 0\n ENV = auto()\n INFO = auto()\n HEALTH = auto()"
},
{
"path": "pyctuator/environment/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "pyctuator/environment/custom_environment_provider.py",
"chars": 1716,
"preview": "from typing import Callable, Dict\n\nfrom pyctuator.environment.environment_provider import PropertiesSource, EnvironmentP"
},
{
"path": "pyctuator/environment/environment_provider.py",
"chars": 588,
"preview": "from abc import ABC, abstractmethod\nfrom dataclasses import dataclass\n\nfrom typing import Mapping, Optional, List, Any, "
},
{
"path": "pyctuator/environment/os_env_variables_impl.py",
"chars": 544,
"preview": "import os\nfrom typing import Dict, Callable\n\nfrom pyctuator.environment.environment_provider import PropertiesSource, Pr"
},
{
"path": "pyctuator/environment/scrubber.py",
"chars": 1044,
"preview": "import re\nfrom typing import Any, Dict, Pattern\n\ndefault_keys_to_scrub = re.compile(\"^(.*[^A-Za-z])?key([^A-Za-z].*)?$|."
},
{
"path": "pyctuator/health/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "pyctuator/health/composite_health_provider.py",
"chars": 1819,
"preview": "from dataclasses import dataclass\nfrom typing import Mapping\n\nfrom pyctuator.health.health_provider import HealthProvide"
},
{
"path": "pyctuator/health/db_health_provider.py",
"chars": 1458,
"preview": "import importlib.util\nfrom dataclasses import dataclass\nfrom typing import Optional\n\nfrom sqlalchemy.engine import Engin"
},
{
"path": "pyctuator/health/diskspace_health_impl.py",
"chars": 1264,
"preview": "# pylint: disable=import-outside-toplevel\nimport importlib.util\nfrom dataclasses import dataclass\n\nfrom pyctuator.health"
},
{
"path": "pyctuator/health/health_provider.py",
"chars": 1324,
"preview": "import abc\nfrom abc import ABC\nfrom dataclasses import dataclass\nfrom enum import Enum\nfrom http import HTTPStatus\n\nfrom"
},
{
"path": "pyctuator/health/redis_health_provider.py",
"chars": 1375,
"preview": "import importlib.util\nfrom dataclasses import dataclass\nfrom typing import Optional\n\nfrom redis import Redis\n\nfrom pyctu"
},
{
"path": "pyctuator/httptrace/__init__.py",
"chars": 629,
"preview": "from dataclasses import dataclass\nfrom datetime import datetime\nfrom typing import List, Mapping, Optional\n\n\n@dataclass\n"
},
{
"path": "pyctuator/httptrace/http_header_scrubber.py",
"chars": 356,
"preview": "import re\n\n_keys_to_scrub = re.compile(\n \"^(.*[^A-Za-z])?key([^A-Za-z].*)?$|\"\n \".*secret.*|\"\n \".*password.*|\"\n "
},
{
"path": "pyctuator/httptrace/http_tracer.py",
"chars": 926,
"preview": "import collections\nfrom typing import List, Mapping\nfrom pyctuator.httptrace.http_header_scrubber import scrub_header_va"
},
{
"path": "pyctuator/impl/__init__.py",
"chars": 83,
"preview": "SBA_V2_CONTENT_TYPE = \"application/vnd.spring-boot.actuator.v2+json;charset=UTF-8\"\n"
},
{
"path": "pyctuator/impl/aiohttp_pyctuator.py",
"chars": 8414,
"preview": "import dataclasses\nimport json\nfrom collections import defaultdict\nfrom datetime import datetime\nfrom functools import p"
},
{
"path": "pyctuator/impl/fastapi_pyctuator.py",
"chars": 8823,
"preview": "from collections import defaultdict\nfrom datetime import datetime\nfrom http import HTTPStatus\nfrom typing import Mapping"
},
{
"path": "pyctuator/impl/flask_pyctuator.py",
"chars": 6972,
"preview": "import json\nfrom collections import defaultdict\nfrom datetime import datetime, date\nfrom http import HTTPStatus\nfrom typ"
},
{
"path": "pyctuator/impl/pyctuator_impl.py",
"chars": 5867,
"preview": "import dataclasses\nfrom dataclasses import dataclass\nfrom datetime import datetime\nfrom typing import List, Dict, Mappin"
},
{
"path": "pyctuator/impl/pyctuator_router.py",
"chars": 1750,
"preview": "from abc import ABC\nfrom dataclasses import dataclass\nfrom typing import Any, Optional, Mapping\n\nfrom pyctuator.endpoint"
},
{
"path": "pyctuator/impl/spring_boot_admin_registration.py",
"chars": 6381,
"preview": "import http.client\nimport json\nimport logging\nimport os\nimport ssl\nimport threading\nimport urllib.parse\nfrom base64 impo"
},
{
"path": "pyctuator/impl/tornado_pyctuator.py",
"chars": 9056,
"preview": "import dataclasses\r\nimport json\r\nfrom datetime import datetime, timedelta\r\nfrom functools import partial\r\nfrom http impo"
},
{
"path": "pyctuator/logfile/logfile.py",
"chars": 2411,
"preview": "import logging\nimport re\nfrom typing import Optional, Tuple\n\nlogfile_request_range_pattern = re.compile(\"bytes=(\\\\d*)-(\\"
},
{
"path": "pyctuator/logging/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "pyctuator/logging/pyctuator_logging.py",
"chars": 2631,
"preview": "import logging\nfrom dataclasses import dataclass\nfrom typing import Dict, List, Optional\n\n\n@dataclass\nclass LoggerLevels"
},
{
"path": "pyctuator/metrics/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "pyctuator/metrics/memory_metrics_impl.py",
"chars": 1153,
"preview": "# pylint: disable=import-outside-toplevel\nimport importlib.util\nfrom typing import List\n\nfrom pyctuator.metrics.metrics_"
},
{
"path": "pyctuator/metrics/metrics_provider.py",
"chars": 831,
"preview": "from abc import ABC, abstractmethod\nfrom dataclasses import dataclass\n\nfrom typing import List, Optional\n\n\n@dataclass\ncl"
},
{
"path": "pyctuator/metrics/thread_metrics_impl.py",
"chars": 935,
"preview": "# pylint: disable=import-outside-toplevel\nimport importlib.util\nfrom typing import List\n\nfrom pyctuator.metrics.metrics_"
},
{
"path": "pyctuator/py.typed",
"chars": 0,
"preview": ""
},
{
"path": "pyctuator/pyctuator.py",
"chars": 13713,
"preview": "# pylint: disable=import-outside-toplevel\nimport atexit\nimport importlib.util\nimport logging\nimport ssl\nfrom datetime im"
},
{
"path": "pyctuator/threads/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "pyctuator/threads/thread_dump_provider.py",
"chars": 2456,
"preview": "import sys\nimport threading\nfrom threading import Thread\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom"
},
{
"path": "pyproject.toml",
"chars": 2298,
"preview": "[tool.poetry]\nname = \"pyctuator\"\nversion = \"1.2.0\"\ndescription = \"A Python implementation of the Spring Actuator API for"
},
{
"path": "tests/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/aiohttp_test_server.py",
"chars": 3537,
"preview": "import asyncio\nimport logging\nimport threading\nimport time\n\nfrom aiohttp import web\n\nfrom pyctuator.endpoints import End"
},
{
"path": "tests/conftest.py",
"chars": 6880,
"preview": "import logging\nimport random\nimport secrets\nimport threading\nimport time\nfrom abc import ABC, abstractmethod\nfrom datacl"
},
{
"path": "tests/environment/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/environment/test_custom_environment_provider.py",
"chars": 1218,
"preview": "from typing import Dict\n\nfrom pyctuator.environment.custom_environment_provider import CustomEnvironmentProvider\nfrom py"
},
{
"path": "tests/environment/test_scrubber.py",
"chars": 1894,
"preview": "import re\n\nfrom pyctuator.environment.scrubber import SecretScrubber\n\n\ndef test_scrub_secrets() -> None:\n with_secret"
},
{
"path": "tests/fast_api_test_server.py",
"chars": 3070,
"preview": "import logging\nimport threading\nimport time\nfrom typing import Optional\n\nfrom fastapi import FastAPI\nfrom starlette.requ"
},
{
"path": "tests/flask_test_server.py",
"chars": 3140,
"preview": "import logging\nimport threading\nimport time\nfrom wsgiref.simple_server import make_server\n\nimport requests\nfrom flask im"
},
{
"path": "tests/health/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/health/test_composite_health_provider.py",
"chars": 2234,
"preview": "from dataclasses import dataclass\n\nfrom pyctuator.health.composite_health_provider import CompositeHealthProvider, Compo"
},
{
"path": "tests/health/test_db_health_provider.py",
"chars": 2223,
"preview": "# pylint: disable=import-outside-toplevel\nimport importlib.util\nimport os\n\nimport pytest\n\n\n@pytest.fixture\ndef require_s"
},
{
"path": "tests/health/test_health_status.py",
"chars": 2383,
"preview": "import pytest\n\nfrom pyctuator.endpoints import Endpoints\nfrom pyctuator.health.health_provider import HealthStatus, Stat"
},
{
"path": "tests/health/test_redis_health_provider.py",
"chars": 1567,
"preview": "# pylint: disable=import-outside-toplevel\n\nimport importlib.util\nimport os\n\nimport pytest\n\n\n@pytest.fixture\ndef require_"
},
{
"path": "tests/httptrace/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/httptrace/test_http_header_scrubber.py",
"chars": 614,
"preview": "import pytest\nfrom pyctuator.httptrace.http_header_scrubber import scrub_header_value\n\n\n@pytest.mark.parametrize(\"key,va"
},
{
"path": "tests/httptrace/test_tornado_pyctuator.py",
"chars": 415,
"preview": "from tornado.httputil import HTTPHeaders\n\nfrom pyctuator.impl.tornado_pyctuator import get_headers\n\n\ndef test_get_header"
},
{
"path": "tests/logfile/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/logfile/test_logfile.py",
"chars": 2196,
"preview": "# pylint: disable=protected-access\nimport logging\n\nfrom pyctuator.logfile.logfile import PyctuatorLogfile # type: ignor"
},
{
"path": "tests/test_disabled_endpoints.py",
"chars": 3110,
"preview": "import logging\nfrom http import HTTPStatus\nfrom typing import Generator\n\nimport pytest\nimport requests\n\nfrom pyctuator.e"
},
{
"path": "tests/test_pyctuator_e2e.py",
"chars": 16094,
"preview": "import importlib.util\nimport json\nimport logging\nimport os\nimport random\nimport time\nfrom dataclasses import dataclass, "
},
{
"path": "tests/test_spring_boot_admin_registration.py",
"chars": 3436,
"preview": "import time\nfrom datetime import datetime\nfrom typing import Optional, Any\n\nimport pytest\n\nfrom pyctuator.auth import Au"
},
{
"path": "tests/tornado_test_server.py",
"chars": 3532,
"preview": "import logging\nimport threading\nimport time\nfrom typing import Optional\n\nfrom tornado import ioloop\nfrom tornado.httpser"
}
]
About this extraction
This page contains the full source code of the SolarEdgeTech/pyctuator GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 86 files (215.4 KB), approximately 51.8k tokens, and a symbol index with 321 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.