Repository: otto-de/tesla-microservice
Branch: master
Commit: b40ff9ba3c30
Files: 46
Total size: 92.6 KB
Directory structure:
gitextract_11uzdn14/
├── .gitignore
├── .tool-versions
├── .travis.yml
├── CHANGES.md
├── Dockerfile
├── LICENSE
├── MAINTAINERS
├── OSSMETADATA
├── README.md
├── dev-resources/
│ └── logback.xml
├── project.clj
├── resources/
│ ├── default.edn
│ └── default.properties
├── src/
│ ├── data_readers.clj
│ └── de/
│ └── otto/
│ └── tesla/
│ ├── middleware/
│ │ └── exceptions.clj
│ ├── stateful/
│ │ ├── app_status.clj
│ │ ├── auth.clj
│ │ ├── configuring.clj
│ │ ├── handler.clj
│ │ ├── health.clj
│ │ ├── keep_alive.clj
│ │ ├── metering.clj
│ │ └── scheduler.clj
│ ├── system.clj
│ └── util/
│ ├── env_var_reader.clj
│ ├── keyword.clj
│ └── sanitize.clj
├── test/
│ └── de/
│ └── otto/
│ └── tesla/
│ ├── reporter/
│ │ └── prometheus_test.clj
│ ├── stateful/
│ │ ├── app_status_test.clj
│ │ ├── configuring_test.clj
│ │ ├── handler_test.clj
│ │ ├── health_test.clj
│ │ ├── keep_alive_test.clj
│ │ └── scheduler_test.clj
│ ├── system_test.clj
│ └── util/
│ ├── env_var_reader_test.clj
│ ├── keyword_test.clj
│ ├── sanitize_test.clj
│ └── test_utils_test.clj
├── test-resources/
│ ├── local.edn
│ ├── local.properties
│ ├── logback-test.xml
│ ├── private.edn
│ ├── test.edn
│ └── version.properties
└── test-utils/
└── de/
└── otto/
└── tesla/
└── util/
└── test_utils.clj
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.idea
.lein*
target
.nrepl-port
*.iml
pom.xml
pom.xml.asc
/.clj-kondo/
/.lsp/
================================================
FILE: .tool-versions
================================================
terraform 1.8.5
awscli 2.16.10
tfenv 0.4.0
================================================
FILE: .travis.yml
================================================
language: clojure
================================================
FILE: CHANGES.md
================================================
## Changes
_tesla-microservice_ is used for a number of different services now. Still it is a work in progress. This section will document changes and give instructions on breaking ones. Likely you will find corresponding changes in [tesla-examples](https://github.com/otto-de/tesla-examples).
### 0.15.0
Changed securing of internal endpoints. Moved from providing separate auth-functions
to components app-status and metering to an auth-middleware provided to the base-system,
see [Securing internal info endpoints](https://github.com/otto-de/tesla-microservice#securing-internal-info-endpoints)
### 0.14.0
This release cleans up remnants of past eperiments and unused functionality. This leads to breaking changes.
- Remove ```register-timed-handler``` from ```handler``` namespace. Use ```goo/timing-middleware``` instead.
- Remove ```register-response-fn``` from ```handler``` namespace. This was only used internally.
- Move internally used middlewares to separate namespaces.
- Remove support for reporting via graphite from ```metering``` namespace. Only support prometheus reporting via goo and iapetos for the moment.
- Remove ```SchedulerPool``` protocol from ```scheduler``` namespace. Use ```(:pool scheduler)``` instead of ```(SchedulerPool/pool scheduler)```
### 0.11.0
Utilize the iapetos library as main metrics library. Tesla-Microservice is now able to report to graphite as well as prometheus.
For configuration of graphite and prometheus reporters please see the updated README.
de.otto.tesla.metrics.prometheus.core now provides some useful instrumentation functions/macros.
### 0.8.0
You are now able to override the name of the base config file via the runtime config. The following example will make the
configuring component disgregard ```default.edn``` and use ```not-default.edn``` instead. This might be useful when deploying several applications from one repo.
```edn
{
:default-cfg-file-name "not-default"
}
```
### 0.6.0
The behaviour of loading configuration changed.
* When using configuration via `properties` files, system properties and environment variables are not loaded by default any more. Use `:merge-env-to-properties-config true` in runtime config to achieve prior behaviour.
* For the config-file `application.edn`/`application.properties` (name can be overriden by env-var `$CONFIG_FILE`)
is now with preference loaded as a resource from classpath. If the resource is not found, it is tried to load it as a file.
### 0.5.0
The scheduler is now part of the tesla-base-system.
Per default no threads are kept in the thread-pool it manages.
### 0.4.0
The scheduler does not have the `de.otto.tesla.stateful.scheduler/schedule` function anymore.
Instead it only wraps the overtone pool and provides it via `de.otto.tesla.stateful.scheduler/pool`.
The pool then can be used with the overtone API like that:
```clj
(overtone.at-at/every 100 #(println "Hello world") (de.otto.tesla.stateful.scheduler/pool scheduler) :desc "HelloWord Task")
```
### 0.1.24
Config can be provided via EDN-files.
Those files are looked up and merged:
* `default.edn`
* `{your-custom}.edn`
* `local.edn`
The `{your-custom.edn}` can be specified via a ENV-variable named `$CONFIG_FILE`. All
EDN-config-files have to be located somewhere in the class path.
Even though the old properties-files are considered deprecated and will go away with
future releases, you can still use them, if you specify `:property-file-preferred` in the
runtime-config of your system:
```edn
{
:property-file-preferred true
}
```
### 0.1.17
Fix wrapping of middleware to not apply to all routes in the application, which created problems with POST-request.
### 0.1.16
Speedup of unit-tests (and possibly runtime behaviour) by simpler implmentation of the `:keep-alive`-component.
### 0.1.15
The function ```de.otto.tesla.system/start-system``` is renamed to ```start```, ```de.otto.tesla.system/empty-system``` is renamed to ```base-system```.
_tesla-microservice_ does not come with an embedded jetty server out of the box anymore.
To go on with jetty as before, add the new dependency in ```project.clj```:
```clojure
[de.otto/tesla-microservice "0.1.15"]
[de.otto/tesla-jetty "0.1.0"]
```
Add the server to your system before you start it. Pass any additional dependencies of the server (```:example-page``` in this case).
```clojure
(system/start (serving-with-jetty/add-server (example-system {}) :example-page))
```
A working example for this is in the [simple-example](https://github.com/otto-de/tesla-examples/tree/master/simple-example).
You can also use the ```->```-threading macro as demonstrated in the [mongo-example](https://github.com/otto-de/tesla-examples/tree/master/mongo-example).
### 0.1.14
The `routes`-component was abandoned in favour of the `handler`-component.
In the ring library, handlers are the thing to push around (wrapping routes and middleware). You can choose your routing library now. Instead of [compojure](https://github.com/weavejester/compojure) you could also use e.g. [bidi](https://github.com/juxt/bidi).
Change components relying on the old ```routes```-component should be trivial: Instead of adding a vector of (compojure)-routes using ```de.otto.tesla.stateful.routes/register-routes```,
```clojure
(routes/register-routes
routes
[(c/GET "/test" [] (test-fn))])
```
just add a single ring handler using ```de.otto.tesla.stateful.handler/register-handler``` like this:
```clojure
(handlers/register-handler
handler
(c/routes (c/GET "/test" [] (test-fn))))
```
Add multiple routes like this:
```clojure
(handlers/register-handler
handler
(c/routes (c/GET "/route1" [] (test-fn))
(c/GET "/route1" [] (test-fn2))))
```
Note that the keyword for the dependency changed from ```:routes``` to ```:handler``` in the base system.
### 0.1.13
Specific logging-dependencies and the escaping-messageconverter have been removed. You now have to (read: you are able to) configure logging yourself in your app. To exactly restore the old behaviour add these dependencies to you own application:
```clojure
[org.slf4j/slf4j-api "1.7.12"]
[ch.qos.logback/logback-core "1.1.3"]
[ch.qos.logback/logback-classic "1.1.3"]
[de.otto/escaping-messageconverter "0.1.1"]
```
in your ```logback.xml``` replace
```xml
<conversionRule conversionWord="mescaped"
converterClass="de.otto.tesla.util.escapingmessageconverter" />
```
with
```xml
<conversionRule conversionWord="mescaped"
converterClass="de.otto.util.escapingmessageconverter" />
```
================================================
FILE: Dockerfile
================================================
# This is an example Dockerfile for running a tesla-microservice app in a docker container.
#
# Instructions:
# 1. build uber jar:
# ./lein.sh clean
# ./lein.sh uberjar
# 2. build docker image
# docker build -t tesla-example:latest .
# 3. run docker container
# docker run -d -p 8080:8080 tesla-example:latest
FROM centos:6
MAINTAINER Felix Bechstein <felix.bechstein@otto.de>
EXPOSE 8080
# prepare image
RUN yum install -y java-1.8.0-openjdk-headless
USER daemon
# set command line
CMD ["java", "-Dlog_level=info", "-jar", "/tesla-microservice-standalone.jar"]
# instead of logging to stdout, you may log to file in /log. create volume or mount host volume to /log
# RUN mkdir /log && chown daemon /log
# CMD ["java", "-Dlog_level=info", "-Dlog_appender=fileAppender", "-Dlog_location=/log", "-jar", "/tesla-microservice-standalone.jar"]
# drop in uber jar
ADD target/tesla-microservice-*-standalone.jar /tesla-microservice-standalone.jar
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: MAINTAINERS
================================================
Team Tesla <tesla@otto.de>
================================================
FILE: OSSMETADATA
================================================
osslifecycle=active
================================================
FILE: README.md
================================================
# tesla-microservice
> "If Edison had a needle to find in a haystack, he would proceed at once with the diligence of the bee to examine straw after straw until he found the object of his search." - Nikola Tesla
This is the common basis for some of otto.de's microservices. It is written in clojure using the [component framework](https://github.com/stuartsierra/component).
[](http://clojars.org/de.otto/tesla-microservice)
[](https://travis-ci.org/otto-de/tesla-microservice)
[](http://jarkeeper.com/otto-de/tesla-microservice)
## Breaking changes
_tesla-microservice_ is used for a number of different services now. Still it is a work in progress. See [CHANGES.md](./CHANGES.md) for instructions on breaking changes.
## Features included
* Load configuration from filesystem.
* Aggregate a status.
* Execute functions with a scheduler
* Reply to a health check.
* Deliver a json status report.
* Report to graphite using the metrics library.
* Manage handlers using ring.
* Optional auto-hot-reloading of changed source files
* Shutdown gracefully. If necessary delayed, so load-balancers have time to notice.
## Examples
* A growing set of example applications can be found at [tesla-examples](https://github.com/otto-de/tesla-examples).
* David & Germán created an example application based, among other, on tesla-microservice. They wrote a very instructive [blog post about it](http://blog.agilityfeat.com/2015/03/clojure-walking-skeleton/)
* Moritz created [tesla-pubsub-service](https://bitbucket.org/DerGuteMoritz/tesla-pubsub-service). It showcases how to connect components via core.async channels. Also the embedded jetty was replaced by immutant.
### Scheduler
The scheduler wraps a thread-pool which can be used for scheduling tasks. It is based on [overtones at-at](https://github.com/overtone/at-at) project.
To actually use it you have to pass the `:scheduler` as a dependency to the component in which it should be used.
Afterwards you can schedule tasks using the overtone api like this:
```clj
(overtone.at-at/every 100 #(println "Hello world") (de.otto.tesla.stateful.scheduler/pool scheduler) :desc "HelloWord Task")
```
The overtone-pool wrapped by the scheduler can be configured by the config-entry `:scheduler`. (See `overtone.at-at/mk-pool`)
By default the pool holds no threads.
### app-status
The app-status indicates the current status of your microservice. To use it you can register a status function to it.
Here is a simple example for a function that checks if an atom is empty or not.
```clj
(de.otto.tesla.stateful.app-status/register-status-fun app-status #(status atom))
```
The `app-status` is injected under the keyword :app-status from the base system.
```clj
(defn status [atom]
(let [status (if @atom :error :ok)
message (if @atom "Atom is empty" "Atom is not empty")]
(de.otto.status/status-detail :status-id status message)))
```
For further information and usages take a look at the: [status library](https://github.com/otto-de/status)
## Choosing a server
As of version ```0.1.15``` there is no server included any more directly in _tesla-microservice_.
This gives you the freedom to a) not use any server at all (e.g. for embedded use) b) choose another server e.g. a non-blocking one like httpkit or immutant. The available options are:
* [tesla-jetty](https://github.com/otto-de/tesla-jetty): The tried and tested embedded jetty.
* [tesla-httpkit](https://github.com/otto-de/tesla-httpkit): The non-blocking httpkit.
## Configuring
Applications build with `tesla-microservices` can be configured via
`edn`-files, that have to be located in the class path.
For backwards compatibility, it is also possible to load config from `properties`-files.
See below for noteworthy differences.
### Order of loading and merging
1. A file named `default.edn` is loaded as a resource from classpath if present.
2. A file either named `application.edn` or overridden by the ENV-variable `$CONFIG_FILE`
is loaded as a resource or, if that is not possible, from the filesystem.
3. A file name `local.edn` is loaded from classpath if present.
The configuration hash-map in those files is loaded and merged in the
specified order. Which mean configurations for the same key is overridden
by the latter occurrence.
### ENV-variables
In contrast to former versions of `tesla-microservice` ENV-variables are not
merged into the configuration.
But you can easily specify ENV-variables, that should be accessible in
your configuration:
```edn
{
:my-app-secret #ts/env [:my-env-dep-app-secret "default"]
}
```
ENV-variables are read with [environ](https://github.com/weavejester/environ). To see
which keyword represents which ENV-var have a look in their docs.
### Configuring via properties files
For backwards compatibility, it is also possible to load config from `properties`-files.
You'll have to pass `{:property-file-preferred true}` as a runtime config to the base-system.
It is not possible to load individual environment variables when using properties config.
Adding `:merge-env-to-properties-config true` to the runtime config will add all system properties
and environment variables, overiding any config from files.
### Reporters
Applications utilizing Tesla-Microservice can use [iapetos prometheus client](https://github.com/xsc/iapetos) for monitoring.
Metrics are send by reporters which can be configured using the `:metrics` keyword.
Each configured reporter will start at system startup automatically.
See example configuration below for all supported reporters.
```clojure
:metrics {:graphite {:host "localhost"
:port "2003"
:prefix "my.prefix"
:interval-in-s 60
:include-hostname :first-part}
:prometheus {:metrics-path "/metrics"}}
```
## Automatic hot-reloading of changed source files
Restarting the whole system after a small change can be cumbersome.
A _tesla-microservice_ can detect changes to your source files and
load them into a running server. Add this to your config, to check
for changes on each request to your system:
```edn
{:handler {:hot-reload? true}}
```
_Note_: This should only be enabled in development mode.
Use your `local.edn` to enable this feature safely.
You can add a `private.edn` as well for personal configurations. This file should be added to your `.gitignore`.
## Securing internal info endpoints
The Tesla-Microservice comes with endpoints that hold information about the internal state of your application.
Those endpoints can be the app-status or even metrics (Prometheus, see above).
To secure those endpoints you can provide an authentication-middleware to the base-system.
E.g.:
```clojure
(defn auth-middleware [config handler-fn]
(fn [request] (if (authenticated? config request)
(handler-fn request)
{:status 401 :body "access denied"})))
(defn example-system [runtime-config]
(-> (de.otto.tesla.system/base-system runtime-config auth-middleware)))
```
## Addons
The basis included is stripped to the very minimum. Additional functionality is available as addons:
* [tesla-zookeeper-observer](https://github.com/otto-de/tesla-zookeeper-observer): Read only access to zookeeper.
* [tesla-mongo-connect](https://github.com/otto-de/tesla-mongo-connect): Read/write access to mongodb.
* [tesla-cachefile](https://github.com/otto-de/tesla-cachefile): Read and write a cachefile. Locally or in hdfs.
More features will be released at a later time as separate addons.
## FAQ
**Q:** Is it any good? **A:** Yes.
**Q:** Why tesla? **A:** It's a reference to the ingenious scientist and inventor.
**Q:** Are there alternatives? **A:** Yes. You might want to look at [modularity.org](https://modularity.org/), [system](https://github.com/danielsz/system) and [duct](https://github.com/weavejester/duct).
## Initial Contributors
Christian Stamm, Felix Bechstein, Ralf Sigmund, Kai Brandes, Florian Weyandt
## License
Released under [Apache License 2.0](http://www.apache.org/licenses/LICENSE-2.0) license.
================================================
FILE: dev-resources/logback.xml
================================================
<configuration>
<contextName>tesla</contextName>
<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<append>true</append>
<encoder>
<pattern>%d{ISO8601} %-5p logger=%c thread=%t msg="%m"%n</pattern>
</encoder>
</appender>
<appender name="fileAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${log_location}/de.otto.tesla.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${log_location}/de.otto.tesla.log.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>5</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d{ISO8601} %-5p logger=%c thread=%t msg="%m"%n</pattern>
</encoder>
</appender>
<root level="${log_level:-info}">
<appender-ref ref="${log_appender:-consoleAppender}"/>
</root>
</configuration>
================================================
FILE: project.clj
================================================
(defproject de.otto/tesla-microservice "0.17.9-SNAPSHOT"
:description "basic microservice."
:url "https://github.com/otto-de/tesla-microservice"
:license {:name "Apache License 2.0"
:url "http://www.apache.org/license/LICENSE-2.0.html"}
:scm {:name "git"
:url "https://github.com/otto-de/tesla-microservice"}
:repositories [["releases" {:url "https://repo.clojars.org"
:creds :gpg}]]
:dependencies [[org.clojure/data.json "2.5.1"]
[org.clojure/tools.logging "1.3.0"]
[de.otto/status "0.1.3"]
[de.otto/goo "1.2.12"]
[clojure.java-time "1.4.3"]
[clojurewerkz/propertied "1.3.0"]
[com.stuartsierra/component "1.1.0"]
[compojure "1.7.1"]
[environ "1.2.0"]
[overtone/at-at "1.4.65"]
[ring/ring-core "1.13.0"]
[ring/ring-devel "1.13.0"]
[ring-basic-authentication "1.2.0"]]
:exclusions [org.clojure/clojure
org.slf4j/slf4j-nop
org.slf4j/slf4j-log4j12
log4j
commons-logging/commons-logging]
:lein-release {:deploy-via :clojars}
:filespecs [{:type :path :path "test-utils"}]
:test-selectors {:default (constantly true)
:integration :integration
:unit :unit
:all (constantly true)}
:profiles {:uberjar {:aot :all}
:dev {:dependencies [[org.clojure/clojure "1.12.0"]
[org.slf4j/slf4j-api "2.0.16"]
[ch.qos.logback/logback-core "1.5.14"]
[ch.qos.logback/logback-classic "1.5.14"]
[ring-mock "0.1.5"]
[org.clojure/data.codec "0.2.0"]]
:plugins [[lein-release/lein-release "1.0.9"]]}}
:test-paths ["test" "test-resources" "test-utils"])
================================================
FILE: resources/default.edn
================================================
{
:status-url "/status"
:health-url "/health"
; metering
; take a look at de.otto.tesla.stateful.metering
; for all available metrics reporters and their configs
:metric {:console {:interval-in-s 120}}
:scheduler {:cpu-count 0}
:server-port "8080"
:jetty-options {:send-server-version? false}
; use this if you need a grace-time during which
; the system reports unhealthy before shutting down.
;:wait-ms-on-stop "10000"
}
================================================
FILE: resources/default.properties
================================================
status.url=/status
health.url=/health
### metering
metering.reporter=console
console.interval.seconds=100
### activate graphite like this:
#metering.reporter=graphite
#graphite.host=localhost
#graphite.port=2003
#graphite.prefix=my-app-prefix
#graphite.interval.seconds=60
server.port=8080
## use this if you need a grace-time during which
## the system reports unhealthy before shutting down.
#wait.ms.on.stop=10000
================================================
FILE: src/data_readers.clj
================================================
{
ts/env de.otto.tesla.util.env_var_reader/read-env-var
}
================================================
FILE: src/de/otto/tesla/middleware/exceptions.clj
================================================
(ns de.otto.tesla.middleware.exceptions
(:require [clojure.tools.logging :as log]))
(defn exceptions-to-500 [handler]
(fn [request]
(try
(handler request)
(catch Exception e
(log/error e "Will return 500 to client because of this error.")
{:status 500
:body (.getMessage e)}))))
================================================
FILE: src/de/otto/tesla/stateful/app_status.clj
================================================
(ns de.otto.tesla.stateful.app-status
"This component renders a status page consisting of instance- and configuration-info
as well as dynamic status of other components"
(:require [com.stuartsierra.component :as component]
[compojure.core :as c]
[clojure.data.json :as json :only [write-str]]
[clojure.tools.logging :as log]
[clojure.string :as str]
[java-time.api :as jt]
[environ.core :as env]
[de.otto.tesla.stateful.handler :as handlers]
[de.otto.status :as s]
[ring.middleware.basic-authentication :as ba]
[de.otto.tesla.util.sanitize :as san]
[de.otto.tesla.stateful.configuring :as configuring]
[de.otto.goo.goo :as goo]
[de.otto.tesla.stateful.auth :as auth]))
(defn keyword-to-status [kw]
(str/upper-case (name kw)))
(defn status-details-to-json [details]
(into {} (map
(fn [[k v]]
{k (update-in v [:status] keyword-to-status)})
details)))
(defn system-infos [config]
{:systemTime (jt/format :iso-date (jt/local-date-time))
:hostname (configuring/external-hostname config)
:port (configuring/external-port config)})
(defn aggregation-strategy [config]
(if (= (get-in config [:status-aggregation]) "forgiving")
s/forgiving-strategy
s/strict-strategy))
(defn create-complete-status [self]
(let [config (get-in self [:config :config])
version-info (get-in self [:config :version])
aggregate-strategy (:status-aggregation self)
extra-info {:name (:name config)
:version (:version version-info)
:git (:commit version-info)
:configuration (san/hide-passwds config)}]
(assoc
(s/aggregate-status :application
aggregate-strategy
@(:status-functions self)
extra-info)
:system (system-infos (:config self)))))
(defn status-response-body [self]
(-> (create-complete-status self)
(update-in [:application :statusDetails] status-details-to-json)
(update-in [:application :status] keyword-to-status)))
;; This should apply to the specification at
;; http://spec.otto.de/media_types/application_vnd_otto_monitoring_status_json.html .
;; Right now it applies only partially.
(defn status-response [self _]
{:status 200
:headers {"Content-Type" "application/json"}
:body (json/write-str (status-response-body self))})
(defn register-status-fun [self fun]
(swap! (:status-functions self) #(conj % fun)))
(defn path-filter [self handler]
(let [status-path (get-in self [:config :config :status-url] "/status")]
(c/GET status-path request (handler request))))
(defn mk-handler [{:keys [auth] :as self}]
((->> (partial status-response self)
(goo/timing-middleware)
(auth/wrap-auth auth)
(partial path-filter self))))
(defrecord ApplicationStatus [config handler auth]
component/Lifecycle
(start [self]
(log/info "-> starting Application Status")
(let [new-self (assoc self
:status-aggregation (aggregation-strategy (:config config))
:status-functions (atom []))]
(handlers/register-handler handler (mk-handler new-self))
(goo/register-gauge! :build/info {:labels [:version :revision] :description "Constant '1' value labeled by version and revision of the service."})
(goo/inc! :build/info {:version (-> config :version :version) :revision (-> config :version :commit)})
new-self))
(stop [self]
(log/info "<- stopping Application Status")
self))
(defn new-app-status
([]
(map->ApplicationStatus {})))
================================================
FILE: src/de/otto/tesla/stateful/auth.clj
================================================
(ns de.otto.tesla.stateful.auth
"This component handles authentication."
(:require [clojure.tools.logging :as log]
[com.stuartsierra.component :as component]))
(defn no-auth-middleware [_config handler-fn]
(log/warn "You are using no authentication...Is this desired?")
(fn [request]
(handler-fn request)))
(defn wrap-auth [self handler-fn]
((:auth-mw self) handler-fn))
(defrecord Auth [config auth-mw]
component/Lifecycle
(start [self]
(log/info "-> starting AuthMiddleware")
(let [auth-mw (partial (or auth-mw no-auth-middleware) (:config config))
new-self (assoc self :auth-mw auth-mw)]
new-self))
(stop [self]
(log/info "<- stopping AuthMiddleware")
self))
(defn new-auth
([auth-mw]
(map->Auth {:auth-mw auth-mw})))
================================================
FILE: src/de/otto/tesla/stateful/configuring.clj
================================================
(ns de.otto.tesla.stateful.configuring
"This component is responsible for loading the configuration."
(:require [com.stuartsierra.component :as component]
[clojurewerkz.propertied.properties :as p]
[clojure.tools.logging :as log]
[clojure.java.io :as io]
[de.otto.tesla.util.keyword :as kwutil]
[environ.core :as env :only [env]]
[de.otto.tesla.util.env_var_reader :only [read-env-var]]
[de.otto.tesla.util.sanitize :as san])
(:import (java.io PushbackReader)))
(defn deep-merge
"Recursively merges maps. If vals are not maps, the last value wins."
[& vals]
(if (every? map? vals)
(apply merge-with deep-merge vals)
(last vals)))
(defn- load-properties-from-resource [resource]
(kwutil/sanitize-keywords
(p/properties->map
(p/load-from resource) false)))
(defn resolve-file [name type]
(cond
(= :resource type) (io/resource name)
(and (= :file type) (.exists (io/file name))) (io/file name)
(and (= :resource-or-file type) (io/resource name)) (io/resource name)
(and (= :resource-or-file type) (.exists (io/file name))) (io/file name)))
(defn- load-properties [name type]
(when-let [resource (resolve-file name type)]
(load-properties-from-resource resource)))
(defn- load-edn [name type]
(when-let [resource (resolve-file name type)]
(log/debugf "Reading %s" name)
(-> resource
(io/reader)
(PushbackReader.)
(read))))
(defn load-merge [load-fn merge-fn ending runtime-config]
(let [default-cfg-name (or (:default-cfg-file-name runtime-config) "default")
defaults (load-fn (str default-cfg-name ending) :resource)
application (load-fn (or (:config-file env/env) (str "application" ending)) :resource-or-file)
local (load-fn (str "local" ending) :resource)
private (load-fn (str "private" ending) :resource)
configs (filter some? [defaults application local private runtime-config])]
(apply merge-fn configs)))
(defn load-config-from-edn-files [runtime-config]
(load-merge load-edn deep-merge ".edn" runtime-config))
(defn load-config-from-properties-files [runtime-config]
(let [loaded (load-merge load-properties merge ".properties" runtime-config)]
(if (:merge-env-to-properties-config runtime-config)
(merge loaded env/env)
loaded)))
(defn load-and-merge [runtime-config]
(if-not (:property-file-preferred runtime-config)
(load-config-from-edn-files runtime-config)
(load-config-from-properties-files runtime-config)))
;; Load config on startup.
(defrecord Configuring [runtime-config]
component/Lifecycle
(start [self]
(log/info "-> loading configuration.")
(log/info runtime-config)
(let [config (load-and-merge runtime-config)]
(log/info "-> using configuration:\n" (with-out-str (clojure.pprint/pprint (san/hide-passwds config))))
(assoc self :config config
:version (load-properties "version.properties" :resource))))
(stop [self]
(log/info "<- stopping configuration.")
self))
(defn new-config [runtime-config]
(map->Configuring {:runtime-config runtime-config}))
;; The hostname and port visble from the outside are different for
;; different environments.
;; These methods default to Marathon defaults.
(defn external-hostname [{:keys [config]}]
(or (:host-name config)
(:host env/env) (:host-name env/env) (:hostname env/env)
"localhost"))
(defn server-port [config]
(:server-port config))
;; see above
(defn external-port [{:keys [config]}]
(or (server-port config)
(:port0 env/env) (:host-port env/env) (:server-port env/env)))
================================================
FILE: src/de/otto/tesla/stateful/handler.clj
================================================
(ns de.otto.tesla.stateful.handler
"This component is responsible for collecting HTTP handlers in order to provide them to a server component."
(:require [com.stuartsierra.component :as component]
[de.otto.tesla.middleware.exceptions :as ex]
[clojure.tools.logging :as log]
[ring.middleware.reload :refer [wrap-reload]]))
(defn- single-handler-fn [{:keys [registered-handlers]}]
(fn [request]
(some (fn [h] (h request)) @registered-handlers)))
(defn register-handler [{:keys [registered-handlers]} new-handler-fn]
(swap! registered-handlers conj (ex/exceptions-to-500 new-handler-fn)))
(defn handler [{:keys [config] :as self}]
(if (get-in config [:config :handler :hot-reload?])
(wrap-reload (single-handler-fn self))
(single-handler-fn self)))
(defrecord Handler [config]
component/Lifecycle
(start [self]
(log/info "-> starting Handler")
(assoc self :registered-handlers (atom [])))
(stop [self]
(log/info "<- stopping Handler")
self))
(defn new-handler []
(map->Handler {}))
================================================
FILE: src/de/otto/tesla/stateful/health.clj
================================================
(ns de.otto.tesla.stateful.health
"This component provides a health-check endpoint which can be used to orchestrate app shutdown with load balancers."
(:require [com.stuartsierra.component :as component]
[compojure.core :as c]
[clojure.tools.logging :as log]
[de.otto.tesla.stateful.handler :as handler]
[de.otto.goo.goo :as goo]))
(def healthy-response {:status 200
:headers {"Content-Type" "text/plain"}
:body "HEALTHY"})
(def unhealthy-response {:status 423
:headers {"Content-Type" "text/plain"}
:body "UNHEALTHY"})
(defn health-response [self _]
(if @(:locked self)
unhealthy-response
healthy-response))
(defn path-filter [self handler]
(let [health-path (get-in self [:config :config :health-url] "/health")]
(c/GET health-path request (handler request))))
(defn make-handler
[self]
(->> (partial health-response self)
goo/timing-middleware
(path-filter self)))
(defn lock-application [self]
(goo/update! :health/locked 0)
(reset! (:locked self) true))
(defrecord Health [config handler]
component/Lifecycle
(start [self]
(log/info "-> Starting healthcheck.")
(let [new-self (assoc self :locked (atom false))]
(handler/register-handler handler (make-handler new-self)) ;; TODO: use config directly
(goo/register-gauge! :health/locked {})
(goo/update! :health/locked 1)
new-self))
(stop [self]
(log/info "<- Stopping Healthcheck")
self))
(defn new-health []
(map->Health {}))
================================================
FILE: src/de/otto/tesla/stateful/keep_alive.clj
================================================
(ns de.otto.tesla.stateful.keep-alive
"This component is responsible for keeping the system alive by creating a non-deamonized noop thread."
(:require [clojure.tools.logging :as log]
[com.stuartsierra.component :as component])
(:import (java.util.concurrent CountDownLatch)))
(defn exit-keep-alive []
(log/info "<- stopping keepalive thread: " (.getName (Thread/currentThread))))
(defn enter-keep-alive []
(log/info "-> starting keepalive thread: " (.getName (Thread/currentThread))))
(defn wait-for-count-down-latch [cdl]
(enter-keep-alive)
(.await cdl)
(exit-keep-alive))
(defn start-keep-alive-thread [cd-latch]
(doto (Thread. ^Runnable (partial wait-for-count-down-latch cd-latch) "tesla-ms-keep-alive")
(.start)))
(defrecord KeepAlive [cd-latch]
component/Lifecycle
(start [self]
(log/info "-> starting keepalive")
(let [cd-latch (CountDownLatch. 1)]
(assoc self
:thread (start-keep-alive-thread cd-latch)
:cd-latch cd-latch)))
(stop [self]
(log/info "<- stopping keepalive")
(.countDown cd-latch)
(dissoc self :thread)))
(defn new-keep-alive []
(map->KeepAlive {}))
================================================
FILE: src/de/otto/tesla/stateful/metering.clj
================================================
(ns de.otto.tesla.stateful.metering
"This component handles exporting of (prometheus) metrics"
(:require
[com.stuartsierra.component :as component]
[clojure.tools.logging :as log]
[de.otto.goo.goo :as goo]
[de.otto.tesla.stateful.handler :as handlers]
[compojure.core :as c]
[overtone.at-at :as at]
[de.otto.tesla.stateful.auth :as auth]))
(defn write-to-console []
(log/info "Metrics Reporting:\n" (goo/text-format)))
(defn start-console-reporter [console-config scheduler]
(let [interval-in-ms (* 1000 (:interval-in-s console-config))]
(log/info "Starting metrics console reporter")
(at/every interval-in-ms write-to-console (:pool scheduler) :desc "Console-Reporter")))
(defn metrics-response [_]
(fn [_request] {:status 200
:headers {"Content-Type" "text/plain"}
:body (goo/text-format)}))
(defn- path-filter [metrics-path handler]
(c/GET metrics-path request (handler request)))
(defn register-metrics-endpoint [{metrics-path :metrics-path} {:keys [handler auth]}]
(log/info "Register metrics prometheus endpoint")
(handlers/register-handler handler ((->> (metrics-response handler)
(goo/timing-middleware)
(auth/wrap-auth auth)
(partial path-filter metrics-path)))))
(defn- start-reporter! [{:keys [scheduler] :as self} [reporter-type reporter-config]]
(case reporter-type
:console (start-console-reporter reporter-config scheduler)
:prometheus (register-metrics-endpoint reporter-config self)))
(defn- start-reporters! [{:keys [config] :as self}]
(let [available-reporters (get-in config [:config :metrics])]
(run! (partial start-reporter! self) available-reporters)))
(defrecord Metering [config handler scheduler auth]
component/Lifecycle
(start [self]
(log/info "-> starting metering.")
(goo/register-counter! :metering/errors {:labels [:error :metric-name]})
(assoc self :reporters (start-reporters! self)))
(stop [self]
(log/info "<- stopping metering")
self))
(defn new-metering
([]
(map->Metering {})))
================================================
FILE: src/de/otto/tesla/stateful/scheduler.clj
================================================
(ns de.otto.tesla.stateful.scheduler
"This components maintains a thread pool which can be used to schedule activities."
(:require [com.stuartsierra.component :as c]
[clojure.tools.logging :as log]
[overtone.at-at :as ot]
[de.otto.tesla.stateful.app-status :as app-status])
(:import (java.util.concurrent ScheduledThreadPoolExecutor)))
(defn- as-seq [v]
(apply concat v))
(defn- new-ot-pool [config]
(let [pool-config (get-in config [:config :scheduler])]
(apply ot/mk-pool (as-seq pool-config))))
(defn- as-readable-time [l]
(.format (java.text.SimpleDateFormat. "EEEE HH':'mm':'ss's'") l))
(defn- job-details [{:keys [id created-at initial-delay desc ms-period scheduled?]}]
[(keyword (str id)) {:desc desc
:createdAt (as-readable-time created-at)
:initialDelay initial-delay
:msPeriod ms-period
:scheduled? @scheduled?}])
(defn- pool-details [{:keys [pool-atom]}]
(when pool-atom
(let [^ScheduledThreadPoolExecutor thread-pool (:thread-pool @pool-atom)]
{:active (.getActiveCount thread-pool)
:queueSize (.size (.getQueue thread-pool))
:poolSize (.getPoolSize thread-pool)})))
(defn- scheduler-app-status [{:keys [pool]}]
{:scheduler {:status :ok
:poolInfo (pool-details pool)
:scheduledJobs (into {} (map job-details (ot/scheduled-jobs pool)))}})
(defn exception-to-log [desc f]
(try (f)
(catch Exception e
(log/error e (str "Exception during scheduled job: " desc)))))
(defn every
"Calls fun every ms-period, and takes an optional initial-delay for
the first call in ms. Returns a scheduled-fn which may be cancelled
with cancel. All exceptions are catched and logged.
Default options are
{:initial-delay 0 :desc \"\"}"
[{:keys [pool]} ms-period fun & {:keys [initial-delay desc]
:or {initial-delay 0
desc ""}}]
(ot/every ms-period (partial exception-to-log desc fun) pool :initial-delay initial-delay :desc desc))
(defn interspaced
"Calls fun repeatedly with an interspacing of ms-period, i.e. the next
call of fun will happen ms-period milliseconds after the completion
of the previous call. Also takes an optional initial-delay for the
first call in ms. Returns a scheduled-fn which may be cancelled with
cancel. All exceptions are catched and logged.
Default options are
{:initial-delay 0 :desc \"\"}"
[{:keys [pool]} ms-period fun & {:keys [initial-delay desc]
:or {initial-delay 0
desc ""}}]
(ot/interspaced ms-period (partial exception-to-log desc fun) pool :initial-delay initial-delay :desc desc))
(defrecord Scheduler [config app-status]
c/Lifecycle
(start [self]
(log/info "-> Start Scheduler")
(let [new-self (assoc self :pool (new-ot-pool config))]
(app-status/register-status-fun app-status (partial scheduler-app-status new-self))
new-self))
(stop [self]
(log/info "<- Stop Scheduler")
(when-let [pool (:pool self)]
(ot/stop-and-reset-pool! pool))
self))
(defn new-scheduler []
(map->Scheduler {}))
================================================
FILE: src/de/otto/tesla/system.clj
================================================
(ns de.otto.tesla.system
(:require [com.stuartsierra.component :as c]
[de.otto.tesla.stateful.app-status :as app-status]
[de.otto.tesla.stateful.health :as health]
[de.otto.goo.goo :as goo]
[de.otto.tesla.stateful.configuring :as configuring]
[de.otto.tesla.stateful.metering :as metering]
[de.otto.tesla.stateful.keep-alive :as keep-alive]
[de.otto.tesla.stateful.scheduler :as scheduler]
[clojure.tools.logging :as log]
[environ.core :as env :only [env]]
[de.otto.tesla.stateful.handler :as handler]
[de.otto.tesla.stateful.auth :as auth])
(:import (clojure.lang ExceptionInfo)))
(defn wait! [system]
(when-let [wait-time (get-in system [:config :config :wait-ms-on-stop])]
(try
(log/info "<- Waiting " wait-time " milliseconds.")
(Thread/sleep (Long/parseLong wait-time))
(catch Exception e (log/error e)))))
(defn- exit [code]
(System/exit code))
(defn- try-stop [system]
(try
(c/stop system)
(log/info "System stopped. Bye.")
(catch Exception ex
(log/error ex "Error on stopping the system.")
(exit 1))))
(defn stop [system]
(when-let [sdt (:sdt system)]
(.removeShutdownHook (Runtime/getRuntime) sdt))
(when-let [health (:health system)]
(log/info "<- System will be stopped. Setting lock.")
(health/lock-application health)
(wait! system))
(log/info "<- Stopping system.")
(try-stop system))
(defn- try-start [system]
(try
(c/start system)
(catch ExceptionInfo e
(log/error (c/ex-without-components e) "Going to shut down because of this error.")
(-> e (ex-data) :system (try-stop)))))
(defn start [system]
(log/info "-> Starting system.")
(let [start-timestamp (System/currentTimeMillis)
started (try-start system)]
(log/info "-> System completely started.")
(goo/register-counter! :system-startups {:description "Counts startups."})
(goo/register-counter! :system-startup-duration-seconds {:description "Measures the startup duration of the system."})
(goo/inc! :system-startups)
(goo/inc! :system-startup-duration-seconds (/ (- (System/currentTimeMillis) start-timestamp) 1000))
(if (map? started)
(let [sdt (Thread. ^Runnable (partial stop started))]
(.addShutdownHook (Runtime/getRuntime) sdt)
(assoc started :sdt sdt))
started)))
(map? [])
(defn base-system [runtime-config & [auth-mw]]
(c/system-map
:keep-alive (keep-alive/new-keep-alive)
:config (c/using (configuring/new-config runtime-config) [:keep-alive])
:handler (c/using (handler/new-handler) [:config])
:metering (c/using (metering/new-metering) [:config :handler :scheduler :auth])
:health (c/using (health/new-health) [:config :handler])
:app-status (c/using (app-status/new-app-status) [:config :handler :auth])
:scheduler (c/using (scheduler/new-scheduler) [:config :app-status])
:auth (c/using (auth/new-auth auth-mw) [:config])))
================================================
FILE: src/de/otto/tesla/util/env_var_reader.clj
================================================
(ns de.otto.tesla.util.env_var_reader
(:require [environ.core :as env]))
(defn read-env-var
([[env-var-key fallback]]
(or
(get env/env env-var-key fallback)
"")))
================================================
FILE: src/de/otto/tesla/util/keyword.clj
================================================
(ns de.otto.tesla.util.keyword
(:require [clojure.string :as str]))
;implementation is copied from environ 1.0.0
(defn- keywordize [s]
(-> (str/lower-case s)
(str/replace "_" "-")
(str/replace "." "-")
(keyword)))
(defn sanitize-keywords [m]
(->> m
(map (fn [[k v]] [(keywordize k) v]))
(into {})))
;; removed keywordize-keys, provided by clojure.walk/keywordize-keys
================================================
FILE: src/de/otto/tesla/util/sanitize.clj
================================================
(ns de.otto.tesla.util.sanitize)
(def checklist ["password" "pw" "passwd" "private" "secret" "token"])
(defn hide-passwd [k v]
(if (some true? (map #(.contains (name k) %) checklist))
"***"
v))
(defn hide-passwds [map]
(reduce
(fn [new-map [k v]] (assoc new-map k (if (map? v) (hide-passwds v) (hide-passwd k v))))
{} map))
================================================
FILE: test/de/otto/tesla/reporter/prometheus_test.clj
================================================
(ns de.otto.tesla.reporter.prometheus-test
(:require [clojure.test :refer :all]
[de.otto.tesla.system :as system]
[com.stuartsierra.component :as c]
[clojure.data.codec.base64 :as b64]
[de.otto.tesla.stateful.metering :as metering]
[de.otto.tesla.stateful.handler :as handler]
[ring.mock.request :as mock]
[ring.middleware.basic-authentication :as ba]))
(defn- to-base64 [original]
(String. ^bytes (b64/encode (.getBytes original)) "UTF-8"))
(defn- auth-header [request user password]
(mock/header request "authorization" (str "Basic " (to-base64 (str user ":" password)))))
(defn system [runtime-config auth-middleware]
(-> (system/base-system runtime-config auth-middleware)
(dissoc :server)))
(defn- handlers [runtime-config & [auth-middleware]]
(let [system (system runtime-config auth-middleware)
started-system (c/start-system system)]
(handler/handler (:handler started-system))))
(defn- rc-metrics-request [system-handler user password]
(-> (mock/request :get "/metrics")
(auth-header user password)
(system-handler)
:status))
(deftest authentication
(let [config {:metrics {:prometheus {:metrics-path "/metrics"}}}
auth-fun (fn [usr pw]
(and
(= "some-user" usr) (= "some-password" pw)))
auth-middleware (fn [_config handler]
(ba/wrap-basic-authentication handler auth-fun))
system-handler (handlers config auth-middleware)]
(testing "it should allow access if authentication succeeds"
(is (= 200 (rc-metrics-request system-handler "some-user" "some-password"))))
(testing "it should deny access if authentication fails"
(is (= 401 (rc-metrics-request system-handler "some-user" "wrong password"))))))
================================================
FILE: test/de/otto/tesla/stateful/app_status_test.clj
================================================
(ns de.otto.tesla.stateful.app-status-test
(:require [clojure.test :refer :all]
[de.otto.tesla.stateful.app-status :as app-status]
[com.stuartsierra.component :as c]
[environ.core :as env]
[clojure.data.json :as json]
[clojure.tools.logging :as log]
[de.otto.tesla.util.test-utils :as u]
[de.otto.tesla.system :as system]
[de.otto.tesla.stateful.handler :as handler]
[ring.mock.request :as mock]
[de.otto.status :as s]
[de.otto.goo.goo :as goo]
[ring.middleware.basic-authentication :as ba]))
(defn- serverless-system [runtime-config & [auth-middleware]]
(dissoc
(system/base-system runtime-config auth-middleware)
:server))
(deftest ^:unit should-have-system-status-for-runtime-config
(u/with-started [system (serverless-system {:host-name "bar" :server-port "0123"})]
(let [status (:app-status system)
system-status (:system (app-status/status-response-body status))]
(is (= (:hostname system-status) "bar"))
(is (= (:port system-status) "0123"))
(is (not (nil? (:systemTime system-status)))))))
(deftest ^:unit host-name-and-port-on-app-status
(with-redefs [env/env {:host-name "foo" :server-port "1234"}]
(testing "should add host and port from env to app-status in property-file case"
(u/with-started [system (serverless-system {:property-file-preferred true :merge-env-to-properties-config true})]
(let [status (:app-status system)
system-status (:system (app-status/status-response-body status))]
(is (= (:hostname system-status) "foo"))
(is (= (:port system-status) "1234"))
(is (not (nil? (:systemTime system-status)))))))
(testing "should add host and port from env to app-status in edn-file case"
(u/with-started [system (serverless-system {} nil)]
(let [status (:app-status system)
system-status (:system (app-status/status-response-body status))]
(is (= (:hostname system-status) "foo"))
(is (= (:port system-status) "9991"))
(is (not (nil? (:systemTime system-status)))))))))
(defrecord MockStatusSource [response]
c/Lifecycle
(start [self]
(app-status/register-status-fun (:app-status self) #(:response self))
self)
(stop [self]
self))
(defn- mock-status-system [response]
(assoc (serverless-system {})
:mock-status
(c/using (map->MockStatusSource {:response response}) [:app-status])))
(deftest ^:unit should-show-applicationstatus
(u/with-started [started (mock-status-system {:mock {:status :ok
:message "nevermind"}})]
(let [status (:app-status started)
page (app-status/status-response status {})
_ (log/info page)
application-body (get (json/read-str (:body page)) "application")]
(testing "it shows OK as application status"
(is (= (get application-body "status")
"OK")))
(testing "it shows the substatus"
(is (= (get application-body "statusDetails")
{"mock" {"message" "nevermind"
"status" "OK"}
"scheduler" {"poolInfo" {"active" 0
"poolSize" 0
"queueSize" 0}
"scheduledJobs" {}
"status" "OK"}}))))))
(deftest ^:unit should-show-warning-as-application-status
(u/with-started [started (mock-status-system {:mock {:status :warning
:message "nevermind"}})]
(let [status (:app-status started)
page (app-status/status-response status {})
applicationStatus (get (get (json/read-str (:body page)) "application") "status")]
(is (= applicationStatus "WARNING")))))
(deftest ^:integration should-serve-status-under-configured-url
(testing "use the default url"
(u/with-started [started (serverless-system {})]
(let [handlers (handler/handler (:handler started))]
(is (= (:status (handlers (mock/request :get "/status")))
200)))))
(testing "use the configuration url"
(u/with-started [started (serverless-system {:status-url "/my-status"})]
(let [handlers (handler/handler (:handler started))]
(is (= (:status (handlers (mock/request :get "/my-status")))
200)))))
(testing "default should be overridden"
(u/with-started [started (serverless-system {:status-url "/my-status"})]
(let [handlers (handler/handler (:handler started))]
(is (= (handlers (mock/request :get "/status"))
nil)))))
(testing "response should be metered"
(goo/clear-default-registry!)
(u/with-started [started (serverless-system {})]
(let [handlers (handler/handler (:handler started))]
(handlers (mock/request :get "/status"))
(u/eventually (= 1.0
(last (.-buckets (.get ((goo/snapshot) :http/duration_in_s {:path "/status" :method :get :rc 200}))))))))))
(deftest should-add-version-properties-to-status
(testing "it should add the version properties"
(u/with-started [started (serverless-system {})]
(let [handlers (handler/handler (:handler started))
request (mock/request :get "/status")
status-map (json/read-json (:body (handlers request)))]
(is (= (get-in status-map [:application :version]) "test.version"))
(is (= (get-in status-map [:application :git]) "test.githash"))))))
(deftest determine-status-strategy
(testing "it should use strict stategy if none is configured"
(let [config {:status-aggregation nil}]
(is (= (app-status/aggregation-strategy config) s/strict-strategy))))
(testing "it should use forgiving stategy if forgiving is configured"
(let [config {:status-aggregation "forgiving"}]
(is (= (app-status/aggregation-strategy config) s/forgiving-strategy))))
(testing "it should use strict stategy if something else is configured"
(let [config {:status-aggregation "unknown"}]
(is (= (app-status/aggregation-strategy config) s/strict-strategy)))))
(defn start-authenticated-system [user password]
(let [config {}
auth-fn (fn [usr pw] (and (= user usr) (= password pw)))
auth-middleware (fn [_config handler] (ba/wrap-basic-authentication handler auth-fn))
system (serverless-system config auth-middleware)
started-system (c/start-system system)]
(handler/handler (:handler started-system))))
(deftest authentication
(let [handlers (start-authenticated-system "some-user" "some-password")]
(testing "it should allow access if authentication succeeds"
(is (= 200 (:status (handlers (mock/header (mock/request :get "/status") "authorization" "Basic c29tZS11c2VyOnNvbWUtcGFzc3dvcmQ="))))))
(testing "it should deny access if authentication fails"
(is (= 401 (:status (handlers (mock/header (mock/request :get "/status") "authorization" "Basic c29tZS11c2VyOnNvbWUtcGEzc3dvcmQ="))))))))
================================================
FILE: test/de/otto/tesla/stateful/configuring_test.clj
================================================
(ns de.otto.tesla.stateful.configuring-test
(:require [clojure.test :refer :all]
[de.otto.tesla.stateful.configuring :as configuring]
[com.stuartsierra.component :as component]
[clojure.java.io :as io]
[de.otto.tesla.util.test-utils :as u]
[environ.core :as env]))
(defn- test-system [rt-conf]
(-> (component/system-map
:conf (configuring/new-config rt-conf))))
(deftest referencing-env-properties
(testing "should return env-property if referenced in edn-config"
(with-redefs [env/env {:prop-without-fallback "prop-value"}]
(u/with-started [started (test-system {})
conf (get-in started [:conf :config])]
(is (= (:prop-without-fallback conf) "prop-value")))))
(testing "should return empty if env prop does not exist and fallback not provided"
(with-redefs [env/env {}]
(u/with-started [started (test-system {})
conf (get-in started [:conf :config])]
(is (= (:prop-without-fallback conf) ""))))))
(deftest ^:unit should-read-property-from-default-config
(testing "should be possible to prefer reading configs from property files"
(u/with-started [started (test-system {:property-file-preferred true})]
(let [conf (get-in started [:conf :config])]
(is (= (:foo-prop conf) "baz"))
(is (= (get-in conf [:foo :edn]) nil))))))
(deftest ^:unit should-read-property-from-default-edn-file
(u/with-started [started (test-system {})]
(let [edn-conf (get-in started [:conf :config])]
(is (= (:foo-prop edn-conf) nil))
(is (= (get-in edn-conf [:foo :edn]) "baz")))))
(deftest ^:unit should-read-property-from-private-edn-file
(u/with-started [started (test-system {})]
(let [conf (get-in started [:conf :config])]
(is (= (:very conf) :private)))))
(deftest ^:unit should-read-property-from-custom-edn-file
(with-redefs [env/env {:config-file "./test-resources/test.edn"}]
(u/with-started [started (test-system {})]
(let [edn-conf (get-in started [:conf :config])]
(is (= (get-in edn-conf [:health-url]) "/test/health"))
(is (= (get-in edn-conf [:foo :local]) true))
(is (= (get-in edn-conf [:foo :edn]) "baz"))))))
(deftest ^:unit should-ignore-missing-custom-edn-file
(with-redefs [env/env {:config-file "non-existing.edn"}]
(u/with-started [started (test-system {:runtime 123})]
(let [edn-conf (get-in started [:conf :config])]
(is (= (get-in edn-conf [:runtime]) 123))
(is (= (get-in edn-conf [:health-url]) "/health"))
(is (= (get-in edn-conf [:foo :edn]) "baz"))))))
(deftest ^:unit should-read-property-from-runtime-config
(u/with-started [started (test-system {:foo-rt "bat" :foo {:nested 123}})]
(let [edn-conf (get-in started [:conf :config])]
(is (= (:foo-prop edn-conf) nil))
(is (= (:foo-rt edn-conf) "bat"))
(is (= (get-in edn-conf [:foo :edn]) "baz"))
(is (= (get-in edn-conf [:foo :nested]) 123)))))
(deftest ^:unit should-read-default-properties
(testing "should read default properties from property-files"
(let [loaded-properties (configuring/load-config-from-properties-files {})]
(is (not (nil? (:server-port loaded-properties))))
(is (not (nil? (:metering-reporter loaded-properties))))))
(testing "should read default properties from edn-property-files"
(let [loaded-properties (configuring/load-config-from-edn-files {})]
(is (= "9991"
(:server-port loaded-properties)))
(is (nil? (:metering-reporter loaded-properties))))))
(deftest ^:unit determine-hostname-from-config-and-env-with-defined-precedence
(testing "it prefers a explicitly configured :host-name"
(with-redefs [env/env {:host "host" :host-name "host-name" :hostname "hostname"}]
(u/with-started [started (test-system {:host-name "configured"})]
(is (= "configured"
(configuring/external-hostname (:conf started)))))))
(testing "it falls back to env-vars and prefers $HOST"
(with-redefs [env/env {:host "host" :host-name "host-name" :hostname "hostname"}]
(u/with-started [started (test-system {})]
(is (= "host"
(configuring/external-hostname (:conf started)))))))
(testing "it falls back to env-vars and prefers $HOST_NAME"
(with-redefs [env/env {:host-name "host-name" :hostname "hostname"}]
(u/with-started [started (test-system {})]
(is (= "host-name"
(configuring/external-hostname (:conf started)))))))
(testing "it falls back to env-vars and looks finally for $HOSTNAME"
(with-redefs [env/env {:hostname "hostname"}]
(u/with-started [started (test-system {})]
(is (= "hostname"
(configuring/external-hostname (:conf started)))))))
(testing "it eventually falls back to localhost"
(with-redefs [env/env {:host-name nil :hostname nil :host nil}]
(u/with-started [started (test-system {})]
(is (= "localhost"
(configuring/external-hostname (:conf started))))))))
(deftest ^:unit determine-hostport-from-config-and-env-with-defined-precedence
(testing "it prefers a explicitly configured :hostname"
(with-redefs [configuring/server-port (constantly "configured")]
(with-redefs [env/env {:port0 "0" :host-port "1" :server-port "2"}]
(is (= "configured"
(configuring/external-port {}))))))
(with-redefs [configuring/server-port (constantly nil)]
(testing "it falls back to env-vars and prefers $PORT0"
(with-redefs [env/env {:port0 "0" :host-port "1" :server-port "2"}]
(u/with-started [started (test-system {})]
(is (= "0"
(configuring/external-port {}))))))
(testing "it falls back to env-vars and prefers $HOST_PORT"
(with-redefs [env/env {:host-port "1" :server-port "2"}]
(u/with-started [started (test-system {})]
(is (= "1"
(configuring/external-port {})))))))
(testing "it falls back to env-vars and finally takes $SERVER_PORT"
(with-redefs [env/env {:server-port "2"}]
(u/with-started [started (test-system {})]
(is (= "2"
(configuring/external-port {})))))))
(deftest ^:integration should-read-properties-from-file
(spit "application.properties" "foooo=barrrr")
(is (= (:foooo (configuring/load-config-from-properties-files {}))
"barrrr"))
(io/delete-file "application.properties"))
(deftest ^:integration should-prefer-configured-conf-file
(spit "application.properties" "foooo=value")
(spit "other.properties" "foooo=other-value")
(with-redefs-fn {#'env/env {:config-file "other.properties"}}
#(is (= (:foooo (configuring/load-config-from-properties-files {}))
"other-value")))
(io/delete-file "other.properties")
(io/delete-file "application.properties"))
(deftest ^:unit deep-merge-test
(testing "simple cases"
(is (= {:a 1 :b 2}
(configuring/deep-merge {:a 1}
{:b 2})))
(is (= {:a 1 :b 2}
(configuring/deep-merge {:a 1 :b 1}
{:b 2})))
(is (= {:a 2 :b 2}
(configuring/deep-merge {:a 1 :b 1}
{:a 2 :b 2})))
(is (= {:a 3 :b 2 :c 3}
(configuring/deep-merge {:a 1 :b 1}
{:b 2}
{:a 3 :c 3}))))
(testing "nested maps"
(is (= {:a {:b {:c 1 :d 2}}}
(configuring/deep-merge {:a {:b {:c 1}}}
{:a {:b {:d 2}}})))
(is (= {:a {:b {:c 2 :d 2} :f 3}}
(configuring/deep-merge {:a {:b {:c 1}}}
{:a {:b {:c 2 :d 2}}}
{:a {:f 3}}))))
(testing "collections as values"
(is (= {:a [4 5 6]}
(configuring/deep-merge {:a [1 2 3]} {:a [4 5 6]}))))
(testing "reseting values"
(is (= {:a nil}
(configuring/deep-merge {:a 1} {:a nil}))))
(testing "missing things"
(is (= {:a 1}
(configuring/deep-merge {:a 1} {})))
(is (= nil
(configuring/deep-merge {:a 1} nil)))
(is (= {:a 1}
(apply configuring/deep-merge (filter some? [{:a 1} nil]))))))
================================================
FILE: test/de/otto/tesla/stateful/handler_test.clj
================================================
(ns de.otto.tesla.stateful.handler-test
(:require [clojure.test :refer :all]
[de.otto.tesla.stateful.handler :as handler]
[com.stuartsierra.component :as component]))
(deftest registering-handlers
(testing "should register a handler and return a single handling-function"
(let [handler (-> (handler/new-handler) (component/start))]
(handler/register-handler handler (fn [r] (when (= r :ping) :pong)))
(handler/register-handler handler (fn [r] (when (= r :pong) :ping)))
(is (= 2 (count @(:registered-handlers handler))))
(is (= :pong ((handler/handler handler) :ping)))
(is (= :ping ((handler/handler handler) :pong))))))
================================================
FILE: test/de/otto/tesla/stateful/health_test.clj
================================================
(ns de.otto.tesla.stateful.health-test
(:require [clojure.test :refer :all]
[de.otto.tesla.stateful.health :as health]
[de.otto.tesla.util.test-utils :as u]
[de.otto.tesla.system :as system]
[de.otto.tesla.stateful.handler :as handler]
[ring.mock.request :as mock]
[de.otto.goo.goo :as goo]))
(defn- serverless-system [runtime-config]
(dissoc
(system/base-system runtime-config)
:server))
(deftest ^:unit should-turn-unhealthy-when-locked
(u/with-started [started (serverless-system {})]
(testing "it is still healthy when not yet locked"
(let [handlers (handler/handler (:handler started))
response (handlers (mock/request :get "/health"))]
(are [key value] (= value (get response key))
:body "HEALTHY"
:status 200)
(is (= 1.0 (-> :health/locked ((goo/snapshot)) (.get))))))
(testing "when locked, it is unhealthy"
(let [handlers (handler/handler (:handler started))
_ (health/lock-application (:health started))
response (handlers (mock/request :get "/health"))]
(are [key value] (= value (get response key))
:body "UNHEALTHY"
:status 423)
(is (= 0.0 (-> :health/locked ((goo/snapshot)) (.get))))))))
(deftest ^:integration should-serve-health-under-configured-url
(testing "use the default url"
(u/with-started [started (serverless-system {})]
(let [handlers (handler/handler (:handler started))]
(is (= (:body (handlers (mock/request :get "/health")))
"HEALTHY"
)))))
(testing "use the configuration url"
(u/with-started [started (serverless-system {:health-url "/my-health"})]
(let [handlers (handler/handler (:handler started))]
(is (= (:body (handlers (mock/request :get "/my-health")))
"HEALTHY"))))))
================================================
FILE: test/de/otto/tesla/stateful/keep_alive_test.clj
================================================
(ns de.otto.tesla.stateful.keep-alive-test
(:require
[clojure.test :refer [deftest testing is]]
[de.otto.tesla.stateful.keep-alive :as kalive]
[de.otto.tesla.util.test-utils :refer [eventually with-started]]
[clojure.tools.logging :as log]))
(deftest starting-and-stopping-the-keepalive-component
(testing "should start and stop keepalive-thread"
(let [state (atom :not-started)]
(with-redefs [kalive/enter-keep-alive (fn []
(log/info "ENTER test keepalive")
(reset! state :entered))
kalive/exit-keep-alive (fn []
(log/info "EXIT test keepalive")
(reset! state :exited))]
(is (= :not-started @state))
(with-started [_ (kalive/new-keep-alive)]
(eventually (= :entered @state)))
(eventually (= :exited @state))))))
================================================
FILE: test/de/otto/tesla/stateful/scheduler_test.clj
================================================
(ns de.otto.tesla.stateful.scheduler-test
(:require [clojure.test :refer :all]
[de.otto.tesla.system :as system]
[de.otto.tesla.stateful.scheduler :as schedule]
[de.otto.tesla.util.test-utils :as u]
[overtone.at-at :as at]
[de.otto.tesla.util.test-utils :refer [eventually]]
[com.stuartsierra.component :as c]
[de.otto.tesla.stateful.handler :as handler]
[ring.mock.request :as mock]
[clojure.data.json :as json]
[de.otto.tesla.stateful.scheduler :as scheduler])
(:import (java.util.concurrent ScheduledThreadPoolExecutor)))
(defn- serverless-system [runtime-config]
(-> (system/base-system runtime-config)
(dissoc :server)
(assoc :scheduler (c/using (schedule/new-scheduler) [:config :app-status]))))
(deftest ^:unit should-call-function-at-scheduled-rate
(testing "Function gets called every 20 ms"
(u/with-started [system (serverless-system {:scheduler {:cpu-count 1}})]
(let [scheduler (:scheduler system)
calls (atom 0)]
(at/every 20 #(swap! calls inc) (:pool scheduler))
(eventually (= @calls 3)))))
(testing "Function gets called every 20 ms with initial delay"
(u/with-started [system (serverless-system {:scheduler {:cpu-count 1}})]
(let [scheduler (:scheduler system)
calls (atom 0)]
(at/every 20 #(swap! calls inc) (:pool scheduler) :initial-delay 20)
(eventually (= @calls 2)))))
(testing "Function gets called every 20 ms AFTER the function last returned"
(u/with-started [system (serverless-system {:scheduler {:cpu-count 1}})]
(let [scheduler (:scheduler system)
calls (atom 0)]
(at/interspaced 20 #((Thread/sleep 10) (swap! calls inc)) (:pool scheduler))
(eventually (= @calls 1))))))
(defn assert-map-args! [args-assert-val]
(fn [& {:as args}] (is (= args args-assert-val))))
(deftest ^:unit configuring-the-schedule
(testing "should pass nr cpus to pool if specified"
(let [config {:scheduler {:cpu-count 2
:stop-delayed? false
:stop-periodic? true}}]
(with-redefs [at/stop-and-reset-pool! (constantly nil)
at/mk-pool (assert-map-args! {:cpu-count 2
:stop-delayed? false
:stop-periodic? true})]
(u/with-started [system (serverless-system config)]
(is ())))))
(testing "should pass nothing to pool if nothing is specified"
(let [config {:some-other :property}]
(with-redefs [at/stop-and-reset-pool! (constantly nil)
at/mk-pool (assert-map-args! {:cpu-count 0})]
(u/with-started [system (serverless-system config)]
(is ()))))))
(deftest ^:unit scheduler-app-status
(with-redefs [schedule/as-readable-time (constantly "mock-time")]
(u/with-started [system (serverless-system {:host-name "bar" :server-port "0123"
:scheduler {:cpu-count 2}})]
(let [{:keys [scheduler handler]} system
handler-fn (handler/handler handler)]
(testing "should register and return status-details in app-status"
(at/every 20 #(Thread/sleep 10) (:pool scheduler) :desc "Job 1")
(at/interspaced 20 #(Thread/sleep 10) (:pool scheduler) :desc "Job 2")
(eventually (= {:poolInfo {:active 0
:poolSize 2
:queueSize 2}
:scheduledJobs {:1 {:createdAt "mock-time"
:desc "Job 1"
:initialDelay 0
:msPeriod 20
:scheduled? true}
:2 {:createdAt "mock-time"
:desc "Job 2"
:initialDelay 0
:msPeriod 20
:scheduled? true}}
:status "OK"}
(-> (mock/request :get "/status")
(handler-fn)
:body
(json/read-str :key-fn keyword)
(get-in [:application :statusDetails :scheduler])))))))))
(deftest ^:unit scheduler-default-conf
(testing "should startup pool with core-pool-size of 0 if nothing else is configured"
(u/with-started [system (serverless-system {})]
(let [{:keys [scheduler]} system
^ScheduledThreadPoolExecutor thread-pool (:thread-pool @(:pool-atom (:pool scheduler)))]
(is (= 0 (.getCorePoolSize thread-pool))))))
(testing "should startup pool with configured core-pool-size"
(u/with-started [system (serverless-system {:scheduler {:cpu-count 2}})]
(let [{:keys [scheduler]} system
^ScheduledThreadPoolExecutor thread-pool (:thread-pool @(:pool-atom (:pool scheduler)))]
(is (= 2 (.getCorePoolSize thread-pool)))))))
================================================
FILE: test/de/otto/tesla/system_test.clj
================================================
(ns de.otto.tesla.system-test
(:require [clojure.test :refer :all]
[com.stuartsierra.component :as c]
[de.otto.tesla.util.test-utils :as u :refer [eventually]]
[de.otto.tesla.system :as system]
[ring.mock.request :as mock]
[de.otto.tesla.stateful.handler :as handler]
[de.otto.tesla.stateful.configuring :as configuring]
[environ.core :as env]
[overtone.at-at :as at]))
(deftest ^:unit should-start-base-system-and-shut-it-down
(testing "start then shutdown using own method"
(let [system-stop-calls (atom 0)
started (system/start (system/base-system {}))]
(with-redefs [c/stop-system (fn [_] (swap! system-stop-calls inc))]
(system/stop started)
(is (= 1 @system-stop-calls))))))
(defrecord BombOnStartup []
c/Lifecycle
(start [_self]
(throw (Exception. "boom!")))
(stop [self]
self))
(defn new-bomb-on-startup []
(map->BombOnStartup {}))
(deftest ^:unit should-shutdown-on-error-while-starting
(let [exploding-system (assoc (system/base-system {}) :bomb (c/using (new-bomb-on-startup) [:health]))
system-stop-calls (atom 0)]
(with-redefs [c/stop-system (fn [_] (swap! system-stop-calls inc))]
(system/start exploding-system)
(is (= 1 @system-stop-calls)))))
(defrecord BombEverytime []
c/Lifecycle
(start [_self]
(throw (Exception. "boom!")))
(stop [_self]
(throw (Exception. "boom!"))))
(defn new-bomb-on-everytime []
(map->BombEverytime {}))
(deftest should-exit-jvm-if-stopping-system-fails
(let [exploding-system (assoc (system/base-system {}) :bomb (c/using (new-bomb-on-everytime) [:health]))
system-exit-calls (atom [])]
(with-redefs [system/exit #(swap! system-exit-calls conj %)]
(system/start exploding-system)
(is (= [1] @system-exit-calls)))))
(defrecord BombOnShutdown []
c/Lifecycle
(start [self]
self)
(stop [_self]
(throw (Exception. "boom!"))))
(defn new-bomb-on-shutdown []
(map->BombOnShutdown {}))
(deftest ^:unit should-shutdown-on-error-while-stopping
(let [exploding-system-on-stop (assoc (system/base-system {}) :bomb (c/using (new-bomb-on-shutdown) [:health]))
system-exit-calls (atom [])]
(with-redefs [system/exit #(swap! system-exit-calls conj %)]
(system/stop (system/start exploding-system-on-stop))
(is (= [1] @system-exit-calls)))))
(deftest should-lock-application-on-shutdown
(testing "the lock is set"
(u/with-started
[started (system/base-system {:wait-ms-on-stop 10})]
(let [healthcomp (:health started)]
(with-redefs [system/exit #(println "System exit would be called with code " %)]
(system/stop started))
(is (= @(:locked healthcomp) true)))))
(testing "it waits on stop"
(u/with-started
[started (system/base-system {:wait-seconds-on-stop 1})]
(let [has-waited (atom false)]
(with-redefs [system/wait! (fn [_] (reset! has-waited true))
system/exit #(println "System exit would be called with code " %)]
(let [healthcomp (:health started)
_ (system/stop started)]
(is (= @(:locked healthcomp) true))
(is (= @has-waited true))))))))
(deftest ^:integration should-substitute-env-variables-while-reading
(with-redefs [env/env {:my-custom-status-url "/custom/status/path" :prop-without-fallback "some prop value"}]
(u/with-started [started (system/base-system {})]
(testing "should load the status-path property from edn"
(is (= "/custom/status/path"
(:status-url (configuring/load-config-from-edn-files {})))))
(testing "should point to edn-configured custom status url"
(let [handlers (handler/handler (:handler started))
response (handlers (mock/request :get "/custom/status/path"))]
(is (= 200 (:status response)))))))
(u/with-started [started (system/base-system {})]
(testing "should fallback to default for status path"
(is (= "/status"
(:status-url (configuring/load-config-from-edn-files {})))))))
(deftest the-scheduler-in-the-base-system
(testing "should schedule and execute task NOW"
(u/with-started [started (system/base-system {})]
(let [work-done (atom :no-work-done)
{:keys [scheduler]} started]
(at/after 0 #(reset! work-done :work-done!) (:pool scheduler))
(eventually (= :work-done! @work-done))))))
================================================
FILE: test/de/otto/tesla/util/env_var_reader_test.clj
================================================
(ns de.otto.tesla.util.env-var-reader-test
(:require [de.otto.tesla.util.env_var_reader :as env-reader]
[clojure.test :refer :all]
[environ.core :as env]))
(deftest ^:unit should-map-keys-to-sanitized-keywords
(with-redefs [env/env {"my-var-key" "my-var-value"}]
(is (= "my-var-value"
(env-reader/read-env-var ["my-var-key"])))))
(deftest ^:unit should-use-fallback-if-env-does-not-have-key
(with-redefs [env/env {}]
(is (= "default"
(env-reader/read-env-var ["my-var-key" "default"])))))
(deftest ^:unit should-return-empty-if-no-fallback-and-key-not-in-env
(with-redefs [env/env {}]
(is (= ""
(env-reader/read-env-var ["my-var-key"])))))
================================================
FILE: test/de/otto/tesla/util/keyword_test.clj
================================================
(ns de.otto.tesla.util.keyword-test
(:require [clojure.test :refer :all]
[de.otto.tesla.util.keyword :as kwutil]))
(deftest ^:unit should-map-keys-to-sanitized-keywords
(is (= (kwutil/sanitize-keywords {"thats.a.property" "a value"})
{:thats-a-property "a value"})))
================================================
FILE: test/de/otto/tesla/util/sanitize_test.clj
================================================
(ns de.otto.tesla.util.sanitize-test
(:require [clojure.test :refer :all]
[de.otto.tesla.util.sanitize :as san]))
(deftest ^:unit should-sanitize-passwords
(is (= (san/hide-passwds {:somerandomstuff "not-so-secret"
:somerandomstuff-passwd-somerandomstuff "secret"
:somerandomstuff-pw-somerandomstuff "longersecret"
:my-private-stuff "sonotpublic"
:my-secret-stuff "psstsecret"
:nested {:some-passwd "secret"}})
{:somerandomstuff "not-so-secret"
:somerandomstuff-passwd-somerandomstuff "***"
:somerandomstuff-pw-somerandomstuff "***"
:my-private-stuff "***"
:my-secret-stuff "***"
:nested {:some-passwd "***"}
})))
================================================
FILE: test/de/otto/tesla/util/test_utils_test.clj
================================================
(ns de.otto.tesla.util.test-utils-test
(:require
[clojure.test :refer :all]
[de.otto.tesla.util.test-utils :as u]
[com.stuartsierra.component :as c]))
(deftest eventually
(testing "should eventually get the right random number"
(u/eventually (= 1 (rand-int 2))))
(testing "should eventually return the right response"
(let [start-time (System/currentTimeMillis)]
(u/eventually (= :expected-response (#(let [waited-200msec? (> (- (System/currentTimeMillis) start-time) 200)]
(if waited-200msec?
:expected-response
:wrong-response))))))))
(defrecord StartableComponent [state]
c/Lifecycle
(start [self] (reset! state :started) self)
(stop [self] (reset! state :stopped) self))
(deftest with-started
(testing "should start and stop something"
(let [state (atom :not-running)]
(is (= :not-running @state))
(u/with-started [started (->StartableComponent state)]
(is (= :started @state)))
(is (= :stopped @state)))))
(deftest testing-the-mock-request
(testing "should create mock-request"
(is (= (u/mock-request :get "url" {})
{:headers {"host" "localhost"}
:query-string nil
:remote-addr "localhost"
:request-method :get
:scheme :http
:server-name "localhost"
:server-port 80
:uri "url"})))
(testing "should create mock-request"
(is (= (u/mock-request :get "url" {:headers {"content-type" "application/json"}})
{:headers {"host" "localhost"
"content-type" "application/json"}
:query-string nil
:remote-addr "localhost"
:request-method :get
:scheme :http
:server-name "localhost"
:server-port 80
:uri "url"}))))
================================================
FILE: test-resources/local.edn
================================================
{:server-port "9991"
:hostname #ts/env [:host-name "localhost"]
:status-url #ts/env [:my-custom-status-url "/status"]
:prop-without-fallback #ts/env [:prop-without-fallback]
:metric {:console {:interval-in-s 120}}
:foo {:edn "baz"}
:very :local}
================================================
FILE: test-resources/local.properties
================================================
foo.prop=baz
server.port=9991
================================================
FILE: test-resources/logback-test.xml
================================================
<configuration>
<contextName>tesla</contextName>
<appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
<append>true</append>
<encoder>
<pattern>%d{HH:mm:ss.SSS} %-5p %c{5} %t "%m"%n</pattern>
</encoder>
</appender>
<root level="${log_level:-error}">
<appender-ref ref="consoleAppender"/>
</root>
</configuration>
================================================
FILE: test-resources/private.edn
================================================
; this is checked in for testing reasons.
; In your projects you should add 'private.edn' to .gitignore.
{:very :private}
================================================
FILE: test-resources/test.edn
================================================
{:health-url "/test/health"
:foo {:local true}}
================================================
FILE: test-resources/version.properties
================================================
version=test.version
commit=test.githash
================================================
FILE: test-utils/de/otto/tesla/util/test_utils.clj
================================================
(ns de.otto.tesla.util.test-utils
(:require
[com.stuartsierra.component :as comp]
[ring.mock.request :as mock]))
(defmacro eventually
"Generic assertion macro, that waits for a predicate test to become true.
'form' is a predicate test, that clojure.test/is can understand
'timeout': optional; in ms; how long to wait in total (defaults to 5000)
'interval' optional; in ms; how long to pause between tries (defaults to 10)
Example:
Since this will fail half of the time ...
(is (= 1 (rand-int 2)))
... use this:
(eventually (= 1 (rand-int 2)))"
[form & {:keys [timeout interval]
:or {timeout 5000
interval 10}}]
`(let [start-time# (System/currentTimeMillis)]
(loop []
(let [last-stats# (atom nil)
pass?# (with-redefs [clojure.test/do-report (fn [s#] (reset! last-stats# s#))]
(clojure.test/is ~form))
took-too-long?# (> (- (System/currentTimeMillis) start-time#) ~timeout)]
(if (or pass?# took-too-long?#)
(clojure.test/do-report @last-stats#)
(do
(Thread/sleep ~interval)
(recur)))))))
(defmacro with-started
"bindings => [name init ...]
Evaluates body in a try expression with names bound to the values
of the inits after (.start init) has been called on them. Finally
a clause calls (.stop name) on each name in reverse order."
[bindings & body]
(if (and
(vector? bindings) "a vector for its binding"
(even? (count bindings)) "an even number of forms in binding vector")
(cond
(= (count bindings) 0) `(do ~@body)
(symbol? (bindings 0)) `(let [~(bindings 0) (comp/start ~(bindings 1))]
(try
(with-started ~(subvec bindings 2) ~@body)
(finally
(comp/stop ~(bindings 0)))))
:else (throw (IllegalArgumentException.
"with-started-system only allows Symbols in bindings")))
(throw (IllegalArgumentException.
"not a vector or bindings-count is not even"))))
(defn mock-request
"merges additional arguments into a mock-request"
[method url args]
(let [request (mock/request method url)]
(merge-with merge request args)))
gitextract_11uzdn14/
├── .gitignore
├── .tool-versions
├── .travis.yml
├── CHANGES.md
├── Dockerfile
├── LICENSE
├── MAINTAINERS
├── OSSMETADATA
├── README.md
├── dev-resources/
│ └── logback.xml
├── project.clj
├── resources/
│ ├── default.edn
│ └── default.properties
├── src/
│ ├── data_readers.clj
│ └── de/
│ └── otto/
│ └── tesla/
│ ├── middleware/
│ │ └── exceptions.clj
│ ├── stateful/
│ │ ├── app_status.clj
│ │ ├── auth.clj
│ │ ├── configuring.clj
│ │ ├── handler.clj
│ │ ├── health.clj
│ │ ├── keep_alive.clj
│ │ ├── metering.clj
│ │ └── scheduler.clj
│ ├── system.clj
│ └── util/
│ ├── env_var_reader.clj
│ ├── keyword.clj
│ └── sanitize.clj
├── test/
│ └── de/
│ └── otto/
│ └── tesla/
│ ├── reporter/
│ │ └── prometheus_test.clj
│ ├── stateful/
│ │ ├── app_status_test.clj
│ │ ├── configuring_test.clj
│ │ ├── handler_test.clj
│ │ ├── health_test.clj
│ │ ├── keep_alive_test.clj
│ │ └── scheduler_test.clj
│ ├── system_test.clj
│ └── util/
│ ├── env_var_reader_test.clj
│ ├── keyword_test.clj
│ ├── sanitize_test.clj
│ └── test_utils_test.clj
├── test-resources/
│ ├── local.edn
│ ├── local.properties
│ ├── logback-test.xml
│ ├── private.edn
│ ├── test.edn
│ └── version.properties
└── test-utils/
└── de/
└── otto/
└── tesla/
└── util/
└── test_utils.clj
Condensed preview — 46 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (101K chars).
[
{
"path": ".gitignore",
"chars": 78,
"preview": ".idea\n.lein*\ntarget\n.nrepl-port\n*.iml\npom.xml\npom.xml.asc\n/.clj-kondo/\n/.lsp/\n"
},
{
"path": ".tool-versions",
"chars": 43,
"preview": "terraform 1.8.5\nawscli 2.16.10\ntfenv 0.4.0\n"
},
{
"path": ".travis.yml",
"chars": 18,
"preview": "language: clojure\n"
},
{
"path": "CHANGES.md",
"chars": 6695,
"preview": "## Changes\n\n_tesla-microservice_ is used for a number of different services now. Still it is a work in progress. This se"
},
{
"path": "Dockerfile",
"chars": 969,
"preview": "# This is an example Dockerfile for running a tesla-microservice app in a docker container.\n#\n# Instructions:\n# 1. build"
},
{
"path": "LICENSE",
"chars": 11325,
"preview": "Apache License\n Version 2.0, January 2004\n http://www.apache.org/licens"
},
{
"path": "MAINTAINERS",
"chars": 27,
"preview": "Team Tesla <tesla@otto.de>\n"
},
{
"path": "OSSMETADATA",
"chars": 20,
"preview": "osslifecycle=active\n"
},
{
"path": "README.md",
"chars": 8470,
"preview": "# tesla-microservice\n\n> \"If Edison had a needle to find in a haystack, he would proceed at once with the diligence of th"
},
{
"path": "dev-resources/logback.xml",
"chars": 1168,
"preview": "<configuration>\n <contextName>tesla</contextName>\n\n <appender name=\"consoleAppender\" class=\"ch.qos.logback.core.Co"
},
{
"path": "project.clj",
"chars": 2080,
"preview": "(defproject de.otto/tesla-microservice \"0.17.9-SNAPSHOT\"\n :description \"basic microservice.\"\n :url \"https://github.com"
},
{
"path": "resources/default.edn",
"chars": 459,
"preview": "{\n :status-url \"/status\"\n :health-url \"/health\"\n\n ; metering\n ; take a look at de.otto.tesla.stateful.metering\n ; "
},
{
"path": "resources/default.properties",
"chars": 421,
"preview": "\nstatus.url=/status\nhealth.url=/health\n\n### metering\nmetering.reporter=console\nconsole.interval.seconds=100\n\n### activat"
},
{
"path": "src/data_readers.clj",
"chars": 60,
"preview": "{\n ts/env de.otto.tesla.util.env_var_reader/read-env-var\n }\n"
},
{
"path": "src/de/otto/tesla/middleware/exceptions.clj",
"chars": 326,
"preview": "(ns de.otto.tesla.middleware.exceptions\n (:require [clojure.tools.logging :as log]))\n\n(defn exceptions-to-500 [handler]"
},
{
"path": "src/de/otto/tesla/stateful/app_status.clj",
"chars": 3844,
"preview": "(ns de.otto.tesla.stateful.app-status\n \"This component renders a status page consisting of instance- and configuration-"
},
{
"path": "src/de/otto/tesla/stateful/auth.clj",
"chars": 795,
"preview": "(ns de.otto.tesla.stateful.auth\n \"This component handles authentication.\"\n (:require [clojure.tools.logging :as log]\n "
},
{
"path": "src/de/otto/tesla/stateful/configuring.clj",
"chars": 3690,
"preview": "(ns de.otto.tesla.stateful.configuring\n \"This component is responsible for loading the configuration.\"\n (:require [com"
},
{
"path": "src/de/otto/tesla/stateful/handler.clj",
"chars": 1065,
"preview": "(ns de.otto.tesla.stateful.handler\n \"This component is responsible for collecting HTTP handlers in order to provide the"
},
{
"path": "src/de/otto/tesla/stateful/health.clj",
"chars": 1633,
"preview": "(ns de.otto.tesla.stateful.health\n \"This component provides a health-check endpoint which can be used to orchestrate ap"
},
{
"path": "src/de/otto/tesla/stateful/keep_alive.clj",
"chars": 1163,
"preview": "(ns de.otto.tesla.stateful.keep-alive\n \"This component is responsible for keeping the system alive by creating a non-de"
},
{
"path": "src/de/otto/tesla/stateful/metering.clj",
"chars": 2195,
"preview": "(ns de.otto.tesla.stateful.metering\n \"This component handles exporting of (prometheus) metrics\"\n (:require\n [com.st"
},
{
"path": "src/de/otto/tesla/stateful/scheduler.clj",
"chars": 3395,
"preview": "(ns de.otto.tesla.stateful.scheduler\n \"This components maintains a thread pool which can be used to schedule activities"
},
{
"path": "src/de/otto/tesla/system.clj",
"chars": 3059,
"preview": "(ns de.otto.tesla.system\n (:require [com.stuartsierra.component :as c]\n [de.otto.tesla.stateful.app-status :"
},
{
"path": "src/de/otto/tesla/util/env_var_reader.clj",
"chars": 181,
"preview": "(ns de.otto.tesla.util.env_var_reader\n (:require [environ.core :as env]))\n\n(defn read-env-var\n ([[env-var-key fallback"
},
{
"path": "src/de/otto/tesla/util/keyword.clj",
"chars": 409,
"preview": "(ns de.otto.tesla.util.keyword\n (:require [clojure.string :as str]))\n\n;implementation is copied from environ 1.0.0\n(def"
},
{
"path": "src/de/otto/tesla/util/sanitize.clj",
"chars": 347,
"preview": "(ns de.otto.tesla.util.sanitize)\n\n(def checklist [\"password\" \"pw\" \"passwd\" \"private\" \"secret\" \"token\"])\n\n(defn hide-pass"
},
{
"path": "test/de/otto/tesla/reporter/prometheus_test.clj",
"chars": 1901,
"preview": "(ns de.otto.tesla.reporter.prometheus-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.system :as "
},
{
"path": "test/de/otto/tesla/stateful/app_status_test.clj",
"chars": 7990,
"preview": "(ns de.otto.tesla.stateful.app-status-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.stateful.ap"
},
{
"path": "test/de/otto/tesla/stateful/configuring_test.clj",
"chars": 8866,
"preview": "(ns de.otto.tesla.stateful.configuring-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.stateful.c"
},
{
"path": "test/de/otto/tesla/stateful/handler_test.clj",
"chars": 688,
"preview": "(ns de.otto.tesla.stateful.handler-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.stateful.handl"
},
{
"path": "test/de/otto/tesla/stateful/health_test.clj",
"chars": 2288,
"preview": "(ns de.otto.tesla.stateful.health-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.stateful.health"
},
{
"path": "test/de/otto/tesla/stateful/keep_alive_test.clj",
"chars": 980,
"preview": "(ns de.otto.tesla.stateful.keep-alive-test\n (:require\n [clojure.test :refer [deftest testing is]]\n [de.otto.tesla"
},
{
"path": "test/de/otto/tesla/stateful/scheduler_test.clj",
"chars": 6028,
"preview": "(ns de.otto.tesla.stateful.scheduler-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.system :as s"
},
{
"path": "test/de/otto/tesla/system_test.clj",
"chars": 4714,
"preview": "(ns de.otto.tesla.system-test\n (:require [clojure.test :refer :all]\n [com.stuartsierra.component :as c]\n "
},
{
"path": "test/de/otto/tesla/util/env_var_reader_test.clj",
"chars": 719,
"preview": "(ns de.otto.tesla.util.env-var-reader-test\n (:require [de.otto.tesla.util.env_var_reader :as env-reader]\n [c"
},
{
"path": "test/de/otto/tesla/util/keyword_test.clj",
"chars": 292,
"preview": "(ns de.otto.tesla.util.keyword-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.util.keyword :as k"
},
{
"path": "test/de/otto/tesla/util/sanitize_test.clj",
"chars": 1048,
"preview": "(ns de.otto.tesla.util.sanitize-test\n (:require [clojure.test :refer :all]\n [de.otto.tesla.util.sanitize :as"
},
{
"path": "test/de/otto/tesla/util/test_utils_test.clj",
"chars": 2047,
"preview": "(ns de.otto.tesla.util.test-utils-test\n (:require\n [clojure.test :refer :all]\n [de.otto.tesla.util.test-utils :as"
},
{
"path": "test-resources/local.edn",
"chars": 336,
"preview": "{:server-port \"9991\"\n :hostname #ts/env [:host-name \"localhost\"]\n :status-url #ts/env "
},
{
"path": "test-resources/local.properties",
"chars": 29,
"preview": "foo.prop=baz\nserver.port=9991"
},
{
"path": "test-resources/logback-test.xml",
"chars": 408,
"preview": "<configuration>\n\n <contextName>tesla</contextName>\n\n\n\n <appender name=\"consoleAppender\" class=\"ch.qos.logback.core"
},
{
"path": "test-resources/private.edn",
"chars": 122,
"preview": "; this is checked in for testing reasons.\n; In your projects you should add 'private.edn' to .gitignore.\n\n{:very :privat"
},
{
"path": "test-resources/test.edn",
"chars": 55,
"preview": "{:health-url \"/test/health\"\n :foo {:local true}}"
},
{
"path": "test-resources/version.properties",
"chars": 40,
"preview": "version=test.version\ncommit=test.githash"
},
{
"path": "test-utils/de/otto/tesla/util/test_utils.clj",
"chars": 2340,
"preview": "(ns de.otto.tesla.util.test-utils\n (:require\n [com.stuartsierra.component :as comp]\n [ring.mock.request :as mock]"
}
]
About this extraction
This page contains the full source code of the otto-de/tesla-microservice GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 46 files (92.6 KB), approximately 24.2k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.