Repository: github/scientist
Branch: main
Commit: 504a396e987f
Files: 26
Total size: 80.0 KB
Directory structure:
gitextract_1_21f9qu/
├── .github/
│ ├── dependabot.yml
│ └── workflows/
│ └── ci.yml
├── .gitignore
├── CONTRIBUTING.md
├── Gemfile
├── LICENSE.txt
├── README.md
├── Rakefile
├── doc/
│ └── changelog.md
├── lib/
│ ├── scientist/
│ │ ├── default.rb
│ │ ├── errors.rb
│ │ ├── experiment.rb
│ │ ├── observation.rb
│ │ ├── result.rb
│ │ └── version.rb
│ └── scientist.rb
├── scientist.gemspec
├── script/
│ ├── bootstrap
│ ├── release
│ └── test
└── test/
├── scientist/
│ ├── default_test.rb
│ ├── experiment_test.rb
│ ├── observation_test.rb
│ └── result_test.rb
├── scientist_test.rb
└── test_helper.rb
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/dependabot.yml
================================================
version: 2
updates:
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: "weekly"
================================================
FILE: .github/workflows/ci.yml
================================================
name: CI
on: [push, pull_request]
permissions:
actions: none
checks: write
contents: read
deployments: none
issues: none
packages: none
pull-requests: none
repository-projects: none
security-events: none
statuses: write
jobs:
build:
name: ruby-${{ matrix.ruby_version }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
ruby_version:
- "2.6"
- "2.7"
- "3.0"
- "3.1"
- "3.2"
- "3.3"
steps:
- uses: actions/checkout@v4
- uses: ruby/setup-ruby@v1
with:
ruby-version: ${{ matrix.ruby_version }}
bundler-cache: true
- run: bundle exec rake
================================================
FILE: .gitignore
================================================
/*.gem
/.bundle
/.ruby-version
/Gemfile.lock
/coverage
================================================
FILE: CONTRIBUTING.md
================================================
## Contributing
[fork]: https://github.com/github/scientist/fork
[pr]: https://github.com/github/scientist/compare
Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE.txt).
## Submitting a pull request
0. [Fork][fork] and clone the repository
1. Create a new branch: `git checkout -b my-branch-name`
2. Make your change, push to your fork and [submit a pull request][pr]
3. Pat your self on the back and wait for your pull request to be reviewed and merged.
Here are a few things you can do that will increase the likelihood of your pull request being accepted:
- Keep your change as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests.
- Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
## Resources
- [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
- [Using Pull Requests](https://help.github.com/articles/about-pull-requests/)
- [GitHub Help](https://help.github.com)
================================================
FILE: Gemfile
================================================
source "https://rubygems.org"
gemspec
================================================
FILE: LICENSE.txt
================================================
Copyright 2013, 2014 GitHub, Inc.
MIT License
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: README.md
================================================
# Scientist!
A Ruby library for carefully refactoring critical paths. [](https://github.com/github/scientist/actions/workflows/ci.yml)
## How do I science?
Let's pretend you're changing the way you handle permissions in a large web app. Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load.
```ruby
require "scientist"
class MyWidget
def allows?(user)
experiment = Scientist::Default.new "widget-permissions"
experiment.use { model.check_user(user).valid? } # old way
experiment.try { user.can?(:read, model) } # new way
experiment.run
end
end
```
Wrap a `use` block around the code's original behavior, and wrap `try` around the new behavior. `experiment.run` will always return whatever the `use` block returns, but it does a bunch of stuff behind the scenes:
* It decides whether or not to run the `try` block,
* Randomizes the order in which `use` and `try` blocks are run,
* Measures the wall time and cpu time of all behaviors in seconds,
* Compares the result of `try` to the result of `use`,
* Swallow and record exceptions raised in the `try` block when overriding `raised`, and
* Publishes all this information.
The `use` block is called the **control**. The `try` block is called the **candidate**.
Creating an experiment is wordy, but when you include the `Scientist` module, the `science` helper will instantiate an experiment and call `run` for you:
```ruby
require "scientist"
class MyWidget
include Scientist
def allows?(user)
science "widget-permissions" do |experiment|
experiment.use { model.check_user(user).valid? } # old way
experiment.try { user.can?(:read, model) } # new way
end # returns the control value
end
end
```
If you don't declare any `try` blocks, none of the Scientist machinery is invoked and the control value is always returned.
## Making science useful
The examples above will run, but they're not really *doing* anything. The `try` blocks don't run yet and none of the results get published. Replace the default experiment implementation to control execution and reporting:
```ruby
require "scientist/experiment"
class MyExperiment
include Scientist::Experiment
attr_accessor :name
def initialize(name)
@name = name
end
def enabled?
# see "Ramping up experiments" below
true
end
def raised(operation, error)
# see "In a Scientist callback" below
p "Operation '#{operation}' failed with error '#{error.inspect}'"
super # will re-raise
end
def publish(result)
# see "Publishing results" below
p result
end
end
```
When `Scientist::Experiment` is included in a class, it automatically sets it as the default implementation via `Scientist::Experiment.set_default`. This `set_default` call is skipped if you include `Scientist::Experiment` in a module.
Now calls to the `science` helper will load instances of `MyExperiment`.
### Controlling comparison
Scientist compares control and candidate values using `==`. To override this behavior, use `compare` to define how to compare observed values instead:
```ruby
class MyWidget
include Scientist
def users
science "users" do |e|
e.use { User.all } # returns User instances
e.try { UserService.list } # returns UserService::User instances
e.compare do |control, candidate|
control.map(&:login) == candidate.map(&:login)
end
end
end
end
```
If either the control block or candidate block raises an error, Scientist compares the two observations' classes and messages using `==`. To override this behavior, use `compare_errors` to define how to compare observed errors instead:
```ruby
class MyWidget
include Scientist
def slug_from_login(login)
science "slug_from_login" do |e|
e.use { User.slug_from_login login } # returns String instance or ArgumentError
e.try { UserService.slug_from_login login } # returns String instance or ArgumentError
compare_error_message_and_class = -> (control, candidate) do
control.class == candidate.class &&
control.message == candidate.message
end
compare_argument_errors = -> (control, candidate) do
control.class == ArgumentError &&
candidate.class == ArgumentError &&
control.message.start_with?("Input has invalid characters") &&
candidate.message.start_with?("Invalid characters in input")
end
e.compare_errors do |control, candidate|
compare_error_message_and_class.call(control, candidate) ||
compare_argument_errors.call(control, candidate)
end
end
end
end
```
### Adding context
Results aren't very useful without some way to identify them. Use the `context` method to add to or retrieve the context for an experiment:
```ruby
science "widget-permissions" do |e|
e.context :user => user
e.use { model.check_user(user).valid? }
e.try { user.can?(:read, model) }
end
```
`context` takes a Symbol-keyed Hash of extra data. The data is available in `Experiment#publish` via the `context` method. If you're using the `science` helper a lot in a class, you can provide a default context:
```ruby
class MyWidget
include Scientist
def allows?(user)
science "widget-permissions" do |e|
e.context :user => user
e.use { model.check_user(user).valid? }
e.try { user.can?(:read, model) }
end
end
def destroy
science "widget-destruction" do |e|
e.use { old_scary_destroy }
e.try { new_safe_destroy }
end
end
def default_scientist_context
{ :widget => self }
end
end
```
The `widget-permissions` and `widget-destruction` experiments will both have a `:widget` key in their contexts.
### Expensive setup
If an experiment requires expensive setup that should only occur when the experiment is going to be run, define it with the `before_run` method:
```ruby
# Code under test modifies this in-place. We want to copy it for the
# candidate code, but only when needed:
value_for_original_code = big_object
value_for_new_code = nil
science "expensive-but-worthwhile" do |e|
e.before_run do
value_for_new_code = big_object.deep_copy
end
e.use { original_code(value_for_original_code) }
e.try { new_code(value_for_new_code) }
end
```
### Keeping it clean
Sometimes you don't want to store the full value for later analysis. For example, an experiment may return `User` instances, but when researching a mismatch, all you care about is the logins. You can define how to clean these values in an experiment:
```ruby
class MyWidget
include Scientist
def users
science "users" do |e|
e.use { User.all }
e.try { UserService.list }
e.clean do |value|
value.map(&:login).sort
end
end
end
end
```
And this cleaned value is available in observations in the final published result:
```ruby
class MyExperiment
include Scientist::Experiment
# ...
def publish(result)
result.control.value # [<User alice>, <User bob>, <User carol>]
result.control.cleaned_value # ["alice", "bob", "carol"]
end
end
```
Note that the `#clean` method will discard the previous cleaner block if you call it again. If for some reason you need to access the currently configured cleaner block, `Scientist::Experiment#cleaner` will return the block without further ado. _(This probably won't come up in normal usage, but comes in handy if you're writing, say, a custom experiment runner that provides default cleaners.)_
The `#clean` method will not be used for comparison of the results, so in the following example it is not possible to remove the `#compare` method without the experiment failing:
```ruby
def user_ids
science "user_ids" do
e.use { [1,2,3] }
e.try { [1,3,2] }
e.clean { |value| value.sort }
e.compare { |a, b| a.sort == b.sort }
end
end
```
### Ignoring mismatches
During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell an experiment whether or not to ignore a mismatch using the `ignore` method. You may include more than one block if needed:
```ruby
def admin?(user)
science "widget-permissions" do |e|
e.use { model.check_user(user).admin? }
e.try { user.can?(:admin, model) }
e.ignore { user.staff? } # user is staff, always an admin in the new system
e.ignore do |control, candidate|
# new system doesn't handle unconfirmed users yet:
control && !candidate && !user.confirmed_email?
end
end
end
```
The ignore blocks are only called if the *values* don't match. Unless a `compare_errors` comparator is defined, two cases are considered mismatches: a) one observation raising an exception and the other not, b) observations raising exceptions with different classes or messages.
### Enabling/disabling experiments
Sometimes you don't want an experiment to run. Say, disabling a new codepath for anyone who isn't staff. You can disable an experiment by setting a `run_if` block. If this returns `false`, the experiment will merely return the control value. Otherwise, it defers to the experiment's configured `enabled?` method.
```ruby
class DashboardController
include Scientist
def dashboard_items
science "dashboard-items" do |e|
# only run this experiment for staff members
e.run_if { current_user.staff? }
# ...
end
end
```
### Ramping up experiments
As a scientist, you know it's always important to be able to turn your experiment off, lest it run amok and result in villagers with pitchforks on your doorstep. In order to control whether or not an experiment is enabled, you must include the `enabled?` method in your `Scientist::Experiment` implementation.
```ruby
class MyExperiment
include Scientist::Experiment
attr_accessor :name, :percent_enabled
def initialize(name)
@name = name
@percent_enabled = 100
end
def enabled?
percent_enabled > 0 && rand(100) < percent_enabled
end
# ...
end
```
This code will be invoked for every method with an experiment every time, so be sensitive about its performance. For example, you can store an experiment in the database but wrap it in various levels of caching such as memcache or per-request thread-locals.
### Publishing results
What good is science if you can't publish your results?
You must implement the `publish(result)` method, and can publish data however you like. For example, timing data can be sent to graphite, and mismatches can be placed in a capped collection in redis for debugging later.
The `publish` method is given a `Scientist::Result` instance with its associated `Scientist::Observation`s:
```ruby
class MyExperiment
include Scientist::Experiment
# ...
def publish(result)
# Wall time
# Store the timing for the control value,
$statsd.timing "science.#{name}.control", result.control.duration
# for the candidate (only the first, see "Breaking the rules" below,
$statsd.timing "science.#{name}.candidate", result.candidates.first.duration
# CPU time
# Store the timing for the control value,
$statsd.timing "science.cpu.#{name}.control", result.control.cpu_time
# for the candidate (only the first, see "Breaking the rules" below,
$statsd.timing "science.cpu.#{name}.candidate", result.candidates.first.cpu_time
# and counts for match/ignore/mismatch:
if result.matched?
$statsd.increment "science.#{name}.matched"
elsif result.ignored?
$statsd.increment "science.#{name}.ignored"
else
$statsd.increment "science.#{name}.mismatched"
# Finally, store mismatches in redis so they can be retrieved and examined
# later on, for debugging and research.
store_mismatch_data(result)
end
end
def store_mismatch_data(result)
payload = {
:name => name,
:context => context,
:control => observation_payload(result.control),
:candidate => observation_payload(result.candidates.first),
:execution_order => result.observations.map(&:name)
}
key = "science.#{name}.mismatch"
$redis.lpush key, payload
$redis.ltrim key, 0, 1000
end
def observation_payload(observation)
if observation.raised?
{
:exception => observation.exception.class,
:message => observation.exception.message,
:backtrace => observation.exception.backtrace
}
else
{
# see "Keeping it clean" above
:value => observation.cleaned_value
}
end
end
end
```
### Testing
When running your test suite, it's helpful to know that the experimental results always match. To help with testing, Scientist defines a `raise_on_mismatches` class attribute when you include `Scientist::Experiment`. Only do this in your test suite!
To raise on mismatches:
```ruby
class MyExperiment
include Scientist::Experiment
# ... implementation
end
MyExperiment.raise_on_mismatches = true
```
Scientist will raise a `Scientist::Experiment::MismatchError` exception if any observations don't match.
#### Custom mismatch errors
To instruct Scientist to raise a custom error instead of the default `Scientist::Experiment::MismatchError`:
```ruby
class CustomMismatchError < Scientist::Experiment::MismatchError
def to_s
message = "There was a mismatch! Here's the diff:"
diffs = result.candidates.map do |candidate|
Diff.new(result.control, candidate)
end.join("\n")
"#{message}\n#{diffs}"
end
end
```
```ruby
science "widget-permissions" do |e|
e.use { Report.find(id) }
e.try { ReportService.new.fetch(id) }
e.raise_with CustomMismatchError
end
```
This allows for pre-processing on mismatch error exception messages.
### Handling errors
#### In candidate code
Scientist rescues and tracks _all_ exceptions raised in a `try` or `use` block, including some where rescuing may cause unexpected behavior (like `SystemExit` or `ScriptError`). To rescue a more restrictive set of exceptions, modify the `RESCUES` list:
```ruby
# default is [Exception]
Scientist::Observation::RESCUES.replace [StandardError]
```
**Timeout ⏲️**: If you're introducing a candidate that could possibly timeout, use caution. ⚠️ While Scientist rescues all exceptions that occur in the candidate block, it *does not* protect you from timeouts, as doing so would be complicated. It would likely require running the candidate code in a background job and tracking the time of a request. We feel the cost of this complexity would outweigh the benefit, so make sure that your code doesn't cause timeouts. This risk can be reduced by running the experiment on a low percentage so that users can (most likely) bypass the experiment by refreshing the page if they hit a timeout. See [Ramping up experiments](#ramping-up-experiments) below for how details on how to set the percentage for your experiment.
#### In a Scientist callback
If an exception is raised within any of Scientist's internal helpers, like `publish`, `compare`, or `clean`, the `raised` method is called with the symbol name of the internal operation that failed and the exception that was raised. The default behavior of `Scientist::Default` is to simply re-raise the exception. Since this halts the experiment entirely, it's often a better idea to handle this error and continue so the experiment as a whole isn't canceled entirely:
```ruby
class MyExperiment
include Scientist::Experiment
# ...
def raised(operation, error)
InternalErrorTracker.track! "science failure in #{name}: #{operation}", error
end
end
```
The operations that may be handled here are:
* `:clean` - an exception is raised in a `clean` block
* `:compare` - an exception is raised in a `compare` block
* `:enabled` - an exception is raised in the `enabled?` method
* `:ignore` - an exception is raised in an `ignore` block
* `:publish` - an exception is raised in the `publish` method
* `:run_if` - an exception is raised in a `run_if` block
### Designing an experiment
Because `enabled?` and `run_if` determine when a candidate runs, it's impossible to guarantee that it will run every time. For this reason, Scientist is only safe for wrapping methods that aren't changing data.
When using Scientist, we've found it most useful to modify both the existing and new systems simultaneously anywhere writes happen, and verify the results at read time with `science`. `raise_on_mismatches` has also been useful to ensure that the correct data was written during tests, and reviewing published mismatches has helped us find any situations we overlooked with our production data at runtime. When writing to and reading from two systems, it's also useful to write some data reconciliation scripts to verify and clean up production data alongside any running experiments.
#### Noise and error rates
Keep in mind that Scientist's `try` and `use` blocks run sequentially in random order. As such, any data upon which your code depends may change before the second block is invoked, potentially yielding a mismatch between the candidate and control return values. To calibrate your expectations with respect to [false negatives](https://en.wikipedia.org/wiki/Type_I_and_type_II_errors) arising from systemic conditions external to your proposed changes, consider starting with an experiment in which both the `try` and `use` blocks invoke the control method. Then proceed with introducing a candidate.
### Finishing an experiment
As your candidate behavior converges on the controls, you'll start thinking about removing an experiment and using the new behavior.
* If there are any ignore blocks, the candidate behavior is *guaranteed* to be different. If this is unacceptable, you'll need to remove the ignore blocks and resolve any ongoing mismatches in behavior until the observations match perfectly every time.
* When removing a read-behavior experiment, it's a good idea to keep any write-side duplication between an old and new system in place until well after the new behavior has been in production, in case you need to roll back.
## Breaking the rules
Sometimes scientists just gotta do weird stuff. We understand.
### Ignoring results entirely
Science is useful even when all you care about is the timing data or even whether or not a new code path blew up. If you have the ability to incrementally control how often an experiment runs via your `enabled?` method, you can use it to silently and carefully test new code paths and ignore the results altogether. You can do this by setting `ignore { true }`, or for greater efficiency, `compare { true }`.
This will still log mismatches if any exceptions are raised, but will disregard the values entirely.
### Trying more than one thing
It's not usually a good idea to try more than one alternative simultaneously. Behavior isn't guaranteed to be isolated and reporting + visualization get quite a bit harder. Still, it's sometimes useful.
To try more than one alternative at once, add names to some `try` blocks:
```ruby
require "scientist"
class MyWidget
include Scientist
def allows?(user)
science "widget-permissions" do |e|
e.use { model.check_user(user).valid? } # old way
e.try("api") { user.can?(:read, model) } # new service API
e.try("raw-sql") { user.can_sql?(:read, model) } # raw query
end
end
end
```
When the experiment runs, all candidate behaviors are tested and each candidate observation is compared with the control in turn.
### No control, just candidates
Define the candidates with named `try` blocks, omit a `use`, and pass a candidate name to `run`:
```ruby
experiment = MyExperiment.new("various-ways") do |e|
e.try("first-way") { ... }
e.try("second-way") { ... }
end
experiment.run("second-way")
```
The `science` helper also knows this trick:
```ruby
science "various-ways", run: "first-way" do |e|
e.try("first-way") { ... }
e.try("second-way") { ... }
end
```
#### Providing fake timing data
If you're writing tests that depend on specific timing values, you can provide canned durations using the `fabricate_durations_for_testing_purposes` method, and Scientist will report these in `Scientist::Observation#duration` and `Scientist::Observation#cpu_time` instead of the actual execution times.
```ruby
science "absolutely-nothing-suspicious-happening-here" do |e|
e.use { ... } # "control"
e.try { ... } # "candidate"
e.fabricate_durations_for_testing_purposes({
"control" => { "duration" => 1.0, "cpu_time" => 0.9 },
"candidate" => { "duration" => 0.5, "cpu_time" => 0.4 }
})
end
```
`fabricate_durations_for_testing_purposes` takes a Hash of duration & cpu_time values, keyed by behavior names. (By default, Scientist uses `"control"` and `"candidate"`, but if you override these as shown in [Trying more than one thing](#trying-more-than-one-thing) or [No control, just candidates](#no-control-just-candidates), use matching names here.) If a name is not provided, the actual execution time will be reported instead.
We should mention these durations will be used both for the `duration` field and the `cpu_time` field.
_Like `Scientist::Experiment#cleaner`, this probably won't come up in normal usage. It's here to make it easier to test code that extends Scientist._
### Without including Scientist
If you need to use Scientist in a place where you aren't able to include the Scientist module, you can call `Scientist.run`:
```ruby
Scientist.run "widget-permissions" do |e|
e.use { model.check_user(user).valid? }
e.try { user.can?(:read, model) }
end
```
## Hacking
Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs the unit tests. All development dependencies are installed automatically. Scientist requires Ruby 2.3 or newer.
## Wrappers
- [RealGeeks/lab_tech](https://github.com/RealGeeks/lab_tech) is a Rails engine for using this library by controlling, storing, and analyzing experiment results with ActiveRecord.
## Alternatives
- [daylerees/scientist](https://github.com/daylerees/scientist) (PHP)
- [scientistproject/scientist.net](https://github.com/scientistproject/Scientist.net) (.NET)
- [joealcorn/laboratory](https://github.com/joealcorn/laboratory) (Python)
- [rawls238/Scientist4J](https://github.com/rawls238/Scientist4J) (Java)
- [tomiaijo/scientist](https://github.com/tomiaijo/scientist) (C++)
- [trello/scientist](https://github.com/trello/scientist) (node.js)
- [ziyasal/scientist.js](https://github.com/ziyasal/scientist.js) (node.js, ES6)
- [TrueWill/tzientist](https://github.com/TrueWill/tzientist) (node.js, TypeScript)
- [TrueWill/paleontologist](https://github.com/TrueWill/paleontologist) (Deno, TypeScript)
- [yeller/laboratory](https://github.com/yeller/laboratory) (Clojure)
- [lancew/Scientist](https://github.com/lancew/Scientist) (Perl 5)
- [lancew/ScientistP6](https://github.com/lancew/ScientistP6) (Perl 6)
- [MadcapJake/Test-Lab](https://github.com/MadcapJake/Test-Lab) (Perl 6)
- [cwbriones/scientist](https://github.com/cwbriones/scientist) (Elixir)
- [calavera/go-scientist](https://github.com/calavera/go-scientist) (Go)
- [jelmersnoeck/experiment](https://github.com/jelmersnoeck/experiment) (Go)
- [spoptchev/scientist](https://github.com/spoptchev/scientist) (Kotlin / Java)
- [junkpiano/scientist](https://github.com/junkpiano/scientist) (Swift)
- [serverless scientist](http://serverlessscientist.com/) (AWS Lambda)
- [fightmegg/scientist](https://github.com/fightmegg/scientist) (TypeScript, Browser / Node.js)
- [MisterSpex/misterspex-scientist](https://github.com/MisterSpex/misterspex-scientist) (Java, no dependencies)
## Maintainers
[@jbarnette](https://github.com/jbarnette),
[@jesseplusplus](https://github.com/jesseplusplus),
[@rick](https://github.com/rick),
and [@zerowidth](https://github.com/zerowidth)
================================================
FILE: Rakefile
================================================
require "rake/testtask"
task default: "test"
Rake::TestTask.new do |t|
t.test_files = FileList['test/test_helper.rb', 'test/**/*_test.rb']
end
================================================
FILE: doc/changelog.md
================================================
# Changes
## v1.6.5 (16 December 2024)
- New: measure CPU time alongside wall time for experiments #275
## v1.6.4 (5 April 2023)
- New: GitHub Actions for CI #171
- New: add ruby 3.1 support #175
- Fix: `compare_errors` in docs #178
- Fix: remove outdated travis configs #179
- Fix: typos #191
- New: add support for `after_run` blocks #211
## v1.6.3 (9 December 2021)
- Fix: improve marshaling implementation #169
## v1.6.2 (4 November 2021)
- New: make `MismatchError` marshalable #168
## v1.6.1 (22 October 2021)
- Fix: moving supported ruby versions from <=2.3 to >=2.6 #150
- Fix: update docs to explain timeout handling #159
- New: add support for comparing errors #77
## v1.6.0 (8 March 2021)
- Fix: clarify unit for observations #124
- New: enable support for truffleruby #143
- Fix: don't default experiment when included in a module #144
## v1.5.0 (8 September 2020)
- Fix: clearer explanation of exception handling #110
- Fix: remove unused attribute from `Scientist::Observation` #119
- New: Added internal extension point for generating experinet results #121
- New: Add `Scientist::Experiment.register` helper #104
## v1.4.0 (20 September 2019)
- New: Make `MismatchError` a base `Exception` #107
## v1.3.0 (2 April 2019)
- New: Drop support for ruby <2.3
- Fix: Build new strings instead of modifying frozen ones
- New: Add an accessor for the configured clean block
- New: Add a hook to use fabricated durations instead of actual timing data.
## v1.2.0 (5 July 2018)
- New: Use monotonic clock for duration calculations
- New: Drop support for ruby <2.1 to support monotonic clock
- New: Run CI on Ruby 2.5
## v1.1.2 (9 May 2018)
- New: Add `raise_with` option to allow for custom mismatch errors to be raised
## v1.1.1 (6 February 2018)
- Fix: default experiment no longer runs all `try` paths
- New: Add `Scientist.run` module method for running experiments when an included module isn't available
- New: Add [Noise and error rates](https://github.com/github/scientist#noise-and-error-rates) to `README.md`
## v1.1.0 (29 August 2017)
- New: [Specify which exception types to rescue](https://github.com/github/scientist#in-candidate-code)
- New: List [alternative implementations](https://github.com/github/scientist#alternatives) in `README.md`
- New: Code coverage via [coveralls.io](https://coveralls.io/github/github/scientist)
- New: Run CI on Ruby 2.3 and 2.4
- Fix: `README` typos, examples, and lies
- Fix: `false` results are now passed to the cleaner
================================================
FILE: lib/scientist/default.rb
================================================
require "scientist/experiment"
# A null experiment.
class Scientist::Default
include Scientist::Experiment
attr_reader :name
def initialize(name)
@name = name
end
# Don't run experiments.
def enabled?
false
end
# Don't publish anything.
def publish(result)
end
end
================================================
FILE: lib/scientist/errors.rb
================================================
module Scientist
# Smoking in the bathroom and/or sassing.
class BadBehavior < StandardError
attr_reader :experiment
attr_reader :name
def initialize(experiment, name, message)
@experiment = experiment
@name = name
super message
end
end
class BehaviorMissing < BadBehavior
def initialize(experiment, name)
super experiment, name,
"#{experiment.name} missing #{name} behavior"
end
end
class BehaviorNotUnique < BadBehavior
def initialize(experiment, name)
super experiment, name,
"#{experiment.name} already has #{name} behavior"
end
end
class NoValue < StandardError
attr_reader :observation
def initialize(observation)
@observation = observation
super "#{observation.name} didn't return a value"
end
end
end
================================================
FILE: lib/scientist/experiment.rb
================================================
# This mixin provides shared behavior for experiments. Includers must implement
# `enabled?` and `publish(result)`.
#
# Override Scientist::Experiment.new to set your own class which includes and
# implements Scientist::Experiment's interface.
module Scientist::Experiment
# Whether to raise when the control and candidate mismatch.
# If this is nil, raise_on_mismatches class attribute is used instead.
attr_accessor :raise_on_mismatches
def self.included(base)
self.set_default(base) if base.instance_of?(Class)
base.extend RaiseOnMismatch
end
# Instantiate a new experiment (using the class given to the .set_default method).
def self.new(name)
(@experiment_klass || Scientist::Default).new(name)
end
# Configure Scientist to use the given class for all future experiments
# (must implement the Scientist::Experiment interface).
#
# Called automatically when new experiments are defined.
def self.set_default(klass)
@experiment_klass = klass
end
# A mismatch, raised when raise_on_mismatches is enabled.
class MismatchError < Exception
attr_reader :name, :result
def initialize(name, result)
@name = name
@result = result
super "experiment '#{name}' observations mismatched"
end
# The default formatting is nearly unreadable, so make it useful.
#
# The assumption here is that errors raised in a test environment are
# printed out as strings, rather than using #inspect.
def to_s
super + ":\n" +
format_observation(result.control) + "\n" +
result.candidates.map { |candidate| format_observation(candidate) }.join("\n") +
"\n"
end
def format_observation(observation)
observation.name + ":\n" +
if observation.raised?
lines = observation.exception.backtrace.map { |line| " #{line}" }.join("\n")
" #{observation.exception.inspect}" + "\n" + lines
else
" #{observation.cleaned_value.inspect}"
end
end
end
module RaiseOnMismatch
# Set this flag to raise on experiment mismatches.
#
# This causes all science mismatches to raise a MismatchError. This is
# intended for test environments and should not be enabled in a production
# environment.
#
# bool - true/false - whether to raise when the control and candidate mismatch.
def raise_on_mismatches=(bool)
@raise_on_mismatches = bool
end
# Whether or not to raise a mismatch error when a mismatch occurs.
def raise_on_mismatches?
@raise_on_mismatches
end
end
# Define a block of code to run before an experiment begins, if the experiment
# is enabled.
#
# The block takes no arguments.
#
# Returns the configured block.
def before_run(&block)
@_scientist_before_run = block
end
# Define a block of code to run after an experiment completes, if the experiment
# is enabled.
#
# The block takes one argument, the Scientist::Result containing experiment results.
#
# Returns the configured block.
def after_run(&block)
@_scientist_after_run = block
end
# A Hash of behavior blocks, keyed by String name. Register behavior blocks
# with the `try` and `use` methods.
def behaviors
@_scientist_behaviors ||= {}
end
# A block to clean an observed value for publishing or storing.
#
# The block takes one argument, the observed value which will be cleaned.
#
# Returns the configured block.
def clean(&block)
@_scientist_cleaner = block
end
# Accessor for the clean block, if one is available.
#
# Returns the configured block, or nil.
def cleaner
@_scientist_cleaner
end
# Internal: Clean a value with the configured clean block, or return the value
# if no clean block is configured.
#
# Rescues and reports exceptions in the clean block if they occur.
def clean_value(value)
if @_scientist_cleaner
@_scientist_cleaner.call value
else
value
end
rescue StandardError => ex
raised :clean, ex
value
end
# A block which compares two experimental values.
#
# The block must take two arguments, the control value and a candidate value,
# and return true or false.
#
# Returns the block.
def compare(*args, &block)
@_scientist_comparator = block
end
# A block which compares two experimental errors.
#
# The block must take two arguments, the control Error and a candidate Error,
# and return true or false.
#
# Returns the block.
def compare_errors(*args, &block)
@_scientist_error_comparator = block
end
# A Symbol-keyed Hash of extra experiment data.
def context(context = nil)
@_scientist_context ||= {}
@_scientist_context.merge!(context) unless context.nil?
@_scientist_context
end
# Configure this experiment to ignore an observation with the given block.
#
# The block takes two arguments, the control observation and the candidate
# observation which didn't match the control. If the block returns true, the
# mismatch is disregarded.
#
# This can be called more than once with different blocks to use.
def ignore(&block)
@_scientist_ignores ||= []
@_scientist_ignores << block
end
# Internal: ignore a mismatched observation?
#
# Iterates through the configured ignore blocks and calls each of them with
# the given control and mismatched candidate observations.
#
# Returns true or false.
def ignore_mismatched_observation?(control, candidate)
return false unless @_scientist_ignores
@_scientist_ignores.any? do |ignore|
begin
ignore.call control.value, candidate.value
rescue StandardError => ex
raised :ignore, ex
false
end
end
end
# The String name of this experiment. Default is "experiment". See
# Scientist::Default for an example of how to override this default.
def name
"experiment"
end
# Internal: compare two observations, using the configured compare and compare_errors lambdas if present.
def observations_are_equivalent?(a, b)
a.equivalent_to? b, @_scientist_comparator, @_scientist_error_comparator
rescue StandardError => ex
raised :compare, ex
false
end
def raise_with(exception)
@_scientist_custom_mismatch_error = exception
end
# Called when an exception is raised while running an internal operation,
# like :publish. Override this method to track these exceptions. The
# default implementation re-raises the exception.
def raised(operation, error)
raise error
end
# Internal: Run all the behaviors for this experiment, observing each and
# publishing the results. Return the result of the named behavior, default
# "control".
def run(name = nil)
behaviors.freeze
context.freeze
name = (name || "control").to_s
block = behaviors[name]
if block.nil?
raise Scientist::BehaviorMissing.new(self, name)
end
unless should_experiment_run?
return block.call
end
if @_scientist_before_run
@_scientist_before_run.call
end
result = generate_result(name)
if @_scientist_after_run
@_scientist_after_run.call(result)
end
begin
publish(result)
rescue StandardError => ex
raised :publish, ex
end
if raise_on_mismatches? && result.mismatched?
if @_scientist_custom_mismatch_error
raise @_scientist_custom_mismatch_error.new(self.name, result)
else
raise MismatchError.new(self.name, result)
end
end
control = result.control
raise control.exception if control.raised?
control.value
end
# Define a block that determines whether or not the experiment should run.
def run_if(&block)
@_scientist_run_if_block = block
end
# Internal: does a run_if block allow the experiment to run?
#
# Rescues and reports exceptions in a run_if block if they occur.
def run_if_block_allows?
(@_scientist_run_if_block ? @_scientist_run_if_block.call : true)
rescue StandardError => ex
raised :run_if, ex
return false
end
# Internal: determine whether or not an experiment should run.
#
# Rescues and reports exceptions in the enabled method if they occur.
def should_experiment_run?
behaviors.size > 1 && enabled? && run_if_block_allows?
rescue StandardError => ex
raised :enabled, ex
return false
end
# Register a named behavior for this experiment, default "candidate".
def try(name = nil, &block)
name = (name || "candidate").to_s
if behaviors.include?(name)
raise Scientist::BehaviorNotUnique.new(self, name)
end
behaviors[name] = block
end
# Register the control behavior for this experiment.
def use(&block)
try "control", &block
end
# Whether or not to raise a mismatch error when a mismatch occurs.
def raise_on_mismatches?
if raise_on_mismatches.nil?
self.class.raise_on_mismatches?
else
!!raise_on_mismatches
end
end
# Provide predefined durations to use instead of actual timing data.
# This is here solely as a convenience for developers of libraries that extend Scientist.
def fabricate_durations_for_testing_purposes(fabricated_durations = {})
@_scientist_fabricated_durations = fabricated_durations
end
# Internal: Generate the observations and create the result from those and the control.
def generate_result(name)
observations = []
behaviors.keys.shuffle.each do |key|
block = behaviors[key]
fabricated_duration = @_scientist_fabricated_durations && @_scientist_fabricated_durations[key]
observations << Scientist::Observation.new(key, self, fabricated_duration: fabricated_duration, &block)
end
control = observations.detect { |o| o.name == name }
Scientist::Result.new(self, observations, control)
end
private
# In order to support marshaling, we have to make the procs marshalable. Some
# CI providers attempt to marshal Scientist mismatch errors so that they can
# be sent out to different places (logs, etc.) The mismatch errors contain
# code from the experiment. This code contains procs. These procs prevent the
# error from being marshaled. To fix this, we simple exclude the procs from
# the data that we marshal.
def marshal_dump
[@name, @result, @raise_on_mismatches]
end
def marshal_load(array)
@name, @result, @raise_on_mismatches = array
end
end
================================================
FILE: lib/scientist/observation.rb
================================================
# What happened when this named behavior was executed? Immutable.
class Scientist::Observation
# An Array of Exception types to rescue when initializing an observation.
# NOTE: This Array will change to `[StandardError]` in the next major release.
RESCUES = [Exception]
# The experiment this observation is for
attr_reader :experiment
# The String name of the behavior.
attr_reader :name
# The value returned, if any.
attr_reader :value
# The raised exception, if any.
attr_reader :exception
# The Float seconds elapsed.
attr_reader :duration
# The Float CPU time elapsed, in seconds
attr_reader :cpu_time
def initialize(name, experiment, fabricated_duration: nil, &block)
@name = name
@experiment = experiment
start_wall_time, start_cpu_time = capture_times unless fabricated_duration
begin
@value = block.call
rescue *RESCUES => e
@exception = e
end
if fabricated_duration.is_a?(Hash)
@duration = fabricated_duration["duration"]
@cpu_time = fabricated_duration["cpu_time"]
elsif fabricated_duration
@duration = fabricated_duration
@cpu_time = 0.0 # setting a default value
else
end_wall_time, end_cpu_time = capture_times
@duration = end_wall_time - start_wall_time
@cpu_time = end_cpu_time - start_cpu_time
end
freeze
end
# Return a cleaned value suitable for publishing. Uses the experiment's
# defined cleaner block to clean the observed value.
def cleaned_value
experiment.clean_value value unless value.nil?
end
# Is this observation equivalent to another?
#
# other - the other Observation in question
# comparator - an optional comparison proc. This observation's value and the
# other observation's value are passed to this to determine
# their equivalency. Proc should return true/false.
# error_comparator - an optional comparison proc. This observation's Error and the
# other observation's Error are passed to this to determine
# their equivalency. Proc should return true/false.
#
# Returns true if:
#
# * The values of the observation are equal (using `==`)
# * The values of the observations are equal according to a comparison
# proc, if given
# * The exceptions raised by the observations are equal according to the
# error comparison proc, if given.
# * Both observations raised an exception with the same class and message.
#
# Returns false otherwise.
def equivalent_to?(other, comparator=nil, error_comparator=nil)
return false unless other.is_a?(Scientist::Observation)
if raised? || other.raised?
if error_comparator
return error_comparator.call(exception, other.exception)
else
return other.exception.class == exception.class &&
other.exception.message == exception.message
end
end
if comparator
comparator.call(value, other.value)
else
value == other.value
end
end
def hash
[value, exception, self.class].compact.map(&:hash).inject(:^)
end
def raised?
!exception.nil?
end
private
def capture_times
[
Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_second),
Process.clock_gettime(Process::CLOCK_PROCESS_CPUTIME_ID, :float_second)
]
end
end
================================================
FILE: lib/scientist/result.rb
================================================
# The immutable result of running an experiment.
class Scientist::Result
# An Array of candidate Observations.
attr_reader :candidates
# The control Observation to which the rest are compared.
attr_reader :control
# An Experiment.
attr_reader :experiment
# An Array of observations which didn't match the control, but were ignored.
attr_reader :ignored
# An Array of observations which didn't match the control.
attr_reader :mismatched
# An Array of Observations in execution order.
attr_reader :observations
# Internal: Create a new result.
#
# experiment - the Experiment this result is for
# observations: - an Array of Observations, in execution order
# control: - the control Observation
#
def initialize(experiment, observations = [], control = nil)
@experiment = experiment
@observations = observations
@control = control
@candidates = observations - [control]
evaluate_candidates
freeze
end
# Public: the experiment's context
def context
experiment.context
end
# Public: the name of the experiment
def experiment_name
experiment.name
end
# Public: was the result a match between all behaviors?
def matched?
mismatched.empty? && !ignored?
end
# Public: were there mismatches in the behaviors?
def mismatched?
mismatched.any?
end
# Public: were there any ignored mismatches?
def ignored?
ignored.any?
end
# Internal: evaluate the candidates to find mismatched and ignored results
#
# Sets @ignored and @mismatched with the ignored and mismatched candidates.
def evaluate_candidates
mismatched = candidates.reject do |candidate|
experiment.observations_are_equivalent?(control, candidate)
end
@ignored = mismatched.select do |candidate|
experiment.ignore_mismatched_observation? control, candidate
end
@mismatched = mismatched - @ignored
end
end
================================================
FILE: lib/scientist/version.rb
================================================
module Scientist
VERSION = '1.6.5'
end
================================================
FILE: lib/scientist.rb
================================================
# Include this module into any class which requires science experiments in its
# methods. Provides the `science` and `default_scientist_context` methods for
# defining and running experiments.
#
# If you need to run science on class methods, extend this module instead.
#
# If including or extending this module are not an option, call
# `Scientist.run`.
module Scientist
# Define and run a science experiment.
#
# name - a String name for this experiment.
# opts - optional hash with the the named test to run instead of "control",
# :run is the only valid key.
#
# Yields an object which implements the Scientist::Experiment interface.
# See `Scientist::Experiment.new` for how this is defined.
#
# Returns the calculated value of the control experiment, or raises if an
# exception was raised.
def self.run(name, opts = {})
experiment = Experiment.new(name)
yield experiment
test = opts[:run] if opts
experiment.run(test)
end
# Define and run a science experiment.
#
# name - a String name for this experiment.
# opts - optional hash with the the named test to run instead of "control",
# :run is the only valid key.
#
# Yields an object which implements the Scientist::Experiment interface.
# See `Scientist::Experiment.new` for how this is defined. The context from
# the `default_scientist_context` method will be applied to the experiment.
#
# Returns the calculated value of the control experiment, or raises if an
# exception was raised.
def science(name, opts = {})
Scientist.run(name, opts) do |experiment|
experiment.context(default_scientist_context)
yield experiment
end
end
# Public: the default context data for an experiment created and run via the
# `science` helper method. Override this in any class that includes Scientist
# to define your own behavior.
#
# Returns a Hash.
def default_scientist_context
{}
end
end
require "scientist/default"
require "scientist/errors"
require "scientist/experiment"
require "scientist/observation"
require "scientist/result"
require "scientist/version"
================================================
FILE: scientist.gemspec
================================================
$: << "lib" and require "scientist/version"
Gem::Specification.new do |gem|
gem.name = "scientist"
gem.description = "A Ruby library for carefully refactoring critical paths"
gem.version = Scientist::VERSION
gem.authors = ["GitHub Open Source", "John Barnette", "Rick Bradley", "Jesse Toth", "Nathan Witmer"]
gem.email = ["opensource+scientist@github.com", "jbarnette@github.com", "rick@rickbradley.com", "jesseplusplus@github.com","zerowidth@github.com"]
gem.summary = "Carefully test, measure, and track refactored code."
gem.homepage = "https://github.com/github/scientist"
gem.license = "MIT"
gem.required_ruby_version = '>= 2.3'
gem.files = `git ls-files`.split($/)
gem.executables = []
gem.test_files = gem.files.grep(/^test/)
gem.require_paths = ["lib"]
gem.add_development_dependency "minitest", "~> 5.8"
gem.add_development_dependency "rake"
end
================================================
FILE: script/bootstrap
================================================
#!/bin/sh
# Ensure local dependencies are available.
set -e
cd $(dirname "$0")/..
rm -f .bundle/config
bundle install --path .bundle --quiet "$@"
================================================
FILE: script/release
================================================
#!/bin/sh
# Tag and push a release.
set -e
# Make sure we're in the project root.
cd $(dirname "$0")/..
# Build a new gem archive.
rm -rf scientist-*.gem
gem build -q scientist.gemspec
# Make sure we're on the main branch.
(git branch --no-color | grep -q '* main') || {
echo "Only release from the main branch."
exit 1
}
# Figure out what version we're releasing.
tag=v`ls scientist-*.gem | sed 's/^scientist-\(.*\)\.gem$/\1/'`
# Make sure we haven't released this version before.
git fetch -t origin
(git tag -l | grep -q "$tag") && {
echo "Whoops, there's already a '${tag}' tag."
exit 1
}
# Tag it and bag it.
gem push scientist-*.gem && git tag "$tag" &&
git push origin main && git push origin "$tag"
================================================
FILE: script/test
================================================
#!/bin/sh
# Run the unit tests.
set -e
cd $(dirname "$0")/..
script/bootstrap && bundle exec rake test
================================================
FILE: test/scientist/default_test.rb
================================================
describe Scientist::Default do
before do
@ex = Scientist::Default.new "default"
end
it "is always disabled" do
refute @ex.enabled?
end
it "noops publish" do
assert_nil @ex.publish("data")
end
it "is an experiment" do
assert Scientist::Default < Scientist::Experiment
end
it "reraises when an internal action raises" do
assert_raises RuntimeError do
@ex.raised :publish, RuntimeError.new("kaboom")
end
end
end
================================================
FILE: test/scientist/experiment_test.rb
================================================
describe Scientist::Experiment do
class Fake
include Scientist::Experiment
# Undo auto-config magic / preserve default behavior of Scientist::Experiment.new
Scientist::Experiment.set_default(nil)
def initialize(*args)
end
def enabled?
true
end
attr_reader :published_result
def exceptions
@exceptions ||= []
end
def raised(op, exception)
exceptions << [op, exception]
end
def publish(result)
@published_result = result
end
end
before do
@ex = Fake.new
end
it "sets the default on inclusion" do
klass = Class.new do
include Scientist::Experiment
def initialize(name)
end
end
assert_kind_of klass, Scientist::Experiment.new("hello")
Scientist::Experiment.set_default(nil)
end
it "doesn't set the default on inclusion when it's a module" do
Module.new { include Scientist::Experiment }
assert_kind_of Scientist::Default, Scientist::Experiment.new("hello")
end
it "has a default implementation" do
ex = Scientist::Experiment.new("hello")
assert_kind_of Scientist::Default, ex
assert_equal "hello", ex.name
end
it "provides a static default name" do
assert_equal "experiment", Fake.new.name
end
it "requires includers to implement enabled?" do
obj = Object.new
obj.extend Scientist::Experiment
assert_raises NoMethodError do
obj.enabled?
end
end
it "requires includers to implement publish" do
obj = Object.new
obj.extend Scientist::Experiment
assert_raises NoMethodError do
obj.publish("result")
end
end
it "can't be run without a control behavior" do
e = assert_raises Scientist::BehaviorMissing do
@ex.run
end
assert_equal "control", e.name
end
it "is a straight pass-through with only a control behavior" do
@ex.use { "control" }
assert_equal "control", @ex.run
end
it "runs other behaviors but always returns the control" do
@ex.use { "control" }
@ex.try { "candidate" }
assert_equal "control", @ex.run
end
it "complains about duplicate behavior names" do
@ex.use { "control" }
e = assert_raises Scientist::BehaviorNotUnique do
@ex.use { "control-again" }
end
assert_equal @ex, e.experiment
assert_equal "control", e.name
end
it "swallows exceptions raised by candidate behaviors" do
@ex.use { "control" }
@ex.try { raise "candidate" }
assert_equal "control", @ex.run
end
it "passes through exceptions raised by the control behavior" do
@ex.use { raise "control" }
@ex.try { "candidate" }
exception = assert_raises RuntimeError do
@ex.run
end
assert_equal "control", exception.message
end
it "shuffles behaviors before running" do
last = nil
runs = []
@ex.use { last = "control" }
@ex.try { last = "candidate" }
10000.times do
@ex.run
runs << last
end
assert runs.uniq.size > 1
end
it "re-raises exceptions raised during publish by default" do
ex = Scientist::Experiment.new("hello")
assert_kind_of Scientist::Default, ex
def ex.enabled?
true
end
def ex.publish(result)
raise "boomtown"
end
ex.use { "control" }
ex.try { "candidate" }
exception = assert_raises RuntimeError do
ex.run
end
assert_equal "boomtown", exception.message
end
it "reports publishing errors" do
def @ex.publish(result)
raise "boomtown"
end
@ex.use { "control" }
@ex.try { "candidate" }
assert_equal "control", @ex.run
op, exception = @ex.exceptions.pop
assert_equal :publish, op
assert_equal "boomtown", exception.message
end
it "publishes results" do
@ex.use { 1 }
@ex.try { 1 }
assert_equal 1, @ex.run
assert @ex.published_result
end
it "does not publish results when there is only a control value" do
@ex.use { 1 }
assert_equal 1, @ex.run
assert_nil @ex.published_result
end
it "compares results with a comparator block if provided" do
@ex.compare { |a, b| a == b.to_s }
@ex.use { "1" }
@ex.try { 1 }
assert_equal "1", @ex.run
assert @ex.published_result.matched?
end
it "compares errors with an error comparator block if provided" do
@ex.compare_errors { |a, b| a.class == b.class }
@ex.use { raise "foo" }
@ex.try { raise "bar" }
resulting_error = assert_raises RuntimeError do
@ex.run
end
assert_equal "foo", resulting_error.message
assert @ex.published_result.matched?
end
it "knows how to compare two experiments" do
a = Scientist::Observation.new(@ex, "a") { 1 }
b = Scientist::Observation.new(@ex, "b") { 2 }
assert @ex.observations_are_equivalent?(a, a)
refute @ex.observations_are_equivalent?(a, b)
end
it "uses a compare block to determine if observations are equivalent" do
a = Scientist::Observation.new(@ex, "a") { "1" }
b = Scientist::Observation.new(@ex, "b") { 1 }
@ex.compare { |x, y| x == y.to_s }
assert @ex.observations_are_equivalent?(a, b)
end
it "reports errors in a compare block" do
@ex.compare { raise "boomtown" }
@ex.use { "control" }
@ex.try { "candidate" }
assert_equal "control", @ex.run
op, exception = @ex.exceptions.pop
assert_equal :compare, op
assert_equal "boomtown", exception.message
end
it "reports errors in the enabled? method" do
def @ex.enabled?
raise "kaboom"
end
@ex.use { "control" }
@ex.try { "candidate" }
assert_equal "control", @ex.run
op, exception = @ex.exceptions.pop
assert_equal :enabled, op
assert_equal "kaboom", exception.message
end
it "reports errors in a run_if block" do
@ex.run_if { raise "kaboom" }
@ex.use { "control" }
@ex.try { "candidate" }
assert_equal "control", @ex.run
op, exception = @ex.exceptions.pop
assert_equal :run_if, op
assert_equal "kaboom", exception.message
end
it "returns the given value when no clean block is configured" do
assert_equal 10, @ex.clean_value(10)
end
it "provides the clean block when asked for it, in case subclasses wish to override and provide defaults" do
assert_nil @ex.cleaner
cleaner = ->(value) { value.upcase }
@ex.clean(&cleaner)
assert_equal cleaner, @ex.cleaner
end
it "calls the configured clean block with a value when configured" do
@ex.clean do |value|
value.upcase
end
assert_equal "TEST", @ex.clean_value("test")
end
it "reports an error and returns the original value when an error is raised in a clean block" do
@ex.clean { |value| raise "kaboom" }
@ex.use { "control" }
@ex.try { "candidate" }
assert_equal "control", @ex.run
assert_equal "control", @ex.published_result.control.cleaned_value
op, exception = @ex.exceptions.pop
assert_equal :clean, op
assert_equal "kaboom", exception.message
end
describe "#raise_with" do
it "raises custom error if provided" do
CustomError = Class.new(Scientist::Experiment::MismatchError)
@ex.use { 1 }
@ex.try { 2 }
@ex.raise_with(CustomError)
@ex.raise_on_mismatches = true
assert_raises(CustomError) { @ex.run }
end
end
describe "#run_if" do
it "does not run the experiment if the given block returns false" do
candidate_ran = false
run_check_ran = false
@ex.use { 1 }
@ex.try { candidate_ran = true; 1 }
@ex.run_if { run_check_ran = true; false }
@ex.run
assert run_check_ran
refute candidate_ran
end
it "runs the experiment if the given block returns true" do
candidate_ran = false
run_check_ran = false
@ex.use { true }
@ex.try { candidate_ran = true }
@ex.run_if { run_check_ran = true }
@ex.run
assert run_check_ran
assert candidate_ran
end
end
describe "#ignore_mismatched_observation?" do
before do
@a = Scientist::Observation.new(@ex, "a") { 1 }
@b = Scientist::Observation.new(@ex, "b") { 2 }
end
it "does not ignore an observation if no ignores are configured" do
refute @ex.ignore_mismatched_observation?(@a, @b)
end
it "calls a configured ignore block with the given observed values" do
called = false
@ex.ignore do |a, b|
called = true
assert_equal @a.value, a
assert_equal @b.value, b
true
end
assert @ex.ignore_mismatched_observation?(@a, @b)
assert called
end
it "calls multiple ignore blocks to see if any match" do
called_one = called_two = called_three = false
@ex.ignore { |a, b| called_one = true; false }
@ex.ignore { |a, b| called_two = true; false }
@ex.ignore { |a, b| called_three = true; false }
refute @ex.ignore_mismatched_observation?(@a, @b)
assert called_one
assert called_two
assert called_three
end
it "only calls ignore blocks until one matches" do
called_one = called_two = called_three = false
@ex.ignore { |a, b| called_one = true; false }
@ex.ignore { |a, b| called_two = true; true }
@ex.ignore { |a, b| called_three = true; false }
assert @ex.ignore_mismatched_observation?(@a, @b)
assert called_one
assert called_two
refute called_three
end
it "reports exceptions raised in an ignore block and returns false" do
def @ex.exceptions
@exceptions ||= []
end
def @ex.raised(op, exception)
exceptions << [op, exception]
end
@ex.ignore { raise "kaboom" }
refute @ex.ignore_mismatched_observation?(@a, @b)
op, exception = @ex.exceptions.pop
assert_equal :ignore, op
assert_equal "kaboom", exception.message
end
it "skips ignore blocks that raise and tests any remaining blocks if an exception is swallowed" do
def @ex.exceptions
@exceptions ||= []
end
# this swallows the exception rather than re-raising
def @ex.raised(op, exception)
exceptions << [op, exception]
end
@ex.ignore { raise "kaboom" }
@ex.ignore { true }
assert @ex.ignore_mismatched_observation?(@a, @b)
assert_equal 1, @ex.exceptions.size
end
end
describe "raising on mismatches" do
before do
@old_raise_on_mismatches = Fake.raise_on_mismatches?
end
after do
Fake.raise_on_mismatches = @old_raise_on_mismatches
end
it "raises when there is a mismatch if raise on mismatches is enabled" do
Fake.raise_on_mismatches = true
@ex.use { "fine" }
@ex.try { "not fine" }
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
end
it "cleans values when raising on observation mismatch" do
Fake.raise_on_mismatches = true
@ex.use { "fine" }
@ex.try { "not fine" }
@ex.clean { "So Clean" }
err = assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
assert_match /So Clean/, err.message
end
it "doesn't raise when there is a mismatch if raise on mismatches is disabled" do
Fake.raise_on_mismatches = false
@ex.use { "fine" }
@ex.try { "not fine" }
assert_equal "fine", @ex.run
end
it "raises a mismatch error if the control raises and candidate doesn't" do
Fake.raise_on_mismatches = true
@ex.use { raise "control" }
@ex.try { "candidate" }
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
end
it "raises a mismatch error if the candidate raises and the control doesn't" do
Fake.raise_on_mismatches = true
@ex.use { "control" }
@ex.try { raise "candidate" }
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
end
it "allows MismatchError to bubble up through bare rescues" do
Fake.raise_on_mismatches = true
@ex.use { "control" }
@ex.try { "candidate" }
runner = -> {
begin
@ex.run
rescue
# StandardError handled
end
}
assert_raises(Scientist::Experiment::MismatchError) { runner.call }
end
it "can be marshaled" do
Fake.raise_on_mismatches = true
@ex.before_run { "some block" }
@ex.clean { "some block" }
@ex.compare_errors { "some block" }
@ex.ignore { false }
@ex.run_if { "some block" }
@ex.try { "candidate" }
@ex.use { "control" }
@ex.compare { |control, candidate| control == candidate }
mismatch = nil
begin
@ex.run
rescue Scientist::Experiment::MismatchError => e
mismatch = e
end
assert_kind_of(String, Marshal.dump(mismatch))
end
it "can be marshal loaded" do
assert_kind_of(Fake, Marshal.load(Marshal.dump(@ex)))
end
describe "#raise_on_mismatches?" do
it "raises when there is a mismatch if the experiment instance's raise on mismatches is enabled" do
Fake.raise_on_mismatches = false
@ex.raise_on_mismatches = true
@ex.use { "fine" }
@ex.try { "not fine" }
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
end
it "doesn't raise when there is a mismatch if the experiment instance's raise on mismatches is disabled" do
Fake.raise_on_mismatches = true
@ex.raise_on_mismatches = false
@ex.use { "fine" }
@ex.try { "not fine" }
assert_equal "fine", @ex.run
end
it "respects the raise_on_mismatches class attribute by default" do
Fake.raise_on_mismatches = false
@ex.use { "fine" }
@ex.try { "not fine" }
assert_equal "fine", @ex.run
Fake.raise_on_mismatches = true
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
end
end
describe "MismatchError" do
before do
Fake.raise_on_mismatches = true
@ex.use { :foo }
@ex.try { :bar }
begin
@ex.run
rescue Scientist::Experiment::MismatchError => e
@mismatch = e
end
assert @mismatch
end
it "has the name of the experiment" do
assert_equal @ex.name, @mismatch.name
end
it "includes the experiments' results" do
assert_equal @ex.published_result, @mismatch.result
end
it "formats nicely as a string" do
assert_equal <<-STR, @mismatch.to_s
experiment 'experiment' observations mismatched:
control:
:foo
candidate:
:bar
STR
end
it "includes the backtrace when an observation raises" do
mismatch = nil
ex = Fake.new
ex.use { "value" }
ex.try { raise "error" }
begin
ex.run
rescue Scientist::Experiment::MismatchError => e
mismatch = e
end
# Should look like this:
# experiment 'experiment' observations mismatched:
# control:
# "value"
# candidate:
# #<RuntimeError: error>
# test/scientist/experiment_test.rb:447:in `block (5 levels) in <top (required)>'
# ... (more backtrace)
lines = mismatch.to_s.split("\n")
assert_equal "control:", lines[1]
assert_equal " \"value\"", lines[2]
assert_equal "candidate:", lines[3]
assert_equal " #<RuntimeError: error>", lines[4]
assert_match %r(test/scientist/experiment_test.rb:\d+:in `block), lines[5]
end
end
end
describe "before run block" do
it "runs when an experiment is enabled" do
control_ok = candidate_ok = false
before = false
@ex.before_run { before = true }
@ex.use { control_ok = before }
@ex.try { candidate_ok = before }
@ex.run
assert before, "before_run should have run"
assert control_ok, "control should have run after before_run"
assert candidate_ok, "candidate should have run after before_run"
end
it "does not run when an experiment is disabled" do
before = false
def @ex.enabled?
false
end
@ex.before_run { before = true }
@ex.use { "value" }
@ex.try { "value" }
@ex.run
refute before, "before_run should not have run"
end
end
describe "after run block" do
it "runs when an experiment is enabled" do
control_ok = candidate_ok = false
after_result = nil
@ex.after_run { |result| after_result = result }
@ex.use { control_ok = after_result.nil? }
@ex.try { candidate_ok = after_result.nil? }
@ex.run
assert after_result, "after_run should have run"
assert after_result.matched?, "after_run should be called with the result"
assert control_ok, "control should have run before after_run"
assert candidate_ok, "candidate should have run before after_run"
end
it "does not run when an experiment is disabled" do
after_result = nil
def @ex.enabled?
false
end
@ex.after_run { |result| after_result = result }
@ex.use { "value" }
@ex.try { "value" }
@ex.run
refute after_result, "after_run should not have run"
end
end
describe "testing hooks for extending code" do
it "allows a user to provide fabricated durations for testing purposes (old version)" do
@ex.use { true }
@ex.try { true }
@ex.fabricate_durations_for_testing_purposes( "control" => 0.5, "candidate" => 1.0 )
@ex.run
cont = @ex.published_result.control
cand = @ex.published_result.candidates.first
assert_in_delta 0.5, cont.duration, 0.01
assert_in_delta 1.0, cand.duration, 0.01
end
it "allows a user to provide fabricated durations for testing purposes (new version)" do
@ex.use { true }
@ex.try { true }
@ex.fabricate_durations_for_testing_purposes({
"control" => { "duration" => 0.5, "cpu_time" => 0.4 },
"candidate" => { "duration" => 1.0, "cpu_time" => 0.9 }
})
@ex.run
cont = @ex.published_result.control
cand = @ex.published_result.candidates.first
# Wall Time
assert_in_delta 0.5, cont.duration, 0.01
assert_in_delta 1.0, cand.duration, 0.01
# CPU Time
assert_equal 0.4, cont.cpu_time
assert_equal 0.9, cand.cpu_time
end
it "returns actual durations if fabricated ones are omitted for some blocks (old version)" do
@ex.use { true }
@ex.try { sleep 0.1; true }
@ex.fabricate_durations_for_testing_purposes( "control" => 0.5 )
@ex.run
cont = @ex.published_result.control
cand = @ex.published_result.candidates.first
assert_in_delta 0.5, cont.duration, 0.01
assert_in_delta 0.1, cand.duration, 0.01
end
it "returns actual durations if fabricated ones are omitted for some blocks (new version)" do
@ex.use { true }
@ex.try do
start_time = Time.now
while Time.now - start_time < 0.1
# Perform some CPU-intensive work
(1..1000).each { |i| i * i }
end
true
end
@ex.fabricate_durations_for_testing_purposes({ "control" => { "duration" => 0.5, "cpu_time" => 0.4 }})
@ex.run
cont = @ex.published_result.control
cand = @ex.published_result.candidates.first
# Fabricated durations
assert_in_delta 0.5, cont.duration, 0.01
assert_in_delta 0.4, cont.cpu_time, 0.01
# Measured durations
assert_in_delta 0.1, cand.duration, 0.01
assert_in_delta 0.1, cand.cpu_time, 0.01
end
end
end
================================================
FILE: test/scientist/observation_test.rb
================================================
describe Scientist::Observation do
before do
@experiment = Scientist::Experiment.new "test"
end
it "observes and records the execution of a block" do
ob = Scientist::Observation.new("test", @experiment) do
start_time = Time.now
while Time.now - start_time < 0.1
# Perform some CPU-intensive work
(1..1000).each { |i| i * i }
end
"ret"
end
assert_equal "ret", ob.value
refute ob.raised?
assert_in_delta 0.1, ob.duration, 0.01
assert_in_delta 0.1, ob.cpu_time, 0.01
end
it "stashes exceptions" do
ob = Scientist::Observation.new("test", @experiment) do
raise "exception"
end
assert ob.raised?
assert_equal "exception", ob.exception.message
assert_nil ob.value
end
describe "::RESCUES" do
before do
@original = Scientist::Observation::RESCUES.dup
end
after do
Scientist::Observation::RESCUES.replace(@original)
end
it "includes all exception types by default" do
ob = Scientist::Observation.new("test", @experiment) do
raise Exception.new("not a StandardError")
end
assert ob.raised?
assert_instance_of Exception, ob.exception
end
it "can customize rescued types" do
Scientist::Observation::RESCUES.replace [StandardError]
ex = assert_raises Exception do
Scientist::Observation.new("test", @experiment) do
raise Exception.new("not a StandardError")
end
end
assert_equal "not a StandardError", ex.message
end
end
it "compares values" do
a = Scientist::Observation.new("test", @experiment) { 1 }
b = Scientist::Observation.new("test", @experiment) { 1 }
assert a.equivalent_to?(b)
x = Scientist::Observation.new("test", @experiment) { 1 }
y = Scientist::Observation.new("test", @experiment) { 2 }
refute x.equivalent_to?(y)
end
it "compares exception messages" do
a = Scientist::Observation.new("test", @experiment) { raise "error" }
b = Scientist::Observation.new("test", @experiment) { raise "error" }
assert a.equivalent_to?(b)
x = Scientist::Observation.new("test", @experiment) { raise "error" }
y = Scientist::Observation.new("test", @experiment) { raise "ERROR" }
refute x.equivalent_to?(y)
end
FirstError = Class.new(StandardError)
SecondError = Class.new(StandardError)
it "compares exception classes" do
x = Scientist::Observation.new("test", @experiment) { raise FirstError, "error" }
y = Scientist::Observation.new("test", @experiment) { raise SecondError, "error" }
z = Scientist::Observation.new("test", @experiment) { raise FirstError, "error" }
assert x.equivalent_to?(z)
refute x.equivalent_to?(y)
end
it "compares values using a comparator proc" do
a = Scientist::Observation.new("test", @experiment) { 1 }
b = Scientist::Observation.new("test", @experiment) { "1" }
refute a.equivalent_to?(b)
compare_on_string = -> (x, y) { x.to_s == y.to_s }
assert a.equivalent_to?(b, compare_on_string)
yielded = []
compare_appends = -> (x, y) do
yielded << x
yielded << y
true
end
a.equivalent_to?(b, compare_appends)
assert_equal [a.value, b.value], yielded
end
it "compares exceptions using an error comparator proc" do
x = Scientist::Observation.new("test", @experiment) { raise FirstError, "error" }
y = Scientist::Observation.new("test", @experiment) { raise SecondError, "error" }
z = Scientist::Observation.new("test", @experiment) { raise FirstError, "ERROR" }
refute x.equivalent_to?(z)
refute x.equivalent_to?(y)
compare_on_class = -> (error, other_error) {
error.class == other_error.class
}
compare_on_message = -> (error, other_error) {
error.message == other_error.message
}
assert x.equivalent_to?(z, nil, compare_on_class)
assert x.equivalent_to?(y, nil, compare_on_message)
end
describe "#cleaned_value" do
it "returns the observation's value by default" do
a = Scientist::Observation.new("test", @experiment) { 1 }
assert_equal 1, a.cleaned_value
end
it "uses the experiment's clean block to clean a value when configured" do
@experiment.clean { |value| value.upcase }
a = Scientist::Observation.new("test", @experiment) { "test" }
assert_equal "TEST", a.cleaned_value
end
it "doesn't clean nil values" do
@experiment.clean { |value| "foo" }
a = Scientist::Observation.new("test", @experiment) { nil }
assert_nil a.cleaned_value
end
it "returns false boolean values" do
a = Scientist::Observation.new("test", @experiment) { false }
assert_equal false, a.cleaned_value
end
it "cleans false values" do
@experiment.clean { |value| value.to_s.upcase }
a = Scientist::Observation.new("test", @experiment) { false }
assert_equal "FALSE", a.cleaned_value
end
end
end
================================================
FILE: test/scientist/result_test.rb
================================================
describe Scientist::Result do
before do
@experiment = Scientist::Experiment.new "experiment"
end
it "is immutable" do
control = Scientist::Observation.new("control", @experiment)
candidate = Scientist::Observation.new("candidate", @experiment)
result = Scientist::Result.new @experiment, [control, candidate], control
assert result.frozen?
end
it "evaluates its observations" do
a = Scientist::Observation.new("a", @experiment) { 1 }
b = Scientist::Observation.new("b", @experiment) { 1 }
assert a.equivalent_to?(b)
result = Scientist::Result.new @experiment, [a, b], a
assert result.matched?
refute result.mismatched?
assert_equal [], result.mismatched
x = Scientist::Observation.new("x", @experiment) { 1 }
y = Scientist::Observation.new("y", @experiment) { 2 }
z = Scientist::Observation.new("z", @experiment) { 3 }
result = Scientist::Result.new @experiment, [x, y, z], x
refute result.matched?
assert result.mismatched?
assert_equal [y, z], result.mismatched
end
it "has no mismatches if there is only a control observation" do
a = Scientist::Observation.new("a", @experiment) { 1 }
result = Scientist::Result.new @experiment, [a], a
assert result.matched?
end
it "evaluates observations using the experiment's compare block" do
a = Scientist::Observation.new("a", @experiment) { "1" }
b = Scientist::Observation.new("b", @experiment) { 1 }
@experiment.compare { |x, y| x == y.to_s }
result = Scientist::Result.new @experiment, [a, b], a
assert result.matched?, result.mismatched
end
it "does not ignore any mismatches when nothing's ignored" do
x = Scientist::Observation.new("x", @experiment) { 1 }
y = Scientist::Observation.new("y", @experiment) { 2 }
result = Scientist::Result.new @experiment, [x, y], x
assert result.mismatched?
refute result.ignored?
end
it "uses the experiment's ignore block to ignore mismatched observations" do
x = Scientist::Observation.new("x", @experiment) { 1 }
y = Scientist::Observation.new("y", @experiment) { 2 }
called = false
@experiment.ignore { called = true }
result = Scientist::Result.new @experiment, [x, y], x
refute result.mismatched?
refute result.matched?
assert result.ignored?
assert_equal [], result.mismatched
assert_equal [y], result.ignored
assert called
end
it "partitions observations into mismatched and ignored when applicable" do
x = Scientist::Observation.new("x", @experiment) { :x }
y = Scientist::Observation.new("y", @experiment) { :y }
z = Scientist::Observation.new("z", @experiment) { :z }
@experiment.ignore { |control, candidate| candidate == :y }
result = Scientist::Result.new @experiment, [x, y, z], x
assert result.mismatched?
assert result.ignored?
assert_equal [y], result.ignored
assert_equal [z], result.mismatched
end
it "knows the experiment's name" do
a = Scientist::Observation.new("a", @experiment) { 1 }
b = Scientist::Observation.new("b", @experiment) { 1 }
result = Scientist::Result.new @experiment, [a, b], a
assert_equal @experiment.name, result.experiment_name
end
it "has the context from an experiment" do
@experiment.context :foo => :bar
a = Scientist::Observation.new("a", @experiment) { 1 }
b = Scientist::Observation.new("b", @experiment) { 1 }
result = Scientist::Result.new @experiment, [a, b], a
assert_equal({:foo => :bar}, result.context)
end
end
================================================
FILE: test/scientist_test.rb
================================================
describe Scientist do
it "has a version or whatever" do
assert Scientist::VERSION
end
it "provides a helper to instantiate and run experiments" do
obj = Object.new
obj.extend(Scientist)
r = obj.science "test" do |e|
e.use { :control }
e.try { :candidate }
end
assert_equal :control, r
end
it "provides a module method to instantiate and run experiments" do
r = Scientist.run "test" do |e|
e.use { :control }
e.try { :candidate }
end
assert_equal :control, r
end
it "provides an empty default_scientist_context" do
obj = Object.new
obj.extend(Scientist)
assert_equal Hash.new, obj.default_scientist_context
end
it "respects default_scientist_context" do
obj = Object.new
obj.extend(Scientist)
def obj.default_scientist_context
{ :default => true }
end
experiment = nil
obj.science "test" do |e|
experiment = e
e.context :inline => true
e.use { }
end
refute_nil experiment
assert_equal true, experiment.context[:default]
assert_equal true, experiment.context[:inline]
end
it "runs the named test instead of the control" do
obj = Object.new
obj.extend(Scientist)
behaviors_executed = []
result = obj.science "test", run: "first-way" do |e|
e.try("first-way") { behaviors_executed << "first-way" ; true }
e.try("second-way") { behaviors_executed << "second-way" ; true }
end
assert_equal true, result
assert_equal [ "first-way" ], behaviors_executed
end
it "runs control when there is a nil named test" do
obj = Object.new
obj.extend(Scientist)
behaviors_executed = []
result = obj.science "test", nil do |e|
e.use { behaviors_executed << "control" ; true }
e.try("second-way") { behaviors_executed << "second-way" ; true }
end
assert_equal true, result
assert_equal [ "control" ], behaviors_executed
end
end
================================================
FILE: test/test_helper.rb
================================================
require "minitest/autorun"
$LOAD_PATH.unshift(File.expand_path(File.join(__dir__, "../lib")))
require "scientist"
gitextract_1_21f9qu/
├── .github/
│ ├── dependabot.yml
│ └── workflows/
│ └── ci.yml
├── .gitignore
├── CONTRIBUTING.md
├── Gemfile
├── LICENSE.txt
├── README.md
├── Rakefile
├── doc/
│ └── changelog.md
├── lib/
│ ├── scientist/
│ │ ├── default.rb
│ │ ├── errors.rb
│ │ ├── experiment.rb
│ │ ├── observation.rb
│ │ ├── result.rb
│ │ └── version.rb
│ └── scientist.rb
├── scientist.gemspec
├── script/
│ ├── bootstrap
│ ├── release
│ └── test
└── test/
├── scientist/
│ ├── default_test.rb
│ ├── experiment_test.rb
│ ├── observation_test.rb
│ └── result_test.rb
├── scientist_test.rb
└── test_helper.rb
SYMBOL INDEX (88 symbols across 9 files)
FILE: lib/scientist.rb
type Scientist (line 9) | module Scientist
function run (line 21) | def self.run(name, opts = {})
function science (line 42) | def science(name, opts = {})
function default_scientist_context (line 55) | def default_scientist_context
FILE: lib/scientist/default.rb
class Scientist::Default (line 4) | class Scientist::Default
method initialize (line 9) | def initialize(name)
method enabled? (line 14) | def enabled?
method publish (line 19) | def publish(result)
FILE: lib/scientist/errors.rb
type Scientist (line 1) | module Scientist
class BadBehavior (line 4) | class BadBehavior < StandardError
method initialize (line 8) | def initialize(experiment, name, message)
class BehaviorMissing (line 16) | class BehaviorMissing < BadBehavior
method initialize (line 17) | def initialize(experiment, name)
class BehaviorNotUnique (line 23) | class BehaviorNotUnique < BadBehavior
method initialize (line 24) | def initialize(experiment, name)
class NoValue (line 30) | class NoValue < StandardError
method initialize (line 33) | def initialize(observation)
FILE: lib/scientist/experiment.rb
type Scientist::Experiment (line 6) | module Scientist::Experiment
function included (line 12) | def self.included(base)
function new (line 18) | def self.new(name)
function set_default (line 26) | def self.set_default(klass)
class MismatchError (line 31) | class MismatchError < Exception
method initialize (line 34) | def initialize(name, result)
method to_s (line 44) | def to_s
method format_observation (line 51) | def format_observation(observation)
type RaiseOnMismatch (line 62) | module RaiseOnMismatch
function raise_on_mismatches= (line 70) | def raise_on_mismatches=(bool)
function raise_on_mismatches? (line 75) | def raise_on_mismatches?
function before_run (line 86) | def before_run(&block)
function after_run (line 96) | def after_run(&block)
function behaviors (line 102) | def behaviors
function clean (line 111) | def clean(&block)
function cleaner (line 118) | def cleaner
function clean_value (line 126) | def clean_value(value)
function compare (line 143) | def compare(*args, &block)
function compare_errors (line 153) | def compare_errors(*args, &block)
function context (line 158) | def context(context = nil)
function ignore (line 171) | def ignore(&block)
function ignore_mismatched_observation? (line 182) | def ignore_mismatched_observation?(control, candidate)
function name (line 196) | def name
function observations_are_equivalent? (line 201) | def observations_are_equivalent?(a, b)
function raise_with (line 208) | def raise_with(exception)
function raised (line 215) | def raised(operation, error)
function run (line 222) | def run(name = nil)
function run_if (line 267) | def run_if(&block)
function run_if_block_allows? (line 274) | def run_if_block_allows?
function should_experiment_run? (line 284) | def should_experiment_run?
function try (line 292) | def try(name = nil, &block)
function use (line 303) | def use(&block)
function raise_on_mismatches? (line 308) | def raise_on_mismatches?
function fabricate_durations_for_testing_purposes (line 318) | def fabricate_durations_for_testing_purposes(fabricated_durations = {})
function generate_result (line 323) | def generate_result(name)
function marshal_dump (line 344) | def marshal_dump
function marshal_load (line 348) | def marshal_load(array)
FILE: lib/scientist/observation.rb
class Scientist::Observation (line 2) | class Scientist::Observation
method initialize (line 26) | def initialize(name, experiment, fabricated_duration: nil, &block)
method cleaned_value (line 55) | def cleaned_value
method equivalent_to? (line 79) | def equivalent_to?(other, comparator=nil, error_comparator=nil)
method hash (line 98) | def hash
method raised? (line 102) | def raised?
method capture_times (line 108) | def capture_times
FILE: lib/scientist/result.rb
class Scientist::Result (line 2) | class Scientist::Result
method initialize (line 28) | def initialize(experiment, observations = [], control = nil)
method context (line 39) | def context
method experiment_name (line 44) | def experiment_name
method matched? (line 49) | def matched?
method mismatched? (line 54) | def mismatched?
method ignored? (line 59) | def ignored?
method evaluate_candidates (line 66) | def evaluate_candidates
FILE: lib/scientist/version.rb
type Scientist (line 1) | module Scientist
FILE: test/scientist/experiment_test.rb
class Fake (line 2) | class Fake
method initialize (line 8) | def initialize(*args)
method enabled? (line 11) | def enabled?
method exceptions (line 17) | def exceptions
method raised (line 21) | def raised(op, exception)
method publish (line 25) | def publish(result)
function initialize (line 38) | def initialize(name)
function enabled? (line 148) | def ex.enabled?
function publish (line 152) | def ex.publish(result)
function publish (line 167) | def @ex.publish(result)
function enabled? (line 245) | def @ex.enabled?
function exceptions (line 396) | def @ex.exceptions
function raised (line 400) | def @ex.raised(op, exception)
function exceptions (line 414) | def @ex.exceptions
function raised (line 419) | def @ex.raised(op, exception)
function enabled? (line 630) | def @ex.enabled?
function enabled? (line 661) | def @ex.enabled?
FILE: test/scientist_test.rb
function default_scientist_context (line 38) | def obj.default_scientist_context
Condensed preview — 26 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (87K chars).
[
{
"path": ".github/dependabot.yml",
"chars": 116,
"preview": "version: 2\nupdates:\n - package-ecosystem: github-actions\n directory: \"/\"\n schedule:\n interval: \"weekly\"\n"
},
{
"path": ".github/workflows/ci.yml",
"chars": 708,
"preview": "name: CI\non: [push, pull_request]\npermissions:\n actions: none\n checks: write\n contents: read\n deployments: none\n is"
},
{
"path": ".gitignore",
"chars": 55,
"preview": "/*.gem\n/.bundle\n/.ruby-version\n/Gemfile.lock\n/coverage\n"
},
{
"path": "CONTRIBUTING.md",
"chars": 1351,
"preview": "## Contributing\n\n[fork]: https://github.com/github/scientist/fork\n[pr]: https://github.com/github/scientist/compare\n\nHi "
},
{
"path": "Gemfile",
"chars": 39,
"preview": "source \"https://rubygems.org\"\n\ngemspec\n"
},
{
"path": "LICENSE.txt",
"chars": 1071,
"preview": "Copyright 2013, 2014 GitHub, Inc.\n\nMIT License\n\nPermission is hereby granted, free of charge, to any person obtaining\na "
},
{
"path": "README.md",
"chars": 24158,
"preview": "# Scientist!\n\nA Ruby library for carefully refactoring critical paths. [![Build Status](https://github.com/github/scient"
},
{
"path": "Rakefile",
"chars": 147,
"preview": "require \"rake/testtask\"\n\ntask default: \"test\"\n\nRake::TestTask.new do |t|\n t.test_files = FileList['test/test_helper.rb'"
},
{
"path": "doc/changelog.md",
"chars": 2504,
"preview": "# Changes\n\n## v1.6.5 (16 December 2024)\n\n- New: measure CPU time alongside wall time for experiments #275\n\n## v1.6.4 (5 "
},
{
"path": "lib/scientist/default.rb",
"chars": 298,
"preview": "require \"scientist/experiment\"\n\n# A null experiment.\nclass Scientist::Default\n include Scientist::Experiment\n\n attr_re"
},
{
"path": "lib/scientist/errors.rb",
"chars": 836,
"preview": "module Scientist\n\n # Smoking in the bathroom and/or sassing.\n class BadBehavior < StandardError\n attr_reader :exper"
},
{
"path": "lib/scientist/experiment.rb",
"chars": 10425,
"preview": "# This mixin provides shared behavior for experiments. Includers must implement\n# `enabled?` and `publish(result)`.\n#\n# "
},
{
"path": "lib/scientist/observation.rb",
"chars": 3406,
"preview": "# What happened when this named behavior was executed? Immutable.\nclass Scientist::Observation\n\n # An Array of Exceptio"
},
{
"path": "lib/scientist/result.rb",
"chars": 1940,
"preview": "# The immutable result of running an experiment.\nclass Scientist::Result\n\n # An Array of candidate Observations.\n attr"
},
{
"path": "lib/scientist/version.rb",
"chars": 41,
"preview": "module Scientist\n VERSION = '1.6.5'\nend\n"
},
{
"path": "lib/scientist.rb",
"chars": 2137,
"preview": "# Include this module into any class which requires science experiments in its\n# methods. Provides the `science` and `de"
},
{
"path": "scientist.gemspec",
"chars": 950,
"preview": "$: << \"lib\" and require \"scientist/version\"\n\nGem::Specification.new do |gem|\n gem.name = \"scientist\"\n gem.des"
},
{
"path": "script/bootstrap",
"chars": 149,
"preview": "#!/bin/sh\n# Ensure local dependencies are available.\n\nset -e\n\ncd $(dirname \"$0\")/..\nrm -f .bundle/config\n\nbundle install"
},
{
"path": "script/release",
"chars": 731,
"preview": "#!/bin/sh\n# Tag and push a release.\n\nset -e\n\n# Make sure we're in the project root.\n\ncd $(dirname \"$0\")/..\n\n# Build a ne"
},
{
"path": "script/test",
"chars": 107,
"preview": "#!/bin/sh\n# Run the unit tests.\n\nset -e\n\ncd $(dirname \"$0\")/..\n script/bootstrap && bundle exec rake test\n"
},
{
"path": "test/scientist/default_test.rb",
"chars": 466,
"preview": "describe Scientist::Default do\n before do\n @ex = Scientist::Default.new \"default\"\n end\n\n it \"is always disabled\" d"
},
{
"path": "test/scientist/experiment_test.rb",
"chars": 19671,
"preview": "describe Scientist::Experiment do\n class Fake\n include Scientist::Experiment\n\n # Undo auto-config magic / preserv"
},
{
"path": "test/scientist/observation_test.rb",
"chars": 4983,
"preview": "describe Scientist::Observation do\n\n before do\n @experiment = Scientist::Experiment.new \"test\"\n end\n\n it \"observes"
},
{
"path": "test/scientist/result_test.rb",
"chars": 3564,
"preview": "describe Scientist::Result do\n before do\n @experiment = Scientist::Experiment.new \"experiment\"\n end\n\n it \"is immut"
},
{
"path": "test/scientist_test.rb",
"chars": 1969,
"preview": "describe Scientist do\n it \"has a version or whatever\" do\n assert Scientist::VERSION\n end\n\n it \"provides a helper t"
},
{
"path": "test/test_helper.rb",
"chars": 115,
"preview": "require \"minitest/autorun\"\n\n$LOAD_PATH.unshift(File.expand_path(File.join(__dir__, \"../lib\")))\nrequire \"scientist\"\n"
}
]
About this extraction
This page contains the full source code of the github/scientist GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 26 files (80.0 KB), approximately 21.3k tokens, and a symbol index with 88 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.