Showing preview only (2,231K chars total). Download the full file or copy to clipboard to get everything.
Repository: swcarpentry/make-novice
Branch: main
Commit: 2843d7678c4a
Files: 67
Total size: 2.1 MB
Directory structure:
gitextract_fzjncbcy/
├── .editorconfig
├── .github/
│ ├── workbench-docker-version.txt
│ └── workflows/
│ ├── README.md
│ ├── docker_apply_cache.yaml
│ ├── docker_build_deploy.yaml
│ ├── docker_pr_receive.yaml
│ ├── pr-close-signal.yaml
│ ├── pr-comment.yaml
│ ├── pr-post-remove-branch.yaml
│ ├── pr-preflight.yaml
│ ├── update-cache.yaml
│ ├── update-workflows.yaml
│ └── workflows-version.txt
├── .gitignore
├── .mailmap
├── .update-copyright.conf
├── .zenodo.json
├── AUTHORS
├── CITATION
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE.md
├── README.md
├── commands.mk
├── config.yaml
├── episodes/
│ ├── 01-intro.md
│ ├── 02-makefiles.md
│ ├── 03-variables.md
│ ├── 04-dependencies.md
│ ├── 05-patterns.md
│ ├── 06-variables.md
│ ├── 07-functions.md
│ ├── 08-self-doc.md
│ ├── 09-conclusion.md
│ ├── data/
│ │ └── books/
│ │ ├── LICENSE_TEXTS.md
│ │ ├── abyss.txt
│ │ ├── isles.txt
│ │ ├── last.txt
│ │ └── sierra.txt
│ └── files/
│ └── code/
│ ├── 02-makefile/
│ │ └── Makefile
│ ├── 02-makefile-challenge/
│ │ └── Makefile
│ ├── 03-variables/
│ │ └── Makefile
│ ├── 03-variables-challenge/
│ │ └── Makefile
│ ├── 04-dependencies/
│ │ └── Makefile
│ ├── 05-patterns/
│ │ └── Makefile
│ ├── 06-variables/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 06-variables-challenge/
│ │ └── Makefile
│ ├── 07-functions/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 08-self-doc/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 09-conclusion-challenge-1/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 09-conclusion-challenge-2/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── countwords.py
│ ├── plotcounts.py
│ └── testzipf.py
├── index.md
├── instructors/
│ └── instructor-notes.md
├── learners/
│ ├── discuss.md
│ ├── reference.md
│ └── setup.md
├── profiles/
│ └── learner-profiles.md
├── requirements.txt
└── site/
└── README.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .editorconfig
================================================
root = true
[*]
charset = utf-8
insert_final_newline = true
trim_trailing_whitespace = true
[*.md]
indent_size = 2
indent_style = space
max_line_length = 100 # Please keep this in sync with bin/lesson_check.py!
trim_trailing_whitespace = false # keep trailing spaces in markdown - 2+ spaces are translated to a hard break (<br/>)
[*.r]
max_line_length = 80
[*.py]
indent_size = 4
indent_style = space
max_line_length = 79
[*.sh]
end_of_line = lf
[Makefile]
indent_style = tab
================================================
FILE: .github/workbench-docker-version.txt
================================================
v0.2.4
================================================
FILE: .github/workflows/README.md
================================================
# Workflow Documentation
## Managing Workflow Updates
By using prebuilt Docker containers that are managed by the Carpentries core Workbench maintainers, these workflows are designed to be rarely updated.
However, is important to be able to keep them up-to-date when appropriate.
You can do this locally using your own R and Workbench installation, or via the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) GitHub Action.
### Updating locally
In a terminal/git bash, navigate to the lesson folder where you want to update the workflows.
Then, start an R session and:
```r
# Install/Update sandpaper
options(repos = c(carpentries = "https://carpentries.r-universe.dev/", CRAN = "https://cloud.r-project.org"))
install.packages("sandpaper")
# update the workflows in your lesson
library("sandpaper")
sandpaper::update_github_workflows()
quit()
```
And then in a bash prompt/git bash terminal:
```bash
$ git add .github/workflows
$ git commit -m "Manual update to docker workflows"
$ git push origin main
```
> [!NOTE]
> For non-renv lessons, this is all the setup you need!
>
> For renv-enabled lessons:
> - Cancel any "01 Maintain: Build and Deploy Site" workflow currently running
> - Run the "02 Maintain: Check for Updated Packages" workflow and merge any PR opened to update the renv lockfile
> - This should automatically run the "03 Maintain: Apply Package Cache" workflow to install packages and build the cache
> - A successful cache buid should then trigger the "01 Maintain: Build and Deploy Site" workflow
### Updating using GitHub
#### Official lessons
"Official" lessons are those in the lesson program repositories, Incubator, or Lab.
They need no extra setup as this is all managed for you as part of the Carpentries GitHub organisations.
To update the workflows, either:
- wait for the scheduled run of the "04 Maintain: Update Workflow Files" at approximately midnight every Tuesday
- go to the Actions tab on GitHub, click "04 Maintain: Update Workflow Files" on the left, then "Run Workflow" on the right
Once complete, this will raise a PR with any changes to the workflows that are needed.
If you are happy with the changes made, you can merge the PR into your lesson repository.
#### Your own lessons
This presumes you:
- already have a lesson repository available on GitHub
- have enabled workflows in the lesson repo
- have set up a SANDPAPER_WORKFLOW personal access token (PAT) in the lesson repo
To go through these steps, please follow the [Forking a Workbench Lesson](https://docs.carpentries.org/resources/curriculum/lesson-forks.html#forking-a-workbench-lesson-repository)
documentation.
Once set up, run the "04 Maintain: Update Workflow Files" (`update-workflows.yaml`) action.
This will raise a PR with any changes to the workflows that are needed.
If you are happy with the changes made, you can merge the PR into your lesson repository.
## Package Caches for RMarkdown Lessons
In summary, generating a reusable package cache is achieved by running the "02 Maintain: Check for Updated Packages" workflow, and then the "03 Maintain: Apply Package Cache" workflow.
> [!NOTE]
> Caching is only relevant for lessons that use Rmd files and renv to manage R packages.
> If you are building basic markdown documents, caching will not apply to you, and the only
> workflow that needs to be run is "01 Maintain: Build and Deploy Site".
### Caching
The two cache management workflows are separated to ensure that once you have a successful build with a working renv cache, this cache is stored and will be reused by the Workbench Docker container.
This means that lesson builds will be faster once an renv cache is created and reused by the Docker container.
Another major bonus of this setup is that you can keep using this cache indefinitely to build your lesson.
This is important if you need very specific versions of R packages ("pinning").
If and when you want to perform an update to the cache, you can re-run the "02 Maintain: Check for Updated Packages" and verify that your lesson still builds with the new packages.
If all looks good, re-run the "03 Maintain: Apply Package Cache" workflow, and this will write a new renv cache file to GitHub.
In any case, the renv cache is invalidated by new versions of the `renv.lock` file.
This happens:
- if you update your lockfile locally by using the `sandpaper::update_cache()` function, and then push it to the lesson repository
- when you run the "02 Maintain: Check for Updated Packages" and there are new packages to install
More information on managing local renv caches for lessons can be found in the [Sandpaper packages vignettes](https://carpentries.github.io/sandpaper/articles/building-with-renv.html).
#### Using different package cache versions
There are times when you may want to go back to a previous renv package cache file:
- if you run "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" and the cache generation fails for some reason
- if there is a new R package that produces incorrect or broken lesson output
Cache files will have the following name format, where IMAGE is the workbench-docker image version, and HASHSUM is the `renv.lock` lockfile MD5 hash:
```
IMAGE HASHSUM
[ | ] [ | ]
v0.2.4_renv-2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4
```
Copy the hashsum part of the desired cache file you want to use, e.g. `2e499eb706112971b2cffceb49b55a6efe49f3ed75cd6579b10ff224489daca4`.
Then either:
1. Add a repository variable called CACHE_VERSION, and paste in the hash
- Go to ...
2. Run the "01 Maintain: Build and Deploy Site" manually, supplying the CACHE_VERSION input
- Go to ...
If you have no caches listed, make sure to run the "02 Maintain: Check for Updated Packages" and "03 Maintain: Apply Package Cache" to create a new renv cache file.
> [!NOTE]
> If you are maintaining an official lesson, caches are saved in an AWS S3 bucket owned by the Carpentries.
> Once a successful cache has been saved, these will be listed in the outputs of the "01 Maintain: Build and Deploy Site" workflow.
>
> If you are developing a lesson in your own repository, caches are saved on GitHub.
> You can see available caches by going to the Actions tab, and clicking Caches on the left hand side.
## User Settings
Input level variables are documented in the `carpentries/actions` repository READMEs for each composite action.
Specific repository level variables can be set that will force particular options across all workflow runs.
### 01 Maintain: Build and Deploy Site (docker_build_deploy.yaml)
Repository-level variables for this workflow are:
- WORKBENCH_TAG
- The workbench-docker release version to use for a given build
- This can be set to a specific version number to force all builds to use a given container version
- Default is unset or `latest`
- BUILD_RESET
- Force a reset of previously build markdown files
- Setting this variable value to `true` will force sandpaper to delete any previously build markdown files
- Default is unset or `false`
- AUTO_MERGE_WORKBENCH_VERSION_UPDATE
- Control merge behaviour of the workbench-docker version update PR
- When a new workbench Docker image version is detected, usually after a sandpaper, varnish, or pegboard update, its version number will be incremented
- If a newer version is available, a PR will be auto-generated that updates the `.github/workbench-docker-version.txt` file, and this PR will be auto-merged
- To not auto-merge this PR and to choose when to update the Docker version used, set this to `false`.
- Default is unset or `true`
- LANG_CODE
- Two-letter language code that triggers the use of Joel Nitta's {dovetail} package for lesson translation
- This is used in the internationalisation repos of the main Carpentry lesson programs
- Default is unset or `''`
### 02 Maintain: Check for Updated Packages (update-cache.yaml)
Repository-level variables for this workflow are:
- LOCKFILE_CACHE_GEN
- Passed to the `generate-cache` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action
- A temporary renv cache is generated when this workflow runs
- If this option is set to `false`, no temporary cache will be generated
- Default is `true`
- FORCE_RENV_INIT
- Passed to the `force-renv-init` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action
- renv initialises a cache based on a given lockfile
- If this lockfile is particularly old or packages have broken/unresolvable dependencies, then builds will fail
- If this option is set to `true`, a full renv reinitialisation will occur, "wiping the slate clean"
- This option is useful if you're using Bioconductor packages which often break when new Bioconductor releases happen
- Default is `false`
- UPDATE_PACKAGES
- Passed to the `update` input of the [update-lockfile](https://github.com/carpentries/actions/tree/main/update-lockfile) action
- If set to `false` only package hydration will happen and no package update checks will occur
- Default is `true`
### 03 Maintain: Apply Package Cache (docker_apply_cache.yaml)
Repository-level variables for this workflow are:
- WORKBENCH_TAG
- The workbench-docker release version to use for a given build
- This can be set to a specific version number to force all builds to use a given container version
- Default is unset or `latest`
### 04 Maintain: Update Workflow Files (update-workflows.yaml)
There are no repository variables for this workflow.
## Pull Request and Review Management
Because our lessons execute code, pull requests are a security risk for any lesson and thus have security measures associted with them.
**Do not merge any pull requests that do not pass checks and do not have bots commented on them.**
This series of workflows all go together and are described in the following diagram and the below sections:
### Pre Flight Pull Request Validation (pr-preflight.yaml)
This workflow runs every time a pull request is created and its purpose is to validate that the pull request is okay to run.
This means the following things:
1. The pull request does not contain modified workflow files
2. If the pull request contains modified workflow files, it does not contain modified content files
(such as a situation where @carpentries-bot will make an automated pull request)
3. The pull request does not contain an invalid commit hash
(e.g. from a fork that was made before a lesson was transitioned from styles to use the Workbench).
Once the checks are finished, a comment is issued to the pull request, which will allow maintainers to determine if it is safe to run the "Receive Pull Request" workflow from new contributors.
### Receive Pull Request (docker_pr_receive.yaml)
**Note of caution:** This workflow runs arbitrary code by anyone who creates a pull request.
GitHub has safeguarded the token used in this workflow to have no privileges in the repository, but we have taken precautions to protect against spoofing.
This workflow is triggered with every push to a pull request.
If this workflow is already running and a new push is sent to the pull request, the workflow running from the previous push will be cancelled and a new workflow run will be started.
The first step of this workflow is to check if it is valid (e.g. that no workflow files have been modified):
- If there are workflow files that have been modified, a comment is made that indicates that the workflow will not continue.
- If both a workflow file and lesson content is modified, an error will occur and the workflow will not continue.
The second step (if valid) is to build the generated content from the pull request.
This builds the content and uploads three artifacts:
1. The pull request number (pr)
2. A summary of changes after the rendering process (diff)
3. The rendered files (build)
The artifacts produced are used by the "Comment on Pull Request" workflow.
### Comment on Pull Request (pr-comment.yaml)
This workflow is triggered if the `docker_pr_receive.yaml` workflow is successful.
The steps in this workflow are:
1. Test if the workflow is valid and comment the validity of the workflow to the pull request.
2. If it is valid: create an orphan branch with two commits: the current state of the repository and the proposed changes.
3. If it is valid: update the pull request comment with the summary of changes
Importantly: if the pull request is invalid, the branch is not created so any malicious code is not published.
From here, the maintainer can request changes from the author and eventually either merge or reject the PR.
When this happens, if the PR was valid, the preview branch needs to be deleted.
### Send Close PR Signal (pr-close-signal.yaml)
Triggered any time a pull request is closed.
This emits an artifact that is the pull request number for the next action.
### Remove Pull Request Branch (pr-post-remove-branch.yaml)
Tiggered by `pr-close-signal.yaml`.
This removes the temporary branch associated with the pull request (if it was created).
================================================
FILE: .github/workflows/docker_apply_cache.yaml
================================================
name: "03 Maintain: Apply Package Cache"
description: "Generate the package cache for the lesson after a pull request has been merged or via manual trigger, and cache in S3 or GitHub"
on:
workflow_dispatch:
inputs:
name:
description: 'Who triggered this build?'
required: true
default: 'Maintainer (via GitHub)'
pull_request:
types:
- closed
branches:
- main
# queue cache runs
concurrency:
group: docker-apply-cache
cancel-in-progress: false
jobs:
preflight:
name: "Preflight: PR or Manual Trigger?"
runs-on: ubuntu-latest
outputs:
do-apply: ${{ steps.check.outputs.merged_or_manual }}
steps:
- name: "Should we run cache application?"
id: check
run: |
if [[ "${{ github.event_name }}" == "workflow_dispatch" ||
("${{ github.ref }}" == "refs/heads/main" && "${{ github.event.action }}" == "closed" && "${{ github.event.pull_request.merged }}" == "true") ]]; then
echo "merged_or_manual=true" >> $GITHUB_OUTPUT
else
echo "This was not a manual trigger and no PR was merged. No action taken."
echo "merged_or_manual=false" >> $GITHUB_OUTPUT
fi
shell: bash
check-renv:
name: "Check If We Need {renv}"
runs-on: ubuntu-latest
needs: preflight
if: needs.preflight.outputs.do-apply == 'true'
permissions:
id-token: write
outputs:
renv-needed: ${{ steps.check-for-renv.outputs.renv-needed }}
renv-cache-hashsum: ${{ steps.check-for-renv.outputs.renv-cache-hashsum }}
renv-cache-available: ${{ steps.check-for-renv.outputs.renv-cache-available }}
steps:
- name: "Check for renv"
id: check-for-renv
uses: carpentries/actions/renv-checks@main
with:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }}
token: ${{ secrets.GITHUB_TOKEN }}
no-renv-cache-used:
name: "No renv cache used"
runs-on: ubuntu-latest
needs: check-renv
if: needs.check-renv.outputs.renv-needed != 'true'
steps:
- name: "No renv cache needed"
run: echo "No renv cache needed for this lesson"
renv-cache-available:
name: "renv cache available"
runs-on: ubuntu-latest
needs: check-renv
if: needs.check-renv.outputs.renv-cache-available == 'true'
steps:
- name: "renv cache available"
run: echo "renv cache available for this lesson"
update-renv-cache:
name: "Update renv Cache"
runs-on: ubuntu-latest
needs: check-renv
if: |
needs.check-renv.outputs.renv-needed == 'true' &&
needs.check-renv.outputs.renv-cache-available != 'true' &&
(
github.event_name == 'workflow_dispatch' ||
(
github.event.pull_request.merged == true &&
(
(
contains(
join(github.event.pull_request.labels.*.name, ','),
'type: package cache'
) &&
github.event.pull_request.head.ref == 'update/packages'
)
||
(
contains(
join(github.event.pull_request.labels.*.name, ','),
'type: workflows'
) &&
github.event.pull_request.head.ref == 'update/workflows'
)
||
(
contains(
join(github.event.pull_request.labels.*.name, ','),
'type: docker version'
) &&
github.event.pull_request.head.ref == 'update/workbench-docker-version'
)
)
)
)
permissions:
checks: write
contents: write
pages: write
id-token: write
container:
image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }}
env:
WORKBENCH_PROFILE: "ci"
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
RENV_PATHS_ROOT: /home/rstudio/lesson/renv
RENV_PROFILE: "lesson-requirements"
RENV_VERSION: ${{ needs.check-renv.outputs.renv-cache-hashsum }}
RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library"
volumes:
- ${{ github.workspace }}:/home/rstudio/lesson
options: --cpus 2
steps:
- uses: actions/checkout@v6
- name: "Debugging Info"
run: |
echo "Current Directory: $(pwd)"
ls -lah /home/rstudio/.workbench
ls -lah $(pwd)
Rscript -e 'sessionInfo()'
shell: bash
- name: "Mark Repository as Safe"
run: |
git config --global --add safe.directory $(pwd)
shell: bash
- name: "Ensure sandpaper is loadable"
run: |
.libPaths()
library(sandpaper)
shell: Rscript {0}
- name: "Setup Lesson Dependencies"
run: |
Rscript /home/rstudio/.workbench/setup_lesson_deps.R
shell: bash
- name: "Fortify renv Cache"
run: |
Rscript /home/rstudio/.workbench/fortify_renv_cache.R
shell: bash
- name: "Get Container Version Used"
id: wb-vers
uses: carpentries/actions/container-version@main
with:
WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }}
renv-needed: ${{ needs.check-renv.outputs.renv-needed }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: "Validate Current Org and Workflow"
id: validate-org-workflow
uses: carpentries/actions/validate-org-workflow@main
with:
repo: ${{ github.repository }}
workflow: ${{ github.workflow }}
- name: "Configure AWS credentials via OIDC"
id: aws-creds
env:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
if: |
steps.validate-org-workflow.outputs.is_valid == 'true' &&
env.role-to-assume != '' &&
env.aws-region != ''
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ env.role-to-assume }}
aws-region: ${{ env.aws-region }}
output-credentials: true
- name: "Upload cache object to S3"
id: upload-cache
uses: tespkg/actions-cache@v1.10.0
with:
accessKey: ${{ steps.aws-creds.outputs.aws-access-key-id }}
secretKey: ${{ steps.aws-creds.outputs.aws-secret-access-key }}
sessionToken: ${{ steps.aws-creds.outputs.aws-session-token }}
bucket: workbench-docker-caches
path: |
/home/rstudio/lesson/renv
/usr/local/lib/R/site-library
key: ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-${{ needs.check-renv.outputs.renv-cache-hashsum }}
restore-keys:
${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-
record-cache-result:
name: "Record Caching Status"
runs-on: ubuntu-latest
needs: [check-renv, update-renv-cache]
if: always()
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: "Record cache result"
run: |
echo "${{ needs.update-renv-cache.result == 'success' || needs.check-renv.outputs.renv-cache-available == 'true' || 'false' }}" > ${{ github.workspace }}/apply-cache-result
shell: bash
- name: "Upload cache result"
uses: actions/upload-artifact@v7
with:
name: apply-cache-result
path: ${{ github.workspace }}/apply-cache-result
================================================
FILE: .github/workflows/docker_build_deploy.yaml
================================================
name: "01 Maintain: Build and Deploy Site"
description: "Build and deploy the lesson site using the carpentries/workbench-docker container"
on:
push:
branches:
- 'main'
- 'l10n_main'
paths-ignore:
- '.github/workflows/**.yaml'
- '.github/workbench-docker-version.txt'
schedule:
- cron: '0 0 * * 2'
workflow_run:
workflows: ["03 Maintain: Apply Package Cache"]
types:
- completed
workflow_dispatch:
inputs:
name:
description: 'Who triggered this build?'
required: true
default: 'Maintainer (via GitHub)'
CACHE_VERSION:
description: 'Optional renv cache version override'
required: false
default: ''
reset:
description: 'Reset cached markdown files'
required: true
default: false
type: boolean
force-skip-manage-deps:
description: 'Skip build-time dependency management'
required: true
default: false
type: boolean
# only one build/deploy at a time
concurrency:
group: docker-build-deploy
cancel-in-progress: true
jobs:
preflight:
name: "Preflight: Schedule, Push, or PR?"
runs-on: ubuntu-latest
outputs:
do-build: ${{ steps.build-check.outputs.do-build }}
renv-needed: ${{ steps.build-check.outputs.renv-needed }}
renv-cache-hashsum: ${{ steps.build-check.outputs.renv-cache-hashsum }}
workbench-container-file-exists: ${{ steps.wb-vers.outputs.workbench-container-file-exists }}
wb-vers: ${{ steps.wb-vers.outputs.container-version }}
last-wb-vers: ${{ steps.wb-vers.outputs.last-container-version }}
workbench-update: ${{ steps.wb-vers.outputs.workbench-update }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: "Should we run build and deploy?"
id: build-check
uses: carpentries/actions/build-preflight@main
- name: "Checkout Lesson"
if: steps.build-check.outputs.do-build == 'true'
uses: actions/checkout@v6
- name: "Get container version info"
id: wb-vers
if: steps.build-check.outputs.do-build == 'true'
uses: carpentries/actions/container-version@main
with:
WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }}
renv-needed: ${{ steps.build-check.outputs.renv-needed }}
token: ${{ secrets.GITHUB_TOKEN }}
full-build:
name: "Build Full Site"
runs-on: ubuntu-latest
needs: preflight
if: |
needs.preflight.outputs.do-build == 'true' &&
needs.preflight.outputs.workbench-update != 'true'
env:
RENV_EXISTS: ${{ needs.preflight.outputs.renv-needed }}
RENV_HASH: ${{ needs.preflight.outputs.renv-cache-hashsum }}
permissions:
checks: write
contents: write
pages: write
id-token: write
container:
image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }}
env:
WORKBENCH_PROFILE: "ci"
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
RENV_PATHS_ROOT: /home/rstudio/lesson/renv
RENV_PROFILE: "lesson-requirements"
RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library"
volumes:
- ${{ github.workspace }}:/home/rstudio/lesson
options: --cpus 1
steps:
- uses: actions/checkout@v6
- name: "Debugging Info"
run: |
cd /home/rstudio/lesson
echo "Current Directory: $(pwd)"
echo "RENV_HASH is $RENV_HASH"
ls -lah /home/rstudio/.workbench
ls -lah $(pwd)
Rscript -e 'sessionInfo()'
shell: bash
- name: "Mark Repository as Safe"
run: |
git config --global --add safe.directory $(pwd)
shell: bash
- name: "Setup Lesson Dependencies"
id: build-container-deps
uses: carpentries/actions/build-container-deps@main
with:
CACHE_VERSION: ${{ vars.CACHE_VERSION || github.event.inputs.CACHE_VERSION || '' }}
WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG || 'latest' }}
LESSON_PATH: ${{ vars.LESSON_PATH || '/home/rstudio/lesson' }}
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: "Run Container and Build Site"
id: build-and-deploy
uses: carpentries/actions/build-and-deploy@main
with:
reset: ${{ vars.BUILD_RESET || github.event.inputs.reset || 'false' }}
skip-manage-deps: ${{ github.event.inputs.force-skip-manage-deps == 'true' || steps.build-container-deps.outputs.renv-cache-available || steps.build-container-deps.outputs.backup-cache-used || 'false' }}
lang-code: ${{ vars.LANG_CODE || '' }}
update-container-version:
name: "Update container version used"
runs-on: ubuntu-latest
needs: [preflight]
permissions:
actions: write
contents: write
pull-requests: write
id-token: write
if: |
needs.preflight.outputs.do-build == 'true' &&
(
needs.preflight.outputs.workbench-container-file-exists == 'false' ||
needs.preflight.outputs.workbench-update == 'true'
)
steps:
- name: "Record container version used"
uses: carpentries/actions/record-container-version@main
with:
CONTAINER_VER: ${{ needs.preflight.outputs.wb-vers }}
AUTO_MERGE: ${{ vars.AUTO_MERGE_CONTAINER_VERSION_UPDATE || 'true' }}
token: ${{ secrets.GITHUB_TOKEN }}
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
================================================
FILE: .github/workflows/docker_pr_receive.yaml
================================================
name: "Bot: Receive Pull Request"
description: "Receive a pull request and build the markdown source files"
on:
pull_request:
types:
[opened, synchronize, reopened]
workflow_dispatch:
inputs:
pr_number:
type: number
required: true
concurrency:
group: ${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
pull-requests: write
jobs:
preflight:
name: "Preflight: md-outputs exists?"
runs-on: ubuntu-latest
outputs:
branch-exists: ${{ steps.check.outputs.exists }}
steps:
- name: "Checkout Lesson"
uses: actions/checkout@v6
- name: "Check if md-outputs branch exists"
id: check
run: |
# 💡 Checking for md-outputs branch #
if [[ -n $(git ls-remote --exit-code --heads origin md-outputs) ]]; then
echo "exists=true" >> $GITHUB_OUTPUT
else
echo "exists=false" >> $GITHUB_OUTPUT
echo "::error::md-outputs branch required but does not exist."
echo "::error::Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows."
echo "## ❌ ERROR: md-outputs branch required" >> $GITHUB_STEP_SUMMARY
echo "Please merge any open package update PRs to trigger the '03 Maintain: Apply Package Cache' and '01: Maintain: Build and Deploy Site' workflows." >> $GITHUB_STEP_SUMMARY
exit 1
fi
shell: bash
test-pr:
name: "Record PR number"
if: |
github.event.action != 'closed' &&
needs.preflight.outputs.branch-exists == 'true'
runs-on: ubuntu-latest
needs: preflight
outputs:
is_valid: ${{ steps.check-pr.outputs.VALID }}
pr_number: ${{ env.NR }}
pr_branch: ${{ env.PR_BRANCH }}
steps:
- name: "Grab PR"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
if [[ "${{ github.event_name }}" == "pull_request" ]] ; then
PR_NUMBER=${{ github.event.number }}
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]] ; then
PR_NUMBER=${{ inputs.pr_number }}
fi
echo $PR_NUMBER > ${{ github.workspace }}/NR
echo "NR=$PR_NUMBER" >> $GITHUB_ENV
echo "PR_BRANCH=$(gh -R ${{ github.repository }} pr view $PR_NUMBER --json headRefName --jq '.headRefName')" >> $GITHUB_ENV
shell: bash
- name: "Upload PR number"
id: upload
if: always()
uses: actions/upload-artifact@v7
with:
name: pr
path: ${{ github.workspace }}/NR
- name: "Get Invalid Hashes File"
id: hash
run: |
echo "json<<EOF
$(curl -sL https://files.carpentries.org/invalid-hashes.json)
EOF" >> $GITHUB_OUTPUT
shell: bash
- name: "Debug Hashes Output"
run: |
echo "${{ steps.hash.outputs.json }}"
shell: bash
- name: "Check PR"
id: check-pr
uses: carpentries/actions/check-valid-pr@main
with:
pr: ${{ env.NR }}
invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }}
check-renv:
name: "Check If We Need {renv}"
runs-on: ubuntu-latest
outputs:
renv-needed: ${{ steps.renv-check.outputs.renv-needed }}
renv-cache-hashsum: ${{ steps.renv-check.outputs.renv-cache-hashsum }}
steps:
- name: "Checkout Lesson"
uses: actions/checkout@v6
- name: "Is renv required?"
id: renv-check
uses: carpentries/actions/renv-checks@main
with:
CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }}
skip-cache-check: true
build-md-source:
name: "Build markdown source files if valid"
needs:
- test-pr
- check-renv
runs-on: ubuntu-latest
if: needs.test-pr.outputs.is_valid == 'true'
env:
CHIVE: ${{ github.workspace }}/site/chive
PR: ${{ github.workspace }}/site/pr
GHWMD: ${{ github.workspace }}/site/built
PR_BRANCH: ${{ needs.test-pr.outputs.pr_branch }}
PR_NUMBER: ${{ needs.test-pr.outputs.pr_number }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
permissions:
checks: write
contents: write
pages: write
id-token: write
container:
image: ghcr.io/carpentries/workbench-docker:${{ vars.WORKBENCH_TAG || 'latest' }}
env:
WORKBENCH_PROFILE: "ci"
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RENV_PATHS_ROOT: /home/rstudio/lesson/renv
RENV_PROFILE: "lesson-requirements"
RENV_CONFIG_EXTERNAL_LIBRARIES: "/usr/local/lib/R/site-library"
volumes:
- ${{ github.workspace }}:/home/rstudio/lesson
options: --cpus 2
outputs:
workbench-update: ${{ steps.wb-vers.outputs.workbench-update }}
build-site: ${{ steps.build-site.outcome }}
steps:
- uses: actions/checkout@v6
- name: "Check Out Staging Branch"
uses: actions/checkout@v6
with:
ref: md-outputs
path: ${{ env.GHWMD }}
- name: Mark Repository as Safe
run: |
git config --global --add safe.directory $(pwd)
git config --global --add safe.directory /home/rstudio/lesson
shell: bash
- name: "Ensure sandpaper is loadable"
run: |
.libPaths()
library(sandpaper)
shell: Rscript {0}
- name: Setup Lesson Dependencies
run: |
Rscript /home/rstudio/.workbench/setup_lesson_deps.R
shell: bash
- name: Get Container Version Used
id: wb-vers
if: needs.check-renv.outputs.renv-needed == 'true'
uses: carpentries/actions/container-version@main
with:
WORKBENCH_TAG: ${{ vars.WORKBENCH_TAG }}
renv-needed: ${{ needs.check-renv.outputs.renv-needed }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: "Validate Current Org and Workflow"
id: validate-org-workflow
if: needs.check-renv.outputs.renv-needed == 'true'
uses: carpentries/actions/validate-org-workflow@main
with:
repo: ${{ github.repository }}
workflow: ${{ github.workflow }}
- name: Configure AWS credentials via OIDC
id: aws-creds
env:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
if: |
steps.validate-org-workflow.outputs.is_valid == 'true' &&
needs.check-renv.outputs.renv-needed == 'true' &&
env.role-to-assume != '' &&
env.aws-region != ''
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ env.role-to-assume }}
aws-region: ${{ env.aws-region }}
output-credentials: true
- name: Get cache object from S3
id: s3-cache
uses: tespkg/actions-cache/restore@v1.10.0
if: needs.check-renv.outputs.renv-needed == 'true'
with:
# insecure: false # optional, use http instead of https. default false
accessKey: ${{ steps.aws-creds.outputs.aws-access-key-id }}
secretKey: ${{ steps.aws-creds.outputs.aws-secret-access-key }}
sessionToken: ${{ steps.aws-creds.outputs.aws-session-token }}
bucket: workbench-docker-caches
path: |
/home/rstudio/lesson/renv
/usr/local/lib/R/site-library
key: ${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-${{ needs.check-renv.outputs.renv-cache-hashsum }}
restore-keys:
${{ github.repository }}/${{ steps.wb-vers.outputs.container-version }}_renv-
- name: "Fortify renv Cache"
if: |
needs.check-renv.outputs.renv-needed == 'true' &&
steps.s3-cache.outputs.cache-hit != 'true'
run: |
Rscript /home/rstudio/.workbench/fortify_renv_cache.R
shell: bash
- name: "Validate and Build Markdown"
id: build-site
run: |
sandpaper::package_cache_trigger(TRUE)
sandpaper::validate_lesson(path = '/home/rstudio/lesson')
sandpaper:::build_markdown(path = '/home/rstudio/lesson', quiet = FALSE)
shell: Rscript {0}
- name: "Generate Artifacts"
id: generate-artifacts
run: |
sandpaper:::ci_bundle_pr_artifacts(
repo = '${{ github.repository }}',
pr_number = '${{ env.PR_NUMBER }}',
path_md = '/home/rstudio/lesson/site/built',
path_pr = '/home/rstudio/lesson/site/pr',
path_archive = '/home/rstudio/lesson/site/chive',
branch = 'md-outputs'
)
shell: Rscript {0}
- name: "Upload PR"
uses: actions/upload-artifact@v7
with:
name: pr
path: ${{ env.PR }}
overwrite: true
- name: "Upload Diff"
uses: actions/upload-artifact@v7
with:
name: diff
path: ${{ env.CHIVE }}
retention-days: 1
- name: "Upload Build"
uses: actions/upload-artifact@v7
with:
name: built
path: ${{ env.GHWMD }}
retention-days: 1
- name: "Teardown"
run: sandpaper::reset_site()
shell: Rscript {0}
================================================
FILE: .github/workflows/pr-close-signal.yaml
================================================
name: "Bot: Send Close Pull Request Signal"
on:
pull_request:
types:
[closed]
jobs:
send-close-signal:
name: "Send closing signal"
runs-on: ubuntu-22.04
if: ${{ github.event.action == 'closed' }}
steps:
- name: "Create PRtifact"
run: |
mkdir -p ./pr
printf ${{ github.event.number }} > ./pr/NUM
- name: Upload Diff
uses: actions/upload-artifact@v7
with:
name: pr
path: ./pr
================================================
FILE: .github/workflows/pr-comment.yaml
================================================
name: "Bot: Comment on the Pull Request"
description: "Comment on the pull request with the results of the markdown generation"
on:
workflow_run:
workflows: ["Bot: Receive Pull Request"]
types:
- completed
jobs:
# Pull requests are valid if:
# - they match the sha of the workflow run head commit
# - they are open
# - no .github files were committed, except for .github/workbench-docker-version.txt
test-pr:
name: "Test if pull request is valid"
runs-on: ubuntu-latest
outputs:
is_valid: ${{ steps.check-pr.outputs.VALID }}
payload: ${{ steps.check-pr.outputs.payload }}
number: ${{ steps.get-pr.outputs.NUM }}
msg: ${{ steps.check-pr.outputs.MSG }}
steps:
- name: "Download PR artifact"
id: dl
uses: carpentries/actions/download-workflow-artifact@main
with:
run: ${{ github.event.workflow_run.id }}
name: 'pr'
- name: "Get PR Number"
if: ${{ steps.dl.outputs.success == 'true' }}
id: get-pr
run: |
unzip pr.zip
echo "NUM=$(<./NR)" >> $GITHUB_OUTPUT
- name: "Fail if PR number was not present"
id: bad-pr
if: ${{ steps.dl.outputs.success != 'true' }}
run: |
echo '::error::A pull request number was not recorded. The pull request that triggered this workflow is likely malicious.'
exit 1
- name: "Checkout Lesson"
uses: actions/checkout@v6
- name: "Verify committed files"
id: changed-files
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
## Get list of changed files in the PR ##
ONLY_VERSION=$(gh pr view ${{ steps.get-pr.outputs.NUM }} --json files --jq '
.files |
length == 1 and
.[0].path == ".github/workbench-docker-version.txt"
')
if [[ "$ONLY_VERSION" == "true" ]]; then
echo "only_version_file=true" >> $GITHUB_OUTPUT
else
echo "only_version_file=false" >> $GITHUB_OUTPUT
fi
shell: bash
- name: "Skip checks for Workbench version file updates"
if: steps.changed-files.outputs.only_version_file == 'true'
run: |
echo "# 🔧 Wait for Next Cache Update #"
echo "Only workbench-docker-version.txt changed."
exit 0
shell: bash
- name: "Get Invalid Hashes File"
id: hash
run: |
echo "json<<EOF
$(curl -sL https://files.carpentries.org/invalid-hashes.json)
EOF" >> $GITHUB_OUTPUT
- name: "Check PR"
id: check-pr
if: ${{ steps.dl.outputs.success == 'true' }}
uses: carpentries/actions/check-valid-pr@main
with:
pr: ${{ steps.get-pr.outputs.NUM }}
sha: ${{ github.event.workflow_run.head_sha }}
headroom: 3 # if it's within the last three commits, we can keep going, because it's likely rapid-fire
invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }}
fail_on_error: true
- name: "Comment result of validation"
id: comment-diff
if: always()
uses: carpentries/actions/comment-diff@main
with:
pr: ${{ steps.get-pr.outputs.NUM }}
body: ${{ steps.check-pr.outputs.MSG }}
# Create an orphan branch on this repository with two commits
# - the current HEAD of the md-outputs branch
# - the output from running the current HEAD of the pull request through
# the md generator
create-branch:
name: "Create Git Branch"
needs: test-pr
runs-on: ubuntu-latest
if: needs.test-pr.outputs.is_valid == 'true'
env:
NR: ${{ needs.test-pr.outputs.number }}
permissions:
contents: write
steps:
- name: "Checkout md outputs"
uses: actions/checkout@v6
with:
ref: md-outputs
path: built
fetch-depth: 1
- name: "Download built markdown"
id: dl
uses: carpentries/actions/download-workflow-artifact@main
with:
run: ${{ github.event.workflow_run.id }}
name: 'built'
- if: steps.dl.outputs.success == 'true'
run: unzip built.zip
- name: "Create orphan and push"
if: steps.dl.outputs.success == 'true'
run: |
cd built/
git config --local user.email "actions@github.com"
git config --local user.name "GitHub Actions"
CURR_HEAD=$(git rev-parse HEAD)
git checkout --orphan md-outputs-PR-${NR}
git add -A
git commit -m "source commit: ${CURR_HEAD}"
ls -A | grep -v '^.git$' | xargs -I _ rm -r '_'
cd ..
unzip -o -d built built.zip
cd built
git add -A
git commit --allow-empty -m "differences for PR #${NR}"
git push -u --force --set-upstream origin md-outputs-PR-${NR}
# Comment on the Pull Request with a link to the branch and the diff
comment-pr:
name: "Comment on Pull Request"
needs: [test-pr, create-branch]
runs-on: ubuntu-latest
if: needs.test-pr.outputs.is_valid == 'true'
env:
NR: ${{ needs.test-pr.outputs.number }}
permissions:
pull-requests: write
steps:
- name: "Download comment artifact"
id: dl
uses: carpentries/actions/download-workflow-artifact@main
with:
run: ${{ github.event.workflow_run.id }}
name: 'diff'
- if: steps.dl.outputs.success == 'true'
run: unzip ${{ github.workspace }}/diff.zip
- name: "Comment on PR"
id: comment-diff
if: steps.dl.outputs.success == 'true'
uses: carpentries/actions/comment-diff@main
with:
pr: ${{ env.NR }}
path: ${{ github.workspace }}/diff.md
# Comment if the PR is open and matches the SHA, but the workflow files have
# changed
comment-changed-workflow:
name: "Comment if workflow files have changed"
needs: test-pr
runs-on: ubuntu-latest
if: |
always() &&
needs.test-pr.outputs.is_valid == 'false'
env:
NR: ${{ needs.test-pr.outputs.number }}
body: ${{ needs.test-pr.outputs.msg }}
permissions:
pull-requests: write
steps:
- name: "Check for spoofing"
id: dl
uses: carpentries/actions/download-workflow-artifact@main
with:
run: ${{ github.event.workflow_run.id }}
name: 'built'
- name: "Alert if spoofed"
id: spoof
if: steps.dl.outputs.success == 'true'
run: |
echo 'body<<EOF' >> $GITHUB_ENV
echo '' >> $GITHUB_ENV
echo '## :x: DANGER :x:' >> $GITHUB_ENV
echo 'This pull request has modified workflows that created output. Close this now.' >> $GITHUB_ENV
echo '' >> $GITHUB_ENV
echo 'EOF' >> $GITHUB_ENV
- name: "Comment on PR"
id: comment-diff
uses: carpentries/actions/comment-diff@main
with:
pr: ${{ env.NR }}
body: ${{ env.body }}
================================================
FILE: .github/workflows/pr-post-remove-branch.yaml
================================================
name: "Bot: Remove Temporary PR Branch"
on:
workflow_run:
workflows: ["Bot: Send Close Pull Request Signal"]
types:
- completed
jobs:
delete:
name: "Delete branch from Pull Request"
runs-on: ubuntu-22.04
if: >
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.conclusion == 'success'
permissions:
contents: write
steps:
- name: 'Download artifact'
uses: carpentries/actions/download-workflow-artifact@main
with:
run: ${{ github.event.workflow_run.id }}
name: pr
- name: "Get PR Number"
id: get-pr
run: |
unzip pr.zip
echo "NUM=$(<./NUM)" >> $GITHUB_OUTPUT
- name: 'Remove branch'
uses: carpentries/actions/remove-branch@main
with:
pr: ${{ steps.get-pr.outputs.NUM }}
================================================
FILE: .github/workflows/pr-preflight.yaml
================================================
name: "Pull Request Preflight Check"
on:
pull_request_target:
branches:
["main"]
types:
["opened", "synchronize", "reopened"]
jobs:
test-pr:
name: "Test if pull request is valid"
if: ${{ github.event.action != 'closed' }}
runs-on: ubuntu-latest
outputs:
is_valid: ${{ steps.check-pr.outputs.VALID }}
permissions:
pull-requests: write
steps:
- name: "Get Invalid Hashes File"
id: hash
run: |
echo "json<<EOF
$(curl -sL https://files.carpentries.org/invalid-hashes.json)
EOF" >> $GITHUB_OUTPUT
- name: "Check PR"
id: check-pr
uses: carpentries/actions/check-valid-pr@main
with:
pr: ${{ github.event.number }}
invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }}
fail_on_error: true
- name: "Comment result of validation"
id: comment-diff
if: ${{ always() }}
uses: carpentries/actions/comment-diff@main
with:
pr: ${{ github.event.number }}
body: ${{ steps.check-pr.outputs.MSG }}
================================================
FILE: .github/workflows/update-cache.yaml
================================================
name: "02 Maintain: Check for Updated Packages"
description: "Check for updated R packages and create a pull request to update the lesson's renv lockfile and package cache"
on:
schedule:
- cron: '0 0 * * 2'
workflow_dispatch:
inputs:
name:
description: 'Who triggered this build?'
required: true
default: 'Maintainer (via GitHub)'
force-renv-init:
description: 'Force full lockfile update?'
required: false
default: false
type: boolean
update-packages:
description: 'Install any package updates?'
required: false
default: true
type: boolean
generate-cache:
description: 'Generate separate package cache?'
required: false
default: false
type: boolean
env:
LOCKFILE_CACHE_GEN: ${{ vars.LOCKFILE_CACHE_GEN || github.event.inputs.generate-cache || 'false' }}
FORCE_RENV_INIT: ${{ vars.FORCE_RENV_INIT || github.event.inputs.force-renv-init || 'false' }}
UPDATE_PACKAGES: ${{ vars.UPDATE_PACKAGES || github.event.inputs.update-packages || 'true' }}
jobs:
preflight:
name: "Preflight: Manual or Scheduled Trigger?"
runs-on: ubuntu-latest
outputs:
ok: ${{ steps.check.outputs.ok }}
steps:
- id: check
run: |
if [[ "${{ github.event_name }}" == 'workflow_dispatch' ]]; then
echo "ok=true" >> $GITHUB_OUTPUT
echo "Running on request"
# using single brackets here to avoid 08 being interpreted as octal
# https://github.com/carpentries/sandpaper/issues/250
elif [ `date +%d` -le 7 ]; then
# If the Tuesday lands in the first week of the month, run it
echo "ok=true" >> $GITHUB_OUTPUT
echo "Running on schedule"
else
echo "ok=false" >> $GITHUB_OUTPUT
echo "Not Running Today"
fi
shell: bash
check-renv:
name: "Check If We Need {renv}"
runs-on: ubuntu-latest
needs: preflight
if: ${{ needs.preflight.outputs.ok == 'true' }}
outputs:
renv-needed: ${{ steps.renv-check.outputs.renv-needed }}
steps:
- name: "Checkout Lesson"
uses: actions/checkout@v6
- name: "Is renv required?"
id: renv-check
uses: carpentries/actions/renv-checks@main
with:
CACHE_VERSION: ${{ inputs.CACHE_VERSION || '' }}
skip-cache-check: true
update_cache:
name: "Create Package Update Pull Request"
runs-on: ubuntu-22.04
needs: check-renv
permissions:
contents: write
pull-requests: write
actions: write
issues: write
id-token: write
if: needs.check-renv.outputs.renv-needed == 'true'
env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RENV_PATHS_ROOT: ~/.local/share/renv/
steps:
- name: "Checkout Lesson"
uses: actions/checkout@v6
- name: "Set up R"
uses: r-lib/actions/setup-r@v2
with:
use-public-rspm: true
install-r: false
- name: "Update {renv} deps and determine if a PR is needed"
id: update
uses: carpentries/actions/update-lockfile@main
with:
update: ${{ env.UPDATE_PACKAGES }}
force-renv-init: ${{ env.FORCE_RENV_INIT }}
generate-cache: ${{ env.LOCKFILE_CACHE_GEN }}
cache-version: ${{ secrets.CACHE_VERSION }}
- name: "Validate Current Org and Workflow"
id: validate-org-workflow
uses: carpentries/actions/validate-org-workflow@main
with:
repo: ${{ github.repository }}
workflow: ${{ github.workflow }}
- name: "Configure AWS credentials via OIDC"
env:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
if: |
steps.validate-org-workflow.outputs.is_valid == 'true' &&
env.role-to-assume != '' &&
env.aws-region != ''
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ env.role-to-assume }}
aws-region: ${{ env.aws-region }}
- name: "Set PAT from AWS Secrets Manager"
env:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
if: |
steps.validate-org-workflow.outputs.is_valid == 'true' &&
env.role-to-assume != '' &&
env.aws-region != ''
id: set-pat
run: |
SECRET=$(aws secretsmanager get-secret-value \
--secret-id carpentries-bot/github-pat \
--query SecretString --output text)
PAT=$(echo "$SECRET" | jq -r .[])
echo "::add-mask::$PAT"
echo "pat=$PAT" >> "$GITHUB_OUTPUT"
shell: bash
# Create the PR with the following roles in order of preference:
# - Carpentries Bot classic PAT fetched from AWS (will only work in official Carpentries repos)
# - repo-scoped SANDPAPER_WORKFLOW classic PAT (will work in all scenarios)
# - default GITHUB_TOKEN (will work suitably, but workflows need to be triggered)
- name: "Create Pull Request"
id: cpr
if: |
steps.update.outputs.n > 0
uses: carpentries/create-pull-request@main
with:
token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }}
delete-branch: true
branch: "update/packages"
commit-message: "[actions] update ${{ steps.update.outputs.n }} packages"
title: "Update ${{ steps.update.outputs.n }} packages"
body: |
:robot: This is an automated build
This will update ${{ steps.update.outputs.n }} packages in your lesson with the following versions:
```
${{ steps.update.outputs.report }}
```
:stopwatch: In a few minutes, a comment will appear that will show you how the output has changed based on these updates.
If you want to inspect these changes locally, you can use the following code to check out a new branch:
```bash
git fetch origin update/packages
git checkout update/packages
```
- Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }}
[1]: https://github.com/carpentries/create-pull-request/tree/main
labels: "type: package cache"
draft: false
- name: "Skip PR creation"
if: steps.update.outputs.n == 0
run: |
echo "No updates needed, skipping PR creation"
shell: bash
================================================
FILE: .github/workflows/update-workflows.yaml
================================================
name: "04 Maintain: Update Workflow Files"
description: "Update workflow files from the carpentries/sandpaper repository"
on:
schedule:
- cron: '0 0 * * 2'
workflow_dispatch:
inputs:
name:
description: 'Who triggered this build (enter github username to tag yourself)?'
required: true
default: 'weekly run'
version:
description: 'Workflows version number (e.g. 0.0.1), branch name (e.g. main), or "latest"'
required: false
default: 'latest'
clean:
description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)'
required: false
default: '.yaml'
jobs:
update_workflow:
name: "Update Workflow"
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
id-token: write
steps:
- name: "Checkout Repository"
uses: actions/checkout@v6
- name: "Validate Current Org and Workflow"
id: validate-org-workflow
uses: carpentries/actions/validate-org-workflow@main
with:
repo: ${{ github.repository }}
workflow: ${{ github.workflow }}
- name: Configure AWS credentials via OIDC
env:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
if: |
steps.validate-org-workflow.outputs.is_valid == 'true' &&
env.role-to-assume != '' &&
env.aws-region != ''
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ env.role-to-assume }}
aws-region: ${{ env.aws-region }}
- name: Set PAT from AWS Secrets Manager
id: set-pat
env:
role-to-assume: ${{ secrets.AWS_GH_OIDC_ARN }}
aws-region: ${{ secrets.AWS_GH_OIDC_REGION }}
if: |
steps.validate-org-workflow.outputs.is_valid == 'true' &&
env.role-to-assume != '' &&
env.aws-region != ''
run: |
SECRET=$(aws secretsmanager get-secret-value \
--secret-id carpentries-bot/github-pat \
--query SecretString --output text)
PAT=$(echo "$SECRET" | jq -r .[])
echo "::add-mask::$PAT"
echo "pat=$PAT" >> "$GITHUB_OUTPUT"
shell: bash
- name: "Validate token"
id: validate-token
uses: carpentries/actions/check-valid-credentials@main
with:
token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }}
- name: "No Token Found: Skipping Workflow Update"
if: ${{ steps.validate-token.outputs.wf == 'false' }}
run: |
echo "❗No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows."
echo "## ❌ Workflow Update Failed" >> $GITHUB_STEP_SUMMARY
echo "No valid SANDPAPER_WORKFLOW token or PAT from AWS found, cannot update workflows." >> $GITHUB_STEP_SUMMARY
shell: bash
- name: Update Workflows
id: update
if: ${{ steps.validate-token.outputs.wf == 'true' }}
uses: carpentries/actions/update-workflows@main
with:
version: ${{ github.event.inputs.version || 'latest' }}
clean: ${{ github.event.inputs.clean || '.yaml' }}
- name: Create Pull Request
id: cpr
if: |
steps.update.outputs.new &&
steps.validate-token.outputs.wf == 'true'
uses: carpentries/create-pull-request@main
with:
token: ${{ steps.set-pat.outputs.pat || secrets.SANDPAPER_WORKFLOW }}
delete-branch: true
branch: "update/workflows"
commit-message: "[actions] update sandpaper workflow to version ${{ steps.update.outputs.new }}"
title: "Update Workflows to Version ${{ steps.update.outputs.new }}"
body: |
:robot: This is an automated build
Update Workflows from sandpaper version ${{ steps.update.outputs.old }} -> ${{ steps.update.outputs.new }}
- Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }}
[1]: https://github.com/carpentries/create-pull-request/tree/main
labels: "type: workflows"
draft: false
================================================
FILE: .github/workflows/workflows-version.txt
================================================
1.0.1
================================================
FILE: .gitignore
================================================
# sandpaper files
episodes/*html
site/*
!site/README.md
# History files
.Rhistory
.Rapp.history
# Session Data files
.RData
# User-specific files
.Ruserdata
# Example code in package build process
*-Ex.R
# Output files from R CMD build
/*.tar.gz
# Output files from R CMD check
/*.Rcheck/
# RStudio files
.Rproj.user/
# produced vignettes
vignettes/*.html
vignettes/*.pdf
# OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3
.httr-oauth
# knitr and R markdown default cache directories
*_cache/
/cache/
# Temporary files created by R markdown
*.utf8.md
*.knit.md
# R Environment Variables
.Renviron
# pkgdown site
docs/
# translation temp files
po/*~
# renv detritus
renv/sandbox/
*.pyc
*~
.DS_Store
.ipynb_checkpoints
.sass-cache
.jekyll-cache/
.jekyll-metadata
__pycache__
_site
.Rproj.user
.bundle/
.vendor/
vendor/
.docker-vendor/
Gemfile.lock
.*history
================================================
FILE: .mailmap
================================================
Abigail Cabunoc Mayes <abigail.cabunoc@gmail.com>
Abigail Cabunoc Mayes <abigail.cabunoc@gmail.com> <abigail.cabunoc@oicr.on.ca>
Deborah Digges <deborah.gertrude.digges@gmail.com>
Erin Becker <erinstellabecker@gmail.com>
Evan P. Williamson <evanpeterw@gmail.com>
François Michonneau <francois.michonneau@gmail.com>
Greg Wilson <gvwilson@software-carpentry.org> <gvwilson@third-bit.com>
James Allen <james@sharelatex.com> <jamesallen0108@gmail.com>
Jason Sherman <jsherman@ou.edu> <jsnshrmn@users.noreply.github.com>
Lex Nederbragt <lex.nederbragt@bio.uio.no> <lex.nederbragt@ibv.uio.no>
Maxim Belkin <maxim.belkin@gmail.com> <maxim-belkin@users.noreply.github.com>
Mike Jackson <m.jackson@software.ac.uk> <michaelj@epcc.ed.ac.uk>
Pier-Luc St-Onge <plstonge@users.noreply.github.com>
Raniere Silva <raniere@rgaiacs.com> <ra092767@ime.unicamp.br>
Raniere Silva <raniere@rgaiacs.com> <raniere@ime.unicamp.br>
Rémi Emonet <remi@heeere.com> <remi.emonet@reverse--com.heeere>
Rémi Emonet <remi@heeere.com> <twitwi@users.noreply.github.com>
Timothée Poisot <t.poisot@gmail.com> <tim@poisotlab.io>
================================================
FILE: .update-copyright.conf
================================================
[project]
vcs: Git
[files]
authors: yes
files: no
================================================
FILE: .zenodo.json
================================================
{
"contributors": [
{
"type": "Editor",
"name": "Gerard Capes"
}
],
"creators": [
{
"name": "Gerard Capes"
},
{
"name": "Matthias Rüster"
},
{
"name": "Maciej Cytowski"
},
{
"name": "Manuel Haussmann"
},
{
"name": "Nicolas Quiniou-Briand"
},
{
"name": "Tomas Stary"
}
],
"license": {
"id": "CC-BY-4.0"
}
}
================================================
FILE: AUTHORS
================================================
Pete Bachant
Maxime Boissonneault
Gerard Capes
Deborah Digges
Andrew Fraser
Luiz Irber
Mike Jackson
Gang Liu
Lex Nederbragt
Adam Richie-Halford
Jason Sherman
Raniere Silva
Byron Smith
Pier-Luc St-Onge
Andy Teucher
Greg Wilson
David E. Bernholdt
Juan Fung
Radovan Bast
================================================
FILE: CITATION
================================================
Please cite as:
Mike Jackson (ed.): "Software Carpentry: Automation and Make."
Version 2016.06, June 2016,
https://github.com/swcarpentry/make-novice, 10.5281/zenodo.57473.
================================================
FILE: CODE_OF_CONDUCT.md
================================================
---
title: "Contributor Code of Conduct"
---
As contributors and maintainers of this project,
we pledge to follow the [The Carpentries Code of Conduct][coc].
Instances of abusive, harassing, or otherwise unacceptable behavior
may be reported by following our [reporting guidelines][coc-reporting].
[coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html
[coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html
================================================
FILE: CONTRIBUTING.md
================================================
## Contributing
[The Carpentries][cp-site] ([Software Carpentry][swc-site], [Data
Carpentry][dc-site], and [Library Carpentry][lc-site]) are open source
projects, and we welcome contributions of all kinds: new lessons, fixes to
existing material, bug reports, and reviews of proposed changes are all
welcome.
### Contributor Agreement
By contributing, you agree that we may redistribute your work under [our
license](LICENSE.md). In exchange, we will address your issues and/or assess
your change proposal as promptly as we can, and help you become a member of our
community. Everyone involved in [The Carpentries][cp-site] agrees to abide by
our [code of conduct](CODE_OF_CONDUCT.md).
### How to Contribute
The easiest way to get started is to file an issue to tell us about a spelling
mistake, some awkward wording, or a factual error. This is a good way to
introduce yourself and to meet some of our community members.
1. If you do not have a [GitHub][github] account, you can [send us comments by
email][contact]. However, we will be able to respond more quickly if you use
one of the other methods described below.
2. If you have a [GitHub][github] account, or are willing to [create
one][github-join], but do not know how to use Git, you can report problems
or suggest improvements by [creating an issue][repo-issues]. This allows us
to assign the item to someone and to respond to it in a threaded discussion.
3. If you are comfortable with Git, and would like to add or change material,
you can submit a pull request (PR). Instructions for doing this are
[included below](#using-github). For inspiration about changes that need to
be made, check out the [list of open issues][issues] across the Carpentries.
Note: if you want to build the website locally, please refer to [The Workbench
documentation][template-doc].
### Where to Contribute
1. If you wish to change this lesson, add issues and pull requests here.
2. If you wish to change the template used for workshop websites, please refer
to [The Workbench documentation][template-doc].
### What to Contribute
There are many ways to contribute, from writing new exercises and improving
existing ones to updating or filling in the documentation and submitting [bug
reports][issues] about things that do not work, are not clear, or are missing.
If you are looking for ideas, please see [the list of issues for this
repository][repo-issues], or the issues for [Data Carpentry][dc-issues],
[Library Carpentry][lc-issues], and [Software Carpentry][swc-issues] projects.
Comments on issues and reviews of pull requests are just as welcome: we are
smarter together than we are on our own. **Reviews from novices and newcomers
are particularly valuable**: it's easy for people who have been using these
lessons for a while to forget how impenetrable some of this material can be, so
fresh eyes are always welcome.
### What *Not* to Contribute
Our lessons already contain more material than we can cover in a typical
workshop, so we are usually *not* looking for more concepts or tools to add to
them. As a rule, if you want to introduce a new idea, you must (a) estimate how
long it will take to teach and (b) explain what you would take out to make room
for it. The first encourages contributors to be honest about requirements; the
second, to think hard about priorities.
We are also not looking for exercises or other material that only run on one
platform. Our workshops typically contain a mixture of Windows, macOS, and
Linux users; in order to be usable, our lessons must run equally well on all
three.
### Using GitHub
If you choose to contribute via GitHub, you may want to look at [How to
Contribute to an Open Source Project on GitHub][how-contribute]. In brief, we
use [GitHub flow][github-flow] to manage changes:
1. Create a new branch in your desktop copy of this repository for each
significant change.
2. Commit the change in that branch.
3. Push that branch to your fork of this repository on GitHub.
4. Submit a pull request from that branch to the [upstream repository][repo].
5. If you receive feedback, make changes on your desktop and push to your
branch on GitHub: the pull request will update automatically.
NB: The published copy of the lesson is usually in the `main` branch.
Each lesson has a team of maintainers who review issues and pull requests or
encourage others to do so. The maintainers are community volunteers, and have
final say over what gets merged into the lesson.
### Other Resources
The Carpentries is a global organisation with volunteers and learners all over
the world. We share values of inclusivity and a passion for sharing knowledge,
teaching and learning. There are several ways to connect with The Carpentries
community listed at <https://carpentries.org/connect/> including via social
media, slack, newsletters, and email lists. You can also [reach us by
email][contact].
[repo]: https://example.com/FIXME
[repo-issues]: https://example.com/FIXME/issues
[contact]: mailto:team@carpentries.org
[cp-site]: https://carpentries.org/
[dc-issues]: https://github.com/issues?q=user%3Adatacarpentry
[dc-lessons]: https://datacarpentry.org/lessons/
[dc-site]: https://datacarpentry.org/
[discuss-list]: https://lists.software-carpentry.org/listinfo/discuss
[github]: https://github.com
[github-flow]: https://guides.github.com/introduction/flow/
[github-join]: https://github.com/join
[how-contribute]: https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github
[issues]: https://carpentries.org/help-wanted-issues/
[lc-issues]: https://github.com/issues?q=user%3ALibraryCarpentry
[swc-issues]: https://github.com/issues?q=user%3Aswcarpentry
[swc-lessons]: https://software-carpentry.org/lessons/
[swc-site]: https://software-carpentry.org/
[lc-site]: https://librarycarpentry.org/
[template-doc]: https://carpentries.github.io/workbench/
================================================
FILE: LICENSE.md
================================================
---
title: "Licenses"
---
## Instructional Material
All Carpentries (Software Carpentry, Data Carpentry, and Library Carpentry)
instructional material is made available under the [Creative Commons
Attribution license][cc-by-human]. The following is a human-readable summary of
(and not a substitute for) the [full legal text of the CC BY 4.0
license][cc-by-legal].
You are free:
- to **Share**---copy and redistribute the material in any medium or format
- to **Adapt**---remix, transform, and build upon the material
for any purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license
terms.
Under the following terms:
- **Attribution**---You must give appropriate credit (mentioning that your work
is derived from work that is Copyright (c) The Carpentries and, where
practical, linking to <https://carpentries.org/>), provide a [link to the
license][cc-by-human], and indicate if changes were made. You may do so in
any reasonable manner, but not in any way that suggests the licensor endorses
you or your use.
- **No additional restrictions**---You may not apply legal terms or
technological measures that legally restrict others from doing anything the
license permits. With the understanding that:
Notices:
* You do not have to comply with the license for elements of the material in
the public domain or where your use is permitted by an applicable exception
or limitation.
* No warranties are given. The license may not give you all of the permissions
necessary for your intended use. For example, other rights such as publicity,
privacy, or moral rights may limit how you use the material.
## Software
Except where otherwise noted, the example programs and other software provided
by The Carpentries are made available under the [OSI][osi]-approved [MIT
license][mit-license].
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
## Trademark
"The Carpentries", "Software Carpentry", "Data Carpentry", and "Library
Carpentry" and their respective logos are registered trademarks of
[The Carpentries, Inc.][carpentries].
[cc-by-human]: https://creativecommons.org/licenses/by/4.0/
[cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode
[mit-license]: https://opensource.org/licenses/mit-license.html
[carpentries]: https://carpentries.org
[osi]: https://opensource.org
================================================
FILE: README.md
================================================
[](https://doi.org/10.5281/zenodo.3265286)
[](https://slack-invite.carpentries.org/)
[](https://carpentries.slack.com/messages/C9X2YCPT5)
# make-novice
An introduction to Make using reproducible papers as a motivating example.
Please see [https://swcarpentry.github.io/make-novice/](https://swcarpentry.github.io/make-novice/) for a rendered version
of this material, [the lesson template documentation][lesson-example]
for instructions on formatting, building, and submitting material,
or run `make` in this directory for a list of helpful commands.
Maintainer(s):
- [Gerard Capes][capes-gerard]
[lesson-example]: https://swcarpentry.github.com/lesson-example/
[capes-gerard]: https://carpentries.org/instructors/#gcapes
================================================
FILE: commands.mk
================================================
## ----------------------------------------
MAKE2PNG = make2graph | grep -v Makefile | grep -v config.mk | grep -v commands.mk | dot -Tpng -o
FIGS = 02-makefile 02-makefile-challenge 04-dependencies 07-functions 09-conclusion-challenge-1
PNGS = $(patsubst %,fig/%.png,$(FIGS))
.PHONY: $(PNGS)
## diagrams : rebuild diagrams of Makefiles.
diagrams: $(PNGS)
fig/02-makefile.png: build
cp code/02-makefile/* $<
cd build && make -Bnd dats | $(MAKE2PNG) "$(CURDIR)/$@"
fig/02-makefile-challenge.png: build
cp code/02-makefile-challenge/* $<
cd build && make dats && make -Bnd results.txt | $(MAKE2PNG) "$(CURDIR)/$@"
fig/04-dependencies.png: build
cp code/04-dependencies/* $<
cd build && make dats && make -Bnd results.txt | $(MAKE2PNG) "$(CURDIR)/$@"
fig/07-functions.png: build
cp code/07-functions/* $<
cd build && make -Bnd results.txt | $(MAKE2PNG) "$(CURDIR)/$@"
fig/09-conclusion-challenge-1.png: build
cp code/09-conclusion-challenge-1/* $<
cd build && make -Bnd | $(MAKE2PNG) "$(CURDIR)/$@"
build:
mkdir -p $@
cp code/*.py $@
cp -r data/books $@
================================================
FILE: config.yaml
================================================
#------------------------------------------------------------
# Values for this lesson.
#------------------------------------------------------------
# Which carpentry is this (swc, dc, lc, or cp)?
# swc: Software Carpentry
# dc: Data Carpentry
# lc: Library Carpentry
# cp: Carpentries (to use for instructor training for instance)
# incubator: The Carpentries Incubator
carpentry: 'swc'
# Overall title for pages.
title: 'Automation and Make'
# Date the lesson was created (YYYY-MM-DD, this is empty by default)
created: '2015-06-18'
# Comma-separated list of keywords for the lesson
keywords: 'software, data, lesson, The Carpentries'
# Life cycle stage of the lesson
# possible values: pre-alpha, alpha, beta, stable
life_cycle: 'stable'
# License of the lesson materials (recommended CC-BY 4.0)
license: 'CC-BY 4.0'
# Link to the source repository for this lesson
source: 'https://github.com/swcarpentry/make-novice'
# Default branch of your lesson
branch: 'main'
# Who to contact if there are any issues
contact: 'team@carpentries.org'
# Navigation ------------------------------------------------
#
# Use the following menu items to specify the order of
# individual pages in each dropdown section. Leave blank to
# include all pages in the folder.
#
# Example -------------
#
# episodes:
# - introduction.md
# - first-steps.md
#
# learners:
# - setup.md
#
# instructors:
# - instructor-notes.md
#
# profiles:
# - one-learner.md
# - another-learner.md
# Order of episodes in your lesson
episodes:
- 01-intro.md
- 02-makefiles.md
- 03-variables.md
- 04-dependencies.md
- 05-patterns.md
- 06-variables.md
- 07-functions.md
- 08-self-doc.md
- 09-conclusion.md
# Information for Learners
learners:
# Information for Instructors
instructors:
# Learner Profiles
profiles:
# Customisation ---------------------------------------------
#
# This space below is where custom yaml items (e.g. pinning
# sandpaper and varnish versions) should live
url: 'https://swcarpentry.github.io/make-novice'
analytics: carpentries
lang: en
================================================
FILE: episodes/01-intro.md
================================================
---
title: Introduction
teaching: 25
exercises: 0
---
::::::::::::::::::::::::::::::::::::::: objectives
- Explain what Make is for.
- Explain why Make differs from shell scripts.
- Name other popular build tools.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How can I make my results easier to reproduce?
::::::::::::::::::::::::::::::::::::::::::::::::::
Let's imagine that we're interested in
testing Zipf's Law in some of our favorite books.
::::::::::::::::::::::::::::::::::::::::: callout
## Zipf's Law
The most frequently-occurring word occurs approximately twice as
often as the second most frequent word. This is [Zipf's Law][zipfs-law].
::::::::::::::::::::::::::::::::::::::::::::::::::
We've compiled our raw data i.e. the books we want to analyze
and have prepared several Python scripts that together make up our
analysis pipeline.
Let's take quick look at one of the books using the command
`head books/isles.txt`.
Our directory has the Python scripts and data files we will be working with:
```output
|- books
| |- abyss.txt
| |- isles.txt
| |- last.txt
| |- LICENSE_TEXTS.md
| |- sierra.txt
|- plotcounts.py
|- countwords.py
|- testzipf.py
```
The first step is to count the frequency of each word in a book.
For this purpose we will use a python script `countwords.py` which takes two command line arguments.
The first argument is the input file (`books/isles.txt`) and the second is the output file that is generated (here `isles.dat`) by processing the input.
```bash
$ python countwords.py books/isles.txt isles.dat
```
Let's take a quick peek at the result.
```bash
$ head -5 isles.dat
```
This shows us the top 5 lines in the output file:
```output
the 3822 6.7371760973
of 2460 4.33632998414
and 1723 3.03719372466
to 1479 2.60708619778
a 1308 2.30565838181
```
We can see that the file consists of one row per word.
Each row shows the word itself, the number of occurrences of that
word, and the number of occurrences as a percentage of the total
number of words in the text file.
We can do the same thing for a different book:
```bash
$ python countwords.py books/abyss.txt abyss.dat
$ head -5 abyss.dat
```
```output
the 4044 6.35449402891
and 2807 4.41074795726
of 1907 2.99654305468
a 1594 2.50471401634
to 1515 2.38057825267
```
Let's visualize the results.
The script `plotcounts.py` reads in a data file and plots the 10 most
frequently occurring words as a text-based bar plot:
```bash
$ python plotcounts.py isles.dat ascii
```
```output
the ########################################################################
of ##############################################
and ################################
to ############################
a #########################
in ###################
is #################
that ############
by ###########
it ###########
```
`plotcounts.py` can also show the plot graphically:
```bash
$ python plotcounts.py isles.dat show
```
Close the window to exit the plot.
`plotcounts.py` can also create the plot as an image file (e.g. a PNG file):
```bash
$ python plotcounts.py isles.dat isles.png
```
Finally, let's test Zipf's law for these books:
```bash
$ python testzipf.py abyss.dat isles.dat
```
```output
Book First Second Ratio
abyss 4044 2807 1.44
isles 3822 2460 1.55
```
So we're not too far off from Zipf's law.
Together these scripts implement a common workflow:
1. Read a data file.
2. Perform an analysis on this data file.
3. Write the analysis results to a new file.
4. Plot a graph of the analysis results.
5. Save the graph as an image, so we can put it in a paper.
6. Make a summary table of the analyses
Running `countwords.py` and `plotcounts.py` at the shell prompt, as we
have been doing, is fine for one or two files. If, however, we had 5
or 10 or 20 text files,
or if the number of steps in the pipeline were to expand, this could turn into
a lot of work.
Plus, no one wants to sit and wait for a command to finish, even just for 30
seconds.
The most common solution to the tedium of data processing is to write
a shell script that runs the whole pipeline from start to finish.
So to reproduce the tasks that we have just done we create a new file
named `run_pipeline.sh` in which we place the commands one by one.
Using a text editor of your choice (e.g. for nano use the command `nano run_pipeline.sh`) copy and paste the following text and save it.
```bash
# USAGE: bash run_pipeline.sh
# to produce plots for isles and abyss
# and the summary table for the Zipf's law tests
python countwords.py books/isles.txt isles.dat
python countwords.py books/abyss.txt abyss.dat
python plotcounts.py isles.dat isles.png
python plotcounts.py abyss.dat abyss.png
# Generate summary table
python testzipf.py abyss.dat isles.dat > results.txt
```
Run the script and check that the output is the same as before:
```bash
$ bash run_pipeline.sh
$ cat results.txt
```
This shell script solves several problems in computational reproducibility:
1. It explicitly documents our pipeline,
making communication with colleagues (and our future selves) more efficient.
2. It allows us to type a single command, `bash run_pipeline.sh`, to
reproduce the full analysis.
3. It prevents us from *repeating* typos or mistakes.
You might not get it right the first time, but once you fix something
it'll stay fixed.
Despite these benefits it has a few shortcomings.
Let's adjust the width of the bars in our plot produced by `plotcounts.py`.
Edit `plotcounts.py` so that the bars are 0.8 units wide instead of 1 unit.
(Hint: replace `width = 1.0` with `width = 0.8` in the definition of
`plot_word_counts`.)
Now we want to recreate our figures.
We *could* just `bash run_pipeline.sh` again.
That would work, but it could also be a big pain if counting words takes
more than a few seconds.
The word counting routine hasn't changed; we shouldn't need to recreate
those files.
Alternatively, we could manually rerun the plotting for each word-count file.
(Experienced shell scripters can make this easier on themselves using a
for-loop.)
```bash
for book in abyss isles; do
python plotcounts.py $book.dat $book.png
done
```
With this approach, however,
we don't get many of the benefits of having a shell script in the first place.
Another popular option is to comment out a subset of the lines in
`run_pipeline.sh`:
```bash
# USAGE: bash run_pipeline.sh
# to produce plots for isles and abyss
# and the summary table for the Zipf's law tests.
# These lines are commented out because they don't need to be rerun.
#python countwords.py books/isles.txt isles.dat
#python countwords.py books/abyss.txt abyss.dat
python plotcounts.py isles.dat isles.png
python plotcounts.py abyss.dat abyss.png
# Generate summary table
# This line is also commented out because it doesn't need to be rerun.
#python testzipf.py abyss.dat isles.dat > results.txt
```
Then, we would run our modified shell script using `bash run_pipeline.sh`.
But commenting out these lines, and subsequently uncommenting them,
can be a hassle and source of errors in complicated pipelines.
What we really want is an executable *description* of our pipeline that
allows software to do the tricky part for us:
figuring out what steps need to be rerun.
For our pipeline Make can execute the commands needed to run our
analysis and plot our results. Like shell scripts it allows us to
execute complex sequences of commands via a single shell
command. Unlike shell scripts it explicitly records the dependencies
between files - what files are needed to create what other files -
and so can determine when to recreate our data files or
image files, if our text files change. Make can be used for any
commands that follow the general pattern of processing files to create
new files, for example:
- Run analysis scripts on raw data files to get data files that
summarize the raw data (e.g. creating files with word counts from book text).
- Run visualization scripts on data files to produce plots
(e.g. creating images of word counts).
- Parse and combine text files and plots to create papers.
- Compile source code into executable programs or libraries.
There are now many build tools available, for example [Apache
ANT][apache-ant], [doit], and [nmake] for Windows.
Which is best for you depends on your requirements,
intended usage, and operating system. However, they all share the same
fundamental concepts as Make.
Also, you might come across build generation scripts e.g. [GNU
Autoconf][autoconf] and [CMake][cmake]. Those tools do not run the
pipelines directly, but rather generate files for use with the build
tools.
::::::::::::::::::::::::::::::::::::::::: callout
## Why Use Make if it is Almost 40 Years Old?
Make development was started by Stuart Feldman in 1977 as a Bell
Labs summer intern. Since then it has been undergoing an active
development and several implementations are available. Since it
solves a common issue of workflow management, it remains in
widespread use even today.
Researchers working with legacy codes in C or FORTRAN, which are
very common in high-performance computing, will, very likely
encounter Make.
Researchers can use Make for implementing reproducible
research workflows, automating data analysis and visualisation
(using Python or R) and combining tables and plots with text to
produce reports and papers for publication.
Make's fundamental concepts are common across build tools.
::::::::::::::::::::::::::::::::::::::::::::::::::
[GNU Make][gnu-make] is a free-libre, fast, [well-documented][gnu-make-documentation],
and very popular Make implementation. From now on, we will focus on it, and when we say
Make, we mean GNU Make.
[zipfs-law]: https://en.wikipedia.org/wiki/Zipf%27s_law
[apache-ant]: https://ant.apache.org/
[doit]: https://pydoit.org/
[nmake]: https://docs.microsoft.com/en-us/cpp/build/reference/nmake-reference
[autoconf]: https://www.gnu.org/software/autoconf/autoconf.html
[cmake]: https://www.cmake.org/
[gnu-make]: https://www.gnu.org/software/make/
[gnu-make-documentation]: https://www.gnu.org/software/make/#documentation
:::::::::::::::::::::::::::::::::::::::: keypoints
- Make allows us to specify what depends on what and how to update things that are out of date.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/02-makefiles.md
================================================
---
title: Makefiles
teaching: 30
exercises: 10
---
::::::::::::::::::::::::::::::::::::::: objectives
- Recognize the key parts of a Makefile, rules, targets, dependencies and actions.
- Write a simple Makefile.
- Run Make from the shell.
- Explain when and why to mark targets as `.PHONY`.
- Explain constraints on dependencies.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How do I write a simple Makefile?
::::::::::::::::::::::::::::::::::::::::::::::::::
Create a file, called `Makefile`, with the following content:
```make
# Count words.
isles.dat : books/isles.txt
python countwords.py books/isles.txt isles.dat
```
This is a [build file](../learners/reference.md#build-file), which for
Make is called a [Makefile](../learners/reference.md#makefile) - a file
executed by Make. Note how it resembles one of the lines from our shell script.
Let us go through each line in turn:
- `#` denotes a *comment*. Any text from `#` to the end of the line is
ignored by Make but could be very helpful for anyone reading your Makefile.
- `isles.dat` is a [target](../learners/reference.md#target), a file to be
created, or built.
- `books/isles.txt` is a [dependency](../learners/reference.md#dependency), a
file that is needed to build or update the target. Targets can have
zero or more dependencies.
- A colon, `:`, separates targets from dependencies.
- `python countwords.py books/isles.txt isles.dat` is an
[action](../learners/reference.md#action), a command to run to build or
update the target using the dependencies. Targets can have zero or more
actions. These actions form a recipe to build the target
from its dependencies and are executed similarly to a shell script.
- Actions are indented using a single TAB character, *not* 8 spaces. This
is a legacy of Make's 1970's origins. If the difference between
spaces and a TAB character isn't obvious in your editor, try moving
your cursor from one side of the TAB to the other. It should jump
four or more spaces.
- Together, the target, dependencies, and actions form a
[rule](../learners/reference.md#rule).
Our rule above describes how to build the target `isles.dat` using the
action `python countwords.py` and the dependency `books/isles.txt`.
Information that was implicit in our shell script - that we are
generating a file called `isles.dat` and that creating this file
requires `books/isles.txt` - is now made explicit by Make's syntax.
Let's first ensure we start from scratch and delete the `.dat` and `.png`
files we created earlier:
```bash
$ rm *.dat *.png
```
By default, Make looks for a Makefile, called `Makefile`, and we can
run Make as follows:
```bash
$ make
```
By default, Make prints out the actions it executes:
```output
python countwords.py books/isles.txt isles.dat
```
If we see,
```error
Makefile:3: *** missing separator. Stop.
```
then we have used a space instead of a TAB characters to indent one of
our actions.
Let's see if we got what we expected.
```bash
head -5 isles.dat
```
The first 5 lines of `isles.dat` should look exactly like before.
::::::::::::::::::::::::::::::::::::::::: callout
## Makefiles Do Not Have to be Called `Makefile`
We don't have to call our Makefile `Makefile`. However, if we call it
something else we need to tell Make where to find it. This we can do
using `-f` flag. For example, if our Makefile is named `MyOtherMakefile`:
```bash
$ make -f MyOtherMakefile
```
Sometimes, the suffix `.mk` will be used to identify Makefiles that
are not called `Makefile` e.g. `install.mk`, `common.mk` etc.
::::::::::::::::::::::::::::::::::::::::::::::::::
When we re-run our Makefile, Make now informs us that:
```output
make: `isles.dat' is up to date.
```
This is because our target, `isles.dat`, has now been created, and
Make will not create it again. To see how this works, let's pretend to
update one of the text files. Rather than opening the file in an
editor, we can use the shell `touch` command to update its timestamp
(which would happen if we did edit the file):
```bash
$ touch books/isles.txt
```
If we compare the timestamps of `books/isles.txt` and `isles.dat`,
```bash
$ ls -l books/isles.txt isles.dat
```
then we see that `isles.dat`, the target, is now older
than `books/isles.txt`, its dependency:
```output
-rw-r--r-- 1 mjj Administ 323972 Jun 12 10:35 books/isles.txt
-rw-r--r-- 1 mjj Administ 182273 Jun 12 09:58 isles.dat
```
If we run Make again,
```bash
$ make
```
then it recreates `isles.dat`:
```output
python countwords.py books/isles.txt isles.dat
```
When it is asked to build a target, Make checks the 'last modification
time' of both the target and its dependencies. If any dependency has
been updated since the target, then the actions are re-run to update
the target. Using this approach, Make knows to only rebuild the files
that, either directly or indirectly, depend on the file that
changed. This is called an [incremental
build](../learners/reference.md#incremental-build).
::::::::::::::::::::::::::::::::::::::::: callout
## Makefiles as Documentation
By explicitly recording the inputs to and outputs from steps in our
analysis and the dependencies between files, Makefiles act as a type
of documentation, reducing the number of things we have to remember.
::::::::::::::::::::::::::::::::::::::::::::::::::
Let's add another rule to the end of `Makefile`:
```make
abyss.dat : books/abyss.txt
python countwords.py books/abyss.txt abyss.dat
```
If we run Make,
```bash
$ make
```
then we get:
```output
make: `isles.dat' is up to date.
```
Nothing happens because Make attempts to build the first target it
finds in the Makefile, the
[default target](../learners/reference.md#default-target), which is
`isles.dat` which is already up-to-date. We need to explicitly tell Make we want
to build `abyss.dat`:
```bash
$ make abyss.dat
```
Now, we get:
```output
python countwords.py books/abyss.txt abyss.dat
```
::::::::::::::::::::::::::::::::::::::::: callout
## "Up to Date" Versus "Nothing to be Done"
If we ask Make to build a file that already exists and is up to
date, then Make informs us that:
```output
make: `isles.dat' is up to date.
```
If we ask Make to build a file that exists but for which there is
no rule in our Makefile, then we get message like:
```bash
$ make countwords.py
```
```output
make: Nothing to be done for `countwords.py'.
```
`up to date` means that the Makefile has a rule with one or more actions
whose target is the name of a file (or directory) and the file is up to date.
`Nothing to be done` means that
the file exists but either :
- the Makefile has no rule for it, or
- the Makefile has a rule for it, but that rule has no actions
::::::::::::::::::::::::::::::::::::::::::::::::::
We may want to remove all our data files so we can explicitly recreate
them all. We can introduce a new target, and associated rule, to do
this. We will call it `clean`, as this is a common name for rules that
delete auto-generated files, like our `.dat` files:
```make
clean :
rm -f *.dat
```
This is an example of a rule that has no dependencies. `clean` has no
dependencies on any `.dat` file as it makes no sense to create these
just to remove them. We just want to remove the data files whether or
not they exist. If we run Make and specify this target,
```bash
$ make clean
```
then we get:
```output
rm -f *.dat
```
There is no actual thing built called `clean`. Rather, it is a
short-hand that we can use to execute a useful sequence of
actions. Such targets, though very useful, can lead to problems. For
example, let us recreate our data files, create a directory called
`clean`, then run Make:
```bash
$ make isles.dat abyss.dat
$ mkdir clean
$ make clean
```
We get:
```output
make: `clean' is up to date.
```
Make finds a file (or directory) called `clean` and, as its `clean`
rule has no dependencies, assumes that `clean` has been built and is
up-to-date and so does not execute the rule's actions. As we are using
`clean` as a short-hand, we need to tell Make to always execute this
rule if we run `make clean`, by telling Make that this is a
[phony target](../learners/reference.md#phony-target), that it does not build
anything. This we do by marking the target as `.PHONY`:
```make
.PHONY : clean
clean :
rm -f *.dat
```
If we run Make,
```bash
$ make clean
```
then we get:
```output
rm -f *.dat
```
We can add a similar command to create all the data files. We can put
this at the top of our Makefile so that it is the [default
target](../learners/reference.md#default-target), which is executed by default
if no target is given to the `make` command:
```make
.PHONY : dats
dats : isles.dat abyss.dat
```
This is an example of a rule that has dependencies that are targets of
other rules. When Make runs, it will check to see if the dependencies
exist and, if not, will see if rules are available that will create
these. If such rules exist it will invoke these first, otherwise
Make will raise an error.
::::::::::::::::::::::::::::::::::::::::: callout
## Dependencies
The order of rebuilding dependencies is arbitrary. You should not
assume that they will be built in the order in which they are
listed.
Dependencies must form a directed acyclic graph. A target cannot
depend on a dependency which itself, or one of its dependencies,
depends on that target.
::::::::::::::::::::::::::::::::::::::::::::::::::
This rule (`dats`) is also an example of a rule that has no actions. It is used
purely to trigger the build of its dependencies, if needed.
If we run,
```bash
$ make dats
```
then Make creates the data files:
```output
python countwords.py books/isles.txt isles.dat
python countwords.py books/abyss.txt abyss.dat
```
If we run `make dats` again, then Make will see that the dependencies (`isles.dat`
and `abyss.dat`) are already up to date.
Given the target `dats` has no actions, there is `nothing to be done`:
```bash
$ make dats
```
```output
make: Nothing to be done for `dats'.
```
Our Makefile now looks like this:
```make
# Count words.
.PHONY : dats
dats : isles.dat abyss.dat
isles.dat : books/isles.txt
python countwords.py books/isles.txt isles.dat
abyss.dat : books/abyss.txt
python countwords.py books/abyss.txt abyss.dat
.PHONY : clean
clean :
rm -f *.dat
```
The following figure shows a graph of the dependencies embodied within
our Makefile, involved in building the `dats` target:
{alt='Dependencies represented within the Makefile'}
::::::::::::::::::::::::::::::::::::::: challenge
## Write Two New Rules
1. Write a new rule for `last.dat`, created from `books/last.txt`.
2. Update the `dats` rule with this target.
3. Write a new rule for `results.txt`, which creates the summary
table. The rule needs to:
- Depend upon each of the three `.dat` files.
- Invoke the action `python testzipf.py abyss.dat isles.dat last.dat > results.txt`.
4. Put this rule at the top of the Makefile so that it is the default target.
5. Update `clean` so that it removes `results.txt`.
The starting Makefile is [here](files/code/02-makefile/Makefile).
::::::::::::::: solution
## Solution
See [this file](files/code/02-makefile-challenge/Makefile) for a solution.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
The following figure shows the dependencies embodied within our
Makefile, involved in building the `results.txt` target:
{alt='results.txt dependencies represented within the Makefile'}
:::::::::::::::::::::::::::::::::::::::: keypoints
- Use `#` for comments in Makefiles.
- Write rules as `target: dependencies`.
- Specify update actions in a tab-indented block under the rule.
- Use `.PHONY` to mark targets that don't correspond to files.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/03-variables.md
================================================
---
title: Automatic Variables
teaching: 10
exercises: 5
---
::::::::::::::::::::::::::::::::::::::: objectives
- Use Make automatic variables to remove duplication in a Makefile.
- Explain why shell wildcards in dependencies can cause problems.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How can I abbreviate the rules in my Makefiles?
::::::::::::::::::::::::::::::::::::::::::::::::::
After the exercise at the end of the previous episode, our Makefile looked like
this:
```make
# Generate summary table.
results.txt : isles.dat abyss.dat last.dat
python testzipf.py abyss.dat isles.dat last.dat > results.txt
# Count words.
.PHONY : dats
dats : isles.dat abyss.dat last.dat
isles.dat : books/isles.txt
python countwords.py books/isles.txt isles.dat
abyss.dat : books/abyss.txt
python countwords.py books/abyss.txt abyss.dat
last.dat : books/last.txt
python countwords.py books/last.txt last.dat
.PHONY : clean
clean :
rm -f *.dat
rm -f results.txt
```
Our Makefile has a lot of duplication. For example, the names of text
files and data files are repeated in many places throughout the
Makefile. Makefiles are a form of code and, in any code, repeated code
can lead to problems e.g. we rename a data file in one part of the
Makefile but forget to rename it elsewhere.
::::::::::::::::::::::::::::::::::::::::: callout
## D.R.Y. (Don't Repeat Yourself)
In many programming languages, the bulk of the language features are
there to allow the programmer to describe long-winded computational
routines as short, expressive, beautiful code. Features in Python
or R or Java, such as user-defined variables and functions are useful in
part because they mean we don't have to write out (or think about)
all of the details over and over again. This good habit of writing
things out only once is known as the "Don't Repeat Yourself"
principle or D.R.Y.
::::::::::::::::::::::::::::::::::::::::::::::::::
Let us set about removing some of the repetition from our Makefile.
In our `results.txt` rule we duplicate the data file names and the
name of the results file name:
```make
results.txt : isles.dat abyss.dat last.dat
python testzipf.py abyss.dat isles.dat last.dat > results.txt
```
Looking at the results file name first, we can replace it in the action
with `$@`:
```make
results.txt : isles.dat abyss.dat last.dat
python testzipf.py abyss.dat isles.dat last.dat > $@
```
`$@` is a Make
[automatic variable](../learners/reference.md#automatic-variable)
which means 'the target of the current rule'. When Make is run it will
replace this variable with the target name.
We can replace the dependencies in the action with `$^`:
```make
results.txt : isles.dat abyss.dat last.dat
python testzipf.py $^ > $@
```
`$^` is another automatic variable which means 'all the dependencies
of the current rule'. Again, when Make is run it will replace this
variable with the dependencies.
Let's update our text files and re-run our rule:
```bash
$ touch books/*.txt
$ make results.txt
```
We get:
```output
python countwords.py books/isles.txt isles.dat
python countwords.py books/abyss.txt abyss.dat
python countwords.py books/last.txt last.dat
python testzipf.py isles.dat abyss.dat last.dat > results.txt
```
::::::::::::::::::::::::::::::::::::::: challenge
## Update Dependencies
What will happen if you now execute:
```bash
$ touch *.dat
$ make results.txt
```
1. nothing
2. all files recreated
3. only `.dat` files recreated
4. only `results.txt` recreated
::::::::::::::: solution
## Solution
`4.` Only `results.txt` recreated.
The rules for `*.dat` are not executed because their corresponding `.txt` files
haven't been modified.
If you run:
```bash
$ touch books/*.txt
$ make results.txt
```
you will find that the `.dat` files as well as `results.txt` are recreated.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
As we saw, `$^` means 'all the dependencies of the current rule'. This
works well for `results.txt` as its action treats all the dependencies
the same - as the input for the `testzipf.py` script.
However, for some rules, we may want to treat the first dependency
differently. For example, our rules for `.dat` use their first (and
only) dependency specifically as the input file to `countwords.py`. If
we add additional dependencies (as we will soon do) then we don't want
these being passed as input files to `countwords.py` as it expects only
one input file to be named when it is invoked.
Make provides an automatic variable for this, `$<` which means 'the
first dependency of the current rule'.
::::::::::::::::::::::::::::::::::::::: challenge
## Rewrite `.dat` Rules to Use Automatic Variables
Rewrite each `.dat` rule to use the automatic variables `$@` ('the
target of the current rule') and `$<` ('the first dependency of the
current rule').
[This file](files/code/03-variables/Makefile) contains
the Makefile immediately before the challenge.
::::::::::::::: solution
## Solution
See [this file](files/code/03-variables-challenge/Makefile)
for a solution to this challenge.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: keypoints
- Use `$@` to refer to the target of the current rule.
- Use `$^` to refer to the dependencies of the current rule.
- Use `$<` to refer to the first dependency of the current rule.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/04-dependencies.md
================================================
---
title: Dependencies on Data and Code
teaching: 15
exercises: 5
---
::::::::::::::::::::::::::::::::::::::: objectives
- Output files are a product not only of input files but of the scripts or code that created the output files.
- Recognize and avoid false dependencies.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How can I write a Makefile to update things when my scripts have changed rather than my input files?
::::::::::::::::::::::::::::::::::::::::::::::::::
Our Makefile now looks like this:
```make
# Generate summary table.
results.txt : isles.dat abyss.dat last.dat
python testzipf.py $^ > $@
# Count words.
.PHONY : dats
dats : isles.dat abyss.dat last.dat
isles.dat : books/isles.txt
python countwords.py $< $@
abyss.dat : books/abyss.txt
python countwords.py $< $@
last.dat : books/last.txt
python countwords.py $< $@
.PHONY : clean
clean :
rm -f *.dat
rm -f results.txt
```
Our data files are produced using not only the input text files but also the
script `countwords.py` that processes the text files and creates the
data files. A change to `countwords.py` (e.g. adding a new column of
summary data or removing an existing one) results in changes to the
`.dat` files it outputs. So, let's pretend to edit `countwords.py`,
using `touch`, and re-run Make:
```bash
$ make dats
$ touch countwords.py
$ make dats
```
Nothing happens! Though we've updated `countwords.py` our data files
are not updated because our rules for creating `.dat` files don't
record any dependencies on `countwords.py`.
We need to add `countwords.py` as a dependency of each of our
data files also:
```make
isles.dat : books/isles.txt countwords.py
python countwords.py $< $@
abyss.dat : books/abyss.txt countwords.py
python countwords.py $< $@
last.dat : books/last.txt countwords.py
python countwords.py $< $@
```
If we pretend to edit `countwords.py` and re-run Make,
```bash
$ touch countwords.py
$ make dats
```
then we get:
```output
python countwords.py books/isles.txt isles.dat
python countwords.py books/abyss.txt abyss.dat
python countwords.py books/last.txt last.dat
```
::::::::::::::::::::::::::::::::::::::::: callout
## Dry run
`make` can show the commands it will execute without actually running them if we pass the `-n` flag:
```bash
$ touch countwords.py
$ make -n dats
```
This gives the same output to the screen as without the `-n` flag, but the commands are not actually run. Using this 'dry-run' mode is a good way to check that you have set up your Makefile properly before actually running the commands in it.
::::::::::::::::::::::::::::::::::::::::::::::::::
The following figure shows a graph of the dependencies, that are
involved in building the target `results.txt`. Notice the recently
added dependencies `countwords.py` and `testzipf.py`. This is how the
Makefile should look after completing the rest of the exercises
in this episode.
{alt='results.txt dependencies after adding countwords.py and testzipf.py as dependencies'}
::::::::::::::::::::::::::::::::::::::::: callout
## Why Don't the `.txt` Files Depend on `countwords.py`?
`.txt` files are input files and as such have no dependencies. To make these
depend on `countwords.py` would introduce a [false
dependency](../learners/reference.md#false-dependency) which is not desirable.
::::::::::::::::::::::::::::::::::::::::::::::::::
Intuitively, we should also add `countwords.py` as a dependency for
`results.txt`, because the final table should be rebuilt if we remake the
`.dat` files. However, it turns out we don't have to do that! Let's see what
happens to `results.txt` when we update `countwords.py`:
```bash
$ touch countwords.py
$ make results.txt
```
then we get:
```output
python countwords.py books/abyss.txt abyss.dat
python countwords.py books/isles.txt isles.dat
python countwords.py books/last.txt last.dat
python testzipf.py abyss.dat isles.dat last.dat > results.txt
```
The whole pipeline is triggered, even the creation of the
`results.txt` file! To understand this, note that according to the
dependency figure, `results.txt` depends on the `.dat` files. The
update of `countwords.py` triggers an update of the `*.dat`
files. Thus, `make` sees that the dependencies (the `.dat` files) are
newer than the target file (`results.txt`) and thus it recreates
`results.txt`. This is an example of the power of `make`: updating a
subset of the files in the pipeline triggers rerunning the appropriate
downstream steps.
::::::::::::::::::::::::::::::::::::::: challenge
## Updating One Input File
What will happen if you now execute:
```bash
$ touch books/last.txt
$ make results.txt
```
1. only `last.dat` is recreated
2. all `.dat` files are recreated
3. only `last.dat` and `results.txt` are recreated
4. all `.dat` and `results.txt` are recreated
::::::::::::::: solution
## Solution
`3.` only `last.dat` and `results.txt` are recreated.
Follow the dependency tree to understand the answer(s).
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::: challenge
## `testzipf.py` as a Dependency of `results.txt`.
What would happen if you added `testzipf.py` as dependency of `results.txt`, and why?
::::::::::::::: solution
## Solution
If you change the rule for the `results.txt` file like this:
```make
results.txt : isles.dat abyss.dat last.dat testzipf.py
python testzipf.py $^ > $@
```
`testzipf.py` becomes a part of `$^`, thus the command becomes
```bash
python testzipf.py abyss.dat isles.dat last.dat testzipf.py > results.txt
```
This results in an error from `testzipf.py` as it tries to parse the
script as if it were a `.dat` file. Try this by running:
```bash
$ make results.txt
```
You'll get
```error
python testzipf.py abyss.dat isles.dat last.dat testzipf.py > results.txt
Traceback (most recent call last):
File "testzipf.py", line 19, in <module>
counts = load_word_counts(input_file)
File "path/to/testzipf.py", line 39, in load_word_counts
counts.append((fields[0], int(fields[1]), float(fields[2])))
IndexError: list index out of range
make: *** [results.txt] Error 1
```
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
We still have to add the `testzipf.py` script as dependency to
`results.txt`.
Given the answer to the challenge above,
we need to make a couple of small changes so that we can still use automatic variables.
We'll move `testzipf.py` to be the first dependency and then edit the action
so that we pass all the dependencies as arguments to python using `$^`.
```make
results.txt : testzipf.py isles.dat abyss.dat last.dat
python $^ > $@
```
::::::::::::::::::::::::::::::::::::::::: callout
## Where We Are
[This Makefile](files/code/04-dependencies/Makefile)
contains everything done so far in this topic.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: keypoints
- Make results depend on processing scripts as well as data files.
- Dependencies are transitive: if A depends on B and B depends on C, a change to C will indirectly trigger an update to A.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/05-patterns.md
================================================
---
title: Pattern Rules
teaching: 10
exercises: 0
---
::::::::::::::::::::::::::::::::::::::: objectives
- Write Make pattern rules.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How can I define rules to operate on similar files?
::::::::::::::::::::::::::::::::::::::::::::::::::
Our Makefile still has repeated content. The rules for each `.dat`
file are identical apart from the text and data file names. We can
replace these rules with a single [pattern
rule](../learners/reference.md#pattern-rule) which can be used to build any
`.dat` file from a `.txt` file in `books/`:
```make
%.dat : countwords.py books/%.txt
python $^ $@
```
`%` is a Make [wildcard](../learners/reference.md#wildcard),
matching any number of any characters.
This rule can be interpreted as:
"In order to build a file named `[something].dat` (the target)
find a file named `books/[that same something].txt` (one of the dependencies)
and run `python [the dependencies] [the target]`."
If we re-run Make,
```bash
$ make clean
$ make dats
```
then we get:
```output
python countwords.py books/isles.txt isles.dat
python countwords.py books/abyss.txt abyss.dat
python countwords.py books/last.txt last.dat
```
Note that we can still use Make to build individual `.dat` targets as before,
and that our new rule will work no matter what stem is being matched.
```bash
$ make sierra.dat
```
which gives the output below:
```output
python countwords.py books/sierra.txt sierra.dat
```
::::::::::::::::::::::::::::::::::::::::: callout
## Using Make Wildcards
The Make `%` wildcard can only be used in a target and in its
dependencies. It cannot be used in actions. In actions, you may
however use `$*`, which will be replaced by the stem with which
the rule matched.
::::::::::::::::::::::::::::::::::::::::::::::::::
Our Makefile is now much shorter and cleaner:
```make
# Generate summary table.
results.txt : testzipf.py isles.dat abyss.dat last.dat
python $^ > $@
# Count words.
.PHONY : dats
dats : isles.dat abyss.dat last.dat
%.dat : countwords.py books/%.txt
python $^ $@
.PHONY : clean
clean :
rm -f *.dat
rm -f results.txt
```
::::::::::::::::::::::::::::::::::::::::: callout
## Where We Are
[This Makefile](files/code/05-patterns/Makefile)
contains all of our work so far.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: keypoints
- Use the wildcard `%` as a placeholder in targets and dependencies.
- Use the special variable `$*` to refer to matching sets of files in actions.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/06-variables.md
================================================
---
title: Variables
teaching: 15
exercises: 5
---
::::::::::::::::::::::::::::::::::::::: objectives
- Use variables in a Makefile.
- Explain the benefits of decoupling configuration from computation.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How can I eliminate redundancy in my Makefiles?
::::::::::::::::::::::::::::::::::::::::::::::::::
Despite our efforts, our Makefile still has repeated content, i.e.
the name of our script -- `countwords.py`, and the program we use to run it --
`python`. If we renamed our script we'd have to update our Makefile in multiple
places.
We can introduce a Make [variable](../learners/reference.md#variable) (called a
[macro](../learners/reference.md#macro) in some versions of Make) to hold our
script name:
```make
COUNT_SRC=countwords.py
```
This is a variable [assignment](../learners/reference.md#assignment) -
`COUNT_SRC` is assigned the value `countwords.py`.
We can do the same thing with the interpreter language used to run the script:
```make
LANGUAGE=python
```
`$(...)` tells Make to replace a variable with its value when Make
is run. This is a variable [reference](../learners/reference.md#reference). At
any place where we want to use the value of a variable we have to
write it, or reference it, in this way.
Here we reference the variables `LANGUAGE` and `COUNT_SRC`. This tells Make to
replace the variable `LANGUAGE` with its value `python`,
and to replace the variable `COUNT_SRC` with its value `countwords.py`.
Defining the variable `LANGUAGE` in this way avoids repeating `python` in our
Makefile, and allows us to easily
change how our script is run (e.g. we might want to use a different
version of Python and need to change `python` to `python2` -- or we might want
to rewrite the script using another language (e.g. switch from Python to R)).
::::::::::::::::::::::::::::::::::::::: challenge
## Use Variables
Update `Makefile` so that the `%.dat` rule
references the variable `COUNT_SRC`.
Then do the same for the `testzipf.py` script
and the `results.txt` rule,
using `ZIPF_SRC` as the variable name.
::::::::::::::: solution
## Solution
[This Makefile](files/code/06-variables-challenge/Makefile)
contains a solution to this challenge.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
We place variables at the top of a Makefile so they are easy to
find and modify. Alternatively, we can pull them out into a new
file that just holds variable definitions (i.e. delete them from
the original Makefile). Let us create `config.mk`:
```make
# Count words script.
LANGUAGE=python
COUNT_SRC=countwords.py
# Test Zipf's rule
ZIPF_SRC=testzipf.py
```
We can then import `config.mk` into `Makefile` using:
```make
include config.mk
```
We can re-run Make to see that everything still works:
```bash
$ make clean
$ make dats
$ make results.txt
```
We have separated the configuration of our Makefile from its rules --
the parts that do all the work. If we want to change our script name
or how it is executed we just need to edit our configuration file, not
our source code in `Makefile`. Decoupling code from configuration in
this way is good programming practice, as it promotes more modular,
flexible and reusable code.
::::::::::::::::::::::::::::::::::::::::: callout
## Where We Are
[This Makefile](files/code/06-variables/Makefile)
and [its accompanying `config.mk`](files/code/06-variables/config.mk)
contain all of our work so far.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: keypoints
- Define variables by assigning values to names.
- Reference variables using `$(...)`.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/07-functions.md
================================================
---
title: Functions
teaching: 20
exercises: 5
---
::::::::::::::::::::::::::::::::::::::: objectives
- Write Makefiles that use functions to match and transform sets of files.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How *else* can I eliminate redundancy in my Makefiles?
::::::::::::::::::::::::::::::::::::::::::::::::::
At this point, we have the following Makefile:
```make
include config.mk
# Generate summary table.
results.txt : $(ZIPF_SRC) isles.dat abyss.dat last.dat
$(LANGUAGE) $^ > $@
# Count words.
.PHONY : dats
dats : isles.dat abyss.dat last.dat
%.dat : $(COUNT_SRC) books/%.txt
$(LANGUAGE) $^ $@
.PHONY : clean
clean :
rm -f *.dat
rm -f results.txt
```
Make has many [functions](../learners/reference.md#function) which can be used
to write more complex rules. One example is `wildcard`. `wildcard` gets a
list of files matching some pattern, which we can then save in a
variable. So, for example, we can get a list of all our text files
(files ending in `.txt`) and save these in a variable by adding this at
the beginning of our makefile:
```make
TXT_FILES=$(wildcard books/*.txt)
```
We can add a `.PHONY` target and rule to show the variable's value:
```make
.PHONY : variables
variables:
@echo TXT_FILES: $(TXT_FILES)
```
::::::::::::::::::::::::::::::::::::::::: callout
## @echo
Make prints actions as it executes them. Using `@` at the start of
an action tells Make not to print this action. So, by using `@echo`
instead of `echo`, we can see the result of `echo` (the variable's
value being printed) but not the `echo` command itself.
::::::::::::::::::::::::::::::::::::::::::::::::::
If we run Make:
```bash
$ make variables
```
We get:
```output
TXT_FILES: books/abyss.txt books/isles.txt books/last.txt books/sierra.txt
```
Note how `sierra.txt` is now included too.
`patsubst` ('pattern substitution') takes a pattern, a replacement string and a
list of names in that order; each name in the list that matches the pattern is
replaced by the replacement string. Again, we can save the result in a
variable. So, for example, we can rewrite our list of text files into
a list of data files (files ending in `.dat`) and save these in a
variable:
```make
DAT_FILES=$(patsubst books/%.txt, %.dat, $(TXT_FILES))
```
We can extend `variables` to show the value of `DAT_FILES` too:
```make
.PHONY : variables
variables:
@echo TXT_FILES: $(TXT_FILES)
@echo DAT_FILES: $(DAT_FILES)
```
If we run Make,
```bash
$ make variables
```
then we get:
```output
TXT_FILES: books/abyss.txt books/isles.txt books/last.txt books/sierra.txt
DAT_FILES: abyss.dat isles.dat last.dat sierra.dat
```
Now, `sierra.txt` is processed too.
With these we can rewrite `clean` and `dats`:
```make
.PHONY : dats
dats : $(DAT_FILES)
.PHONY : clean
clean :
rm -f $(DAT_FILES)
rm -f results.txt
```
Let's check:
```bash
$ make clean
$ make dats
```
We get:
```output
python countwords.py books/abyss.txt abyss.dat
python countwords.py books/isles.txt isles.dat
python countwords.py books/last.txt last.dat
python countwords.py books/sierra.txt sierra.dat
```
We can also rewrite `results.txt`:
```make
results.txt : $(ZIPF_SRC) $(DAT_FILES)
$(LANGUAGE) $^ > $@
```
If we re-run Make:
```bash
$ make clean
$ make results.txt
```
We get:
```output
python countwords.py books/abyss.txt abyss.dat
python countwords.py books/isles.txt isles.dat
python countwords.py books/last.txt last.dat
python countwords.py books/sierra.txt sierra.dat
python testzipf.py last.dat isles.dat abyss.dat sierra.dat > results.txt
```
Let's check the `results.txt` file:
```bash
$ cat results.txt
```
```output
Book First Second Ratio
abyss 4044 2807 1.44
isles 3822 2460 1.55
last 12244 5566 2.20
sierra 4242 2469 1.72
```
So the range of the ratios of occurrences of the two most frequent
words in our books is indeed around 2, as predicted by Zipf's Law,
i.e., the most frequently-occurring word occurs approximately twice as
often as the second most frequent word. Here is our final Makefile:
```make
include config.mk
TXT_FILES=$(wildcard books/*.txt)
DAT_FILES=$(patsubst books/%.txt, %.dat, $(TXT_FILES))
# Generate summary table.
results.txt : $(ZIPF_SRC) $(DAT_FILES)
$(LANGUAGE) $^ > $@
# Count words.
.PHONY : dats
dats : $(DAT_FILES)
%.dat : $(COUNT_SRC) books/%.txt
$(LANGUAGE) $^ $@
.PHONY : clean
clean :
rm -f $(DAT_FILES)
rm -f results.txt
.PHONY : variables
variables:
@echo TXT_FILES: $(TXT_FILES)
@echo DAT_FILES: $(DAT_FILES)
```
Remember, the `config.mk` file contains:
```make
# Count words script.
LANGUAGE=python
COUNT_SRC=countwords.py
# Test Zipf's rule
ZIPF_SRC=testzipf.py
```
The following figure shows the dependencies embodied within our Makefile,
involved in building the `results.txt` target,
now we have introduced our function:
{alt='results.txt dependencies after introducing a function'}
::::::::::::::::::::::::::::::::::::::::: callout
## Where We Are
[This Makefile](files/code/07-functions/Makefile)
and [its accompanying `config.mk`](files/code/07-functions/config.mk)
contain all of our work so far.
::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::: challenge
## Adding more books
We can now do a better job at testing Zipf's rule by adding more books.
The books we have used come from the [Project Gutenberg](https://www.gutenberg.org/) website.
Project Gutenberg offers thousands of free ebooks to download.
**Exercise instructions:**
- go to [Project Gutenberg](https://www.gutenberg.org/) and use the search box to find another book,
for example ['The Picture of Dorian Gray'](https://www.gutenberg.org/ebooks/174) from Oscar Wilde.
- download the 'Plain Text UTF-8' version and save it to the `books` folder;
choose a short name for the file (**that doesn't include spaces**) e.g. "dorian\_gray.txt"
because the filename is going to be used in the `results.txt` file
- optionally, open the file in a text editor and remove extraneous text at the beginning and end
(look for the phrase `END OF THE PROJECT GUTENBERG EBOOK [title]`)
- run `make` and check that the correct commands are run, given the dependency tree
- check the results.txt file to see how this book compares to the others
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: keypoints
- Make is actually a small programming language with many built-in functions.
- Use `wildcard` function to get lists of files matching a pattern.
- Use `patsubst` function to rewrite file names.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/08-self-doc.md
================================================
---
title: Self-Documenting Makefiles
teaching: 10
exercises: 0
---
::::::::::::::::::::::::::::::::::::::: objectives
- Write self-documenting Makefiles with built-in help.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- How should I document a Makefile?
::::::::::::::::::::::::::::::::::::::::::::::::::
Many bash commands, and programs that people have written that can be
run from within bash, support a `--help` flag to display more
information on how to use the commands or programs. In this spirit, it
can be useful, both for ourselves and for others, to provide a `help`
target in our Makefiles. This can provide a summary of the names of
the key targets and what they do, so we don't need to look at the
Makefile itself unless we want to. For our Makefile, running a `help`
target might print:
```bash
$ make help
```
```output
results.txt : Generate Zipf summary table.
dats : Count words in text files.
clean : Remove auto-generated files.
```
So, how would we implement this? We could write a rule like:
```make
.PHONY : help
help :
@echo "results.txt : Generate Zipf summary table."
@echo "dats : Count words in text files."
@echo "clean : Remove auto-generated files."
```
But every time we add or remove a rule, or change the description of a
rule, we would have to update this rule too. It would be better if we
could keep the descriptions of the rules by the rules themselves and
extract these descriptions automatically.
The bash shell can help us here. It provides a command called
[sed][sed-docs] which stands for 'stream editor'. `sed` reads in some
text, does some filtering, and writes out the filtered text.
So, we could write comments for our rules, and mark them up in a way
which `sed` can detect. Since Make uses `#` for comments, we can use
`##` for comments that describe what a rule does and that we want
`sed` to detect. For example:
```make
## results.txt : Generate Zipf summary table.
results.txt : $(ZIPF_SRC) $(DAT_FILES)
$(LANGUAGE) $^ > $@
## dats : Count words in text files.
.PHONY : dats
dats : $(DAT_FILES)
%.dat : $(COUNT_SRC) books/%.txt
$(LANGUAGE) $^ $@
## clean : Remove auto-generated files.
.PHONY : clean
clean :
rm -f $(DAT_FILES)
rm -f results.txt
## variables : Print variables.
.PHONY : variables
variables:
@echo TXT_FILES: $(TXT_FILES)
@echo DAT_FILES: $(DAT_FILES)
```
We use `##` so we can distinguish between comments that we want `sed`
to automatically filter, and other comments that may describe what
other rules do, or that describe variables.
We can then write a `help` target that applies `sed` to our `Makefile`:
```make
.PHONY : help
help : Makefile
@sed -n 's/^##//p' $<
```
This rule depends upon the Makefile itself. It runs `sed` on the first
dependency of the rule, which is our Makefile, and tells `sed` to get
all the lines that begin with `##`, which `sed` then prints for us.
If we now run
```bash
$ make help
```
we get:
```output
results.txt : Generate Zipf summary table.
dats : Count words in text files.
clean : Remove auto-generated files.
variables : Print variables.
```
If we add, change or remove a target or rule, we now only need to
remember to add, update or remove a comment next to the rule. So long
as we respect our convention of using `##` for such comments, then our
`help` rule will take care of detecting these comments and printing
them for us.
::::::::::::::::::::::::::::::::::::::::: callout
## Where We Are
[This Makefile](files/code/08-self-doc/Makefile)
and [its accompanying `config.mk`](files/code/08-self-doc/config.mk)
contain all of our work so far.
::::::::::::::::::::::::::::::::::::::::::::::::::
[sed-docs]: https://www.gnu.org/software/sed/
:::::::::::::::::::::::::::::::::::::::: keypoints
- Document Makefiles by adding specially-formatted comments and a target to extract and format them.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/09-conclusion.md
================================================
---
title: Conclusion
teaching: 5
exercises: 30
---
::::::::::::::::::::::::::::::::::::::: objectives
- Understand advantages of automated build tools such as Make.
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: questions
- What are the advantages and disadvantages of using tools like Make?
::::::::::::::::::::::::::::::::::::::::::::::::::
Automated build tools such as Make can help us in a number of
ways. They help us to automate repetitive commands, hence saving us
time and reducing the likelihood of errors compared with running
these commands manually.
They can also save time by ensuring that automatically-generated
artifacts (such as data files or plots) are only recreated when the
files that were used to create these have changed in some way.
Through their notion of targets, dependencies, and actions, they serve
as a form of documentation, recording dependencies between code,
scripts, tools, configurations, raw data, derived data, plots, and
papers.
::::::::::::::::::::::::::::::::::::::: challenge
## Creating PNGs
Add new rules, update existing rules, and add new variables to:
- Create `.png` files from `.dat` files using `plotcounts.py`.
- Remove all auto-generated files (`.dat`, `.png`,
`results.txt`).
Finally, many Makefiles define a default [phony
target](../learners/reference.md#phony-target) called `all` as first target,
that will build what the Makefile has been written to build (e.g. in
our case, the `.png` files and the `results.txt` file). As others
may assume your Makefile conforms to convention and supports an
`all` target, add an `all` target to your Makefile (Hint: this rule
has the `results.txt` file and the `.png` files as dependencies, but
no actions). With that in place, instead of running `make results.txt`, you should now run `make all`, or just simply
`make`. By default, `make` runs the first target it finds in the
Makefile, in this case your new `all` target.
::::::::::::::: solution
## Solution
[This Makefile](files/code/09-conclusion-challenge-1/Makefile)
and [this `config.mk`](files/code/09-conclusion-challenge-1/config.mk)
contain a solution to this challenge.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
The following figure shows the dependencies involved in building the `all`
target, once we've added support for images:
{alt='results.txt dependencies once images have been added'}
::::::::::::::::::::::::::::::::::::::: challenge
## Creating an Archive
Often it is useful to create an archive file of your project that includes all data, code
and results. An archive file can package many files into a single file that can easily be
downloaded and shared with collaborators. We can add steps to create the archive file inside
the Makefile itself so it's easy to update our archive file as the project changes.
Edit the Makefile to create an archive file of your project. Add new rules, update existing
rules and add new variables to:
- Create a new directory called `zipf_analysis` in the project directory.
- Copy all our code, data, plots, the Zipf summary table, the Makefile and config.mk
to this directory.
The `cp -r` command can be used to copy files and directories
into the new `zipf_analysis` directory:
```bash
$ cp -r [files and directories to copy] zipf_analysis/
```
- Hint: create a new variable for the `books` directory so that it can be
copied to the new `zipf_analysis` directory
- Create an archive, `zipf_analysis.tar.gz`, of this directory. The
bash command `tar` can be used, as follows:
```bash
$ tar -czf zipf_analysis.tar.gz zipf_analysis
```
- Update the target `all` so that it creates `zipf_analysis.tar.gz`.
- Remove `zipf_analysis.tar.gz` when `make clean` is called.
- Print the values of any additional variables you have defined when
`make variables` is called.
::::::::::::::: solution
## Solution
[This Makefile](files/code/09-conclusion-challenge-2/Makefile)
and [this `config.mk`](files/code/09-conclusion-challenge-2/config.mk)
contain a solution to this challenge.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::: challenge
## Archiving the Makefile
Why does the Makefile rule for the archive directory add the Makefile to our archive of code,
data, plots and Zipf summary table?
::::::::::::::: solution
## Solution
Our code files (`countwords.py`, `plotcounts.py`, `testzipf.py`) implement
the individual parts of our workflow. They allow us to create `.dat`
files from `.txt` files, and `results.txt` and `.png` files from `.dat` files.
Our Makefile, however, documents dependencies between
our code, raw data, derived data, and plots, as well as implementing
our workflow as a whole. `config.mk` contains configuration information
for our Makefile, so it must be archived too.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::: challenge
## `touch` the Archive Directory
Why does the Makefile rule for the archive directory `touch` the archive directory after moving our code, data, plots and summary table into it?
::::::::::::::: solution
## Solution
A directory's timestamp is not automatically updated when files are copied into it.
If the code, data, plots, and summary table are updated and copied into the
archive directory, the archive directory's timestamp must be updated with `touch`
so that the rule that makes `zipf_analysis.tar.gz` knows to run again;
without this `touch`, `zipf_analysis.tar.gz` will only be created the first time
the rule is run and will not be updated on subsequent runs even if the contents
of the archive directory have changed.
:::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::
:::::::::::::::::::::::::::::::::::::::: keypoints
- Makefiles save time by automating repetitive work, and save thinking by documenting how to reproduce results.
::::::::::::::::::::::::::::::::::::::::::::::::::
================================================
FILE: episodes/data/books/LICENSE_TEXTS.md
================================================
A Note on the Texts' Licensing
==============================
Each text is from [Project Gutenberg](http://www.gutenberg.org/).
Headers and footers have been removed for the purposes of this
exercise. All the texts are governed by The Full Project Gutenberg
License reproduced below.
The texts and originating URLs are:
* [A Journey to the Western Islands of Scotland by Samuel Johnson](http://www.gutenberg.org/cache/epub/2064/pg2064.txt)
* [The People of the Abyss by Jack London](http://www.gutenberg.org/ebooks/1688)
* [My First Summer in the Sierra by John Muir](http://www.gutenberg.org/cache/epub/32540/pg32540.txt)
* [Scott's Last Expedition Volume I by Robert Falcon Scott](http://www.gutenberg.org/ebooks/11579)
*** START: FULL LICENSE ***
THE FULL PROJECT GUTENBERG LICENSE
PLEASE READ THIS BEFORE YOU DISTRIBUTE OR USE THIS WORK
To protect the Project Gutenberg-tm mission of promoting the free
distribution of electronic works, by using or distributing this work
(or any other work associated in any way with the phrase "Project
Gutenberg"), you agree to comply with all the terms of the Full Project
Gutenberg-tm License (available with this file or online at
http://gutenberg.net/license).
Section 1. General Terms of Use and Redistributing Project Gutenberg-tm
electronic works
1.A. By reading or using any part of this Project Gutenberg-tm
electronic work, you indicate that you have read, understand, agree to
and accept all the terms of this license and intellectual property
(trademark/copyright) agreement. If you do not agree to abide by all
the terms of this agreement, you must cease using and return or destroy
all copies of Project Gutenberg-tm electronic works in your possession.
If you paid a fee for obtaining a copy of or access to a Project
Gutenberg-tm electronic work and you do not agree to be bound by the
terms of this agreement, you may obtain a refund from the person or
entity to whom you paid the fee as set forth in paragraph 1.E.8.
1.B. "Project Gutenberg" is a registered trademark. It may only be
used on or associated in any way with an electronic work by people who
agree to be bound by the terms of this agreement. There are a few
things that you can do with most Project Gutenberg-tm electronic works
even without complying with the full terms of this agreement. See
paragraph 1.C below. There are a lot of things you can do with Project
Gutenberg-tm electronic works if you follow the terms of this agreement
and help preserve free future access to Project Gutenberg-tm electronic
works. See paragraph 1.E below.
1.C. The Project Gutenberg Literary Archive Foundation ("the Foundation"
or PGLAF), owns a compilation copyright in the collection of Project
Gutenberg-tm electronic works. Nearly all the individual works in the
collection are in the public domain in the United States. If an
individual work is in the public domain in the United States and you are
located in the United States, we do not claim a right to prevent you from
copying, distributing, performing, displaying or creating derivative
works based on the work as long as all references to Project Gutenberg
are removed. Of course, we hope that you will support the Project
Gutenberg-tm mission of promoting free access to electronic works by
freely sharing Project Gutenberg-tm works in compliance with the terms of
this agreement for keeping the Project Gutenberg-tm name associated with
the work. You can easily comply with the terms of this agreement by
keeping this work in the same format with its attached full Project
Gutenberg-tm License when you share it without charge with others.
1.D. The copyright laws of the place where you are located also govern
what you can do with this work. Copyright laws in most countries are in
a constant state of change. If you are outside the United States, check
the laws of your country in addition to the terms of this agreement
before downloading, copying, displaying, performing, distributing or
creating derivative works based on this work or any other Project
Gutenberg-tm work. The Foundation makes no representations concerning
the copyright status of any work in any country outside the United
States.
1.E. Unless you have removed all references to Project Gutenberg:
1.E.1. The following sentence, with active links to, or other immediate
access to, the full Project Gutenberg-tm License must appear prominently
whenever any copy of a Project Gutenberg-tm work (any work on which the
phrase "Project Gutenberg" appears, or with which the phrase "Project
Gutenberg" is associated) is accessed, displayed, performed, viewed,
copied or distributed:
This eBook is for the use of anyone anywhere at no cost and with
almost no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included
with this eBook or online at www.gutenberg.net
1.E.2. If an individual Project Gutenberg-tm electronic work is derived
from the public domain (does not contain a notice indicating that it is
posted with permission of the copyright holder), the work can be copied
and distributed to anyone in the United States without paying any fees
or charges. If you are redistributing or providing access to a work
with the phrase "Project Gutenberg" associated with or appearing on the
work, you must comply either with the requirements of paragraphs 1.E.1
through 1.E.7 or obtain permission for the use of the work and the
Project Gutenberg-tm trademark as set forth in paragraphs 1.E.8 or
1.E.9.
1.E.3. If an individual Project Gutenberg-tm electronic work is posted
with the permission of the copyright holder, your use and distribution
must comply with both paragraphs 1.E.1 through 1.E.7 and any additional
terms imposed by the copyright holder. Additional terms will be linked
to the Project Gutenberg-tm License for all works posted with the
permission of the copyright holder found at the beginning of this work.
1.E.4. Do not unlink or detach or remove the full Project Gutenberg-tm
License terms from this work, or any files containing a part of this
work or any other work associated with Project Gutenberg-tm.
1.E.5. Do not copy, display, perform, distribute or redistribute this
electronic work, or any part of this electronic work, without
prominently displaying the sentence set forth in paragraph 1.E.1 with
active links or immediate access to the full terms of the Project
Gutenberg-tm License.
1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form, including any
word processing or hypertext form. However, if you provide access to or
distribute copies of a Project Gutenberg-tm work in a format other than
"Plain Vanilla ASCII" or other format used in the official version
posted on the official Project Gutenberg-tm web site (www.gutenberg.net),
you must, at no additional cost, fee or expense to the user, provide a
copy, a means of exporting a copy, or a means of obtaining a copy upon
request, of the work in its original "Plain Vanilla ASCII" or other
form. Any alternate format must include the full Project Gutenberg-tm
License as specified in paragraph 1.E.1.
1.E.7. Do not charge a fee for access to, viewing, displaying,
performing, copying or distributing any Project Gutenberg-tm works
unless you comply with paragraph 1.E.8 or 1.E.9.
1.E.8. You may charge a reasonable fee for copies of or providing
access to or distributing Project Gutenberg-tm electronic works provided
that
- You pay a royalty fee of 20% of the gross profits you derive from
the use of Project Gutenberg-tm works calculated using the method
you already use to calculate your applicable taxes. The fee is
owed to the owner of the Project Gutenberg-tm trademark, but he
has agreed to donate royalties under this paragraph to the
Project Gutenberg Literary Archive Foundation. Royalty payments
must be paid within 60 days following each date on which you
prepare (or are legally required to prepare) your periodic tax
returns. Royalty payments should be clearly marked as such and
sent to the Project Gutenberg Literary Archive Foundation at the
address specified in Section 4, "Information about donations to
the Project Gutenberg Literary Archive Foundation."
- You provide a full refund of any money paid by a user who notifies
you in writing (or by e-mail) within 30 days of receipt that s/he
does not agree to the terms of the full Project Gutenberg-tm
License. You must require such a user to return or
destroy all copies of the works possessed in a physical medium
and discontinue all use of and all access to other copies of
Project Gutenberg-tm works.
- You provide, in accordance with paragraph 1.F.3, a full refund of any
money paid for a work or a replacement copy, if a defect in the
electronic work is discovered and reported to you within 90 days
of receipt of the work.
- You comply with all other terms of this agreement for free
distribution of Project Gutenberg-tm works.
1.E.9. If you wish to charge a fee or distribute a Project Gutenberg-tm
electronic work or group of works on different terms than are set
forth in this agreement, you must obtain permission in writing from
both the Project Gutenberg Literary Archive Foundation and Michael
Hart, the owner of the Project Gutenberg-tm trademark. Contact the
Foundation as set forth in Section 3 below.
1.F.
1.F.1. Project Gutenberg volunteers and employees expend considerable
effort to identify, do copyright research on, transcribe and proofread
public domain works in creating the Project Gutenberg-tm
collection. Despite these efforts, Project Gutenberg-tm electronic
works, and the medium on which they may be stored, may contain
"Defects," such as, but not limited to, incomplete, inaccurate or
corrupt data, transcription errors, a copyright or other intellectual
property infringement, a defective or damaged disk or other medium, a
computer virus, or computer codes that damage or cannot be read by
your equipment.
1.F.2. LIMITED WARRANTY, DISCLAIMER OF DAMAGES - Except for the "Right
of Replacement or Refund" described in paragraph 1.F.3, the Project
Gutenberg Literary Archive Foundation, the owner of the Project
Gutenberg-tm trademark, and any other party distributing a Project
Gutenberg-tm electronic work under this agreement, disclaim all
liability to you for damages, costs and expenses, including legal
fees. YOU AGREE THAT YOU HAVE NO REMEDIES FOR NEGLIGENCE, STRICT
LIABILITY, BREACH OF WARRANTY OR BREACH OF CONTRACT EXCEPT THOSE
PROVIDED IN PARAGRAPH F3. YOU AGREE THAT THE FOUNDATION, THE
TRADEMARK OWNER, AND ANY DISTRIBUTOR UNDER THIS AGREEMENT WILL NOT BE
LIABLE TO YOU FOR ACTUAL, DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE OR
INCIDENTAL DAMAGES EVEN IF YOU GIVE NOTICE OF THE POSSIBILITY OF SUCH
DAMAGE.
1.F.3. LIMITED RIGHT OF REPLACEMENT OR REFUND - If you discover a
defect in this electronic work within 90 days of receiving it, you can
receive a refund of the money (if any) you paid for it by sending a
written explanation to the person you received the work from. If you
received the work on a physical medium, you must return the medium with
your written explanation. The person or entity that provided you with
the defective work may elect to provide a replacement copy in lieu of a
refund. If you received the work electronically, the person or entity
providing it to you may choose to give you a second opportunity to
receive the work electronically in lieu of a refund. If the second copy
is also defective, you may demand a refund in writing without further
opportunities to fix the problem.
1.F.4. Except for the limited right of replacement or refund set forth
in paragraph 1.F.3, this work is provided to you 'AS-IS' WITH NO OTHER
WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
WARRANTIES OF MERCHANTIBILITY OR FITNESS FOR ANY PURPOSE.
1.F.5. Some states do not allow disclaimers of certain implied
warranties or the exclusion or limitation of certain types of damages.
If any disclaimer or limitation set forth in this agreement violates the
law of the state applicable to this agreement, the agreement shall be
interpreted to make the maximum disclaimer or limitation permitted by
the applicable state law. The invalidity or unenforceability of any
provision of this agreement shall not void the remaining provisions.
1.F.6. INDEMNITY - You agree to indemnify and hold the Foundation, the
trademark owner, any agent or employee of the Foundation, anyone
providing copies of Project Gutenberg-tm electronic works in accordance
with this agreement, and any volunteers associated with the production,
promotion and distribution of Project Gutenberg-tm electronic works,
harmless from all liability, costs and expenses, including legal fees,
that arise directly or indirectly from any of the following which you do
or cause to occur: (a) distribution of this or any Project Gutenberg-tm
work, (b) alteration, modification, or additions or deletions to any
Project Gutenberg-tm work, and (c) any Defect you cause.
Section 2. Information about the Mission of Project Gutenberg-tm
Project Gutenberg-tm is synonymous with the free distribution of
electronic works in formats readable by the widest variety of computers
including obsolete, old, middle-aged and new computers. It exists
because of the efforts of hundreds of volunteers and donations from
people in all walks of life.
Volunteers and financial support to provide volunteers with the
assistance they need, is critical to reaching Project Gutenberg-tm's
goals and ensuring that the Project Gutenberg-tm collection will
remain freely available for generations to come. In 2001, the Project
Gutenberg Literary Archive Foundation was created to provide a secure
and permanent future for Project Gutenberg-tm and future generations.
To learn more about the Project Gutenberg Literary Archive Foundation
and how your efforts and donations can help, see Sections 3 and 4
and the Foundation web page at http://www.pglaf.org.
Section 3. Information about the Project Gutenberg Literary Archive
Foundation
The Project Gutenberg Literary Archive Foundation is a non profit
501(c)(3) educational corporation organized under the laws of the
state of Mississippi and granted tax exempt status by the Internal
Revenue Service. The Foundation's EIN or federal tax identification
number is 64-6221541. Its 501(c)(3) letter is posted at
http://pglaf.org/fundraising. Contributions to the Project Gutenberg
Literary Archive Foundation are tax deductible to the full extent
permitted by U.S. federal laws and your state's laws.
The Foundation's principal office is located at 4557 Melan Dr. S.
Fairbanks, AK, 99712., but its volunteers and employees are scattered
throughout numerous locations. Its business office is located at
809 North 1500 West, Salt Lake City, UT 84116, (801) 596-1887, email
business@pglaf.org. Email contact links and up to date contact
information can be found at the Foundation's web site and official
page at http://pglaf.org
For additional contact information:
Dr. Gregory B. Newby
Chief Executive and Director
gbnewby@pglaf.org
Section 4. Information about Donations to the Project Gutenberg
Literary Archive Foundation
Project Gutenberg-tm depends upon and cannot survive without wide
spread public support and donations to carry out its mission of
increasing the number of public domain and licensed works that can be
freely distributed in machine readable form accessible by the widest
array of equipment including outdated equipment. Many small donations
($1 to $5,000) are particularly important to maintaining tax exempt
status with the IRS.
The Foundation is committed to complying with the laws regulating
charities and charitable donations in all 50 states of the United
States. Compliance requirements are not uniform and it takes a
considerable effort, much paperwork and many fees to meet and keep up
with these requirements. We do not solicit donations in locations
where we have not received written confirmation of compliance. To
SEND DONATIONS or determine the status of compliance for any
particular state visit http://pglaf.org
While we cannot and do not solicit contributions from states where we
have not met the solicitation requirements, we know of no prohibition
against accepting unsolicited donations from donors in such states who
approach us with offers to donate.
International donations are gratefully accepted, but we cannot make
any statements concerning tax treatment of donations received from
outside the United States. U.S. laws alone swamp our small staff.
Please check the Project Gutenberg Web pages for current donation
methods and addresses. Donations are accepted in a number of other
ways including including checks, online payments and credit card
donations. To donate, please visit: http://pglaf.org/donate
Section 5. General Information About Project Gutenberg-tm electronic
works.
Professor Michael S. Hart is the originator of the Project Gutenberg-tm
concept of a library of electronic works that could be freely shared
with anyone. For thirty years, he produced and distributed Project
Gutenberg-tm eBooks with only a loose network of volunteer support.
Project Gutenberg-tm eBooks are often created from several printed
editions, all of which are confirmed as Public Domain in the U.S.
unless a copyright notice is included. Thus, we do not necessarily
keep eBooks in compliance with any particular paper edition.
Most people start at our Web site which has the main PG search facility:
http://www.gutenberg.net
This Web site includes information about Project Gutenberg-tm,
including how to make donations to the Project Gutenberg Literary
Archive Foundation, how to help produce our new eBooks, and how to
subscribe to our email newsletter to hear about new eBooks
================================================
FILE: episodes/data/books/abyss.txt
================================================
THE PEOPLE OF THE ABYSS
The chief priests and rulers cry:-
"O Lord and Master, not ours the guilt,
We build but as our fathers built;
Behold thine images how they stand
Sovereign and sole through all our land.
"Our task is hard--with sword and flame,
To hold thine earth forever the same,
And with sharp crooks of steel to keep,
Still as thou leftest them, thy sheep."
Then Christ sought out an artisan,
A low-browed, stunted, haggard man,
And a motherless girl whose fingers thin
Crushed from her faintly want and sin.
These set he in the midst of them,
And as they drew back their garment hem
For fear of defilement, "Lo, here," said he,
"The images ye have made of me."
JAMES RUSSELL LOWELL.
PREFACE
The experiences related in this volume fell to me in the summer of 1902.
I went down into the under-world of London with an attitude of mind which
I may best liken to that of the explorer. I was open to be convinced by
the evidence of my eyes, rather than by the teachings of those who had
not seen, or by the words of those who had seen and gone before. Further,
I took with me certain simple criteria with which to measure the life of
the under-world. That which made for more life, for physical and
spiritual health, was good; that which made for less life, which hurt,
and dwarfed, and distorted life, was bad.
It will be readily apparent to the reader that I saw much that was bad.
Yet it must not be forgotten that the time of which I write was
considered "good times" in England. The starvation and lack of shelter I
encountered constituted a chronic condition of misery which is never
wiped out, even in the periods of greatest prosperity.
Following the summer in question came a hard winter. Great numbers of
the unemployed formed into processions, as many as a dozen at a time, and
daily marched through the streets of London crying for bread. Mr. Justin
McCarthy, writing in the month of January 1903, to the New York
_Independent_, briefly epitomises the situation as follows:-
"The workhouses have no space left in which to pack the starving
crowds who are craving every day and night at their doors for food and
shelter. All the charitable institutions have exhausted their means
in trying to raise supplies of food for the famishing residents of the
garrets and cellars of London lanes and alleys. The quarters of the
Salvation Army in various parts of London are nightly besieged by
hosts of the unemployed and the hungry for whom neither shelter nor
the means of sustenance can be provided."
It has been urged that the criticism I have passed on things as they are
in England is too pessimistic. I must say, in extenuation, that of
optimists I am the most optimistic. But I measure manhood less by
political aggregations than by individuals. Society grows, while
political machines rack to pieces and become "scrap." For the English,
so far as manhood and womanhood and health and happiness go, I see a
broad and smiling future. But for a great deal of the political
machinery, which at present mismanages for them, I see nothing else than
the scrap heap.
JACK LONDON.
PIEDMONT, CALIFORNIA.
CHAPTER I--THE DESCENT
"But you can't do it, you know," friends said, to whom I applied for
assistance in the matter of sinking myself down into the East End of
London. "You had better see the police for a guide," they added, on
second thought, painfully endeavouring to adjust themselves to the
psychological processes of a madman who had come to them with better
credentials than brains.
"But I don't want to see the police," I protested. "What I wish to do is
to go down into the East End and see things for myself. I wish to know
how those people are living there, and why they are living there, and
what they are living for. In short, I am going to live there myself."
"You don't want to _live_ down there!" everybody said, with
disapprobation writ large upon their faces. "Why, it is said there are
places where a man's life isn't worth tu'pence."
"The very places I wish to see," I broke in.
"But you can't, you know," was the unfailing rejoinder.
"Which is not what I came to see you about," I answered brusquely,
somewhat nettled by their incomprehension. "I am a stranger here, and I
want you to tell me what you know of the East End, in order that I may
have something to start on."
"But we know nothing of the East End. It is over there, somewhere." And
they waved their hands vaguely in the direction where the sun on rare
occasions may be seen to rise.
"Then I shall go to Cook's," I announced.
"Oh yes," they said, with relief. "Cook's will be sure to know."
But O Cook, O Thomas Cook & Son, path-finders and trail-clearers, living
sign-posts to all the world, and bestowers of first aid to bewildered
travellers--unhesitatingly and instantly, with ease and celerity, could
you send me to Darkest Africa or Innermost Thibet, but to the East End of
London, barely a stone's throw distant from Ludgate Circus, you know not
the way!
"You can't do it, you know," said the human emporium of routes and fares
at Cook's Cheapside branch. "It is so--hem--so unusual."
"Consult the police," he concluded authoritatively, when I had persisted.
"We are not accustomed to taking travellers to the East End; we receive
no call to take them there, and we know nothing whatsoever about the
place at all."
"Never mind that," I interposed, to save myself from being swept out of
the office by his flood of negations. "Here's something you can do for
me. I wish you to understand in advance what I intend doing, so that in
case of trouble you may be able to identify me."
"Ah, I see! should you be murdered, we would be in position to identify
the corpse."
He said it so cheerfully and cold-bloodedly that on the instant I saw my
stark and mutilated cadaver stretched upon a slab where cool waters
trickle ceaselessly, and him I saw bending over and sadly and patiently
identifying it as the body of the insane American who _would_ see the
East End.
"No, no," I answered; "merely to identify me in case I get into a scrape
with the 'bobbies.'" This last I said with a thrill; truly, I was
gripping hold of the vernacular.
"That," he said, "is a matter for the consideration of the Chief Office."
"It is so unprecedented, you know," he added apologetically.
The man at the Chief Office hemmed and hawed. "We make it a rule," he
explained, "to give no information concerning our clients."
"But in this case," I urged, "it is the client who requests you to give
the information concerning himself."
Again he hemmed and hawed.
"Of course," I hastily anticipated, "I know it is unprecedented, but--"
"As I was about to remark," he went on steadily, "it is unprecedented,
and I don't think we can do anything for you."
However, I departed with the address of a detective who lived in the East
End, and took my way to the American consul-general. And here, at last,
I found a man with whom I could "do business." There was no hemming and
hawing, no lifted brows, open incredulity, or blank amazement. In one
minute I explained myself and my project, which he accepted as a matter
of course. In the second minute he asked my age, height, and weight, and
looked me over. And in the third minute, as we shook hands at parting,
he said: "All right, Jack. I'll remember you and keep track."
I breathed a sigh of relief. Having burnt my ships behind me, I was now
free to plunge into that human wilderness of which nobody seemed to know
anything. But at once I encountered a new difficulty in the shape of my
cabby, a grey-whiskered and eminently decorous personage who had
imperturbably driven me for several hours about the "City."
"Drive me down to the East End," I ordered, taking my seat.
"Where, sir?" he demanded with frank surprise.
"To the East End, anywhere. Go on."
The hansom pursued an aimless way for several minutes, then came to a
puzzled stop. The aperture above my head was uncovered, and the cabman
peered down perplexedly at me.
"I say," he said, "wot plyce yer wanter go?"
"East End," I repeated. "Nowhere in particular. Just drive me around
anywhere."
"But wot's the haddress, sir?"
"See here!" I thundered. "Drive me down to the East End, and at once!"
It was evident that he did not understand, but he withdrew his head, and
grumblingly started his horse.
Nowhere in the streets of London may one escape the sight of abject
poverty, while five minutes' walk from almost any point will bring one to
a slum; but the region my hansom was now penetrating was one unending
slum. The streets were filled with a new and different race of people,
short of stature, and of wretched or beer-sodden appearance. We rolled
along through miles of bricks and squalor, and from each cross street and
alley flashed long vistas of bricks and misery. Here and there lurched a
drunken man or woman, and the air was obscene with sounds of jangling and
squabbling. At a market, tottery old men and women were searching in the
garbage thrown in the mud for rotten potatoes, beans, and vegetables,
while little children clustered like flies around a festering mass of
fruit, thrusting their arms to the shoulders into the liquid corruption,
and drawing forth morsels but partially decayed, which they devoured on
the spot.
Not a hansom did I meet with in all my drive, while mine was like an
apparition from another and better world, the way the children ran after
it and alongside. And as far as I could see were the solid walls of
brick, the slimy pavements, and the screaming streets; and for the first
time in my life the fear of the crowd smote me. It was like the fear of
the sea; and the miserable multitudes, street upon street, seemed so many
waves of a vast and malodorous sea, lapping about me and threatening to
well up and over me.
"Stepney, sir; Stepney Station," the cabby called down.
I looked about. It was really a railroad station, and he had driven
desperately to it as the one familiar spot he had ever heard of in all
that wilderness.
"Well," I said.
He spluttered unintelligibly, shook his head, and looked very miserable.
"I'm a strynger 'ere," he managed to articulate. "An' if yer don't want
Stepney Station, I'm blessed if I know wotcher do want."
"I'll tell you what I want," I said. "You drive along and keep your eye
out for a shop where old clothes are sold. Now, when you see such a
shop, drive right on till you turn the corner, then stop and let me out."
I could see that he was growing dubious of his fare, but not long
afterwards he pulled up to the curb and informed me that an old-clothes
shop was to be found a bit of the way back.
"Won'tcher py me?" he pleaded. "There's seven an' six owin' me."
"Yes," I laughed, "and it would be the last I'd see of you."
"Lord lumme, but it'll be the last I see of you if yer don't py me," he
retorted.
But a crowd of ragged onlookers had already gathered around the cab, and
I laughed again and walked back to the old-clothes shop.
Here the chief difficulty was in making the shopman understand that I
really and truly wanted old clothes. But after fruitless attempts to
press upon me new and impossible coats and trousers, he began to bring to
light heaps of old ones, looking mysterious the while and hinting darkly.
This he did with the palpable intention of letting me know that he had
"piped my lay," in order to bulldose me, through fear of exposure, into
paying heavily for my purchases. A man in trouble, or a high-class
criminal from across the water, was what he took my measure for--in
either case, a person anxious to avoid the police.
But I disputed with him over the outrageous difference between prices and
values, till I quite disabused him of the notion, and he settled down to
drive a hard bargain with a hard customer. In the end I selected a pair
of stout though well-worn trousers, a frayed jacket with one remaining
button, a pair of brogans which had plainly seen service where coal was
shovelled, a thin leather belt, and a very dirty cloth cap. My
underclothing and socks, however, were new and warm, but of the sort that
any American waif, down in his luck, could acquire in the ordinary course
of events.
"I must sy yer a sharp 'un," he said, with counterfeit admiration, as I
handed over the ten shillings finally agreed upon for the outfit.
"Blimey, if you ain't ben up an' down Petticut Lane afore now. Yer
trouseys is wuth five bob to hany man, an' a docker 'ud give two an' six
for the shoes, to sy nothin' of the coat an' cap an' new stoker's singlet
an' hother things."
"How much will you give me for them?" I demanded suddenly. "I paid you
ten bob for the lot, and I'll sell them back to you, right now, for
eight! Come, it's a go!"
But he grinned and shook his head, and though I had made a good bargain,
I was unpleasantly aware that he had made a better one.
I found the cabby and a policeman with their heads together, but the
latter, after looking me over sharply, and particularly scrutinizing the
bundle under my arm, turned away and left the cabby to wax mutinous by
himself. And not a step would he budge till I paid him the seven
shillings and sixpence owing him. Whereupon he was willing to drive me
to the ends of the earth, apologising profusely for his insistence, and
explaining that one ran across queer customers in London Town.
But he drove me only to Highbury Vale, in North London, where my luggage
was waiting for me. Here, next day, I took off my shoes (not without
regret for their lightness and comfort), and my soft, grey travelling
suit, and, in fact, all my clothing; and proceeded to array myself in the
clothes of the other and unimaginable men, who must have been indeed
unfortunate to have had to part with such rags for the pitiable sums
obtainable from a dealer.
Inside my stoker's singlet, in the armpit, I sewed a gold sovereign (an
emergency sum certainly of modest proportions); and inside my stoker's
singlet I put myself. And then I sat down and moralised upon the fair
years and fat, which had made my skin soft and brought the nerves close
to the surface; for the singlet was rough and raspy as a hair shirt, and
I am confident that the most rigorous of ascetics suffer no more than I
did in the ensuing twenty-four hours.
The remainder of my costume was fairly easy to put on, though the
brogans, or brogues, were quite a problem. As stiff and hard as if made
of wood, it was only after a prolonged pounding of the uppers with my
fists that I was able to get my feet into them at all. Then, with a few
shillings, a knife, a handkerchief, and some brown papers and flake
tobacco stowed away in my pockets, I thumped down the stairs and said
good-bye to my foreboding friends. As I paused out of the door, the
"help," a comely middle-aged woman, could not conquer a grin that twisted
her lips and separated them till the throat, out of involuntary sympathy,
made the uncouth animal noises we are wont to designate as "laughter."
No sooner was I out on the streets than I was impressed by the difference
in status effected by my clothes. All servility vanished from the
demeanour of the common people with whom I came in contact. Presto! in
the twinkling of an eye, so to say, I had become one of them. My frayed
and out-at-elbows jacket was the badge and advertisement of my class,
which was their class. It made me of like kind, and in place of the
fawning and too respectful attention I had hitherto received, I now
shared with them a comradeship. The man in corduroy and dirty
neckerchief no longer addressed me as "sir" or "governor." It was "mate"
now--and a fine and hearty word, with a tingle to it, and a warmth and
gladness, which the other term does not possess. Governor! It smacks of
mastery, and power, and high authority--the tribute of the man who is
under to the man on top, delivered in the hope that he will let up a bit
and ease his weight, which is another way of saying that it is an appeal
for alms.
This brings me to a delight I experienced in my rags and tatters which is
denied the average American abroad. The European traveller from the
States, who is not a Croesus, speedily finds himself reduced to a chronic
state of self-conscious sordidness by the hordes of cringing robbers who
clutter his steps from dawn till dark, and deplete his pocket-book in a
way that puts compound interest to the blush.
In my rags and tatters I escaped the pestilence of tipping, and
encountered men on a basis of equality. Nay, before the day was out I
turned the tables, and said, most gratefully, "Thank you, sir," to a
gentleman whose horse I held, and who dropped a penny into my eager palm.
Other changes I discovered were wrought in my condition by my new garb.
In crossing crowded thoroughfares I found I had to be, if anything, more
lively in avoiding vehicles, and it was strikingly impressed upon me that
my life had cheapened in direct ratio with my clothes. When before I
inquired the way of a policeman, I was usually asked, "Bus or 'ansom,
sir?" But now the query became, "Walk or ride?" Also, at the railway
stations, a third-class ticket was now shoved out to me as a matter of
course.
But there was compensation for it all. For the first time I met the
English lower classes face to face, and knew them for what they were.
When loungers and workmen, at street corners and in public-houses, talked
with me, they talked as one man to another, and they talked as natural
men should talk, without the least idea of getting anything out of me for
what they talked or the way they talked.
And when at last I made into the East End, I was gratified to find that
the fear of the crowd no longer haunted me. I had become a part of it.
The vast and malodorous sea had welled up and over me, or I had slipped
gently into it, and there was nothing fearsome about it--with the one
exception of the stoker's singlet.
CHAPTER II--JOHNNY UPRIGHT
I shall not give you the address of Johnny Upright. Let it suffice that
he lives in the most respectable street in the East End--a street that
would be considered very mean in America, but a veritable oasis in the
desert of East London. It is surrounded on every side by close-packed
squalor and streets jammed by a young and vile and dirty generation; but
its own pavements are comparatively bare of the children who have no
other place to play, while it has an air of desertion, so few are the
people that come and go.
Each house in this street, as in all the streets, is shoulder to shoulder
with its neighbours. To each house there is but one entrance, the front
door; and each house is about eighteen feet wide, with a bit of a brick-
walled yard behind, where, when it is not raining, one may look at a
slate-coloured sky. But it must be understood that this is East End
opulence we are now considering. Some of the people in this street are
even so well-to-do as to keep a "slavey." Johnny Upright keeps one, as I
well know, she being my first acquaintance in this particular portion of
the world.
To Johnny Upright's house I came, and to the door came the "slavey." Now,
mark you, her position in life was pitiable and contemptible, but it was
with pity and contempt that she looked at me. She evinced a plain desire
that our conversation should be short. It was Sunday, and Johnny Upright
was not at home, and that was all there was to it. But I lingered,
discussing whether or not it was all there was to it, till Mrs. Johnny
Upright was attracted to the door, where she scolded the girl for not
having closed it before turning her attention to me.
No, Mr. Johnny Upright was not at home, and further, he saw nobody on
Sunday. It is too bad, said I. Was I looking for work? No, quite the
contrary; in fact, I had come to see Johnny Upright on business which
might be profitable to him.
A change came over the face of things at once. The gentleman in question
was at church, but would be home in an hour or thereabouts, when no doubt
he could be seen.
Would I kindly step in?--no, the lady did not ask me, though I fished for
an invitation by stating that I would go down to the corner and wait in a
public-house. And down to the corner I went, but, it being church time,
the "pub" was closed. A miserable drizzle was falling, and, in lieu of
better, I took a seat on a neighbourly doorstep and waited.
And here to the doorstep came the "slavey," very frowzy and very
perplexed, to tell me that the missus would let me come back and wait in
the kitchen.
"So many people come 'ere lookin' for work," Mrs. Johnny Upright
apologetically explained. "So I 'ope you won't feel bad the way I
spoke."
"Not at all, not at all," I replied in my grandest manner, for the nonce
investing my rags with dignity. "I quite understand, I assure you. I
suppose people looking for work almost worry you to death?"
"That they do," she answered, with an eloquent and expressive glance; and
thereupon ushered me into, not the kitchen, but the dining room--a
favour, I took it, in recompense for my grand manner.
This dining-room, on the same floor as the kitchen, was about four feet
below the level of the ground, and so dark (it was midday) that I had to
wait a space for my eyes to adjust themselves to the gloom. Dirty light
filtered in through a window, the top of which was on a level with a
sidewalk, and in this light I found that I was able to read newspaper
print.
And here, while waiting the coming of Johnny Upright, let me explain my
errand. While living, eating, and sleeping with the people of the East
End, it was my intention to have a port of refuge, not too far distant,
into which could run now and again to assure myself that good clothes and
cleanliness still existed. Also in such port I could receive my mail,
work up my notes, and sally forth occasionally in changed garb to
civilisation.
But this involved a dilemma. A lodging where my property would be safe
implied a landlady apt to be suspicious of a gentleman leading a double
life; while a landlady who would not bother her head over the double life
of her lodgers would imply lodgings where property was unsafe. To avoid
the dilemma was what had brought me to Johnny Upright. A detective of
thirty-odd years' continuous service in the East End, known far and wide
by a name given him by a convicted felon in the dock, he was just the man
to find me an honest landlady, and make her rest easy concerning the
strange comings and goings of which I might be guilty.
His two daughters beat him home from church--and pretty girls they were
in their Sunday dresses; withal it was the certain weak and delicate
prettiness which characterises the Cockney lasses, a prettiness which is
no more than a promise with no grip on time, and doomed to fade quickly
away like the colour from a sunset sky.
They looked me over with frank curiosity, as though I were some sort of a
strange animal, and then ignored me utterly for the rest of my wait. Then
Johnny Upright himself arrived, and I was summoned upstairs to confer
with him.
"Speak loud," he interrupted my opening words. "I've got a bad cold, and
I can't hear well."
Shades of Old Sleuth and Sherlock Holmes! I wondered as to where the
assistant was located whose duty it was to take down whatever information
I might loudly vouchsafe. And to this day, much as I have seen of Johnny
Upright and much as I have puzzled over the incident, I have never been
quite able to make up my mind as to whether or not he had a cold, or had
an assistant planted in the other room. But of one thing I am sure:
though I gave Johnny Upright the facts concerning myself and project, he
withheld judgment till next day, when I dodged into his street
conventionally garbed and in a hansom. Then his greeting was cordial
enough, and I went down into the dining-room to join the family at tea.
"We are humble here," he said, "not given to the flesh, and you must take
us for what we are, in our humble way."
The girls were flushed and embarrassed at greeting me, while he did not
make it any the easier for them.
"Ha! ha!" he roared heartily, slapping the table with his open hand till
the dishes rang. "The girls thought yesterday you had come to ask for a
piece of bread! Ha! ha! ho! ho! ho!"
This they indignantly denied, with snapping eyes and guilty red cheeks,
as though it were an essential of true refinement to be able to discern
under his rags a man who had no need to go ragged.
And then, while I ate bread and marmalade, proceeded a play at cross
purposes, the daughters deeming it an insult to me that I should have
been mistaken for a beggar, and the father considering it as the highest
compliment to my cleverness to succeed in being so mistaken. All of
which I enjoyed, and the bread, the marmalade, and the tea, till the time
came for Johnny Upright to find me a lodging, which he did, not half-a-
dozen doors away, in his own respectable and opulent street, in a house
as like to his own as a pea to its mate.
CHAPTER III--MY LODGING AND SOME OTHERS
From an East London standpoint, the room I rented for six shillings, or a
dollar and a half, per week, was a most comfortable affair. From the
American standpoint, on the other hand, it was rudely furnished,
uncomfortable, and small. By the time I had added an ordinary typewriter
table to its scanty furnishing, I was hard put to turn around; at the
best, I managed to navigate it by a sort of vermicular progression
requiring great dexterity and presence of mind.
Having settled myself, or my property rather, I put on my knockabout
clothes and went out for a walk. Lodgings being fresh in my mind, I
began to look them up, bearing in mind the hypothesis that I was a poor
young man with a wife and large family.
My first discovery was that empty houses were few and far between--so far
between, in fact, that though I walked miles in irregular circles over a
large area, I still remained between. Not one empty house could I find--a
conclusive proof that the district was "saturated."
It being plain that as a poor young man with a family I could rent no
houses at all in this most undesirable region, I next looked for rooms,
unfurnished rooms, in which I could store my wife and babies and
chattels. There were not many, but I found them, usually in the
singular, for one appears to be considered sufficient for a poor man's
family in which to cook and eat and sleep. When I asked for two rooms,
the sublettees looked at me very much in the manner, I imagine, that a
certain personage looked at Oliver Twist when he asked for more.
Not only was one room deemed sufficient for a poor man and his family,
but I learned that many families, occupying single rooms, had so much
space to spare as to be able to take in a lodger or two. When such rooms
can be rented for from three to six shillings per week, it is a fair
conclusion that a lodger with references should obtain floor space for,
say, from eightpence to a shilling. He may even be able to board with
the sublettees for a few shillings more. This, however, I failed to
inquire into--a reprehensible error on my part, considering that I was
working on the basis of a hypothetical family.
Not only did the houses I investigated have no bath-tubs, but I learned
that there were no bath-tubs in all the thousands of houses I had seen.
Under the circumstances, with my wife and babies and a couple of lodgers
suffering from the too great spaciousness of one room, taking a bath in a
tin wash-basin would be an unfeasible undertaking. But, it seems, the
compensation comes in with the saving of soap, so all's well, and God's
still in heaven.
However, I rented no rooms, but returned to my own Johnny Upright's
street. What with my wife, and babies, and lodgers, and the various
cubby-holes into which I had fitted them, my mind's eye had become narrow-
angled, and I could not quite take in all of my own room at once. The
immensity of it was awe-inspiring. Could this be the room I had rented
for six shillings a week? Impossible! But my landlady, knocking at the
door to learn if I were comfortable, dispelled my doubts.
"Oh yes, sir," she said, in reply to a question. "This street is the
very last. All the other streets were like this eight or ten years ago,
and all the people were very respectable. But the others have driven our
kind out. Those in this street are the only ones left. It's shocking,
sir!"
And then she explained the process of saturation, by which the rental
value of a neighbourhood went up, while its tone went down.
"You see, sir, our kind are not used to crowding in the way the others
do. We need more room. The others, the foreigners and lower-class
people, can get five and six families into this house, where we only get
one. So they can pay more rent for the house than we can afford. It
_is_ shocking, sir; and just to think, only a few years ago all this
neighbourhood was just as nice as it could be."
I looked at her. Here was a woman, of the finest grade of the English
working-class, with numerous evidences of refinement, being slowly
engulfed by that noisome and rotten tide of humanity which the powers
that be are pouring eastward out of London Town. Bank, factory, hotel,
and office building must go up, and the city poor folk are a nomadic
breed; so they migrate eastward, wave upon wave, saturating and degrading
neighbourhood by neighbourhood, driving the better class of workers
before them to pioneer, on the rim of the city, or dragging them down, if
not in the first generation, surely in the second and third.
It is only a question of months when Johnny Upright's street must go. He
realises it himself.
"In a couple of years," he says, "my lease expires. My landlord is one
of our kind. He has not put up the rent on any of his houses here, and
this has enabled us to stay. But any day he may sell, or any day he may
die, which is the same thing so far as we are concerned. The house is
bought by a money breeder, who builds a sweat shop on the patch of ground
at the rear where my grapevine is, adds to the house, and rents it a room
to a family. There you are, and Johnny Upright's gone!"
And truly I saw Johnny Upright, and his good wife and fair daughters, and
frowzy slavey, like so many ghosts flitting eastward through the gloom,
the monster city roaring at their heels.
But Johnny Upright is not alone in his flitting. Far, far out, on the
fringe of the city, live the small business men, little managers, and
successful clerks. They dwell in cottages and semi-detached villas, with
bits of flower garden, and elbow room, and breathing space. They inflate
themselves with pride, and throw out their chests when they contemplate
the Abyss from which they have escaped, and they thank God that they are
not as other men. And lo! down upon them comes Johnny Upright and the
monster city at his heels. Tenements spring up like magic, gardens are
built upon, villas are divided and subdivided into many dwellings, and
the black night of London settles down in a greasy pall.
CHAPTER IV--A MAN AND THE ABYSS
"I say, can you let a lodging?"
These words I discharged carelessly over my shoulder at a stout and
elderly woman, of whose fare I was partaking in a greasy coffee-house
down near the Pool and not very far from Limehouse.
"Oh yus," she answered shortly, my appearance possibly not approximating
the standard of affluence required by her house.
I said no more, consuming my rasher of bacon and pint of sickly tea in
silence. Nor did she take further interest in me till I came to pay my
reckoning (fourpence), when I pulled all of ten shillings out of my
pocket. The expected result was produced.
"Yus, sir," she at once volunteered; "I 'ave nice lodgin's you'd likely
tyke a fancy to. Back from a voyage, sir?"
"How much for a room?" I inquired, ignoring her curiosity.
She looked me up and down with frank surprise. "I don't let rooms, not
to my reg'lar lodgers, much less casuals."
"Then I'll have to look along a bit," I said, with marked disappointment.
But the sight of my ten shillings had made her keen. "I can let you have
a nice bed in with two hother men," she urged. "Good, respectable men,
an' steady."
"But I don't want to sleep with two other men," I objected.
"You don't 'ave to. There's three beds in the room, an' hit's not a very
small room."
"How much?" I demanded.
"'Arf a crown a week, two an' six, to a regular lodger. You'll fancy the
men, I'm sure. One works in the ware'ouse, an' 'e's been with me two
years now. An' the hother's bin with me six--six years, sir, an' two
months comin' nex' Saturday. 'E's a scene-shifter," she went on. "A
steady, respectable man, never missin' a night's work in the time 'e's
bin with me. An' 'e likes the 'ouse; 'e says as it's the best 'e can do
in the w'y of lodgin's. I board 'im, an' the hother lodgers too."
"I suppose he's saving money right along," I insinuated innocently.
"Bless you, no! Nor can 'e do as well helsewhere with 'is money."
And I thought of my own spacious West, with room under its sky and
unlimited air for a thousand Londons; and here was this man, a steady and
reliable man, never missing a night's work, frugal and honest, lodging in
one room with two other men, paying two dollars and a half per month for
it, and out of his experience adjudging it to be the best he could do!
And here was I, on the strength of the ten shillings in my pocket, able
to enter in with my rags and take up my bed with him. The human soul is
a lonely thing, but it must be very lonely sometimes when there are three
beds to a room, and casuals with ten shillings are admitted.
"How long have you been here?" I asked.
"Thirteen years, sir; an' don't you think you'll fancy the lodgin'?"
The while she talked she was shuffling ponderously about the small
kitchen in which she cooked the food for her lodgers who were also
boarders. When I first entered, she had been hard at work, nor had she
let up once throughout the conversation. Undoubtedly she was a busy
woman. "Up at half-past five," "to bed the last thing at night,"
"workin' fit ter drop," thirteen years of it, and for reward, grey hairs,
frowzy clothes, stooped shoulders, slatternly figure, unending toil in a
foul and noisome coffee-house that faced on an alley ten feet between the
walls, and a waterside environment that was ugly and sickening, to say
the least.
"You'll be hin hagain to 'ave a look?" she questioned wistfully, as I
went out of the door.
And as I turned and looked at her, I realized to the full the deeper
truth underlying that very wise old maxim: "Virtue is its own reward."
I went back to her. "Have you ever taken a vacation?" I asked.
"Vycytion!"
"A trip to the country for a couple of days, fresh air, a day off, you
know, a rest."
"Lor' lumme!" she laughed, for the first time stopping from her work. "A
vycytion, eh? for the likes o' me? Just fancy, now!--Mind yer
feet!"--this last sharply, and to me, as I stumbled over the rotten
threshold.
Down near the West India Dock I came upon a young fellow staring
disconsolately at the muddy water. A fireman's cap was pulled down
across his eyes, and the fit and sag of his clothes whispered
unmistakably of the sea.
"Hello, mate," I greeted him, sparring for a beginning. "Can you tell me
the way to Wapping?"
"Worked yer way over on a cattle boat?" he countered, fixing my
nationality on the instant.
And thereupon we entered upon a talk that extended itself to a public-
house and a couple of pints of "arf an' arf." This led to closer
intimacy, so that when I brought to light all of a shilling's worth of
coppers (ostensibly my all), and put aside sixpence for a bed, and
sixpence for more arf an' arf, he generously proposed that we drink up
the whole shilling.
"My mate, 'e cut up rough las' night," he explained. "An' the bobbies
got 'm, so you can bunk in wi' me. Wotcher say?"
I said yes, and by the time we had soaked ourselves in a whole shilling's
worth of beer, and slept the night on a miserable bed in a miserable den,
I knew him pretty fairly for what he was. And that in one respect he was
representative of a large body of the lower-class London workman, my
later experience substantiates.
He was London-born, his father a fireman and a drinker before him. As a
child, his home was the streets and the docks. He had never learned to
read, and had never felt the need for it--a vain and useless
accomplishment, he held, at least for a man of his station in life.
He had had a mother and numerous squalling brothers and sisters, all
crammed into a couple of rooms and living on poorer and less regular food
than he could ordinarily rustle for himself. In fact, he never went home
except at periods when he was unfortunate in procuring his own food.
Petty pilfering and begging along the streets and docks, a trip or two to
sea as mess-boy, a few trips more as coal-trimmer, and then a
full-fledged fireman, he had reached the top of his life.
And in the course of this he had also hammered out a philosophy of life,
an ugly and repulsive philosophy, but withal a very logical and sensible
one from his point of view. When I asked him what he lived for, he
immediately answered, "Booze." A voyage to sea (for a man must live and
get the wherewithal), and then the paying off and the big drunk at the
end. After that, haphazard little drunks, sponged in the "pubs" from
mates with a few coppers left, like myself, and when sponging was played
out another trip to sea and a repetition of the beastly cycle.
"But women," I suggested, when he had finished proclaiming booze the sole
end of existence.
"Wimmen!" He thumped his pot upon the bar and orated eloquently. "Wimmen
is a thing my edication 'as learnt me t' let alone. It don't pay, matey;
it don't pay. Wot's a man like me want o' wimmen, eh? jest you tell me.
There was my mar, she was enough, a-bangin' the kids about an' makin' the
ole man mis'rable when 'e come 'ome, w'ich was seldom, I grant. An' fer
w'y? Becos o' mar! She didn't make 'is 'ome 'appy, that was w'y. Then,
there's the other wimmen, 'ow do they treat a pore stoker with a few
shillin's in 'is trouseys? A good drunk is wot 'e's got in 'is pockits,
a good long drunk, an' the wimmen skin 'im out of his money so quick 'e
ain't 'ad 'ardly a glass. I know. I've 'ad my fling, an' I know wot's
wot. An' I tell you, where's wimmen is trouble--screechin' an' carryin'
on, fightin', cuttin', bobbies, magistrates, an' a month's 'ard labour
back of it all, an' no pay-day when you come out."
"But a wife and children," I insisted. "A home of your own, and all
that. Think of it, back from a voyage, little children climbing on your
knee, and the wife happy and smiling, and a kiss for you when she lays
the table, and a kiss all round from the babies when they go to bed, and
the kettle singing and the long talk afterwards of where you've been and
what you've seen, and of her and all the little happenings at home while
you've been away, and--"
"Garn!" he cried, with a playful shove of his fist on my shoulder. "Wot's
yer game, eh? A missus kissin' an' kids clim'in', an' kettle singin',
all on four poun' ten a month w'en you 'ave a ship, an' four nothin' w'en
you 'aven't. I'll tell you wot I'd get on four poun' ten--a missus
rowin', kids squallin', no coal t' make the kettle sing, an' the kettle
up the spout, that's wot I'd get. Enough t' make a bloke bloomin' well
glad to be back t' sea. A missus! Wot for? T' make you mis'rable?
Kids? Jest take my counsel, matey, an' don't 'ave 'em. Look at me! I
can 'ave my beer w'en I like, an' no blessed missus an' kids a-crying for
bread. I'm 'appy, I am, with my beer an' mates like you, an' a good ship
comin', an' another trip to sea. So I say, let's 'ave another pint. Arf
an' arf's good enough for me."
Without going further with the speech of this young fellow of two-and-
twenty, I think I have sufficiently indicated his philosophy of life and
the underlying economic reason for it. Home life he had never known. The
word "home" aroused nothing but unpleasant associations. In the low
wages of his father, and of other men in the same walk in life, he found
sufficient reason for branding wife and children as encumbrances and
causes of masculine misery. An unconscious hedonist, utterly unmoral and
materialistic, he sought the greatest possible happiness for himself, and
found it in drink.
A young sot; a premature wreck; physical inability to do a stoker's work;
the gutter or the workhouse; and the end--he saw it all as clearly as I,
but it held no terrors for him. From the moment of his birth, all the
forces of his environment had tended to harden him, and he viewed his
wretched, inevitable future with a callousness and unconcern I could not
shake.
And yet he was not a bad man. He was not inherently vicious and brutal.
He had normal mentality, and a more than average physique. His eyes were
blue and round, shaded by long lashes, and wide apart. And there was a
laugh in them, and a fund of humour behind. The brow and general
features were good, the mouth and lips sweet, though already developing a
harsh twist. The chin was weak, but not too weak; I have seen men
sitting in the high places with weaker.
His head was shapely, and so gracefully was it poised upon a perfect neck
that I was not surprised by his body that night when he stripped for bed.
I have seen many men strip, in gymnasium and training quarters, men of
good blood and upbringing, but I have never seen one who stripped to
better advantage than this young sot of two-and-twenty, this young god
doomed to rack and ruin in four or five short years, and to pass hence
without posterity to receive the splendid heritage it was his to
bequeath.
It seemed sacrilege to waste such life, and yet I was forced to confess
that he was right in not marrying on four pounds ten in London Town. Just
as the scene-shifter was happier in making both ends meet in a room
shared with two other men, than he would have been had he packed a feeble
family along with a couple of men into a cheaper room, and failed in
making both ends meet.
And day by day I became convinced that not only is it unwise, but it is
criminal for the people of the Abyss to marry. They are the stones by
the builder rejected. There is no place for them, in the social fabric,
while all the forces of society drive them downward till they perish. At
the bottom of the Abyss they are feeble, besotted, and imbecile. If they
reproduce, the life is so cheap that perforce it perishes of itself. The
work of the world goes on above them, and they do not care to take part
in it, nor are they able. Moreover, the work of the world does not need
them. There are plenty, far fitter than they, clinging to the steep
slope above, and struggling frantically to slide no more.
In short, the London Abyss is a vast shambles. Year by year, and decade
after decade, rural England pours in a flood of vigorous strong life,
that not only does not renew itself, but perishes by the third
generation. Competent authorities aver that the London workman whose
parents and grand-parents were born in London is so remarkable a specimen
that he is rarely found.
Mr. A. C. Pigou has said that the aged poor, and the residuum which
compose the "submerged tenth," constitute 71 per cent, of the population
of London. Which is to say that last year, and yesterday, and to-day, at
this very moment, 450,000 of these creatures are dying miserably at the
bottom of the social pit called "London." As to how they die, I shall
take an instance from this morning's paper.
SELF-NEGLECT
Yesterday Dr. Wynn Westcott held an inquest at Shoreditch, respecting
the death of Elizabeth Crews, aged 77 years, of 32 East Street,
Holborn, who died on Wednesday last. Alice Mathieson stated that she
was landlady of the house where deceased lived. Witness last saw her
alive on the previous Monday. She lived quite alone. Mr. Francis
Birch, relieving officer for the Holborn district, stated that
deceased had occupied the room in question for thirty-five years. W
gitextract_fzjncbcy/
├── .editorconfig
├── .github/
│ ├── workbench-docker-version.txt
│ └── workflows/
│ ├── README.md
│ ├── docker_apply_cache.yaml
│ ├── docker_build_deploy.yaml
│ ├── docker_pr_receive.yaml
│ ├── pr-close-signal.yaml
│ ├── pr-comment.yaml
│ ├── pr-post-remove-branch.yaml
│ ├── pr-preflight.yaml
│ ├── update-cache.yaml
│ ├── update-workflows.yaml
│ └── workflows-version.txt
├── .gitignore
├── .mailmap
├── .update-copyright.conf
├── .zenodo.json
├── AUTHORS
├── CITATION
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE.md
├── README.md
├── commands.mk
├── config.yaml
├── episodes/
│ ├── 01-intro.md
│ ├── 02-makefiles.md
│ ├── 03-variables.md
│ ├── 04-dependencies.md
│ ├── 05-patterns.md
│ ├── 06-variables.md
│ ├── 07-functions.md
│ ├── 08-self-doc.md
│ ├── 09-conclusion.md
│ ├── data/
│ │ └── books/
│ │ ├── LICENSE_TEXTS.md
│ │ ├── abyss.txt
│ │ ├── isles.txt
│ │ ├── last.txt
│ │ └── sierra.txt
│ └── files/
│ └── code/
│ ├── 02-makefile/
│ │ └── Makefile
│ ├── 02-makefile-challenge/
│ │ └── Makefile
│ ├── 03-variables/
│ │ └── Makefile
│ ├── 03-variables-challenge/
│ │ └── Makefile
│ ├── 04-dependencies/
│ │ └── Makefile
│ ├── 05-patterns/
│ │ └── Makefile
│ ├── 06-variables/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 06-variables-challenge/
│ │ └── Makefile
│ ├── 07-functions/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 08-self-doc/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 09-conclusion-challenge-1/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── 09-conclusion-challenge-2/
│ │ ├── Makefile
│ │ └── config.mk
│ ├── countwords.py
│ ├── plotcounts.py
│ └── testzipf.py
├── index.md
├── instructors/
│ └── instructor-notes.md
├── learners/
│ ├── discuss.md
│ ├── reference.md
│ └── setup.md
├── profiles/
│ └── learner-profiles.md
├── requirements.txt
└── site/
└── README.md
SYMBOL INDEX (14 symbols across 3 files) FILE: episodes/files/code/countwords.py function load_text (line 8) | def load_text(filename): function save_word_counts (line 18) | def save_word_counts(filename, counts): function load_word_counts (line 28) | def load_word_counts(filename): function update_word_counts (line 43) | def update_word_counts(line, counts): function calculate_word_counts (line 61) | def calculate_word_counts(lines): function word_count_dict_to_tuples (line 74) | def word_count_dict_to_tuples(counts, decrease=True): function filter_word_counts (line 85) | def filter_word_counts(counts, min_length=1): function calculate_percentages (line 97) | def calculate_percentages(counts): function word_count (line 111) | def word_count(input_file, output_file, min_length=1): FILE: episodes/files/code/plotcounts.py function plot_word_counts (line 14) | def plot_word_counts(counts, limit=10): function typeset_labels (line 37) | def typeset_labels(labels=None, gap=5): function get_ascii_bars (line 56) | def get_ascii_bars(values, truncate=True, maxlen=10, symbol='#'): function plot_ascii_bars (line 82) | def plot_ascii_bars(values, labels=None, screenwidth=80, gap=2, truncate... FILE: episodes/files/code/testzipf.py function top_two_word (line 5) | def top_two_word(counts):
Condensed preview — 67 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (2,268K chars).
[
{
"path": ".editorconfig",
"chars": 484,
"preview": "root = true\n\n[*]\ncharset = utf-8\ninsert_final_newline = true\ntrim_trailing_whitespace = true\n\n[*.md]\nindent_size = 2\nind"
},
{
"path": ".github/workbench-docker-version.txt",
"chars": 7,
"preview": "v0.2.4\n"
},
{
"path": ".github/workflows/README.md",
"chars": 13401,
"preview": "# Workflow Documentation\n\n## Managing Workflow Updates\n\nBy using prebuilt Docker containers that are managed by the Carp"
},
{
"path": ".github/workflows/docker_apply_cache.yaml",
"chars": 7690,
"preview": "name: \"03 Maintain: Apply Package Cache\"\ndescription: \"Generate the package cache for the lesson after a pull request ha"
},
{
"path": ".github/workflows/docker_build_deploy.yaml",
"chars": 5681,
"preview": "name: \"01 Maintain: Build and Deploy Site\"\ndescription: \"Build and deploy the lesson site using the carpentries/workbenc"
},
{
"path": ".github/workflows/docker_pr_receive.yaml",
"chars": 9469,
"preview": "name: \"Bot: Receive Pull Request\"\ndescription: \"Receive a pull request and build the markdown source files\"\non:\n pull_r"
},
{
"path": ".github/workflows/pr-close-signal.yaml",
"chars": 482,
"preview": "name: \"Bot: Send Close Pull Request Signal\"\n\non:\n pull_request:\n types:\n [closed]\n\njobs:\n send-close-signal:\n "
},
{
"path": ".github/workflows/pr-comment.yaml",
"chars": 7114,
"preview": "name: \"Bot: Comment on the Pull Request\"\ndescription: \"Comment on the pull request with the results of the markdown gene"
},
{
"path": ".github/workflows/pr-post-remove-branch.yaml",
"chars": 870,
"preview": "name: \"Bot: Remove Temporary PR Branch\"\n\non:\n workflow_run:\n workflows: [\"Bot: Send Close Pull Request Signal\"]\n "
},
{
"path": ".github/workflows/pr-preflight.yaml",
"chars": 1125,
"preview": "name: \"Pull Request Preflight Check\"\n\non:\n pull_request_target:\n branches:\n [\"main\"]\n types:\n [\"opened\""
},
{
"path": ".github/workflows/update-cache.yaml",
"chars": 6734,
"preview": "name: \"02 Maintain: Check for Updated Packages\"\ndescription: \"Check for updated R packages and create a pull request to "
},
{
"path": ".github/workflows/update-workflows.yaml",
"chars": 4278,
"preview": "name: \"04 Maintain: Update Workflow Files\"\ndescription: \"Update workflow files from the carpentries/sandpaper repository"
},
{
"path": ".github/workflows/workflows-version.txt",
"chars": 6,
"preview": "1.0.1\n"
},
{
"path": ".gitignore",
"chars": 875,
"preview": "# sandpaper files\nepisodes/*html\nsite/*\n!site/README.md\n\n# History files\n.Rhistory\n.Rapp.history\n# Session Data files\n.R"
},
{
"path": ".mailmap",
"chars": 1090,
"preview": "Abigail Cabunoc Mayes <abigail.cabunoc@gmail.com>\nAbigail Cabunoc Mayes <abigail.cabunoc@gmail.com> <abigail.cabunoc@oic"
},
{
"path": ".update-copyright.conf",
"chars": 51,
"preview": "[project]\nvcs: Git\n\n[files]\nauthors: yes\nfiles: no\n"
},
{
"path": ".zenodo.json",
"chars": 425,
"preview": "{\n \"contributors\": [\n {\n \"type\": \"Editor\",\n \"name\": \"Gerard Capes\"\n }\n ],\n \"creators\": [\n {\n "
},
{
"path": "AUTHORS",
"chars": 268,
"preview": "Pete Bachant\nMaxime Boissonneault\nGerard Capes\nDeborah Digges\nAndrew Fraser\nLuiz Irber\nMike Jackson\nGang Liu\nLex Nederbr"
},
{
"path": "CITATION",
"chars": 174,
"preview": "Please cite as:\n\nMike Jackson (ed.): \"Software Carpentry: Automation and Make.\"\nVersion 2016.06, June 2016,\nhttps://gith"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 475,
"preview": "---\ntitle: \"Contributor Code of Conduct\"\n---\n\nAs contributors and maintainers of this project,\nwe pledge to follow the ["
},
{
"path": "CONTRIBUTING.md",
"chars": 5919,
"preview": "## Contributing\n\n[The Carpentries][cp-site] ([Software Carpentry][swc-site], [Data\nCarpentry][dc-site], and [Library Car"
},
{
"path": "LICENSE.md",
"chars": 3345,
"preview": "---\ntitle: \"Licenses\"\n---\n\n## Instructional Material\n\nAll Carpentries (Software Carpentry, Data Carpentry, and Library C"
},
{
"path": "README.md",
"chars": 984,
"preview": "[](https://doi.org/10.5281/zenodo.3265286)\n[![Create a Sl"
},
{
"path": "commands.mk",
"chars": 1079,
"preview": "## ----------------------------------------\nMAKE2PNG = make2graph | grep -v Makefile | grep -v config.mk | grep -v comma"
},
{
"path": "config.yaml",
"chars": 2045,
"preview": "#------------------------------------------------------------\n# Values for this lesson.\n#-------------------------------"
},
{
"path": "episodes/01-intro.md",
"chars": 10403,
"preview": "---\ntitle: Introduction\nteaching: 25\nexercises: 0\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Explain wha"
},
{
"path": "episodes/02-makefiles.md",
"chars": 12069,
"preview": "---\ntitle: Makefiles\nteaching: 30\nexercises: 10\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Recognize the"
},
{
"path": "episodes/03-variables.md",
"chars": 5528,
"preview": "---\ntitle: Automatic Variables\nteaching: 10\nexercises: 5\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Use "
},
{
"path": "episodes/04-dependencies.md",
"chars": 7380,
"preview": "---\ntitle: Dependencies on Data and Code\nteaching: 15\nexercises: 5\n---\n\n::::::::::::::::::::::::::::::::::::::: objectiv"
},
{
"path": "episodes/05-patterns.md",
"chars": 2661,
"preview": "---\ntitle: Pattern Rules\nteaching: 10\nexercises: 0\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Write Make"
},
{
"path": "episodes/06-variables.md",
"chars": 3777,
"preview": "---\ntitle: Variables\nteaching: 15\nexercises: 5\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Use variables "
},
{
"path": "episodes/07-functions.md",
"chars": 6755,
"preview": "---\ntitle: Functions\nteaching: 20\nexercises: 5\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Write Makefile"
},
{
"path": "episodes/08-self-doc.md",
"chars": 4028,
"preview": "---\ntitle: Self-Documenting Makefiles\nteaching: 10\nexercises: 0\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n"
},
{
"path": "episodes/09-conclusion.md",
"chars": 6190,
"preview": "---\ntitle: Conclusion\nteaching: 5\nexercises: 30\n---\n\n::::::::::::::::::::::::::::::::::::::: objectives\n\n- Understand ad"
},
{
"path": "episodes/data/books/LICENSE_TEXTS.md",
"chars": 18043,
"preview": "A Note on the Texts' Licensing\n==============================\n\nEach text is from [Project Gutenberg](http://www.gutenber"
},
{
"path": "episodes/data/books/abyss.txt",
"chars": 348929,
"preview": "THE PEOPLE OF THE ABYSS\n\n\nThe chief priests and rulers cry:-\n\n \"O Lord and Master, not ours the guilt,\n We build but"
},
{
"path": "episodes/data/books/isles.txt",
"chars": 323972,
"preview": "A JOURNEY TO THE WESTERN ISLANDS OF SCOTLAND\n\n\nINCH KEITH\n\n\nI had desired to visit the Hebrides, or Western Islands of S"
},
{
"path": "episodes/data/books/last.txt",
"chars": 1005781,
"preview": " SCOTT'S LAST EXPEDITION\n\n IN TWO VOLUMES\n\n VOL. "
},
{
"path": "episodes/data/books/sierra.txt",
"chars": 345623,
"preview": "THE WRITINGS OF JOHN MUIR\n\nSierra Edition\n\nVOLUME II\n\n[Illustration: _The Yosemite Falls, Yosemite National Park_]\n\n\n\n\nM"
},
{
"path": "episodes/files/code/02-makefile/Makefile",
"chars": 248,
"preview": "# Count words.\n.PHONY : dats\ndats : isles.dat abyss.dat\n\nisles.dat : books/isles.txt\n\tpython countwords.py books/isles.t"
},
{
"path": "episodes/files/code/02-makefile-challenge/Makefile",
"chars": 481,
"preview": "# Generate summary table.\nresults.txt : isles.dat abyss.dat last.dat\n\tpython testzipf.py abyss.dat isles.dat last.dat > "
},
{
"path": "episodes/files/code/03-variables/Makefile",
"chars": 423,
"preview": "# Generate summary table.\nresults.txt : *.dat\n\tpython testzipf.py $^ > $@\n\n# Count words.\n.PHONY : dats\ndats : isles.dat"
},
{
"path": "episodes/files/code/03-variables-challenge/Makefile",
"chars": 388,
"preview": "# Generate summary table.\nresults.txt : isles.dat abyss.dat last.dat\n\tpython testzipf.py $^ > $@\n\n# Count words.\n.PHONY "
},
{
"path": "episodes/files/code/04-dependencies/Makefile",
"chars": 430,
"preview": "# Generate summary table.\nresults.txt : testzipf.py isles.dat abyss.dat last.dat\n\tpython $^ > $@\n\n# Count words.\n.PHONY "
},
{
"path": "episodes/files/code/05-patterns/Makefile",
"chars": 268,
"preview": "# Generate summary table.\nresults.txt : testzipf.py isles.dat abyss.dat last.dat\n\tpython $^ > $@\n\n# Count words.\n.PHONY "
},
{
"path": "episodes/files/code/06-variables/Makefile",
"chars": 293,
"preview": "include config.mk\n\n# Generate summary table.\nresults.txt : $(ZIPF_SRC) isles.dat abyss.dat last.dat\n\t$(LANGUAGE) $^ > $@"
},
{
"path": "episodes/files/code/06-variables/config.mk",
"chars": 103,
"preview": "# Count words script.\nLANGUAGE=python\nCOUNT_SRC=countwords.py\n\n# Test Zipf's rule\nZIPF_SRC=testzipf.py\n"
},
{
"path": "episodes/files/code/06-variables-challenge/Makefile",
"chars": 339,
"preview": "LANGUAGE=python\nCOUNT_SRC=countwords.py\nZIPF_SRC=testzipf.py\n\n# Generate summary table.\nresults.txt : $(ZIPF_SRC) isles."
},
{
"path": "episodes/files/code/07-functions/Makefile",
"chars": 454,
"preview": "include config.mk\n\nTXT_FILES=$(wildcard books/*.txt)\nDAT_FILES=$(patsubst books/%.txt, %.dat, $(TXT_FILES))\n\n# Generate "
},
{
"path": "episodes/files/code/07-functions/config.mk",
"chars": 103,
"preview": "# Count words script.\nLANGUAGE=python\nCOUNT_SRC=countwords.py\n\n# Test Zipf's rule\nZIPF_SRC=testzipf.py\n"
},
{
"path": "episodes/files/code/08-self-doc/Makefile",
"chars": 648,
"preview": "include config.mk\n\nTXT_FILES=$(wildcard books/*.txt)\nDAT_FILES=$(patsubst books/%.txt, %.dat, $(TXT_FILES))\n\n## results."
},
{
"path": "episodes/files/code/08-self-doc/config.mk",
"chars": 171,
"preview": "# Count words script.\nLANGUAGE=python\nCOUNT_SRC=countwords.py\nCOUNT_EXE=$(LANGUAGE) $(COUNT_SRC)\n\n# Test Zipf's rule\nZIP"
},
{
"path": "episodes/files/code/09-conclusion-challenge-1/Makefile",
"chars": 975,
"preview": "include config.mk\n\nTXT_FILES=$(wildcard books/*.txt)\nDAT_FILES=$(patsubst books/%.txt, %.dat, $(TXT_FILES))\nPNG_FILES=$("
},
{
"path": "episodes/files/code/09-conclusion-challenge-1/config.mk",
"chars": 154,
"preview": "# Count words script.\nLANGUAGE=python\nCOUNT_SRC=countwords.py\n\n# Plot word counts script.\nPLOT_SRC=plotcounts.py\n\n# Test"
},
{
"path": "episodes/files/code/09-conclusion-challenge-2/Makefile",
"chars": 1485,
"preview": "include config.mk\n\nTXT_DIR=books\nTXT_FILES=$(wildcard $(TXT_DIR)/*.txt)\nDAT_FILES=$(patsubst $(TXT_DIR)/%.txt, %.dat, $("
},
{
"path": "episodes/files/code/09-conclusion-challenge-2/config.mk",
"chars": 155,
"preview": "# Count words script.\nLANGUAGE=python\nCOUNT_SRC=countwords.py\n\n# Plot word counts script.\nPLOT_SRC=plotcounts.py\n\n# Test"
},
{
"path": "episodes/files/code/countwords.py",
"chars": 4164,
"preview": "#!/usr/bin/env python\n\nimport sys\n\nDELIMITERS = \". , ; : ? $ @ ^ < > # % ` ! * - = ( ) [ ] { } / \\\" '\".split()\n\n\ndef loa"
},
{
"path": "episodes/files/code/plotcounts.py",
"chars": 3711,
"preview": "#!/usr/bin/env python\n\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport sys\ntry:\n from collections.abc impor"
},
{
"path": "episodes/files/code/testzipf.py",
"chars": 669,
"preview": "#!/usr/bin/env python\nfrom countwords import load_word_counts\nimport sys\n\ndef top_two_word(counts):\n \"\"\"\n Given a "
},
{
"path": "index.md",
"chars": 1610,
"preview": "---\npermalink: index.html\nsite: sandpaper::sandpaper_site\n---\n\nMake is a tool which can run commands to read files, proc"
},
{
"path": "instructors/instructor-notes.md",
"chars": 4803,
"preview": "---\ntitle: Instructor Notes\n---\n\nMake is a popular tool for automating the building of software -\ncompiling source code "
},
{
"path": "learners/discuss.md",
"chars": 8853,
"preview": "---\ntitle: Discussion\n---\n\n## Parallel Execution\n\nMake can build dependencies in *parallel* sub-processes, via its `--jo"
},
{
"path": "learners/reference.md",
"chars": 10007,
"preview": "---\ntitle: 'FIXME'\n---\n\n## Glossary\n\n## Running Make\n\nTo run Make:\n\n```bash\n$ make\n```\n\nMake will look for a Makefile ca"
},
{
"path": "learners/setup.md",
"chars": 1464,
"preview": "---\ntitle: Setup\n---\n\n## Files\n\nYou need to download some files to follow this lesson:\n\n1. Download [make-lesson.zip][zi"
},
{
"path": "profiles/learner-profiles.md",
"chars": 76,
"preview": "---\ntitle: FIXME\n---\n\nThis is a placeholder file. Please add content here. \n"
},
{
"path": "requirements.txt",
"chars": 24,
"preview": "PyYAML\nupdate-copyright\n"
},
{
"path": "site/README.md",
"chars": 84,
"preview": "This directory contains rendered lesson materials. Please do not edit files\nhere. \n"
}
]
About this extraction
This page contains the full source code of the swcarpentry/make-novice GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 67 files (2.1 MB), approximately 557.6k tokens, and a symbol index with 14 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.