Full Code of spotify/luigi for AI

master 0be72e20a2e6 cached
329 files
2.5 MB
669.9k tokens
5364 symbols
1 requests
Download .txt
Showing preview only (2,674K chars total). Download the full file or copy to clipboard to get everything.
Repository: spotify/luigi
Branch: master
Commit: 0be72e20a2e6
Files: 329
Total size: 2.5 MB

Directory structure:
gitextract_i5ice6bg/

├── .coveragerc
├── .github/
│   ├── CODEOWNERS
│   ├── ISSUE_TEMPLATE.md
│   ├── PULL_REQUEST_TEMPLATE.md
│   ├── stale.yml
│   └── workflows/
│       ├── codeql.yml
│       └── pythonbuild.yml
├── .gitignore
├── .readthedocs.yaml
├── CONTRIBUTING.rst
├── LICENSE
├── README.rst
├── RELEASE-PROCESS.rst
├── SECURITY.md
├── bin/
│   ├── luigi
│   └── luigid
├── catalog-info.yaml
├── codecov.yml
├── doc/
│   ├── .gitignore
│   ├── Makefile
│   ├── central_scheduler.rst
│   ├── conf.py
│   ├── configuration.rst
│   ├── design_and_limitations.rst
│   ├── example_top_artists.rst
│   ├── execution_model.rst
│   ├── index.rst
│   ├── logging.rst
│   ├── luigi_patterns.rst
│   ├── mypy.rst
│   ├── parameters.rst
│   ├── running_luigi.rst
│   ├── tasks.rst
│   └── workflows.rst
├── examples/
│   ├── __init__.py
│   ├── config.toml
│   ├── dynamic_requirements.py
│   ├── elasticsearch_index.py
│   ├── execution_summary_example.py
│   ├── foo.py
│   ├── foo_complex.py
│   ├── ftp_experiment_outputs.py
│   ├── hello_world.py
│   ├── kubernetes.py
│   ├── per_task_retry_policy.py
│   ├── pyspark_wc.py
│   ├── spark_als.py
│   ├── ssh_remote_execution.py
│   ├── terasort.py
│   ├── top_artists.py
│   ├── top_artists_spark.py
│   ├── wordcount.py
│   └── wordcount_hadoop.py
├── luigi/
│   ├── __init__.py
│   ├── __main__.py
│   ├── __version__.py
│   ├── batch_notifier.py
│   ├── cmdline.py
│   ├── cmdline_parser.py
│   ├── configuration/
│   │   ├── __init__.py
│   │   ├── base_parser.py
│   │   ├── cfg_parser.py
│   │   ├── core.py
│   │   └── toml_parser.py
│   ├── contrib/
│   │   ├── __init__.py
│   │   ├── azureblob.py
│   │   ├── batch.py
│   │   ├── beam_dataflow.py
│   │   ├── bigquery.py
│   │   ├── bigquery_avro.py
│   │   ├── datadog_metric.py
│   │   ├── dataproc.py
│   │   ├── docker_runner.py
│   │   ├── dropbox.py
│   │   ├── ecs.py
│   │   ├── esindex.py
│   │   ├── external_daily_snapshot.py
│   │   ├── external_program.py
│   │   ├── ftp.py
│   │   ├── gcp.py
│   │   ├── gcs.py
│   │   ├── hadoop.py
│   │   ├── hadoop_jar.py
│   │   ├── hdfs/
│   │   │   ├── __init__.py
│   │   │   ├── abstract_client.py
│   │   │   ├── clients.py
│   │   │   ├── config.py
│   │   │   ├── error.py
│   │   │   ├── format.py
│   │   │   ├── hadoopcli_clients.py
│   │   │   ├── target.py
│   │   │   └── webhdfs_client.py
│   │   ├── hive.py
│   │   ├── kubernetes.py
│   │   ├── lsf.py
│   │   ├── lsf_runner.py
│   │   ├── mongodb.py
│   │   ├── mssqldb.py
│   │   ├── mysqldb.py
│   │   ├── opener.py
│   │   ├── pai.py
│   │   ├── pig.py
│   │   ├── postgres.py
│   │   ├── presto.py
│   │   ├── prometheus_metric.py
│   │   ├── pyspark_runner.py
│   │   ├── rdbms.py
│   │   ├── redis_store.py
│   │   ├── redshift.py
│   │   ├── s3.py
│   │   ├── salesforce.py
│   │   ├── scalding.py
│   │   ├── sge.py
│   │   ├── sge_runner.py
│   │   ├── simulate.py
│   │   ├── spark.py
│   │   ├── sparkey.py
│   │   ├── sqla.py
│   │   ├── ssh.py
│   │   ├── target.py
│   │   └── webhdfs.py
│   ├── date_interval.py
│   ├── db_task_history.py
│   ├── event.py
│   ├── execution_summary.py
│   ├── format.py
│   ├── freezing.py
│   ├── interface.py
│   ├── local_target.py
│   ├── lock.py
│   ├── metrics.py
│   ├── mock.py
│   ├── mypy.py
│   ├── notifications.py
│   ├── parameter.py
│   ├── process.py
│   ├── py.typed
│   ├── retcodes.py
│   ├── rpc.py
│   ├── safe_extractor.py
│   ├── scheduler.py
│   ├── server.py
│   ├── setup_logging.py
│   ├── static/
│   │   └── visualiser/
│   │       ├── css/
│   │       │   ├── luigi.css
│   │       │   └── tipsy.css
│   │       ├── fonts/
│   │       │   └── FontAwesome.otf
│   │       ├── index.html
│   │       ├── js/
│   │       │   ├── graph.js
│   │       │   ├── luigi.js
│   │       │   ├── test/
│   │       │   │   └── graph_test.js
│   │       │   ├── tipsy.js
│   │       │   ├── util.js
│   │       │   └── visualiserApp.js
│   │       ├── lib/
│   │       │   ├── URI/
│   │       │   │   └── 1.18.2/
│   │       │   │       └── URI.js
│   │       │   ├── datatables/
│   │       │   │   └── images/
│   │       │   │       └── Sorting icons.psd
│   │       │   └── mustache.js
│   │       ├── mockdata/
│   │       │   ├── dep_graph
│   │       │   ├── fetch_error
│   │       │   └── task_list
│   │       └── test.html
│   ├── target.py
│   ├── task.py
│   ├── task_history.py
│   ├── task_register.py
│   ├── task_status.py
│   ├── templates/
│   │   ├── history.html
│   │   ├── layout.html
│   │   ├── menu.html
│   │   ├── recent.html
│   │   └── show.html
│   ├── tools/
│   │   ├── __init__.py
│   │   ├── deps.py
│   │   ├── deps_tree.py
│   │   ├── luigi_grep.py
│   │   └── range.py
│   ├── util.py
│   └── worker.py
├── pyproject.toml
├── scripts/
│   └── ci/
│       ├── conditional_tox.sh
│       ├── install_start_azurite.sh
│       ├── setup_hadoop_env.sh
│       └── stop_azurite.sh
├── test/
│   ├── _mysqldb_test.py
│   ├── _test_ftp.py
│   ├── auto_namespace_test/
│   │   ├── __init__.py
│   │   └── my_namespace_test.py
│   ├── batch_notifier_test.py
│   ├── choice_parameter_test.py
│   ├── clone_test.py
│   ├── cmdline_test.py
│   ├── config_env_test.py
│   ├── config_toml_test.py
│   ├── conftest.py
│   ├── contrib/
│   │   ├── __init__.py
│   │   ├── _webhdfs_test.py
│   │   ├── azureblob_test.py
│   │   ├── batch_test.py
│   │   ├── beam_dataflow_test.py
│   │   ├── bigquery_avro_test.py
│   │   ├── bigquery_gcloud_test.py
│   │   ├── bigquery_test.py
│   │   ├── cascading_test.py
│   │   ├── datadog_metric_test.py
│   │   ├── dataproc_test.py
│   │   ├── docker_runner_test.py
│   │   ├── dropbox_test.py
│   │   ├── ecs_test.py
│   │   ├── esindex_test.py
│   │   ├── external_daily_snapshot_test.py
│   │   ├── external_program_test.py
│   │   ├── gcs_test.py
│   │   ├── hadoop_jar_test.py
│   │   ├── hdfs/
│   │   │   └── webhdfs_client_test.py
│   │   ├── hdfs_test.py
│   │   ├── hive_test.py
│   │   ├── kubernetes_test.py
│   │   ├── lsf_test.py
│   │   ├── mongo_test.py
│   │   ├── mysqldb_test.py
│   │   ├── opener_test.py
│   │   ├── pai_test.py
│   │   ├── pig_test.py
│   │   ├── postgres_test.py
│   │   ├── postgres_with_server_test.py
│   │   ├── presto_test.py
│   │   ├── prometheus_metric_test.py
│   │   ├── rdbms_test.py
│   │   ├── redis_test.py
│   │   ├── redshift_test.py
│   │   ├── s3_test.py
│   │   ├── salesforce_test.py
│   │   ├── scalding_test.py
│   │   ├── sge_test.py
│   │   ├── spark_test.py
│   │   ├── sqla_test.py
│   │   ├── streaming_test.py
│   │   └── test_ssh.py
│   ├── create_packages_archive_root/
│   │   ├── module.py
│   │   └── package/
│   │       ├── __init__.py
│   │       ├── submodule.py
│   │       ├── submodule_with_absolute_import.py
│   │       ├── submodule_without_imports.py
│   │       └── subpackage/
│   │           ├── __init__.py
│   │           └── submodule.py
│   ├── custom_metrics_test.py
│   ├── customized_run_test.py
│   ├── date_interval_test.py
│   ├── date_parameter_test.py
│   ├── db_task_history_test.py
│   ├── decorator_test.py
│   ├── dict_parameter_test.py
│   ├── dynamic_import_test.py
│   ├── event_callbacks_test.py
│   ├── execution_summary_test.py
│   ├── factorial_test.py
│   ├── fib_test.py
│   ├── gcloud-credentials.json.enc
│   ├── hdfs_client_test.py
│   ├── helpers.py
│   ├── helpers_test.py
│   ├── import_test.py
│   ├── instance_test.py
│   ├── instance_wrap_test.py
│   ├── interface_test.py
│   ├── list_parameter_test.py
│   ├── local_target_test.py
│   ├── lock_test.py
│   ├── metrics_test.py
│   ├── mock_test.py
│   ├── most_common_test.py
│   ├── mypy_test.py
│   ├── notifications_test.py
│   ├── numerical_parameter_test.py
│   ├── optional_parameter_test.py
│   ├── other_module.py
│   ├── parameter_test.py
│   ├── priority_test.py
│   ├── range_test.py
│   ├── recursion_test.py
│   ├── remote_scheduler_test.py
│   ├── retcodes_test.py
│   ├── rpc_test.py
│   ├── runtests.py
│   ├── safe_extractor_test.py
│   ├── scheduler_api_test.py
│   ├── scheduler_message_test.py
│   ├── scheduler_parameter_visibilities_test.py
│   ├── scheduler_test.py
│   ├── scheduler_visualisation_test.py
│   ├── server_test.py
│   ├── set_task_name_test.py
│   ├── setup_logging_test.py
│   ├── simulate_test.py
│   ├── subtask_test.py
│   ├── target_test.py
│   ├── task_bulk_complete_test.py
│   ├── task_forwarded_attributes_test.py
│   ├── task_history_test.py
│   ├── task_progress_percentage_test.py
│   ├── task_register_test.py
│   ├── task_running_resources_test.py
│   ├── task_serialize_test.py
│   ├── task_status_message_test.py
│   ├── task_test.py
│   ├── test_sigpipe.py
│   ├── test_ssh.py
│   ├── testconfig/
│   │   ├── core-site.xml
│   │   ├── log4j.properties
│   │   ├── logging.cfg
│   │   ├── luigi.toml
│   │   ├── luigi_local.toml
│   │   ├── luigi_logging.toml
│   │   └── pyproject.toml
│   ├── util_previous_test.py
│   ├── util_test.py
│   ├── visible_parameters_test.py
│   ├── visualiser/
│   │   ├── __init__.py
│   │   ├── phantomjs_test.js
│   │   └── visualiser_test.py
│   ├── worker_external_task_test.py
│   ├── worker_keep_alive_test.py
│   ├── worker_multiprocess_test.py
│   ├── worker_parallel_scheduling_test.py
│   ├── worker_scheduler_com_test.py
│   ├── worker_task_process_test.py
│   ├── worker_task_test.py
│   ├── worker_test.py
│   └── wrap_test.py
└── tox.ini

================================================
FILE CONTENTS
================================================

================================================
FILE: .coveragerc
================================================
[report]
omit =
    luigi/mrrunner.py
    test/_test_time_generated_module*.py
    */python?.?/*
    */site-packages/nose/*
    *__init__*
    *test/*
    */.tox/*
    */setup.py
    */bin/luigidc
    hadoop_test.py
    minicluster.py

[run]
parallel=True
concurrency=multiprocessing


================================================
FILE: .github/CODEOWNERS
================================================
# The following patterns are used to auto-assign review requests
# to specific individuals. Order is important; the last matching
# pattern takes the most precedence.

# These owners will be the default owners for everything in
# the repo. Unless a later match takes precedence,
* @dlstadther @spotify/dataex

# Specific files, directories, paths, or file types can be
# assigned more specificially.
contrib/redshift*.py @dlstadther



================================================
FILE: .github/ISSUE_TEMPLATE.md
================================================
<!---
We use GitHub issues mainly for tracking bugs and feature requests.
Questions for how to use luigi can be sent to the mailing list.

Currently, there are no strict procedures or guidelines for submitting issues.
In short, please just use common sense.

Common sense includes this at bare-minimum:

 * search for similar issues posted before creating a new issue.
 * Use markdown to format all code/logs. Issues which are hard to read
   when rendered on GitHub might be closed with a friendly reminder of this.
 * If applicable, reading relevant parts of the documentation.

Also, add steps to reproduce the bug, if applicable. Sample code would be nice too :)

For more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->


================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
<!--- This template is optional. Please use it as a starting point to help guide PRs -->

<!--- Provide a general summary of your changes in the Title above -->

## Description
<!--- Describe your changes -->

## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->

## Have you tested this? If so, how?
<!--- Valid responses are "I have included unit tests." or --> 
<!--- "I ran my jobs with this code and it works for me." -->

<!---
for more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->


================================================
FILE: .github/stale.yml
================================================
# Number of days of inactivity before an issue becomes stale
daysUntilStale: 120
# Number of days of inactivity before a stale issue is closed
daysUntilClose: 14
# Issues with these labels will never be considered stale
exemptLabels:
  - pinned
  - security
# Label to use when marking an issue as stale
staleLabel: wontfix
# Comment to post when marking an issue as stale. Set to `false` to disable
markComment: >
  This issue has been automatically marked as stale because it has not had
  recent activity. It will be closed if no further activity occurs.
  If closed, you may revisit when your time allows and reopen!
  Thank you for your contributions.
# Comment to post when closing a stale issue. Set to `false` to disable
closeComment: false
# Limit to only `issues` or `pulls`
# only: issues


================================================
FILE: .github/workflows/codeql.yml
================================================
name: "CodeQL"

on:
  push:
    branches: [ 'master' ]
  pull_request:
    # The branches below must be a subset of the branches above
    branches: [ 'master' ]
  schedule:
    - cron: '29 18 * * 0'

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    permissions:
      actions: read
      contents: read
      security-events: write

    strategy:
      fail-fast: false
      matrix:
        language: [ 'python', 'javascript' ]
        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
        # Use only 'java' to analyze code written in Java, Kotlin or both
        # Use only 'javascript' to analyze code written in JavaScript, TypeScript or both
        # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v2
      with:
        languages: ${{ matrix.language }}
        # If you wish to specify custom queries, you can do so here or in a config file.
        # By default, queries listed here will override any specified in a config file.
        # Prefix the list here with "+" to use these queries and those in the config file.

        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
        # queries: security-extended,security-and-quality


    # Autobuild attempts to build any compiled languages (C/C++, C#, Go, Java, or Swift).
    # If this step fails, then you should remove it and run the build manually (see below)
    - name: Autobuild
      uses: github/codeql-action/autobuild@v2

    # ℹ️ Command-line programs to run using the OS shell.
    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun

    #   If the Autobuild fails above, remove it and uncomment the following three lines.
    #   modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance.

    # - run: |
    #     echo "Run, Build Application using script"
    #     ./location_of_script_within_repo/buildscript.sh

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v2
      with:
        category: "/language:${{matrix.language}}"


================================================
FILE: .github/workflows/pythonbuild.yml
================================================
name: Build

on:
  push:
    branches:
      - master
  pull_request:

jobs:
  core:
    runs-on: ubuntu-22.04

    strategy:
      matrix:
        include:
          - tox-env: py310-core
          - tox-env: py311-core
          - tox-env: py312-core
          - tox-env: py313-core

    steps:
      - uses: actions/checkout@v6
      - name: Set up the latest version of uv
        uses: astral-sh/setup-uv@v7
        with:
          enable-cache: true
          cache-dependency-glob: "pyproject.toml"
      - name: Install dependencies
        run: |
          uv tool install --python-preference only-managed --python 3.12 tox --with tox-uv
      - name: Build
        env:
          TOXENV: ${{ matrix.tox-env }}
        run: uvx --with tox-uv tox run
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v5
        with:
          fail_ci_if_error: true
          verbose: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

  mysql:
    runs-on: ubuntu-22.04

    strategy:
      matrix:
        include:
          - tox-env: py310-mysql
          - tox-env: py311-mysql
          - tox-env: py312-mysql
          - tox-env: py313-mysql

    steps:
      - uses: actions/checkout@v6
      - name: Set up the latest version of uv
        uses: astral-sh/setup-uv@v7
        with:
          enable-cache: true
          cache-dependency-glob: "pyproject.toml"
      - name: Install dependencies
        run: |
          uv tool install --python-preference only-managed --python 3.12 tox --with tox-uv
      - name: Setup MySQL DB
        run: |
          sudo /etc/init.d/mysql start
          mysql -e 'create database IF NOT EXISTS luigi_test;' -uroot -proot || true
          mysql -e 'create user 'travis'@'localhost';' -uroot -proot || true
          mysql -e 'grant all privileges ON *.* TO 'travis'@'localhost';' -uroot -proot || true
      - name: Build
        env:
          TOXENV: ${{ matrix.tox-env }}
        run: uvx --with tox-uv tox run
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v5
        with:
          fail_ci_if_error: true
          verbose: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

  postgres:
    runs-on: ubuntu-22.04
    services:
      postgres:
        image: postgres
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: postgres
        ports:
        - 5432:5432
        # Set health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    strategy:
      matrix:
        include:
          - tox-env: py310-postgres
          - tox-env: py311-postgres
          - tox-env: py312-postgres
          - tox-env: py313-postgres

    steps:
      - uses: actions/checkout@v6
      - name: Set up the latest version of uv
        uses: astral-sh/setup-uv@v7
        with:
          enable-cache: true
          cache-dependency-glob: "pyproject.toml"
      - name: Install dependencies
        run: |
          uv tool install --python-preference only-managed --python 3.12 tox --with tox-uv
      - name: Create PSQL database
        run: |
          PGPASSWORD=postgres psql -h localhost -p 5432 -c 'create database spotify;' -U postgres
      - name: Build
        env:
          TOXENV: ${{ matrix.tox-env }}
        run: uvx --with tox-uv tox run
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v5
        with:
          fail_ci_if_error: true
          verbose: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

  base:
    runs-on: ubuntu-22.04
    env:
      AWS_DEFAULT_REGION: us-east-1
      AWS_ACCESS_KEY_ID: accesskey
      AWS_SECRET_ACCESS_KEY: secretkey

    strategy:
      matrix:
        include:
          - tox-env: py310-aws
          - tox-env: py311-aws
          - tox-env: py312-aws
          - tox-env: py313-aws

          - tox-env: py310-unixsocket
            OVERRIDE_SKIP_CI_TESTS: True
          - tox-env: py311-unixsocket
            OVERRIDE_SKIP_CI_TESTS: True
          - tox-env: py312-unixsocket
            OVERRIDE_SKIP_CI_TESTS: True
          - tox-env: py313-unixsocket
            OVERRIDE_SKIP_CI_TESTS: True

          - tox-env: py310-apache
          - tox-env: py311-apache
          - tox-env: py312-apache
          - tox-env: py313-apache

          - tox-env: py310-azureblob
          - tox-env: py311-azureblob
          - tox-env: py312-azureblob
          - tox-env: py313-azureblob

          - tox-env: py310-contrib
          - tox-env: py311-contrib
          - tox-env: py312-contrib
          - tox-env: py313-contrib

    steps:
      - uses: actions/checkout@v6
      - name: Set up the latest version of uv
        uses: astral-sh/setup-uv@v7
        with:
          enable-cache: true
          cache-dependency-glob: "pyproject.toml"
      - name: Install dependencies
        run: |
          uv tool install --python-preference only-managed --python 3.12 tox --with tox-uv
      - name: Build
        env:
          TOXENV: ${{ matrix.tox-env }}
          OVERRIDE_SKIP_CI_TESTS: ${{ matrix.OVERRIDE_SKIP_CI_TESTS }}
        run: uvx --with tox-uv tox run
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v5
        with:
          fail_ci_if_error: true
          verbose: true
        env:
          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

  others:
    runs-on: ubuntu-22.04

    strategy:
      matrix:
        include:
          - tox-env: lint
          - tox-env: docs
          - tox-env: typecheck
    steps:
      - uses: actions/checkout@v6
      - name: Set up the latest version of uv
        uses: astral-sh/setup-uv@v7
        with:
          enable-cache: true
          cache-dependency-glob: "pyproject.toml"
      - name: Install dependencies
        run: |
          uv tool install --python-preference only-managed --python 3.12 tox --with tox-uv
      - name: Build
        env:
          TOXENV: ${{ matrix.tox-env }}
          OVERRIDE_SKIP_CI_TESTS: ${{ matrix.OVERRIDE_SKIP_CI_TESTS }}
        run: uvx --with tox-uv tox run


================================================
FILE: .gitignore
================================================
.coverage.*
doc/api/*.rst
test/gcloud-credentials.json
.hypothesis/

.nicesetup

client.cfg
luigi.cfg

hadoop_test.py
minicluster.py
mrrunner.py
pig_property_file

packages.tar

# Ignore the data files
data
test/data
examples/data

Vagrantfile

*.pickle
*.rej
*.orig

# Created by https://www.gitignore.io

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
# NOTE : lib/ prevents inclusion of static/visualiser/lib
#lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
my_dir

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
doc/_build/

# PyBuilder
target/


### Vim ###
[._]*.s[a-w][a-z]
[._]s[a-w][a-z]
*.un~
Session.vim
.netrwhist
*~


### PyCharm ###
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm

*.iml

## Directory-based project format:
.idea/
# if you remove the above rule, at least ignore the following:

# User-specific stuff:
# .idea/workspace.xml
# .idea/tasks.xml
# .idea/dictionaries

# Sensitive or high-churn files:
# .idea/dataSources.ids
# .idea/dataSources.xml
# .idea/sqlDataSources.xml
# .idea/dynamic.xml
# .idea/uiDesigner.xml

# Gradle:
# .idea/gradle.xml
# .idea/libraries

# Mongo Explorer plugin:
# .idea/mongoSettings.xml

## File-based project format:
*.ipr
*.iws

## Plugin-specific files:

# IntelliJ
out/

# mpeltonen/sbt-idea plugin
.idea_modules/

# JIRA plugin
atlassian-ide-plugin.xml

# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties


### Vagrant ###
.vagrant/


### OSX ###
.DS_Store
.AppleDouble
.LSOverride

# Icon must end with two \r
Icon


# Thumbnails
._*

# Files that might appear on external disk
.Spotlight-V100
.Trashes

# Directories potentially created on remote AFP share
.AppleDB
.AppleDesktop
Network Trash Folder
Temporary Items
.apdisk

.python-version


================================================
FILE: .readthedocs.yaml
================================================
version: 2

build:
  os: ubuntu-24.04
  tools:
    python: "3.13"
  jobs:
    pre_create_environment:
      - asdf plugin add uv
      - asdf install uv latest
      - asdf global uv latest
    create_environment:
      - uv venv "${READTHEDOCS_VIRTUALENV_PATH}"
    install:
      - UV_PROJECT_ENVIRONMENT="${READTHEDOCS_VIRTUALENV_PATH}" uv sync --frozen --group docs

sphinx:
  configuration: doc/conf.py

formats:
  - pdf
  - epub


================================================
FILE: CONTRIBUTING.rst
================================================
Code of conduct
---------------

This project adheres to the `Open Code of Conduct 
<https://github.com/spotify/code-of-conduct/blob/master/code-of-conduct.md>`_.  By 
participating, you are expected to honor this code.

Running the tests
-----------------


We are always happy to receive Pull Requests. When you open a PR, it will
automatically build on Travis. So you're not strictly required to test the
patch locally before submitting it.

If you do want to run the tests locally you'll need to run the commands below
.. code:: bash
   curl -LsSf https://astral.sh/uv/install.sh | sh
   uv tool install tox --with tox-uv

You will need a ``tox --version`` of at least 4.22.

.. code:: bash

    # These commands are pretty fast and will tell if you've
    # broken something major:
    tox run -e flake8
    tox run -e py38-core

    # You can also test particular files for even faster iterations
    tox run -e py38-core -- test/rpc_test.py

    # The visualiser tests require phantomjs to be installed on your path
    tox run -e visualiser

    # And some of the others involve downloading and running Hadoop:
    tox run -e py38-cdh
    tox run -e py39-hdp

Where ``flake8`` is the lint checking, ``py38`` is obviously Python 3.8.
``core`` are tests that do not require external components and ``cdh`` and
``hdp`` are two different hadoop distributions. For most local development it's
usually enough to run the lint checking and a python version for ``core``
and let Travis run for the whole matrix.

For `cdh` and `hdp`, tox will download the hadoop distribution for you. You
however have to have Java installed and the `JAVA_HOME` environment variable
set.

For more details, check out the ``.github/workflows/pythonbuild.yml`` and ``tox.ini`` files.

Writing documentation
=====================

All documentation for Luigi is written in `reStructuredText/Sphinx markup
<http://sphinx-doc.org/domains.html#the-python-domain>`_ and are both in the
code as docstrings and in `.rst`. Pull requests should come with documentation
when appropriate.

You verify that your documentation code compiles by running

.. code:: bash

    tox run -e docs

After that, you can check how it renders locally with your browser

.. code:: bash

    firefox doc/_build/html/index.html


================================================
FILE: LICENSE
================================================
                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright 2012-2021 Spotify AB

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.


================================================
FILE: README.rst
================================================
.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/luigi.png
   :alt: Luigi Logo
   :align: center

.. image:: https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Fspotify%2Fluigi%2Fbadge&label=build&logo=none&%3Fref%3Dmaster&style=flat
    :target: https://actions-badge.atrox.dev/spotify/luigi/goto?ref=master

.. image:: https://img.shields.io/codecov/c/github/spotify/luigi/master.svg?style=flat
    :target: https://codecov.io/gh/spotify/luigi?branch=master

.. image:: https://img.shields.io/pypi/v/luigi.svg?style=flat
   :target: https://pypi.python.org/pypi/luigi

.. image:: https://img.shields.io/pypi/l/luigi.svg?style=flat
   :target: https://pypi.python.org/pypi/luigi

.. image:: https://readthedocs.org/projects/luigi/badge/?version=stable
    :target: https://luigi.readthedocs.io/en/stable/?badge=stable
    :alt: Documentation Status

Luigi is a Python (3.10, 3.11, 3.12, 3.13 tested) package that helps you build complex
pipelines of batch jobs. It handles dependency resolution, workflow management,
visualization, handling failures, command line integration, and much more.

Getting Started
---------------

Run ``pip install luigi`` to install the latest stable version from `PyPI
<https://pypi.python.org/pypi/luigi>`_. `Documentation for the latest release
<https://luigi.readthedocs.io/en/stable/>`__ is hosted on readthedocs.

Run ``pip install luigi[toml]`` to install Luigi with `TOML-based configs
<https://luigi.readthedocs.io/en/stable/configuration.html>`__ support.

For the bleeding edge code, ``pip install
git+https://github.com/spotify/luigi.git``. `Bleeding edge documentation
<https://luigi.readthedocs.io/en/latest/>`__ is also available.

Background
----------

The purpose of Luigi is to address all the plumbing typically associated
with long-running batch processes. You want to chain many tasks,
automate them, and failures *will* happen. These tasks can be anything,
but are typically long running things like
`Hadoop <http://hadoop.apache.org/>`_ jobs, dumping data to/from
databases, running machine learning algorithms, or anything else.

There are other software packages that focus on lower level aspects of
data processing, like `Hive <http://hive.apache.org/>`__,
`Pig <http://pig.apache.org/>`_, or
`Cascading <http://www.cascading.org/>`_. Luigi is not a framework to
replace these. Instead it helps you stitch many tasks together, where
each task can be a `Hive query <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,
a `Hadoop job in Java <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop_jar.html>`_,
a  `Spark job in Scala or Python <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.spark.html>`_,
a Python snippet,
`dumping a table <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.sqla.html>`_
from a database, or anything else. It's easy to build up
long-running pipelines that comprise thousands of tasks and take days or
weeks to complete. Luigi takes care of a lot of the workflow management
so that you can focus on the tasks themselves and their dependencies.

You can build pretty much any task you want, but Luigi also comes with a
*toolbox* of several common task templates that you use. It includes
support for running
`Python mapreduce jobs <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hadoop.html>`_
in Hadoop, as well as
`Hive <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hive.html>`__,
and `Pig <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`__,
jobs. It also comes with
`file system abstractions for HDFS <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.hdfs.html>`_,
and local files that ensures all file system operations are atomic. This
is important because it means your data pipeline will not crash in a
state containing partial data.

Visualiser page
---------------

The Luigi server comes with a web interface too, so you can search and filter
among all your tasks.

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/visualiser_front_page.png
   :alt: Visualiser page

Dependency graph example
------------------------

Just to give you an idea of what Luigi does, this is a screen shot from
something we are running in production. Using Luigi's visualiser, we get
a nice visual overview of the dependency graph of the workflow. Each
node represents a task which has to be run. Green tasks are already
completed whereas yellow tasks are yet to be run. Most of these tasks
are Hadoop jobs, but there are also some things that run locally and
build up data files.

.. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/user_recs.png
   :alt: Dependency graph

Philosophy
----------

Conceptually, Luigi is similar to `GNU
Make <http://www.gnu.org/software/make/>`_ where you have certain tasks
and these tasks in turn may have dependencies on other tasks. There are
also some similarities to `Oozie <http://oozie.apache.org/>`_
and `Azkaban <https://azkaban.github.io/>`_. One major
difference is that Luigi is not just built specifically for Hadoop, and
it's easy to extend it with other kinds of tasks.

Everything in Luigi is in Python. Instead of XML configuration or
similar external data files, the dependency graph is specified *within
Python*. This makes it easy to build up complex dependency graphs of
tasks, where the dependencies can involve date algebra or recursive
references to other versions of the same task. However, the workflow can
trigger things not in Python, such as running
`Pig scripts <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.pig.html>`_
or `scp'ing files <https://luigi.readthedocs.io/en/latest/api/luigi.contrib.ssh.html>`_.

Who uses Luigi?
---------------

We use Luigi internally at `Spotify <https://www.spotify.com>`_ to run
thousands of tasks every day, organized in complex dependency graphs.
Most of these tasks are Hadoop jobs. Luigi provides an infrastructure
that powers all kinds of stuff including recommendations, toplists, A/B
test analysis, external reports, internal dashboards, etc.

Since Luigi is open source and without any registration walls, the exact number
of Luigi users is unknown. But based on the number of unique contributors, we
expect hundreds of enterprises to use it. Some users have written blog posts
or held presentations about Luigi:

* `Spotify <https://www.spotify.com>`_ `(presentation, 2014) <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`__
* `Foursquare <https://foursquare.com/>`_ `(presentation, 2013) <http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentation-17-23199897>`__
* `Mortar Data (Datadog) <https://www.datadoghq.com/>`_ `(documentation / tutorial) <http://help.mortardata.com/technologies/luigi>`__
* `Stripe <https://stripe.com/>`_ `(presentation, 2014) <http://www.slideshare.net/PyData/python-as-part-of-a-production-machine-learning-stack-by-michael-manapat-pydata-sv-2014>`__
* `Buffer <https://buffer.com/>`_ `(blog, 2014) <https://buffer.com/resources/buffers-new-data-architecture/>`__
* `SeatGeek <https://seatgeek.com/>`_ `(blog, 2015) <http://chairnerd.seatgeek.com/building-out-the-seatgeek-data-pipeline/>`__
* `Treasure Data <https://www.treasuredata.com/>`_ `(blog, 2015) <http://blog.treasuredata.com/blog/2015/02/25/managing-the-data-pipeline-with-git-luigi/>`__
* `Growth Intelligence <http://growthintel.com/>`_ `(presentation, 2015) <http://www.slideshare.net/growthintel/a-beginners-guide-to-building-data-pipelines-with-luigi>`__
* `AdRoll <https://www.adroll.com/>`_ `(blog, 2015) <http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html>`__
* 17zuoye `(presentation, 2015) <https://speakerdeck.com/mvj3/luiti-an-offline-task-management-framework>`__
* `Custobar <https://www.custobar.com/>`_ `(presentation, 2016) <http://www.slideshare.net/teemukurppa/managing-data-workflows-with-luigi>`__
* `Blendle <https://launch.blendle.com/>`_ `(presentation) <http://www.anneschuth.nl/wp-content/uploads/sea-anneschuth-streamingblendle.pdf#page=126>`__
* `TrustYou <http://www.trustyou.com/>`_ `(presentation, 2015) <https://speakerdeck.com/mfcabrera/pydata-berlin-2015-processing-hotel-reviews-with-python>`__
* `Groupon <https://www.groupon.com/>`_ / `OrderUp <https://orderup.com>`_ `(alternative implementation) <https://github.com/groupon/luigi-warehouse>`__
* `Red Hat - Marketing Operations <https://www.redhat.com>`_ `(blog, 2017) <https://github.com/rh-marketingops/rh-mo-scc-luigi>`__
* `GetNinjas <https://www.getninjas.com.br/>`_ `(blog, 2017) <https://labs.getninjas.com.br/using-luigi-to-create-and-monitor-pipelines-of-batch-jobs-eb8b3cd2a574>`__
* `voyages-sncf.com <https://www.voyages-sncf.com/>`_ `(presentation, 2017) <https://github.com/voyages-sncf-technologies/meetup-afpy-nantes-luigi>`__
* `Open Targets <https://www.opentargets.org/>`_ `(blog, 2017) <https://blog.opentargets.org/using-containers-with-luigi>`__
* `Leipzig University Library <https://ub.uni-leipzig.de>`_ `(presentation, 2016) <https://de.slideshare.net/MartinCzygan/build-your-own-discovery-index-of-scholary-eresources>`__ / `(project) <https://finc.info/de/datenquellen>`__
* `Synetiq <https://synetiq.net/>`_ `(presentation, 2017) <https://www.youtube.com/watch?v=M4xUQXogSfo>`__
* `Glossier <https://www.glossier.com/>`_ `(blog, 2018) <https://medium.com/glossier/how-to-build-a-data-warehouse-what-weve-learned-so-far-at-glossier-6ff1e1783e31>`__
* `Data Revenue <https://www.datarevenue.com/>`_ `(blog, 2018) <https://www.datarevenue.com/en/blog/how-to-scale-your-machine-learning-pipeline>`_
* `Uppsala University <http://pharmb.io>`_ `(tutorial) <http://uppnex.se/twiki/do/view/Courses/EinfraMPS2015/Luigi.html>`_   / `(presentation, 2015) <https://www.youtube.com/watch?v=f26PqSXZdWM>`_ / `(slides, 2015) <https://www.slideshare.net/SamuelLampa/building-workflows-with-spotifys-luigi>`_ / `(poster, 2015) <https://pharmb.io/poster/2015-sciluigi/>`_ / `(paper, 2016) <https://doi.org/10.1186/s13321-016-0179-6>`_ / `(project) <https://github.com/pharmbio/sciluigi>`_
* `GIPHY <https://giphy.com/>`_ `(blog, 2019) <https://engineering.giphy.com/luigi-the-10x-plumber-containerizing-scaling-luigi-in-kubernetes/>`__
* `xtream <https://xtreamers.io/>`__ `(blog, 2019) <https://towardsdatascience.com/lessons-from-a-real-machine-learning-project-part-1-from-jupyter-to-luigi-bdfd0b050ca5>`__
* `CIAN <https://cian.ru/>`__ `(presentation, 2019) <https://www.highload.ru/moscow/2019/abstracts/6030>`__

Some more companies are using Luigi but haven't had a chance yet to write about it:

* `Schibsted <http://www.schibsted.com/>`_
* `enbrite.ly <http://enbrite.ly/>`_
* `Dow Jones / The Wall Street Journal <http://wsj.com>`_
* `Hotels.com <https://hotels.com>`_
* `Newsela <https://newsela.com>`_
* `Squarespace <https://www.squarespace.com/>`_
* `OAO <https://adops.com/>`_
* `Grovo <https://grovo.com/>`_
* `Weebly <https://www.weebly.com/>`_
* `Deloitte <https://www.Deloitte.co.uk/>`_
* `Stacktome <https://stacktome.com/>`_
* `LINX+Neemu+Chaordic <https://www.chaordic.com.br/>`_
* `Foxberry <https://www.foxberry.com/>`_
* `Okko <https://okko.tv/>`_
* `ISVWorld <http://isvworld.com/>`_
* `Big Data <https://bigdata.com.br/>`_
* `Movio <https://movio.co.nz/>`_
* `Bonnier News <https://www.bonniernews.se/>`_
* `Starsky Robotics <https://www.starsky.io/>`_
* `BaseTIS <https://www.basetis.com/>`_
* `Hopper <https://www.hopper.com/>`_
* `VOYAGE GROUP/Zucks <https://zucks.co.jp/en/>`_
* `Textpert <https://www.textpert.ai/>`_
* `Tracktics <https://www.tracktics.com/>`_
* `Whizar <https://www.whizar.com/>`_
* `xtream <https://www.xtreamers.io/>`__
* `Skyscanner <https://www.skyscanner.net/>`_
* `Jodel <https://www.jodel.com/>`_
* `Mekar <https://mekar.id/en/>`_
* `M3 <https://corporate.m3.com/en/>`_
* `Assist Digital <https://www.assistdigital.com/>`_
* `Meltwater <https://www.meltwater.com/>`_
* `DevSamurai <https://www.devsamurai.com/>`_
* `Veridas <https://veridas.com/>`_
* `Aidentified <https://www.aidentified.com/>`_

We're more than happy to have your company added here. Just send a PR on GitHub.

External links
--------------

* `Mailing List <https://groups.google.com/d/forum/luigi-user/>`_ for discussions and asking questions. (Google Groups)
* `Releases <https://pypi.python.org/pypi/luigi>`_ (PyPI)
* `Source code <https://github.com/spotify/luigi>`_ (GitHub)
* `Hubot Integration <https://github.com/houzz/hubot-luigi>`_ plugin for Slack, Hipchat, etc (GitHub)

Authors
-------

Luigi was built at `Spotify <https://www.spotify.com>`_, mainly by
`Erik Bernhardsson <https://github.com/erikbern>`_ and
`Elias Freider <https://github.com/freider>`_.
`Many other people <https://github.com/spotify/luigi/graphs/contributors>`_
have contributed since open sourcing in late 2012.
`Arash Rouhani <https://github.com/tarrasch>`_ was the chief maintainer from 2015 to 2019, and now
Spotify's Data Team maintains Luigi.


================================================
FILE: RELEASE-PROCESS.rst
================================================
For maintainers of Luigi, who have push access to pypi. Here's how you upload
Luigi to pypi.

#. Make sure [uv](https://github.com/astral-sh/uv) is installed ``curl -LsSf https://astral.sh/uv/install.sh | sh``.
#. Update version number in `luigi/__version__.py`.
#. Commit, perhaps simply with a commit message like ``Version x.y.z``.
#. Push to GitHub at [spotify/luigi](https://github.com/spotify/luigi).
#. Clean up previous distributions by executing ``rm -rf dist``.
#. Build a source distribution by executing ``uv build``.
#. Set pypi token on environment variable ``export UV_PUBLISH_TOKEN="LUIGI_PYPI_TOKEN_HERE"``.
#. Upload to pypi by executing ``uv publish``.
#. Add a tag on github (https://github.com/spotify/luigi/releases),
   including a handwritten changelog, possibly inspired from previous notes.

Currently, Luigi is not released on any particular schedule and it is not
strictly abiding semantic versioning. Whenever possible, bump major version when you make incompatible API changes, minor version when you add functionality in a backwards compatible manner, and patch version when you make backwards compatible bug fixes.


================================================
FILE: SECURITY.md
================================================
# Security Policy

## Reporting a Vulnerability

Please report sensitive security issues via Spotify's [bug-bounty program](https://hackerone.com/spotify) by following this [instruction](https://docs.hackerone.com/programs/security-page.html), rather than GitHub. 


================================================
FILE: bin/luigi
================================================
#!/usr/bin/env python

import sys
import warnings
import luigi.cmdline


def main(argv):
    warnings.warn("'bin/luigi' has moved to console script 'luigi'", DeprecationWarning)
    luigi.cmdline.luigi_run(argv)


if __name__ == '__main__':
    main(sys.argv[1:])


================================================
FILE: bin/luigid
================================================
#!/usr/bin/env python

import sys
import warnings
import luigi.cmdline


def main(argv):
    warnings.warn("'bin/luigid' has moved to console script 'luigid'", DeprecationWarning)
    luigi.cmdline.luigid(argv)


if __name__ == '__main__':
    main(sys.argv[1:])


================================================
FILE: catalog-info.yaml
================================================
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: luigi
spec:
  type: library
  owner: dataex


================================================
FILE: codecov.yml
================================================
codecov:
  require_ci_to_pass: true
  notify:
    wait_for_ci: true

coverage:
  precision: 2
  round: down
  range: "50...70"

  status:
    project:
      default: false  # disable the default status that measures entire project
      core:
        target: 90%
        paths:
          - "luigi/*.py"
    patch:
      default:
        target: 50%
        if_no_uploads: error

    changes:
      default:
        informational: true

  ignore:
    - "examples/"
    - "luigi/tools"  # These are tested as actual run commands without coverage
    # List modules who's tests are not run by CI or are run in a subprocesses (like on cluster).
    - "luigi/contrib/beam_dataflow.py"
    - "luigi/contrib/bigquery.py"
    - "luigi/contrib/bigquery_avro.py"
    - "luigi/contrib/dataproc.py"
    - "luigi/contrib/dropbox.py"
    - "luigi/contrib/ftp.py"
    - "luigi/contrib/gcs.py"
    - "luigi/contrib/hadoop.py"
    - "luigi/contrib/hdfs/"
    - "luigi/contrib/kubernetes.py"
    - "luigi/contrib/mrrunner.py"
    - "luigi/contrib/sparkey.py"
    - "luigi/contrib/webhdfs.py"

# For luigi we do not want any comments
comment: false


================================================
FILE: doc/.gitignore
================================================
_static
_build
_templates


================================================
FILE: doc/Makefile
================================================
# Makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS    =
SPHINXBUILD   = sphinx-build
PAPER         =
BUILDDIR      = _build

# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif

# Internal variables.
PAPEROPT_a4     = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .

.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext

help:
	@echo "Please use \`make <target>' where <target> is one of"
	@echo "  html       to make standalone HTML files"
	@echo "  dirhtml    to make HTML files named index.html in directories"
	@echo "  singlehtml to make a single large HTML file"
	@echo "  pickle     to make pickle files"
	@echo "  json       to make JSON files"
	@echo "  htmlhelp   to make HTML files and a HTML help project"
	@echo "  qthelp     to make HTML files and a qthelp project"
	@echo "  devhelp    to make HTML files and a Devhelp project"
	@echo "  epub       to make an epub"
	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
	@echo "  text       to make text files"
	@echo "  man        to make manual pages"
	@echo "  texinfo    to make Texinfo files"
	@echo "  info       to make Texinfo files and run them through makeinfo"
	@echo "  gettext    to make PO message catalogs"
	@echo "  changes    to make an overview of all changed/added/deprecated items"
	@echo "  xml        to make Docutils-native XML files"
	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
	@echo "  linkcheck  to check all external links for integrity"
	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"

clean:
	rm -rf $(BUILDDIR)/*

html:
	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

dirhtml:
	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."

singlehtml:
	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
	@echo
	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."

pickle:
	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
	@echo
	@echo "Build finished; now you can process the pickle files."

json:
	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
	@echo
	@echo "Build finished; now you can process the JSON files."

htmlhelp:
	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
	@echo
	@echo "Build finished; now you can run HTML Help Workshop with the" \
	      ".hhp project file in $(BUILDDIR)/htmlhelp."

qthelp:
	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
	@echo
	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Luigi.qhcp"
	@echo "To view the help file:"
	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Luigi.qhc"

devhelp:
	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
	@echo
	@echo "Build finished."
	@echo "To view the help file:"
	@echo "# mkdir -p $$HOME/.local/share/devhelp/Luigi"
	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Luigi"
	@echo "# devhelp"

epub:
	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
	@echo
	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."

latex:
	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
	@echo
	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
	@echo "Run \`make' in that directory to run these through (pdf)latex" \
	      "(use \`make latexpdf' here to do that automatically)."

latexpdf:
	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
	@echo "Running LaTeX files through pdflatex..."
	$(MAKE) -C $(BUILDDIR)/latex all-pdf
	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."

latexpdfja:
	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
	@echo "Running LaTeX files through platex and dvipdfmx..."
	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."

text:
	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
	@echo
	@echo "Build finished. The text files are in $(BUILDDIR)/text."

man:
	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
	@echo
	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."

texinfo:
	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
	@echo
	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
	@echo "Run \`make' in that directory to run these through makeinfo" \
	      "(use \`make info' here to do that automatically)."

info:
	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
	@echo "Running Texinfo files through makeinfo..."
	make -C $(BUILDDIR)/texinfo info
	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."

gettext:
	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
	@echo
	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."

changes:
	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
	@echo
	@echo "The overview file is in $(BUILDDIR)/changes."

linkcheck:
	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
	@echo
	@echo "Link check complete; look for any errors in the above output " \
	      "or in $(BUILDDIR)/linkcheck/output.txt."

doctest:
	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
	@echo "Testing of doctests in the sources finished, look at the " \
	      "results in $(BUILDDIR)/doctest/output.txt."

xml:
	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
	@echo
	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."

pseudoxml:
	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
	@echo
	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."


================================================
FILE: doc/central_scheduler.rst
================================================
Using the Central Scheduler
---------------------------

While the ``--local-scheduler`` flag is useful for development purposes,
it's not recommended for production usage.
The centralized scheduler serves two purposes:

-  Make sure two instances of the same task are not running simultaneously
-  Provide visualization of everything that's going on.

Note that the central scheduler does not execute anything for you or
help you with job parallelization.
For running tasks periodically,
the easiest thing to do is to trigger a Python script from cron or
from a continuously running process.
There is no central process that automatically triggers jobs.
This model may seem limited, but
we believe that it makes things far more intuitive and easy to understand.

.. figure:: dependency_graph.png
   :alt: Dependency graph in the visualiser

The luigid server
~~~~~~~~~~~~~~~~~

To run the server as a daemon run:

.. code-block:: console

    $ luigid --background --pidfile <PATH_TO_PIDFILE> --logdir <PATH_TO_LOGDIR> --state-path <PATH_TO_STATEFILE>

Note that this requires ``python-daemon``.
By default, the server starts on AF_INET and AF_INET6 port ``8082``
(which can be changed with the ``--port`` flag) and listens on all IPs. To change the default behavior of listening on all IPs, pass the ``--address`` flag and the IP address to listen on.
To use an AF_UNIX socket use the ``--unix-socket`` flag.

For a full list of configuration options and defaults,
see the :ref:`scheduler configuration section <scheduler-config>`.
Note that ``luigid`` uses the same configuration files as the Luigi client
(i.e. ``luigi.cfg`` or ``/etc/luigi/client.cfg`` by default).

.. _TaskHistory:

Enabling Task History
~~~~~~~~~~~~~~~~~~~~~

Task History is an experimental feature in which
additional information about tasks that have been executed are recorded in a relational database
for historical analysis.
This information is exposed via the Central Scheduler at ``/history``.

To enable the task history,
specify ``record_task_history = True`` in the
``[scheduler]`` section of ``luigi.cfg`` and
specify ``db_connection`` under ``[task_history]``.
The ``db_connection`` string is used to configure the `SQLAlchemy engine
<http://docs.sqlalchemy.org/en/rel_0_9/core/engines.html>`_.
When starting up,
``luigid`` will create all the necessary tables using `create_all
<http://docs.sqlalchemy.org/en/rel_0_9/core/metadata.html#sqlalchemy.schema.MetaData.create_all>`_.

Example configuration

.. code:: ini

    [scheduler]
    record_task_history = True
    state_path = /usr/local/var/luigi-state.pickle

    [task_history]
    db_connection = sqlite:////usr/local/var/luigi-task-hist.db

The task history has the following pages:

* ``/history``
  a reverse-cronological listing of runs from the past 24 hours.
  Example screenshot:

    .. figure:: history.png
       :alt: Recent history screenshot
* ``/history/by_id/{id}``
  detailed information about a run, including:
  parameter values, the host on which it ran, and timing information.
  Example screenshot:

    .. figure:: history_by_id.png
       :alt: By id screenshot
* ``/history/by_name/{name}``
  a listing of all runs of a task with the given task ``{name}``.
  Example screenshot:

    .. figure:: history_by_name.png
       :alt: By name screenshot
* ``/history/by_params/{name}?data=params``
  a listing of all runs of the task ``{name}`` restricted to runs with ``params`` matching the given history.
  The ``params`` is a json blob describing the parameters,
  e.g. ``data={"foo": "bar"}`` looks for a task with ``foo=bar``.
* ``/history/by_task_id/{task_id}``
  the latest run of a task given the ``{task_id}``. It is different from just ``{id}``
  and is a derivative of ``params``. It is available via ``{task_id}`` property of a 
  ``luigi.Task`` instance or via `luigi.task.task_id_str
  <https://luigi.readthedocs.io/en/stable/api/luigi.task.html#luigi.task.task_id_str>`_.
  This kind of representation is useful for concisely recording URLs in a history tree.
  Example screenshot:

    .. figure:: history_by_task_id.png
       :alt: By task_id screenshot
  


================================================
FILE: doc/conf.py
================================================
# -*- coding: utf-8 -*-
#
# Luigi documentation build configuration file, created by
# sphinx-quickstart on Sat Feb  8 00:56:43 2014.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.

import sys
import os
import datetime
from importlib.metadata import Distribution


try:
    import luigi
    import luigi.parameter

    def parameter_repr(self):
        """
        When building documentation, we want Parameter objects to show their
        description in a nice way
        """
        significance = 'Insignificant ' if not self.significant else ''
        class_name = self.__class__.__name__
        has_default = self._default != luigi.parameter._no_value
        default = ' (defaults to {})'.format(self._default) if has_default else ''
        description = (': ' + self.description if self.description else '')
        return significance + class_name + default + description

    luigi.parameter.Parameter.__repr__ = parameter_repr

    def assertIn(needle, haystack):
        """
        We test repr of Parameter objects, since it'll be used for readthedocs
        """
        assert needle in haystack

    # TODO: find a better place to put this!
    assertIn('IntParameter', repr(luigi.IntParameter()))
    assertIn('defaults to 37', repr(luigi.IntParameter(default=37)))
    assertIn('hi mom', repr(luigi.IntParameter(description='hi mom')))
    assertIn('Insignificant BoolParameter', repr(luigi.BoolParameter(significant=False)))
except ImportError:
    pass


# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath(os.path.pardir))

# append the __init__ to class definitions
autoclass_content = 'both'

# -- General configuration ------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
needs_sphinx = '9.0'

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.viewcode',
    'sphinx.ext.autosummary',
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# The suffix of source filenames.
source_suffix = '.rst'

# The encoding of source files.
#source_encoding = 'utf-8-sig'

# The master toctree document.
master_doc = 'index'

# General information about the project.
project = u'Luigi'
authors = u"The Luigi Authors"
copyright = u"2011-{}, {}".format(datetime.datetime.now().year, authors)

# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
__version__ = Distribution.from_name('luigi').version  # assume luigi is already installed
# The short X.Y version.
version = ".".join(__version__.split(".")[0:2])
# The full version, including alpha/beta/rc tags.
release = __version__

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None

# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build', 'README.rst']

# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None

# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True

# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True

# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'

# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []

# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False

autodoc_default_options = {'members': True, 'undoc-members': True}
autosummary_generate = True
autodoc_member_order = 'bysource'

# -- Options for HTML output ----------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.

html_theme = 'sphinx_rtd_theme'

# Theme options are theme-specific and customize the look and feel of a theme
# further.  For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}

# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []

# The name for this set of Sphinx documents.  If None, it defaults to
# "<project> v<release> documentation".
#html_title = None

# A shorter title for the navigation bar.  Default is the same as html_title.
#html_short_title = None

# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
html_logo = 'luigi.png'

# The name of an image file (within the static path) to use as favicon of the
# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']

# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []

# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'

# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True

# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}

# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}

# If false, no module index is generated.
#html_domain_indices = True

# If false, no index is generated.
#html_use_index = True

# If true, the index is split into individual pages for each letter.
#html_split_index = False

# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True

# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True

# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True

# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it.  The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''

# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None

# Output file base name for HTML help builder.
htmlhelp_basename = 'Luigidoc'


# -- Options for LaTeX output ---------------------------------------------

latex_elements = {
    # The paper size ('letterpaper' or 'a4paper').
    #'papersize': 'letterpaper',

    # The font size ('10pt', '11pt' or '12pt').
    #'pointsize': '10pt',

    # Additional stuff for the LaTeX preamble.
    #'preamble': '',
}

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
#  author, documentclass [howto, manual, or own class]).
latex_documents = [
    ('index', 'Luigi.tex', u'Luigi Documentation',
     authors, 'manual'),
]

# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None

# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False

# If true, show page references after internal links.
#latex_show_pagerefs = False

# If true, show URL addresses after external links.
#latex_show_urls = False

# Documents to append as an appendix to all manuals.
#latex_appendices = []

# If false, no module index is generated.
#latex_domain_indices = True


# -- Options for manual page output ---------------------------------------

# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
    ('index', 'luigi', u'Luigi Documentation',
     [authors], 1)
]

# If true, show URL addresses after external links.
#man_show_urls = False


# -- Options for Texinfo output -------------------------------------------

# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
#  dir menu entry, description, category)
texinfo_documents = [
    ('index', 'Luigi', u'Luigi Documentation',
     authors, 'Luigi', 'One line description of project.',
     'Miscellaneous'),
]

# Documents to append as an appendix to all manuals.
#texinfo_appendices = []

# If false, no module index is generated.
#texinfo_domain_indices = True

# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'

# If true, do not generate a @detailmenu in the "Top" node's menu.
#texinfo_no_detailmenu = False

autodoc_mock_imports = ["mypy"]

# sphinx-apidoc --separate generates individual RST files not referenced by any toctree;
# suppress the resulting warnings since this is expected behaviour.
suppress_warnings = ['toc.not_included']

# Some regression introduced
# https://github.com/sphinx-doc/sphinx/issues/2330
# https://github.com/spotify/luigi/pull/1555
highlight_language = "python"


================================================
FILE: doc/configuration.rst
================================================
Configuration
=============

All configuration can be done by adding configuration files.

Supported config parsers:

* ``cfg`` (default), based on Python's standard ConfigParser_. Values may refer to environment variables using ``${ENVVAR}`` syntax.
* ``toml``

.. _ConfigParser: https://docs.python.org/3/library/configparser.html

You can choose right parser via ``LUIGI_CONFIG_PARSER`` environment variable. For example, ``LUIGI_CONFIG_PARSER=toml``.

Default (cfg) parser are looked for in:

* ``/etc/luigi/client.cfg`` (deprecated)
* ``/etc/luigi/luigi.cfg``
* ``client.cfg`` (deprecated)
* ``luigi.cfg``
* ``LUIGI_CONFIG_PATH`` environment variable

`TOML <https://github.com/toml-lang/toml>`_ parser are looked for in:

* ``/etc/luigi/luigi.toml``
* ``luigi.toml``
* ``LUIGI_CONFIG_PATH`` environment variable

Both config lists increase in priority (from low to high). The order only
matters in case of key conflicts (see docs for ConfigParser.read_).
These files are meant for both the client and ``luigid``.
If you decide to specify your own configuration you should make sure
that both the client and ``luigid`` load it properly.

.. _ConfigParser.read: https://docs.python.org/3/library/configparser.html#configparser.ConfigParser.read

The config file is broken into sections, each controlling a different part of the config.

Example cfg config:

.. code:: ini

    [hadoop]
    version=cdh4
    streaming_jar=/usr/lib/hadoop-xyz/hadoop-streaming-xyz-123.jar

    [core]
    scheduler_host=luigi-host.mycompany.foo

Example toml config:

.. code:: python

    [hadoop]
    version = "cdh4"
    streaming_jar = "/usr/lib/hadoop-xyz/hadoop-streaming-xyz-123.jar"

    [core]
    scheduler_host = "luigi-host.mycompany.foo"

Also see `examples/config.toml
<https://github.com/spotify/luigi/blob/master/examples/config.toml>`_
for more complex example.

.. _ParamConfigIngestion:

Parameters from config Ingestion
--------------------------------

All parameters can be overridden from configuration files. For instance if you
have a Task definition:

.. code:: python

    class DailyReport(luigi.contrib.hadoop.JobTask):
        date = luigi.DateParameter(default=datetime.date.today())
        # ...

Then you can override the default value for ``DailyReport().date`` by providing
it in the configuration:

.. code:: ini

    [DailyReport]
    date=2012-01-01

.. _ConfigClasses:

Configuration classes
*********************

Using the :ref:`ParamConfigIngestion` method, we derive the
conventional way to do global configuration. Imagine this configuration.

.. code:: ini

    [mysection]
    option=hello
    intoption=123


We can create a :py:class:`~luigi.Config` class:

.. code:: python

    import luigi

    # Config classes should be camel cased
    class mysection(luigi.Config):
        option = luigi.Parameter(default='world')
        intoption = luigi.IntParameter(default=555)

    mysection().option
    mysection().intoption


Configurable options
--------------------

Luigi comes with a lot of configurable options. Below, we describe each
section and the parameters available within it.


[core]
------

These parameters control core Luigi behavior, such as error e-mails and
interactions between the worker and scheduler.

autoload_range
  .. versionadded:: 2.8.11

  If false, prevents range tasks from autoloading. They can still be loaded
  using ``--module luigi.tools.range``. Defaults to true. Setting this to true
  explicitly disables the deprecation warning.

default_scheduler_host
  Hostname of the machine running the scheduler. Defaults to localhost.

default_scheduler_port
  Port of the remote scheduler api process. Defaults to 8082.

default_scheduler_url
  Full path to remote scheduler. Defaults to ``http://localhost:8082/``.
  For TLS support use the URL scheme: ``https``,
  example: ``https://luigi.example.com:443/``
  (Note: you will have to terminate TLS using an HTTP proxy)
  You can also use this to connect to a local Unix socket using the
  non-standard URI scheme: ``http+unix``
  example: ``http+unix://%2Fvar%2Frun%2Fluigid%2Fluigid.sock/``

hdfs_tmp_dir
  Base directory in which to store temporary files on hdfs. Defaults to
  tempfile.gettempdir()

history_filename
  If set, specifies a filename for Luigi to write stuff (currently just
  job id) to in mapreduce job's output directory. Useful in a
  configuration where no history is stored in the output directory by
  Hadoop.

log_level
  The default log level to use when no logging_conf_file is set. Must be
  a valid name of a `Python log level
  <https://docs.python.org/2/library/logging.html#logging-levels>`_.
  Default is ``DEBUG``.

logging_conf_file
  Location of the logging configuration file.

no_configure_logging
  If true, logging is not configured. Defaults to false.

parallel_scheduling
  If true, the scheduler will compute complete functions of tasks in
  parallel using multiprocessing. This can significantly speed up
  scheduling, but requires that all tasks can be pickled.
  Defaults to false.

parallel_scheduling_processes
  The number of processes to use for parallel scheduling. If not specified
  the default number of processes will be the total number of CPUs available.

rpc_connect_timeout
  Number of seconds to wait before timing out when making an API call.
  Defaults to 10.0

rpc_retry_attempts
  The maximum number of retries to connect the central scheduler before giving up.
  Defaults to 3

rpc_retry_wait
  Number of seconds to wait before the next attempt will be started to
  connect to the central scheduler between two retry attempts.
  Defaults to 30


[cors]
------

.. versionadded:: 2.8.0

These parameters control ``/api/<method>`` ``CORS`` behaviour (see: `W3C Cross-Origin Resource Sharing
<http://www.w3.org/TR/cors/>`_).

enabled
  Enables CORS support.
  Defaults to false.

allowed_origins
  A list of allowed origins. Used only if ``allow_any_origin`` is false.
  Configure in JSON array format, e.g. ["foo", "bar"].
  Defaults to empty.

allow_any_origin
  Accepts requests from any origin.
  Defaults to false.

allow_null_origin
  Allows the request to set ``null`` value of the ``Origin`` header.
  Defaults to false.

max_age
  Content of ``Access-Control-Max-Age``.
  Defaults to 86400 (24 hours).

allowed_methods
  Content of ``Access-Control-Allow-Methods``.
  Defaults to ``GET, OPTIONS``.

allowed_headers
  Content of ``Access-Control-Allow-Headers``.
  Defaults to ``Accept, Content-Type, Origin``.

exposed_headers
  Content of ``Access-Control-Expose-Headers``.
  Defaults to empty string (will NOT be sent as a response header).

allow_credentials
  Indicates that the actual request can include user credentials.
  Defaults to false.

.. _worker-config:

[worker]
--------

These parameters control Luigi worker behavior.

count_uniques
  If true, workers will only count unique pending jobs when deciding
  whether to stay alive. So if a worker can't get a job to run and other
  workers are waiting on all of its pending jobs, the worker will die.
  ``worker_keep_alive`` must be ``true`` for this to have any effect. Defaults
  to false.

keep_alive
  If true, workers will stay alive when they run out of jobs to run, as
  long as they have some pending job waiting to be run. Defaults to
  false.

ping_interval
  Number of seconds to wait between pinging scheduler to let it know
  that the worker is still alive. Defaults to 1.0.

task_limit
  .. versionadded:: 1.0.25

  Maximum number of tasks to schedule per invocation. Upon exceeding it,
  the worker will issue a warning and proceed with the workflow obtained
  thus far. Prevents incidents due to spamming of the scheduler, usually
  accidental. Default: no limit.

task_process_context
  An optional setting allowing Luigi to import a custom context manager
  used to wrap the execution of tasks' run methods. Default: no context manager.

timeout
  .. versionadded:: 1.0.20

  Number of seconds after which to kill a task which has been running
  for too long. This provides a default value for all tasks, which can
  be overridden by setting the ``worker_timeout`` property in any task.
  Default value is 0, meaning no timeout.

wait_interval
  Number of seconds for the worker to wait before asking the scheduler
  for another job after the scheduler has said that it does not have any
  available jobs.

wait_jitter
  Duration of jitter to add to the worker wait interval such that the multiple
  workers do not ask the scheduler for another job at the same time, in seconds.
  Default: 5.0

max_keep_alive_idle_duration
  .. versionadded:: 2.8.4

  Maximum duration in seconds to keep worker alive while in idle state.
  Default: 0 (Indefinitely)

max_reschedules
  The maximum number of times that a job can be automatically
  rescheduled by a worker before it will stop trying. Workers will
  reschedule a job if it is found to not be done when attempting to run
  a dependent job. This defaults to 1.

retry_external_tasks
  If true, incomplete external tasks (i.e. tasks where the ``run()`` method is
  NotImplemented) will be retested for completion while Luigi is running.
  This means that if external dependencies are satisfied after a workflow has
  started, any tasks dependent on that resource will be eligible for running.
  Note: Every time the task remains incomplete, it will count as FAILED, so
  normal retry logic applies (see: ``retry_count`` and ``retry_delay``).
  This setting works best with ``worker_keep_alive: true``.
  If false, external tasks will only be evaluated when Luigi is first invoked.
  In this case, Luigi will not check whether external dependencies are
  satisfied  while a workflow is in progress, so dependent tasks will remain
  PENDING until the workflow is reinvoked.
  Defaults to false for backwards compatibility.

no_install_shutdown_handler
  By default, workers will stop requesting new work and finish running
  pending tasks after receiving a ``SIGUSR1`` signal. This provides a hook
  for gracefully shutting down workers that are in the process of running
  (potentially expensive) tasks. If set to true, Luigi will NOT install
  this shutdown hook on workers. Note this hook does not work on Windows
  operating systems, or when jobs are launched outside the main execution
  thread.
  Defaults to false.

send_failure_email
  Controls whether the worker will send e-mails on task and scheduling
  failures. If set to false, workers will only send e-mails on
  framework errors during scheduling and all other e-mail must be
  handled by the scheduler.
  Defaults to true.

check_unfulfilled_deps
  If true, the worker checks for completeness of dependencies before running a
  task. In case unfulfilled dependencies are detected, an exception is raised
  and the task will not run. This mechanism is useful to detect situations
  where tasks do not create their outputs properly, or when targets were
  removed after the dependency tree was built. It is recommended to disable
  this feature only when the completeness checks are known to be bottlenecks,
  e.g. when the ``exists()`` calls of the dependencies' outputs are
  resource-intensive.
  Defaults to true.

force_multiprocessing
  By default, luigi uses multiprocessing when *more than one* worker process is
  requested. When set to true, multiprocessing is used independent of the
  number of workers.
  Defaults to false.

check_complete_on_run
  By default, luigi tasks are marked as 'done' when they finish running without
  raising an error. When set to true, tasks will also verify that their outputs
  exist when they finish running, and will fail immediately if the outputs are
  missing.
  Defaults to false.

cache_task_completion
  By default, luigi task processes might check the completion status multiple
  times per task which is a safe way to avoid potential inconsistencies. For
  tasks with many dynamic dependencies, yielded in multiple stages, this might
  become expensive, e.g. in case the per-task completion check entails remote
  resources. When set to true, completion checks are cached so that tasks
  declared as complete once are not checked again.
  Defaults to false.


[elasticsearch]
---------------

These parameters control use of elasticsearch

marker_index
  Defaults to "update_log".

marker_doc_type
  Defaults to "entry".


[email]
-------

General parameters

force_send
  If true, e-mails are sent in all run configurations (even if stdout is
  connected to a tty device).  Defaults to False.

format
  Type of e-mail to send. Valid values are "plain", "html" and "none".
  When set to html, tracebacks are wrapped in <pre> tags to get fixed-
  width font. When set to none, no e-mails will be sent.

  Default value is plain.

method
  Valid values are "smtp", "sendgrid", "ses" and "sns". SES and SNS are
  services of Amazon web services. SendGrid is an email delivery service.
  The default value is "smtp".

  In order to send messages through Amazon SNS or SES set up your AWS
  config files or run Luigi on an EC2 instance with proper instance
  profile.

  In order to use sendgrid, fill in your sendgrid API key in the
  `[sendgrid]`_ section.

  In order to use smtp, fill in the appropriate fields in the `[smtp]`_
  section.

prefix
  Optional prefix to add to the subject line of all e-mails. For
  example, setting this to "[LUIGI]" would change the subject line of an
  e-mail from "Luigi: Framework error" to "[LUIGI] Luigi: Framework
  error"

receiver
  Recipient of all error e-mails. If this is not set, no error e-mails
  are sent when Luigi crashes unless the crashed job has owners set. If
  Luigi is run from the command line, no e-mails will be sent unless
  output is redirected to a file.

  Set it to SNS Topic ARN if you want to receive notifications through
  Amazon SNS. Make sure to set method to sns in this case too.

sender
  User name in from field of error e-mails.
  Default value: luigi-client@<server_name>

traceback_max_length
  Maximum length for traceback included in error email. Default is 5000.


[batch_email]
----------------

Parameters controlling the contents of batch notifications sent from the
scheduler

email_interval
  Number of minutes between e-mail sends. Making this larger results in
  fewer, bigger e-mails.
  Defaults to 60.

batch_mode
  Controls how tasks are grouped together in the e-mail. Suppose we have
  the following sequence of failures:

  1. TaskA(a=1, b=1)
  2. TaskA(a=1, b=1)
  3. TaskA(a=2, b=1)
  4. TaskA(a=1, b=2)
  5. TaskB(a=1, b=1)

  For any setting of batch_mode, the batch e-mail will record 5 failures
  and mention them in the subject. The difference is in how they will
  be displayed in the body. Here are example bodies with error_messages
  set to 0.

  "all" only groups together failures for the exact same task:

  - TaskA(a=1, b=1) (2 failures)
  - TaskA(a=1, b=2) (1 failure)
  - TaskA(a=2, b=1) (1 failure)
  - TaskB(a=1, b=1) (1 failure)

  "family" groups together failures for tasks of the same family:

  - TaskA (4 failures)
  - TaskB (1 failure)

  "unbatched_params" groups together tasks that look the same after
  removing batched parameters. So if TaskA has a batch_method set for
  parameter a, we get the following:

  - TaskA(b=1) (3 failures)
  - TaskA(b=2) (1 failure)
  - TaskB(a=1, b=2) (1 failure)

  Defaults to "unbatched_params", which is identical to "all" if you are
  not using batched parameters.

error_lines
  Number of lines to include from each error message in the batch
  e-mail. This can be used to keep e-mails shorter while preserving the
  more useful information usually found near the bottom of stack traces.
  This can be set to 0 to include all lines. If you don't wish to see
  error messages, instead set ``error_messages`` to 0.
  Defaults to 20.

error_messages
  Number of messages to preserve for each task group. As most tasks that
  fail repeatedly do so for similar reasons each time, it's not usually
  necessary to keep every message. This controls how many messages are
  kept for each task or task group. The most recent error messages are
  kept. Set to 0 to not include error messages in the e-mails.
  Defaults to 1.

group_by_error_messages
  Quite often, a system or cluster failure will cause many disparate
  task types to fail for the same reason. This can cause a lot of noise
  in the batch e-mails. This cuts down on the noise by listing items
  with identical error messages together. Error messages are compared
  after limiting by ``error_lines``.
  Defaults to true.


[hadoop]
--------

Parameters controlling basic hadoop tasks

command
  Name of command for running hadoop from the command line. Defaults to
  "hadoop"

python_executable
  Name of command for running python from the command line. Defaults to
  "python"

scheduler
  Type of scheduler to use when scheduling hadoop jobs. Can be "fair" or
  "capacity". Defaults to "fair".

streaming_jar
  Path to your streaming jar. Must be specified to run streaming jobs.

version
  Version of hadoop used in your cluster. Can be "cdh3", "chd4", or
  "apache1". Defaults to "cdh4".


[hdfs]
------

Parameters controlling the use of snakebite to speed up hdfs queries.

client
  Client to use for most hadoop commands. Options are "snakebite",
  "snakebite_with_hadoopcli_fallback", "webhdfs" and "hadoopcli". Snakebite is
  much faster, so use of it is encouraged. webhdfs is fast and works with
  Python 3 as well, but has not been used that much in the wild.
  Both snakebite and webhdfs requires you to install it separately on
  the machine. Defaults to "hadoopcli".

client_version
  Optionally specifies hadoop client version for snakebite.

effective_user
  Optionally specifies the effective user for snakebite.

namenode_host
  The hostname of the namenode. Needed for snakebite if
  snakebite_autoconfig is not set.

namenode_port
  The port used by snakebite on the namenode. Needed for snakebite if
  snakebite_autoconfig is not set.

snakebite_autoconfig
  If true, attempts to automatically detect the host and port of the
  namenode for snakebite queries. Defaults to false.

tmp_dir
  Path to where Luigi will put temporary files on hdfs


[hive]
------

Parameters controlling hive tasks

command
  Name of the command used to run hive on the command line. Defaults to
  "hive".

hiverc_location
  Optional path to hive rc file.

metastore_host
  Hostname for metastore.

metastore_port
  Port for hive to connect to metastore host.

release
  If set to "apache", uses a hive client that better handles apache
  hive output. All other values use the standard client Defaults to
  "cdh4".


[kubernetes]
------------

Parameters controlling Kubernetes Job Tasks

auth_method
  Authorization method to access the cluster.
  Options are "kubeconfig_" or "service-account_"

kubeconfig_path
  Path to kubeconfig file, for cluster authentication.
  It defaults to ``~/.kube/config``, which is the default location when
  using minikube_.
  When auth_method is "service-account" this property is ignored.

max_retrials
  Maximum number of retrials in case of job failure.

.. _service-account: http://kubernetes.io/docs/user-guide/kubeconfig-file
.. _kubeconfig: http://kubernetes.io/docs/user-guide/service-accounts
.. _minikube: http://kubernetes.io/docs/getting-started-guides/minikube


[mysql]
-------

Parameters controlling use of MySQL targets

marker_table
  Table in which to store status of table updates. This table will be
  created if it doesn't already exist. Defaults to "table_updates".


[postgres]
----------

Parameters controlling the use of Postgres targets

local_tmp_dir
  Directory in which to temporarily store data before writing to
  postgres. Uses system default if not specified.

marker_table
  Table in which to store status of table updates. This table will be
  created if it doesn't already exist. Defaults to "table_updates".


[prometheus]
------------

use_task_family_in_labels
  Should task family be used as a prometheus bucket label.
  Default value is true.

task_parameters_to_use_in_labels
  List of task arguments' names used as additional prometheus bucket labels.
  Passed in a form of a json list.


[redshift]
----------

Parameters controlling the use of Redshift targets

marker_table
  Table in which to store status of table updates. This table will be
  created if it doesn't already exist. Defaults to "table_updates".

.. _resources-config:

[resources]
-----------

This section can contain arbitrary keys. Each of these specifies the
amount of a global resource that the scheduler can allow workers to use.
The scheduler will prevent running jobs with resources specified from
exceeding the counts in this section. Unspecified resources are assumed
to have limit 1. Example resources section for a configuration with 2
hive resources and 1 mysql resource:

.. code:: ini

  [resources]
  hive=2
  mysql=1

Note that it was not necessary to specify the 1 for mysql here, but it
is good practice to do so when you have a fixed set of resources.

.. _retcode-config:

[retcode]
---------

Configure return codes for the Luigi binary. In the case of multiple return
codes that could apply, for example a failing task and missing data, the
*numerically greatest* return code is returned.

We recommend that you copy this set of exit codes to your ``luigi.cfg`` file:

.. code:: ini

  [retcode]
  # The following return codes are the recommended exit codes for Luigi
  # They are in increasing level of severity (for most applications)
  already_running=10
  missing_data=20
  not_run=25
  task_failed=30
  scheduling_error=35
  unhandled_exception=40

already_running
  This can happen in two different cases. Either the local lock file was taken
  at the time the invocation starts up. Or, the central scheduler have reported
  that some tasks could not have been run, because other workers are already
  running the tasks.
missing_data
  For when an :py:class:`~luigi.task.ExternalTask` is not complete, and this
  caused the worker to give up.  As an alternative to fiddling with this, see
  the [worker] keep_alive option.
not_run
  For when a task is not granted run permission by the scheduler. Typically
  because of lack of resources, because the task has been already run by
  another worker or because the attempted task is in DISABLED state.
  Connectivity issues with the central scheduler might also cause this.
  This does not include the cases for which a run is not allowed due to missing
  dependencies (missing_data) or due to the fact that another worker is currently
  running the task (already_running).
task_failed
  For signaling that there were last known to have failed. Typically because
  some exception have been raised.
scheduling_error
  For when a task's ``complete()`` or ``requires()`` method fails with an
  exception, or when the limit number of tasks is reached.
unhandled_exception
  For internal Luigi errors.  Defaults to 4, since this type of error
  probably will not recover over time.

If you customize return codes, prefer to set them in range 128 to 255 to avoid
conflicts. Return codes in range 0 to 127 are reserved for possible future use
by Luigi contributors.

[scalding]
----------

Parameters controlling running of scalding jobs

scala_home
  Home directory for scala on your machine. Defaults to either
  SCALA_HOME or /usr/share/scala if SCALA_HOME is unset.

scalding_home
  Home directory for scalding on your machine. Defaults to either
  SCALDING_HOME or /usr/share/scalding if SCALDING_HOME is unset.

scalding_provided
  Provided directory for scalding on your machine. Defaults to either
  SCALDING_HOME/provided or /usr/share/scalding/provided

scalding_libjars
  Libjars directory for scalding on your machine. Defaults to either
  SCALDING_HOME/libjars or /usr/share/scalding/libjars


.. _scheduler-config:

[scheduler]
-----------

Parameters controlling scheduler behavior

batch_emails
  Whether to send batch e-mails for failures and disables rather than
  sending immediate disable e-mails and just relying on workers to send
  immediate batch e-mails.
  Defaults to false.

disable_hard_timeout
  Hard time limit after which tasks will be disabled by the server if
  they fail again, in seconds. It will disable the task if it fails
  **again** after this amount of time. E.g. if this was set to 600
  (i.e. 10 minutes), and the task first failed at 10:00am, the task would
  be disabled if it failed again any time after 10:10am. Note: This setting
  does not consider the values of the ``retry_count`` or
  ``disable_window`` settings.

retry_count
  Number of times a task can fail within ``disable_window`` before
  the scheduler will automatically disable it. If not set, the scheduler
  will not automatically disable jobs.

disable_persist
  Number of seconds for which an automatic scheduler disable lasts.
  Defaults to 86400 (1 day).

disable_window
  Number of seconds during which ``retry_count`` failures must
  occur in order for an automatic disable by the scheduler. The
  scheduler forgets about disables that have occurred longer ago than
  this amount of time. Defaults to 3600 (1 hour).

max_shown_tasks
  .. versionadded:: 1.0.20

  The maximum number of tasks returned in a task_list api call. This
  will restrict the number of tasks shown in task lists in the
  visualiser. Small values can alleviate frozen browsers when there are
  too many done tasks. This defaults to 100000 (one hundred thousand).

max_graph_nodes
  .. versionadded:: 2.0.0

  The maximum number of nodes returned by a dep_graph or
  inverse_dep_graph api call. Small values can greatly speed up graph
  display in the visualiser by limiting the number of nodes shown. Some
  of the nodes that are not sent to the visualiser will still show up as
  dependencies of nodes that were sent. These nodes are given TRUNCATED
  status.

record_task_history
  If true, stores task history in a database. Defaults to false.

remove_delay
  Number of seconds to wait before removing a task that has no
  stakeholders. Defaults to 600 (10 minutes).

retry_delay
  Number of seconds to wait after a task failure to mark it pending
  again. Defaults to 900 (15 minutes).

state_path
  Path in which to store the Luigi scheduler's state. When the scheduler
  is shut down, its state is stored in this path. The scheduler must be
  shut down cleanly for this to work, usually with a kill command. If
  the kill command includes the -9 flag, the scheduler will not be able
  to save its state. When the scheduler is started, it will load the
  state from this path if it exists. This will restore all scheduled
  jobs and other state from when the scheduler last shut down.

  Sometimes this path must be deleted when restarting the scheduler
  after upgrading Luigi, as old state files can become incompatible
  with the new scheduler. When this happens, all workers should be
  restarted after the scheduler both to become compatible with the
  updated code and to reschedule the jobs that the scheduler has now
  forgotten about.

  This defaults to /var/lib/luigi-server/state.pickle

worker_disconnect_delay
  Number of seconds to wait after a worker has stopped pinging the
  scheduler before removing it and marking all of its running tasks as
  failed. Defaults to 60.

pause_enabled
  If false, disables pause/unpause operations and hides the pause toggle from
  the visualiser.

send_messages
  When true, the scheduler is allowed to send messages to running tasks and
  the central scheduler provides a simple prompt per task to send messages.
  Defaults to true.

metrics_collector
  Optional setting allowing Luigi to use a contribution to collect metrics
  about the pipeline to a third-party. By default this uses the default metric
  collector that acts as a shell and does nothing. The currently available
  options are "datadog", "prometheus" and "custom". If it's custom the
  'metrics_custom_import' needs to be set.

metrics_custom_import
  Optional setting allowing Luigi to import a custom subclass of MetricsCollector
  at runtime. The string should be formatted like "module.sub_module.ClassName".


[sendgrid]
----------

These parameters control sending error e-mails through SendGrid.

apikey
  API key of the SendGrid account.


[smtp]
------

These parameters control the smtp server setup.

host
  Hostname for sending mail through smtp. Defaults to localhost.

local_hostname
  If specified, overrides the FQDN of localhost in the HELO/EHLO
  command.

no_tls
  If true, connects to smtp without TLS. Defaults to false.

password
  Password to log in to your smtp server. Must be specified for
  username to have an effect.

port
  Port number for smtp on smtp_host. Defaults to 0.

ssl
  If true, connects to smtp through SSL. Defaults to false.

timeout
  Sets the number of seconds after which smtp attempts should time out.
  Defaults to 10.

username
  Username to log in to your smtp server, if necessary.


[spark]
-------

Parameters controlling the default execution of :py:class:`~luigi.contrib.spark.SparkSubmitTask` and :py:class:`~luigi.contrib.spark.PySparkTask`:

.. deprecated:: 1.1.1
   :py:class:`~luigi.contrib.spark.SparkJob`, :py:class:`~luigi.contrib.spark.Spark1xJob` and :py:class:`~luigi.contrib.spark.PySpark1xJob`
    are deprecated. Please use :py:class:`~luigi.contrib.spark.SparkSubmitTask` or :py:class:`~luigi.contrib.spark.PySparkTask`.

spark_submit
  Command to run in order to submit spark jobs. Default: ``"spark-submit"``

master
  Master url to use for ``spark_submit``. Example: local[*], spark://masterhost:7077. Default: Spark default (Prior to 1.1.1: yarn-client)

deploy_mode
    Whether to launch the driver programs locally ("client") or on one of the worker machines inside the cluster ("cluster"). Default: Spark default

jars
    Comma-separated list of local jars to include on the driver and executor classpaths. Default: Spark default

packages
    Comma-separated list of packages to link to on the driver and executors

py_files
    Comma-separated list of .zip, .egg, or .py files to place on the ``PYTHONPATH`` for Python apps. Default: Spark default

files
    Comma-separated list of files to be placed in the working directory of each executor. Default: Spark default

conf:
    Arbitrary Spark configuration property in the form Prop=Value|Prop2=Value2. Default: Spark default

properties_file
    Path to a file from which to load extra properties. Default: Spark default

driver_memory
    Memory for driver (e.g. 1000M, 2G). Default: Spark default

driver_java_options
    Extra Java options to pass to the driver. Default: Spark default

driver_library_path
    Extra library path entries to pass to the driver. Default: Spark default

driver_class_path
    Extra class path entries to pass to the driver. Default: Spark default

executor_memory
    Memory per executor (e.g. 1000M, 2G). Default: Spark default

*Configuration for Spark submit jobs on Spark standalone with cluster deploy mode only:*

driver_cores
    Cores for driver. Default: Spark default

supervise
    If given, restarts the driver on failure. Default: Spark default

*Configuration for Spark submit jobs on Spark standalone and Mesos only:*

total_executor_cores
    Total cores for all executors. Default: Spark default

*Configuration for Spark submit jobs on YARN only:*

executor_cores
    Number of cores per executor. Default: Spark default

queue
    The YARN queue to submit to. Default: Spark default

num_executors
    Number of executors to launch. Default: Spark default

archives
    Comma separated list of archives to be extracted into the working directory of each executor. Default: Spark default

hadoop_conf_dir
  Location of the hadoop conf dir. Sets HADOOP_CONF_DIR environment variable
  when running spark. Example: /etc/hadoop/conf

*Extra configuration for PySparkTask jobs:*

py_packages
    Comma-separated list of local packages (in your python path) to be distributed to the cluster.

*Parameters controlling the execution of SparkJob jobs (deprecated):*


[task_history]
--------------

Parameters controlling storage of task history in a database

db_connection
  Connection string for connecting to the task history db using
  sqlalchemy.


[execution_summary]
-------------------

Parameters controlling execution summary of a worker

summary_length
  Maximum number of tasks to show in an execution summary.  If the value is 0,
  then all tasks will be displayed.  Default value is 5.


[webhdfs]
---------

port
  The port to use for webhdfs. The normal namenode port is probably on a
  different port from this one.

user
  Perform file system operations as the specified user instead of $USER.  Since
  this parameter is not honored by any of the other hdfs clients, you should
  think twice before setting this parameter.

client_type
  The type of client to use. Default is the "insecure" client that requires no
  authentication. The other option is the "kerberos" client that uses kerberos
  authentication.

[datadog]
---------

api_key
  The api key found in the account settings of Datadog under the API
  sections.
app_key
  The application key found in the account settings of Datadog under the API
  sections.
default_tags
  Optional settings that adds the tag to all the metrics and events sent to
  Datadog. Default value is "application:luigi".
environment
  Allows you to tweak multiple environment to differentiate between production,
  staging or development metrics within Datadog. Default value is "development".
statsd_host
  The host that has the statsd instance to allow Datadog to send statsd metric. Default value is "localhost".
statsd_port
  The port on the host that allows connection to the statsd host. Defaults value is 8125.
metric_namespace
  Optional prefix to add to the beginning of every metric sent to Datadog.
  Default value is "luigi".


Per Task Retry-Policy
---------------------

Luigi also supports defining ``retry_policy`` per task.

.. code-block:: python

    class GenerateWordsFromHdfs(luigi.Task):

       retry_count = 2

        ...

    class GenerateWordsFromRDBM(luigi.Task):

       retry_count = 5

        ...

    class CountLetters(luigi.Task):

        def requires(self):
            return [GenerateWordsFromHdfs()]

        def run():
            yield GenerateWordsFromRDBM()

        ...

If none of retry-policy fields is defined per task, the field value will be **default** value which is defined in luigi config file.

To make luigi sticks to the given retry-policy, be sure you run luigi worker with ``keep_alive`` config. Please check ``keep_alive`` config in :ref:`worker-config` section.

Retry-Policy Fields
-------------------

The fields below are in retry-policy and they can be defined per task.

* ``retry_count``
* ``disable_hard_timeout``
* ``disable_window``


================================================
FILE: doc/design_and_limitations.rst
================================================
Design and limitations
----------------------

Luigi is the successor to a couple of attempts that we weren't fully happy with.
We learned a lot from our mistakes and some design decisions include:

-  Straightforward command-line integration.
-  As little boilerplate as possible.
-  Focus on job scheduling and dependency resolution, not a particular platform.
   In particular, this means no limitation to Hadoop.
   Though Hadoop/HDFS support is built-in and is easy to use,
   this is just one of many types of things you can run.
-  A file system abstraction where code doesn't have to care about where files are located.
-  Atomic file system operations through this abstraction.
   If a task crashes it won't lead to a broken state.
-  The dependencies are decentralized.
   No big config file in XML.
   Each task just specifies which inputs it needs and cross-module dependencies are trivial.
-  A web server that renders the dependency graph and does locking, etc for free.
-  Trivial to extend with new file systems, file formats, and job types.
   You can easily write jobs that inserts a Tokyo Cabinet into Cassandra.
   Adding support for new systems is generally not very hard.
   (Feel free to send us a patch when you're done!)
-  Date algebra included.
-  Lots of unit tests of the most basic stuff.

It wouldn't be fair not to mention some limitations with the current design:

-  Its focus is on batch processing so
   it's probably less useful for near real-time pipelines or continuously running processes.
-  The assumption is that each task is a sizable chunk of work.
   While you can probably schedule a few thousand jobs,
   it's not meant to scale beyond tens of thousands.
-  Luigi does not support distribution of execution.
   When you have workers running thousands of jobs daily, this starts to matter,
   because the worker nodes get overloaded.
   There are some ways to mitigate this (trigger from many nodes, use resources),
   but none of them are ideal.
-  Luigi does not come with built-in triggering, and you still need to rely on something like
   crontab to trigger workflows periodically.

Also, it should be mentioned that Luigi is named after the world's second most famous plumber.


================================================
FILE: doc/example_top_artists.rst
================================================
Example – Top Artists
---------------------

This is a very simplified case of something we do at Spotify a lot.
All user actions are logged to Google Cloud Storage (previously HDFS) where
we run a bunch of processing jobs to transform the data. The processing code itself is implemented
in a scalable data processing framework, such as Scio, Scalding, or Spark, but the jobs
are orchestrated with Luigi.
At some point we might end up with
a smaller data set that we can bulk ingest into Cassandra, Postgres, or
other storage suitable for serving or exploration.

For the purpose of this exercise, we want to aggregate all streams,
find the top 10 artists and then put the results into Postgres.

This example is also available in
`examples/top_artists.py <https://github.com/spotify/luigi/blob/master/examples/top_artists.py>`_.

Step 1 - Aggregate Artist Streams
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: python

    class AggregateArtists(luigi.Task):
        date_interval = luigi.DateIntervalParameter()

        def output(self):
            return luigi.LocalTarget("data/artist_streams_%s.tsv" % self.date_interval)

        def requires(self):
            return [Streams(date) for date in self.date_interval]

        def run(self):
            artist_count = defaultdict(int)

            for input in self.input():
                with input.open('r') as in_file:
                    for line in in_file:
                        timestamp, artist, track = line.strip().split()
                        artist_count[artist] += 1

            with self.output().open('w') as out_file:
                for artist, count in artist_count.iteritems():
                    print(artist, count, file=out_file)

Note that this is just a portion of the file ``examples/top_artists.py``.
In particular, ``Streams`` is defined as a :class:`~luigi.task.Task`,
acting as a dependency for ``AggregateArtists``.
In addition, ``luigi.run()`` is called if the script is executed directly,
allowing it to be run from the command line.

There are several pieces of this snippet that deserve more explanation.

-  Any :class:`~luigi.task.Task` may be customized by instantiating one
   or more :class:`~luigi.parameter.Parameter` objects on the class level.
-  The :func:`~luigi.task.Task.output` method tells Luigi where the result
   of running the task will end up. The path can be some function of the
   parameters.
-  The :func:`~luigi.task.Task.requires` tasks specifies other tasks that
   we need to perform this task. In this case it's an external dump named
   *Streams* which takes the date as the argument.
-  For plain Tasks, the :func:`~luigi.task.Task.run` method implements the
   task. This could be anything, including calling subprocesses, performing
   long running number crunching, etc. For some subclasses of
   :class:`~luigi.task.Task` you don't have to implement the ``run``
   method. For instance, for the :class:`~luigi.contrib.hadoop.JobTask`
   subclass you implement a *mapper* and *reducer* instead.
-  :class:`~luigi.LocalTarget` is a built in class that makes it
   easy to read/write from/to the local filesystem. It also makes all file operations
   atomic, which is nice in case your script crashes for any reason.

Running this Locally
~~~~~~~~~~~~~~~~~~~~

Try running this using eg.

.. code-block:: console

    $ cd examples
    $ luigi --module top_artists AggregateArtists --local-scheduler --date-interval 2012-06

Note that  *top_artists* needs to be in your PYTHONPATH, or else this can produce an error (*ImportError: No module named top_artists*). Add the current working directory to the command PYTHONPATH with:

.. code-block:: console

    $ PYTHONPATH='.' luigi --module top_artists AggregateArtists --local-scheduler --date-interval 2012-06

You can also try to view the manual using ``--help`` which will give you an
overview of the options.

Running the command again will do nothing because the output file is
already created.
In that sense, any task in Luigi is *idempotent*
because running it many times gives the same outcome as running it once.
Note that unlike Makefile, the output will not be recreated when any of
the input files is modified.
You need to delete the output file
manually.

The ``--local-scheduler`` flag tells Luigi not to connect to a scheduler
server. This is not recommended for other purpose than just testing
things.

Step 1b - Aggregate artists with Spark
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

While Luigi can process data inline, it is normally used to orchestrate external programs that
perform the actual processing. In this example, we will demonstrate how top artists instead can be
read from HDFS and calculated with Spark, orchestrated by Luigi.

.. code:: python

    class AggregateArtistsSpark(luigi.contrib.spark.SparkSubmitTask):
        date_interval = luigi.DateIntervalParameter()

        app = 'top_artists_spark.py'
        master = 'local[*]'

        def output(self):
            return luigi.contrib.hdfs.HdfsTarget("data/artist_streams_%s.tsv" % self.date_interval)

        def requires(self):
            return [StreamsHdfs(date) for date in self.date_interval]

        def app_options(self):
            # :func:`~luigi.task.Task.input` returns the targets produced by the tasks in
            # `~luigi.task.Task.requires`.
            return [','.join([p.path for p in self.input()]),
                    self.output().path]


:class:`luigi.contrib.hadoop.SparkSubmitTask` doesn't require you to implement a
:func:`~luigi.task.Task.run` method. Instead, you specify the command line parameters to send
to ``spark-submit``, as well as any other configuration specific to Spark.

Python code for the Spark job is found below.

.. code:: python

    import operator
    import sys
    from pyspark.sql import SparkSession


    def main(argv):
        input_paths = argv[1].split(',')
        output_path = argv[2]

        spark = SparkSession.builder.getOrCreate()

        streams = spark.read.option('sep', '\t').csv(input_paths[0])
        for stream_path in input_paths[1:]:
            streams.union(spark.read.option('sep', '\t').csv(stream_path))

        # The second field is the artist
        counts = streams \
            .map(lambda row: (row[1], 1)) \
            .reduceByKey(operator.add)

        counts.write.option('sep', '\t').csv(output_path)


    if __name__ == '__main__':
        sys.exit(main(sys.argv))


In a typical deployment scenario, the Luigi orchestration definition above as well as the
Pyspark processing code would be packaged into a deployment package, such as a container image. The
processing code does not have to be implemented in Python, any program can be packaged in the
image and run from Luigi.


Step 2 – Find the Top Artists
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

At this point, we've counted the number of streams for each artists,
for the full time period.
We are left with a large file that contains
mappings of artist -> count data, and we want to find the top 10 artists.
Since we only have a few hundred thousand artists, and
calculating artists is nontrivial to parallelize,
we choose to do this not as a Hadoop job, but just as a plain old for-loop in Python.

.. code:: python

    class Top10Artists(luigi.Task):
        date_interval = luigi.DateIntervalParameter()
        use_hadoop = luigi.BoolParameter()

        def requires(self):
            if self.use_hadoop:
                return AggregateArtistsSpark(self.date_interval)
            else:
                return AggregateArtists(self.date_interval)

        def output(self):
            return luigi.LocalTarget("data/top_artists_%s.tsv" % self.date_interval)

        def run(self):
            top_10 = nlargest(10, self._input_iterator())
            with self.output().open('w') as out_file:
                for streams, artist in top_10:
                    print(self.date_interval.date_a, self.date_interval.date_b, artist, streams, file=out_file)

        def _input_iterator(self):
            with self.input().open('r') as in_file:
                for line in in_file:
                    artist, streams = line.strip().split()
                    yield int(streams), int(artist)

The most interesting thing here is that this task (*Top10Artists*)
defines a dependency on the previous task (*AggregateArtists*).
This means that if the output of *AggregateArtists* does not exist,
the task will run before *Top10Artists*.

.. code-block:: console

    $ luigi --module examples.top_artists Top10Artists --local-scheduler --date-interval 2012-07

This will run both tasks.

Step 3 - Insert into Postgres
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This mainly serves as an example of a specific subclass *Task* that
doesn't require any code to be written.
It's also an example of how you can define task templates that
you can reuse for a lot of different tasks.

.. code:: python

    class ArtistToplistToDatabase(luigi.contrib.postgres.CopyToTable):
        date_interval = luigi.DateIntervalParameter()
        use_hadoop = luigi.BoolParameter()

        host = "localhost"
        database = "toplists"
        user = "luigi"
        password = "abc123"  # ;)
        table = "top10"

        columns = [("date_from", "DATE"),
                   ("date_to", "DATE"),
                   ("artist", "TEXT"),
                   ("streams", "INT")]

        def requires(self):
            return Top10Artists(self.date_interval, self.use_hadoop)

Just like previously, this defines a recursive dependency on the
previous task. If you try to build the task, that will also trigger
building all its upstream dependencies.

Using the Central Planner
~~~~~~~~~~~~~~~~~~~~~~~~~

The ``--local-scheduler`` flag tells Luigi not to connect to a central scheduler.
This is recommended in order to get started and or for development purposes.
At the point where you start putting things in production
we strongly recommend running the central scheduler server.
In addition to providing locking
so that the same task is not run by multiple processes at the same time,
this server also provides a pretty nice visualization of your current work flow.

If you drop the ``--local-scheduler`` flag,
your script will try to connect to the central planner,
by default at localhost port 8082.
If you run

.. code-block:: console

    $ luigid

in the background and then run your task without the ``--local-scheduler`` flag,
then your script will now schedule through a centralized server.
You need `Tornado <http://www.tornadoweb.org/>`__ for this to work.

Launching http://localhost:8082 should show something like this:

.. figure:: web_server.png
   :alt: Web server screenshot

Web server screenshot
Looking at the dependency graph
for any of the tasks yields something like this:

.. figure:: aggregate_artists.png
   :alt: Aggregate artists screenshot

Aggregate artists screenshot

In production, you'll want to run the centralized scheduler.
See: :doc:`central_scheduler` for more information.


================================================
FILE: doc/execution_model.rst
================================================
Execution Model
---------------

Luigi has a quite simple model for execution and triggering.

Workers and task execution
~~~~~~~~~~~~~~~~~~~~~~~~~~

The most important aspect is that *no execution is transferred*.
When you run a Luigi workflow,
the worker schedules all tasks, and
also executes the tasks within the process.

    .. figure:: execution_model.png
       :alt: Execution model

The benefit of this scheme is that
it's super easy to debug since all execution takes place in the process.
It also makes deployment a non-event.
During development,
you typically run the Luigi workflow from the command line,
whereas when you deploy it,
you can trigger it using crontab or any other scheduler.

The downside is that Luigi doesn't give you scalability for free.
In practice this is not a problem until you start running thousands of tasks.

Isn't the point of Luigi to automate and schedule these workflows?
To some extent.
Luigi helps you *encode the dependencies* of tasks and build up chains.
Furthermore, Luigi's scheduler makes sure that there's a centralized view of the dependency graph and
that the same job will not be executed by multiple workers simultaneously.

Scheduler
~~~~~~~~~

A client only starts the ``run()`` method of a task when the single-threaded
central scheduler has permitted it. Since the number of tasks is usually very
small (in comparison with the petabytes of data one task is processing), we
can afford the convenience of a simple centralised server.

.. figure:: https://tarrasch.github.io/luigid-basics-jun-2015/img/50.gif
   :alt: Scheduling gif

The gif is from `this presentation
<https://tarrasch.github.io/luigid-basics-jun-2015/>`__, which is about the
client and server interaction.

Triggering tasks
~~~~~~~~~~~~~~~~

Luigi does not include its own triggering, so you have to rely on an external scheduler
such as crontab to actually trigger the workflows.

In practice, it's not a big hurdle because Luigi avoids all the mess typically caused by it.
Scheduling a complex workflow is fairly trivial using eg. crontab.

In the future, Luigi might implement its own triggering.
The dependency on crontab (or any external triggering mechanism) is a bit awkward and it would be nice to avoid.

Trigger example
^^^^^^^^^^^^^^^

For instance, if you have an external data dump that arrives every day and that your workflow depends on it,
you write a workflow that depends on this data dump.
Crontab can then trigger this workflow *every minute* to check if the data has arrived.
If it has, it will run the full dependency graph.

.. code:: python

    # my_tasks.py

    class DataDump(luigi.ExternalTask):
        date = luigi.DateParameter()
        def output(self): return luigi.contrib.hdfs.HdfsTarget(self.date.strftime('/var/log/dump/%Y-%m-%d.txt'))

    class AggregationTask(luigi.Task):
        date = luigi.DateParameter()
        window = luigi.IntParameter()
        def requires(self): return [DataDump(self.date - datetime.timedelta(i)) for i in xrange(self.window)]
        def run(self): run_some_cool_stuff(self.input())
        def output(self): return luigi.contrib.hdfs.HdfsTarget('/aggregated-%s-%d' % (self.date, self.window))

    class RunAll(luigi.Task):
        ''' Dummy task that triggers execution of a other tasks'''
        def requires(self):
            for window in [3, 7, 14]:
                for d in xrange(10): # guarantee that aggregations were run for the past 10 days
                   yield AggregationTask(datetime.date.today() - datetime.timedelta(d), window)

In your cronline you would then have something like

.. code:: console

    30 0 * * * my-user luigi RunAll --module my_tasks


You can trigger this as much as you want from crontab, and
even across multiple machines, because
the central scheduler will make sure at most one of each ``AggregationTask`` task is run simultaneously.
Note that this might actually mean multiple tasks can be run because
there are instances with different parameters, and
this can give you some form of parallelization
(eg. ``AggregationTask(2013-01-09)`` might run in parallel with ``AggregationTask(2013-01-08)``).

Of course,
some Task types (eg. ``HadoopJobTask``) can transfer execution to other places, but
this is up to each Task to define.


================================================
FILE: doc/index.rst
================================================
.. Luigi documentation master file, created by
   sphinx-quickstart on Sat Feb  8 00:56:43 2014.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

.. include:: ../README.rst

Table of Contents
-----------------

.. toctree::
   :maxdepth: 2

   example_top_artists.rst
   workflows.rst
   tasks.rst
   parameters.rst
   running_luigi.rst
   central_scheduler.rst
   execution_model.rst
   luigi_patterns.rst
   configuration.rst
   logging.rst
   design_and_limitations.rst
   mypy.rst

API Reference
-------------

.. autosummary::
   :toctree: api
   :recursive:

   luigi


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`


================================================
FILE: doc/logging.rst
================================================
Configure logging
-----------------


Config options:
~~~~~~~~~~~~~~~

Some config options for config [core] section

log_level
    The default log level to use when no logging_conf_file is set. Must be
    a valid name of a `Python log level
    <https://docs.python.org/3/library/logging.html#logging-levels>`_.
    Default is ``DEBUG``.
logging_conf_file
      Location of the logging configuration file.
no_configure_logging
    If true, logging is not configured. Defaults to false.


Config section
~~~~~~~~~~~~~~

If you're use TOML for configuration file, you can configure logging
via ``logging`` section in this file. See `example
<https://github.com/spotify/luigi/blob/master/examples/config.toml>`_
for more details.

Luigid CLI options:
~~~~~~~~~~~~~~~~~~~

``--background``
    Run daemon in background mode. Disable logging setup
    and set up log level to INFO for root logger.
``--logdir``
    set logging with INFO level and output in ``$logdir/luigi-server.log`` file


Worker CLI options:
~~~~~~~~~~~~~~~~~~~

``--logging-conf-file``
    Configuration file for logging.
``--log-level``
    Default log level.
    Available values: NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL.
    Default DEBUG. See `Python documentation
    <https://docs.python.org/3/library/logging.html#logging-levels>`_
    For information about levels difference.


Configuration options resolution order:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. no_configure_logging option
2. ``--background``
3. ``--logdir``
4. ``--logging-conf-file``
5. logging_conf_file option
6. ``logging`` section
7. ``--log-level``
8. log_level option


================================================
FILE: doc/luigi_patterns.rst
================================================
Luigi Patterns
--------------

Code Reuse
~~~~~~~~~~

One nice thing about Luigi is that it's super easy to depend on tasks defined in other repos.
It's also trivial to have "forks" in the execution path,
where the output of one task may become the input of many other tasks.

Currently, no semantics for "intermediate" output is supported,
meaning that all output will be persisted indefinitely.
The upside of that is that if you try to run X -> Y, and Y crashes,
you can resume with the previously built X.
The downside is that you will have a lot of intermediate results on your file system.
A useful pattern is to put these files in a special directory and
have some kind of periodical garbage collection clean it up.

Triggering Many Tasks
~~~~~~~~~~~~~~~~~~~~~

A convenient pattern is to have a dummy Task at the end of several
dependency chains, so you can trigger a multitude of pipelines by
specifying just one task in command line, similarly to how e.g. `make <http://www.gnu.org/software/make/>`_
works.

.. code:: python

    class AllReports(luigi.WrapperTask):
        date = luigi.DateParameter(default=datetime.date.today())
        def requires(self):
            yield SomeReport(self.date)
            yield SomeOtherReport(self.date)
            yield CropReport(self.date)
            yield TPSReport(self.date)
            yield FooBarBazReport(self.date)

This simple task will not do anything itself, but will invoke a bunch of
other tasks. Per each invocation, Luigi will perform as many of the pending
jobs as possible (those which have all their dependencies present).

You'll need to use :class:`~luigi.task.WrapperTask` for this instead of the usual Task class, because this job will not produce any output of its own, and as such needs a way to indicate when it's complete. This class is used for tasks that only wrap other tasks and that by definition are done if all their requirements exist.

Triggering recurring tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~

A common requirement is to have a daily report (or something else)
produced every night. Sometimes for various reasons tasks will keep
crashing or lacking their required dependencies for more than a day
though, which would lead to a missing deliverable for some date. Oops.

To ensure that the above AllReports task is eventually completed for
every day (value of date parameter), one could e.g. add a loop in
requires method to yield dependencies on the past few days preceding
self.date. Then, so long as Luigi keeps being invoked, the backlog of
jobs would catch up nicely after fixing intermittent problems.

Luigi actually comes with a reusable tool for achieving this, called
:class:`~luigi.tools.range.RangeDailyBase` (resp. :class:`~luigi.tools.range.RangeHourlyBase`). Simply putting

.. code-block:: console

	luigi --module all_reports RangeDailyBase --of AllReports --start 2015-01-01

in your crontab will easily keep gaps from occurring from 2015-01-01
onwards. NB - it will not always loop over everything from 2015-01-01
till current time though, but rather a maximum of 3 months ago by
default - see :class:`~luigi.tools.range.RangeDailyBase` documentation for this and more knobs
for tweaking behavior. See also Monitoring below.

Efficiently triggering recurring tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

RangeDailyBase, described above, is named like that because a more
efficient subclass exists, :class:`~luigi.tools.range.RangeDaily` (resp. :class:`~luigi.tools.range.RangeHourly`), tailored for
hundreds of task classes scheduled concurrently with contiguousness
requirements spanning years (which would incur redundant completeness
checks and scheduler overload using the naive looping approach.) Usage:

.. code-block:: console

	luigi --module all_reports RangeDaily --of AllReports --start 2015-01-01

It has the same knobs as RangeDailyBase, with some added requirements.
Namely the task must implement an efficient bulk_complete method, or
must be writing output to file system Target with date parameter value
consistently represented in the file path.

Backfilling tasks
~~~~~~~~~~~~~~~~~

Also a common use case, sometimes you have tweaked existing recurring
task code and you want to schedule recomputation of it over an interval
of dates for that or another reason. Most conveniently it is achieved
with the above described range tools, just with both start (inclusive)
and stop (exclusive) parameters specified:

.. code-block:: console

	luigi --module all_reports RangeDaily --of AllReportsV2 --start 2014-10-31 --stop 2014-12-25

Propagating parameters with Range
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some tasks you want to recur may include additional parameters which need to be configured.
The Range classes provide a parameter which accepts a :class:`~luigi.parameter.DictParameter`
and passes any parameters onwards for this purpose.

.. code-block:: console

	luigi RangeDaily --of MyTask --start 2014-10-31 --of-params '{"my_string_param": "123", "my_int_param": 123}'

Alternatively, you can specify parameters at the task family level (as described :ref:`here <Parameter-class-level-parameters>`),
however these will not appear in the task name for the upstream Range task which
can have implications in how the scheduler and visualizer handle task instances.

.. code-block:: console

	luigi RangeDaily --of MyTask --start 2014-10-31 --MyTask-my-param 123

.. _batch_method:

Batching multiple parameter values into a single run
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes it'll be faster to run multiple jobs together as a single
batch rather than running them each individually. When this is the case,
you can mark some parameters with a batch_method in their constructor
to tell the worker how to combine multiple values. One common way to do
this is by simply running the maximum value. This is good for tasks that
overwrite older data when a newer one runs. You accomplish this by
setting the batch_method to max, like so:

.. code-block:: python

    class A(luigi.Task):
        date = luigi.DateParameter(batch_method=max)

What's exciting about this is that if you send multiple As to the
scheduler, it can combine them and return one. So if
``A(date=2016-07-28)``, ``A(date=2016-07-29)`` and
``A(date=2016-07-30)`` are all ready to run, you will start running
``A(date=2016-07-30)``. While this is running, the scheduler will show
``A(date=2016-07-28)``, ``A(date=2016-07-29)`` as batch running while
``A(date=2016-07-30)`` is running. When ``A(date=2016-07-30)`` is done
running and becomes FAILED or DONE, the other two tasks will be updated
to the same status.

If you want to limit how big a batch can get, simply set max_batch_size.
So if you have

.. code-block:: python

    class A(luigi.Task):
        date = luigi.DateParameter(batch_method=max)

        max_batch_size = 10

then the scheduler will batch at most 10 jobs together. You probably do
not want to do this with the max batch method, but it can be helpful if
you use other methods. You can use any method that takes a list of
parameter values and returns a single parameter value.

If you have two max batch parameters, you'll get the max values for both
of them. If you have parameters that don't have a batch method, they'll
be aggregated separately. So if you have a class like

.. code-block:: python

    class A(luigi.Task):
        p1 = luigi.IntParameter(batch_method=max)
        p2 = luigi.IntParameter(batch_method=max)
        p3 = luigi.IntParameter()

and you create tasks ``A(p1=1, p2=2, p3=0)``, ``A(p1=2, p2=3, p3=0)``,
``A(p1=3, p2=4, p3=1)``, you'll get them batched as
``A(p1=2, p2=3, p3=0)`` and ``A(p1=3, p2=4, p3=1)``.

Note that batched tasks do not take up :ref:`resources-config`, only the
task that ends up running will use resources. The scheduler only checks
that there are sufficient resources for each task individually before
batching them all together.

Tasks that regularly overwrite the same data source
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you are overwriting of the same data source with every run, you'll
need to ensure that two batches can't run at the same time. You can do
this pretty easily by setting batch_method to max and setting a unique
resource:

.. code-block:: python

    class A(luigi.Task):
        date = luigi.DateParameter(batch_method=max)

        resources = {'overwrite_resource': 1}

Now if you have multiple tasks such as ``A(date=2016-06-01)``,
``A(date=2016-06-02)``, ``A(date=2016-06-03)``, the scheduler will just
tell you to run the highest available one and mark the lower ones as
batch_running. Using a unique resource will prevent multiple tasks from
writing to the same location at the same time if a new one becomes
available while others are running.

Avoiding concurrent writes to a single file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Updating a single file from several tasks is almost always a bad idea, and you
need to be very confident that no other good solution exists before doing this.
If, however, you have no other option, then you will probably at least need to ensure that
no two tasks try to write to the file _simultaneously_.

By turning 'resources' into a Python property, it can return a value dependent on
the task parameters or other dynamic attributes:

.. code-block:: python

    class A(luigi.Task):
        ...

        @property
        def resources(self):
            return { self.important_file_name: 1 }

Since, by default, resources have a usage limit of 1, no two instances of Task A
will now run if they have the same `important_file_name` property.

Decreasing resources of running tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

At scheduling time, the luigi scheduler needs to be aware of the maximum
resource consumption a task might have once it runs. For some tasks, however,
it can be beneficial to decrease the amount of consumed resources between two
steps within their run method (e.g. after some heavy computation). In this
case, a different task waiting for that particular resource can already be
scheduled.

.. code-block:: python

    class A(luigi.Task):

        # set maximum resources a priori
        resources = {"some_resource": 3}

        def run(self):
            # do something
            ...

            # decrease consumption of "some_resource" by one
            self.decrease_running_resources({"some_resource": 1})

            # continue with reduced resources
            ...

Monitoring task pipelines
~~~~~~~~~~~~~~~~~~~~~~~~~

Luigi comes with some existing ways in :py:mod:`luigi.notifications` to receive
notifications whenever tasks crash. Email is the most common way.

The above mentioned range tools for recurring tasks not only implement
reliable scheduling for you, but also emit events which you can use to
set up delay monitoring. That way you can implement alerts for when
jobs are stuck for prolonged periods lacking input data or otherwise
requiring attention.

.. _AtomicWrites:

Atomic Writes Problem
~~~~~~~~~~~~~~~~~~~~~

A very common mistake done by luigi plumbers is to write data partially to the
final destination, that is, not atomically. The problem arises because
completion checks in luigi are exactly as naive as running
:meth:`luigi.target.Target.exists`. And in many cases it just means to check if
a folder exist on disk. During the time we have partially written data, a task
depending on that output would think its input is complete. This can have
devestating effects, as in `the thanksgiving bug
<http://tarrasch.github.io/luigi-budapest-bi-oct-2015/#/21>`__.

The concept can be illustrated by imagining that we deal with data stored on
local disk and by running commands:

.. code-block:: console

    # This the BAD way
    $ mkdir /outputs/final_output
    $ big-slow-calculation > /outputs/final_output/foo.data

As stated earlier, the problem is that only partial data exists for a duration,
yet we consider the data to be :meth:`~luigi.task.Task.complete` because the
output folder already exists. Here is a robust version of this:

.. code-block:: console

    # This is the good way
    $ mkdir /outputs/final_output-tmp-123456
    $ big-slow-calculation > /outputs/final_output-tmp-123456/foo.data
    $ mv --no-target-directory --no-clobber /outputs/final_output{-tmp-123456,}
    $ [[ -d /outputs/final_output-tmp-123456 ]] && rm -r /outputs/final_output-tmp-123456

Indeed, the good way is not as trivial. It involves coming up with a unique
directory name and a pretty complex ``mv`` line, the reason ``mv`` need all
those is because we don't want ``mv`` to move a directory into a potentially
existing directory. A directory could already exist in exceptional cases, for
example when central locking fails and the same task would somehow run twice at
the same time. Lastly, in the exceptional case where the file was never moved,
one might want to remove the temporary directory that never got used.

Note that this was an example where the storage was on local disk. But for
every storage (hard disk file, hdfs file, database table, etc.) this procedure
will look different. But do every luigi user need to implement that complexity?
Nope, thankfully luigi developers are aware of these and luigi comes with many
built-in solutions. In the case of you're dealing with a file system
(:class:`~luigi.target.FileSystemTarget`), you should consider using
:meth:`~luigi.target.FileSystemTarget.temporary_path`. For other targets, you
should ensure that the way you're writing your final output directory is
atomic.

Sending messages to tasks
~~~~~~~~~~~~~~~~~~~~~~~~~

The central scheduler is able to send messages to particular tasks. When a running task accepts
messages, it can access a `multiprocessing.Queue <https://docs.python.org/3/library/multiprocessing.html#pipes-and-queues>`__
object storing incoming messages. You can implement custom behavior to react and respond to
messages:

.. code-block:: python

    class Example(luigi.Task):

        # common task setup
        ...

        # configure the task to accept all incoming messages
        accepts_messages = True

        def run(self):
            # this example runs some loop and listens for the
            # "terminate" message, and responds to all other messages
            for _ in some_loop():
                # check incomming messages
                if not self.scheduler_messages.empty():
                    msg = self.scheduler_messages.get()
                    if msg.content == "terminate":
                        break
                    else:
                        msg.respond("unknown message")

            # finalize
            ...

Messages can be sent right from the scheduler UI which also displays responses (if any). Note that
this feature is only available when the scheduler is configured to send messages (see the :ref:`scheduler-config` config), and the task is configured to accept them.

Gathering custom metrics from tasks' executions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The central scheduler is able to gather custom metrics from tasks' executions with help of
custom metrics collector (see the :ref:`scheduler-config` config). To obtain custom metrics,
you need to implement:

#. Custom metrics collector class inheriting from
   :class:`~luigi.metrics.MetricsCollector` (or derived) and implementing the
   :meth:`~luigi.metrics.MetricsCollector.handle_task_statistics`
   method (default one does nothing). This method will be called for each task
   that has been executed everytime, when
   :meth:`~luigi.worker.TaskStatusReporter.report_task_statistics` is called.
   For instance, following metrics collector adds monitoring tasks' execution
   time and memory usage:

   .. code-block:: python

       class MetricsCollector(PrometheusMetricsCollector):
           def __init__(self, *args, **kwargs):
               super().__init__(*args, **kwargs)
               self.task_run_execution_time = Gauge(
                   'luigi_task_run_execution_time_seconds',
                   'luigi task run method execution time in seconds',
                   self.labels,
                   registry=self.registry
               )
               self.task_execution_memory = Gauge(
                   'luigi_task_max_memory_megabytes',
                   'luigi task run method max memory usage in megabytes',
                   self.labels,
                   registry=self.registry
               )

           def handle_task_statistics(self, task, statistics):
               if "elapsed" in statistics:
                   self.task_run_execution_time.labels(**self._generate_task_labels(task)).set(statistics["elapsed"])
               if "memory" in statistics:
                   self.task_execution_memory.labels(**self._generate_task_labels(task)).set(statistics["memory"])

#. Custom task context manager (see the :ref:`worker-config` config),
   which in `__exit__` method would call
   :meth:`~luigi.worker.TaskStatusReporter.report_task_statistics` method with
   the statistics dictionary. For instance, following task context manager collects
   task execution time and memory usage:

   .. code-block:: python

       class TaskContext:
           def __init__(self, task_process):
               self._task_process = task_process
               self._start = None

           def __enter__(self):
               self._start = time.perf_counter()
               return self

           def __exit__(self, exc_type, exc_val, exc_tb):
               assert self._start is not None
               elapsed = time.perf_counter() - self._start
               used_memory = max(
                   resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss
               )
               logging.getLogger("luigi-interface").info(
                   f'Task {self._task_process.task}: time: {elapsed:.2f}s, memory: {used_memory / 1024:.2f}MB '
               )
               self._task_process.status_reporter.report_task_statistics({"memory": used_memory / 1024, "elapsed": elapsed})


================================================
FILE: doc/mypy.rst
================================================
Mypy plugin
--------------

Mypy plugin provides type checking for ``luigi.Task`` using Mypy.

Require Python 3.8 or later.

How to use
~~~~~~~~~~

Configure Mypy to use this plugin by adding the following to your ``mypy.ini`` file:

.. code:: ini

    [mypy]
    plugins = luigi.mypy

or by adding the following to your ``pyproject.toml`` file:

.. code:: toml

    [tool.mypy]
    plugins = ["luigi.mypy"]

Then, run Mypy as usual.

Examples
~~~~~~~~

For example the following code linted by Mypy:

.. code:: python

    import luigi


    class MyTask(luigi.Task):
        foo: int = luigi.IntParameter()
        bar: str = luigi.Parameter()

    MyTask(foo=1, bar='2')   # OK
    MyTask(foo='1', bar='2')  # Error: Argument 1 to "Foo" has incompatible type "str"; expected "int"


================================================
FILE: doc/parameters.rst
================================================
Parameters
----------

Parameters is the Luigi equivalent of creating a constructor for each Task.
Luigi requires you to declare these parameters by instantiating
:class:`~luigi.parameter.Parameter` objects on the class scope:

.. code:: python

    class DailyReport(luigi.contrib.hadoop.JobTask):
        date = luigi.DateParameter(default=datetime.date.today())
        # ...

By doing this, Luigi can take care of all the boilerplate code that
would normally be needed in the constructor.
Internally, the DailyReport object can now be constructed by running
``DailyReport(datetime.date(2012, 5, 10))`` or just ``DailyReport()``.
Luigi also creates a command line parser that automatically handles the
conversion from strings to Python types.
This way you can invoke the job on the command line eg. by passing ``--date 2012-05-10``.

The parameters are all set to their values on the Task object instance,
i.e.

.. code:: python

    d = DailyReport(datetime.date(2012, 5, 10))
    print(d.date)

will return the same date that the object was constructed with.
Same goes if you invoke Luigi on the command line.

.. _Parameter-instance-caching:

Instance caching
^^^^^^^^^^^^^^^^

Tasks are uniquely identified by their class name and values of their
parameters.
In fact, within the same worker, two tasks of the same class with
parameters of the same values are not just equal, but the same instance:

.. code:: python

    >>> import luigi
    >>> import datetime
    >>> class DateTask(luigi.Task):
    ...   date = luigi.DateParameter()
    ...
    >>> a = datetime.date(2014, 1, 21)
    >>> b = datetime.date(2014, 1, 21)
    >>> a is b
    False
    >>> c = DateTask(date=a)
    >>> d = DateTask(date=b)
    >>> c
    DateTask(date=2014-01-21)
    >>> d
    DateTask(date=2014-01-21)
    >>> c is d
    True

Insignificant parameters
^^^^^^^^^^^^^^^^^^^^^^^^

If a parameter is created with ``significant=False``,
it is ignored as far as the Task signature is concerned.
Tasks created with only insignificant parameters differing have the same signature but
are not the same instance:

.. code:: python

    >>> class DateTask2(DateTask):
    ...   other = luigi.Parameter(significant=False)
    ...
    >>> c = DateTask2(date=a, other="foo")
    >>> d = DateTask2(date=b, other="bar")
    >>> c
    DateTask2(date=2014-01-21)
    >>> d
    DateTask2(date=2014-01-21)
    >>> c.other
    'foo'
    >>> d.other
    'bar'
    >>> c is d
    False
    >>> hash(c) == hash(d)
    True

Parameter visibility
^^^^^^^^^^^^^^^^^^^^

Using :class:`~luigi.parameter.ParameterVisibility` you can configure parameter visibility. By default, all
parameters are public, but you can also set them hidden or private.

.. code:: python

    >>> import luigi
    >>> from luigi.parameter import ParameterVisibility
    
    >>> luigi.Parameter(visibility=ParameterVisibility.PRIVATE)

``ParameterVisibility.PUBLIC`` (default) - visible everywhere

``ParameterVisibility.HIDDEN`` - ignored in WEB-view, but saved into database if save db_history is true

``ParameterVisibility.PRIVATE`` - visible only inside task.

Parameter types
^^^^^^^^^^^^^^^

In the examples above, the *type* of the parameter is determined by using different
subclasses of :class:`~luigi.parameter.Parameter`. There are a few of them, like
:class:`~luigi.parameter.DateParameter`,
:class:`~luigi.parameter.DateIntervalParameter`,
:class:`~luigi.parameter.IntParameter`,
:class:`~luigi.parameter.FloatParameter`, etc.

Python is not a statically typed language and you don't have to specify the types
of any of your parameters.
You can simply use the base class :class:`~luigi.parameter.Parameter` if you don't care.

The reason you would use a subclass like :class:`~luigi.parameter.DateParameter`
is that Luigi needs to know its type for the command line interaction.
That's how it knows how to convert a string provided on the command line to
the corresponding type (i.e. datetime.date instead of a string).

.. _Parameter-class-level-parameters:

Setting parameter value for other classes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

All parameters are also exposed on a class level on the command line interface.
For instance, say you have classes TaskA and TaskB:

.. code:: python

    class TaskA(luigi.Task):
        x = luigi.Parameter()

    class TaskB(luigi.Task):
        y = luigi.Parameter()


You can run ``TaskB`` on the command line: ``luigi TaskB --y 42``.
But you can also set the class value of ``TaskA`` by running
``luigi TaskB --y 42 --TaskA-x 43``.
This sets the value of ``TaskA.x`` to 43 on a *class* level.
It is still possible to override it inside Python if you instantiate ``TaskA(x=44)``.

All parameters can also be set from the configuration file.
For instance, you can put this in the config:

.. code:: ini

    [TaskA]
    x: 45


Just as in the previous case, this will set the value of ``TaskA.x`` to 45 on the *class* level.
And likewise, it is still possible to override it inside Python if you instantiate ``TaskA(x=44)``.

Parameter resolution order
^^^^^^^^^^^^^^^^^^^^^^^^^^

Parameters are resolved in the following order of decreasing priority:

1. Any value passed to the constructor, or task level value set on the command line (applies on an instance level)
2. Any value set on the command line (applies on a class level)
3. Any configuration option (applies on a class level)
4. Any default value provided to the parameter (applies on a class level)

See the :class:`~luigi.parameter.Parameter` class for more information.


================================================
FILE: doc/running_luigi.rst
================================================
Running Luigi
-------------

Running from the Command Line
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The preferred way to run Luigi tasks is through the ``luigi`` command line tool
that will be installed with the pip package.

.. code-block:: python

    # my_module.py, available in your sys.path
    import luigi

    class MyTask(luigi.Task):
        x = luigi.IntParameter()
        y = luigi.IntParameter(default=45)

        def run(self):
            print(self.x + self.y)

Should be run like this

.. code-block:: console

        $ luigi --module my_module MyTask --x 123 --y 456 --local-scheduler

Or alternatively like this:

.. code-block:: console

        $ python -m luigi --module my_module MyTask --x 100 --local-scheduler

Note that if a parameter name contains '_', it should be replaced by '-'.
For example, if MyTask had a parameter called 'my_parameter':

.. code-block:: console

        $ luigi --module my_module MyTask --my-parameter 100 --local-scheduler

.. note:: Please make sure to always place task parameters behind the task family!


Running from Python code
^^^^^^^^^^^^^^^^^^^^^^^^

Another way to start tasks from Python code is using ``luigi.build(tasks, worker_scheduler_factory=None, **env_params)``
from ``luigi.interface`` module.

This way of running luigi tasks is useful if you want to get some dynamic parameters from another
source, such as database, or provide additional logic before you start tasks.

One notable difference is that ``build`` defaults to not using the identical process lock.
If you want to change this behaviour, just pass ``no_lock=False``.


.. code-block:: python

    class MyTask1(luigi.Task):
        x = luigi.IntParameter()
        y = luigi.IntParameter(default=0)

        def run(self):
            print(self.x + self.y)


    class MyTask2(luigi.Task):
        x = luigi.IntParameter()
        y = luigi.IntParameter(default=1)
        z = luigi.IntParameter(default=2)

        def run(self):
            print(self.x * self.y * self.z)


    if __name__ == '__main__':
        luigi.build([MyTask1(x=10), MyTask2(x=15, z=3)])


Also, it is possible to pass additional parameters to ``build`` such as host, port, workers and local_scheduler:

.. code-block:: python

    if __name__ == '__main__':
         luigi.build([MyTask1(x=1)], workers=5, local_scheduler=True)

To achieve some special requirements you can pass to ``build`` your  ``worker_scheduler_factory``
which will return your worker and/or scheduler implementations:

.. code-block:: python

    class MyWorker(Worker):
        # some custom logic


    class MyFactory:
      def create_local_scheduler(self):
          return scheduler.Scheduler(prune_on_get_work=True, record_task_history=False)

      def create_remote_scheduler(self, url):
          return rpc.RemoteScheduler(url)

      def create_worker(self, scheduler, worker_processes, assistant=False):
          # return your worker instance
          return MyWorker(
              scheduler=scheduler, worker_processes=worker_processes, assistant=assistant)


    if __name__ == '__main__':
        luigi.build([MyTask1(x=1)], worker_scheduler_factory=MyFactory())

In some cases (like task queue) it may be useful.



Response of luigi.build()/luigi.run()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- **Default response** By default *luigi.build()/luigi.run()* returns True if there were no scheduling errors. This is the same as the attribute ``LuigiRunResult.scheduling_succeeded``.

- **Detailed response** This is a response of type :class:`~luigi.execution_summary.LuigiRunResult`. This is obtained by passing a keyword argument ``detailed_summary=True`` to *build/run*. This response contains detailed information about the jobs.

  .. code-block:: python

    if __name__ == '__main__':
         luigi_run_result = luigi.build(..., detailed_summary=True)
         print(luigi_run_result.summary_text)


Luigi on Windows
^^^^^^^^^^^^^^^^

Most Luigi functionality works on Windows. Exceptions:

- Specifying multiple worker processes using the ``workers`` argument for
  ``luigi.build``, or using the ``--workers`` command line argument. (Similarly,
  specifying ``--worker-force-multiprocessing``). For most programs, this will
  result in failure (a common sight is ``BrokenPipeError``). The reason is that
  worker processes are assumed to be forked from the main process. Forking is
  `not possible <https://docs.python.org/dev/library/multiprocessing.html#contexts-and-start-methods>`_
  on Windows.
- Running the Luigi central scheduling server as a daemon (i.e. with ``--background``).
  Again, a Unix-only concept.


================================================
FILE: doc/tasks.rst
================================================
Tasks
-----

Tasks are where the execution takes place.
Tasks depend on each other and output targets.

An outline of how a task can look like:

    .. figure:: task_breakdown.png
       :alt: Task breakdown

.. _Task.requires:

Task.requires
~~~~~~~~~~~~~

The :func:`~luigi.task.Task.requires` method is used to specify dependencies on other Task object,
which might even be of the same class.
For instance, an example implementation could be

.. code:: python

    def requires(self):
        return OtherTask(self.date), DailyReport(self.date - datetime.timedelta(1))

In this case, the DailyReport task depends on two inputs created earlier,
one of which is the same class.
requires can return other Tasks in any way wrapped up within dicts/lists/tuples/etc.

Requiring another Task
~~~~~~~~~~~~~~~~~~~~~~

Note that :func:`~luigi.task.Task.requires` can *not* return a :class:`~luigi.target.Target` object.
If you have a simple Target object that is created externally
you can wrap it in a Task class like this:

.. code:: python

    class LogFiles(luigi.ExternalTask):
        def output(self):
            return luigi.contrib.hdfs.HdfsTarget('/log')

This also makes it easier to add parameters:

.. code:: python

    class LogFiles(luigi.ExternalTask):
        date = luigi.DateParameter()
        def output(self):
            return luigi.contrib.hdfs.HdfsTarget(self.date.strftime('/log/%Y-%m-%d'))

.. _Task.output:

Task.output
~~~~~~~~~~~

The :func:`~luigi.task.Task.output` method returns one or more :class:`~luigi.target.Target` objects.
Similarly to requires, you can return them wrapped up in any way that's convenient for you.
However we recommend that any :class:`~luigi.task.Task` only return one single :class:`~luigi.target.Target` in output.
If multiple outputs are returned,
atomicity will be lost unless the :class:`~luigi.task.Task` itself can ensure that each :class:`~luigi.target.Target` is atomically created.
(If atomicity is not of concern, then it is safe to return multiple :class:`~luigi.target.Target` objects.)

.. code:: python

    class DailyReport(luigi.Task):
        date = luigi.DateParameter()
        def output(self):
            return luigi.contrib.hdfs.HdfsTarget(self.date.strftime('/reports/%Y-%m-%d'))
        # ...

.. _Task.run:

Task.run
~~~~~~~~

The :func:`~luigi.task.Task.run` method now contains the actual code that is run.
When you are using Task.requires_ and Task.run_ Luigi breaks down everything into two stages.
First it figures out all dependencies between tasks,
then it runs everything.
The :func:`~luigi.task.Task.input` method is an internal helper method that just replaces all Task objects in requires
with their corresponding output.
An example:

.. code:: python

    class GenerateWords(luigi.Task):

        def output(self):
            return luigi.LocalTarget('words.txt')

        def run(self):

            # write a dummy list of words to output file
            words = [
                    'apple',
                    'banana',
                    'grapefruit'
                    ]

            with self.output().open('w') as f:
                for word in words:
                    f.write('{word}\n'.format(word=word))


    class CountLetters(luigi.Task):

        def requires(self):
            return GenerateWords()

        def output(self):
            return luigi.LocalTarget('letter_counts.txt')

        def run(self):

            # read in file as list
            with self.input().open('r') as infile:
                words = infile.read().splitlines()

            # write each word to output file with its corresponding letter count
            with self.output().open('w') as outfile:
                for word in words:
                    outfile.write(
                            '{word} | {letter_count}\n'.format(
                                word=word,
                                letter_count=len(word)
                                )
                            )

It's useful to note that if you're writing to a binary file, Luigi automatically
strips the ``'b'`` flag due to how atomic writes/reads work. In order to write a binary
file, such as a pickle file, you should instead use ``format=Nop`` when calling
LocalTarget. Following the above example:

.. code:: python

    from luigi.format import Nop

    class GenerateWords(luigi.Task):

        def output(self):
            return luigi.LocalTarget('words.pckl', format=Nop)

        def run(self):
            import pickle

            # write a dummy list of words to output file
            words = [
                    'apple',
                    'banana',
                    'grapefruit'
                    ]

            with self.output().open('w') as f:
                pickle.dump(words, f)


It is your responsibility to ensure that after running :func:`~luigi.task.Task.run`, the task is
complete, i.e. :func:`~luigi.task.Task.complete` returns ``True``. Unless you have overridden
:func:`~luigi.task.Task.complete`, :func:`~luigi.task.Task.run` should generate all the targets
defined as outputs. Luigi verifies that you adhere to the contract before running downstream
dependencies, and reports ``Unfulfilled dependencies at run time`` if a violation is detected.

.. _Task.input:

Task.input
~~~~~~~~~~

As seen in the example above, :func:`~luigi.task.Task.input` is a wrapper around Task.requires_ that
returns the corresponding Target objects instead of Task objects.
Anything returned by Task.requires_ will be transformed, including lists,
nested dicts, etc.
This can be useful if you have many dependencies:

.. code:: python

    class TaskWithManyInputs(luigi.Task):
        def requires(self):
            return {'a': TaskA(), 'b': [TaskB(i) for i in xrange(100)]}

        def run(self):
            f = self.input()['a'].open('r')
            g = [y.open('r') for y in self.input()['b']]


Dynamic dependencies
~~~~~~~~~~~~~~~~~~~~

Sometimes you might not know exactly what other tasks to depend on until runtime.
In that case, Luigi provides a mechanism to specify dynamic dependencies.
If you yield another :class:`~luigi.task.Task` in the Task.run_ method,
the current task will be suspended and the other task will be run.
You can also yield a list of tasks.

.. code:: python

    class MyTask(luigi.Task):
        def run(self):
            other_target = yield OtherTask()

            # dynamic dependencies resolve into targets
            f = other_target.open('r')


This mechanism is an alternative to Task.requires_ in case
you are not able to build up the full dependency graph before running the task.
It does come with some constraints:
the Task.run_ method will resume from scratch each time a new task is yielded.
In other words, you should make sure your Task.run_ method is idempotent.
(This is good practice for all Tasks in Luigi, but especially so for tasks with dynamic dependencies).
As this might entail redundant calls to tasks' :func:`~luigi.task.Task.complete` methods,
you should consider setting the "cache_task_completion" option in the :ref:`worker-config`.
To further control how dynamic task requirements are handled internally by worker nodes,
there is also the option to wrap dependent tasks by :class:`~luigi.task.DynamicRequirements`.

For an example of a workflow using dynamic dependencies, see
`examples/dynamic_requirements.py <https://github.com/spotify/luigi/blob/master/examples/dynamic_requirements.py>`_.


Task status tracking
~~~~~~~~~~~~~~~~~~~~

For long-running or remote tasks it is convenient to see extended status information not only on
the command line or in your logs but also in the GUI of the central scheduler. Luigi implements
dynamic status messages, progress bar and tracking urls which may point to an external monitoring system.
You can set this information using callbacks within Task.run_:

.. code:: python

    class MyTask(luigi.Task):
        def run(self):
            # set a tracking url
            self.set_tracking_url("http://...")

            # set status messages during the workload
            for i in range(100):
                # do some hard work here
                if i % 10 == 0:
                    self.set_status_message("Progress: %d / 100" % i)
                    # displays a progress bar in the scheduler UI
                    self.set_progress_percentage(i)


.. _Events:

Events and callbacks
~~~~~~~~~~~~~~~~~~~~

Luigi has a built-in event system that
allows you to register callbacks to events and trigger them from your own tasks.
You can both hook into some pre-defined events and create your own.
Each event handle is tied to a Task class and
will be triggered only from that class or
a subclass of it.
This allows you to effortlessly subscribe to events only from a specific class (e.g. for hadoop jobs).

.. code:: python

    @luigi.Task.event_handler(luigi.Event.SUCCESS)
    def celebrate_success(task):
        """Will be called directly after a successful execution
           of `run` on any Task subclass (i.e. all luigi Tasks)
        """
        ...

    @luigi.contrib.hadoop.JobTask.event_handler(luigi.Event.FAILURE)
    def mourn_failure(task, exception):
        """Will be called directly after a failed execution
           of `run` on any JobTask subclass
        """
        ...

    luigi.run()


But I just want to run a Hadoop job?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Hadoop code is integrated in the rest of the Luigi code because
we really believe almost all Hadoop jobs benefit from being part of some sort of workflow.
However, in theory, nothing stops you from using the :class:`~luigi.contrib.hadoop.JobTask` class (and also :class:`~luigi.contrib.hdfs.target.HdfsTarget`)
without using the rest of Luigi.
You can simply run it manually using

.. code:: python

    MyJobTask('abc', 123).run()

You can use the hdfs.target.HdfsTarget class anywhere by just instantiating it:

.. code:: python

    t = luigi.contrib.hdfs.target.HdfsTarget('/tmp/test.gz', format=format.Gzip)
    f = t.open('w')
    # ...
    f.close() # needed

.. _Task.priority:

Task priority
~~~~~~~~~~~~~

The scheduler decides which task to run next from
the set of all tasks that have all their dependencies met.
By default, this choice is pretty arbitrary,
which is fine for most workflows and situations.

If you want to have some control on the order of execution of available tasks,
you can set the ``priority`` property of a task,
for example as follows:

.. code:: python

    # A static priority value as a class constant:
    class MyTask(luigi.Task):
        priority = 100
        # ...

    # A dynamic priority value with a "@property" decorated method:
    class OtherTask(luigi.Task):
        @property
        def priority(self):
            if self.date > some_threshold:
                return 80
            else:
                return 40
        # ...

Tasks with a higher priority value will be picked before tasks with a lower priority value.
There is no predefined range of priorities,
you can choose whatever (int or float) values you want to use.
The default value is 0.

Warning: task execution order in Luigi is influenced by both dependencies and priorities, but
in Luigi dependencies come first.
For example:
if there is a task A with priority 1000 but still with unmet dependencies and
a task B with priority 1 without any pending dependencies,
task B will be picked first.

.. _Task.namespaces_famlies_and_ids:

Namespaces, families and ids
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to avoid name clashes and to be able to have an identifier for tasks,
Luigi introduces the concepts *task_namespace*, *task_family* and
*task_id*. The namespace and family operate on class level meanwhile the task
id only exists on instance level. The concepts are best illustrated using code.

.. code:: python

    import luigi
    class MyTask(luigi.Task):
        my_param = luigi.Parameter()
        task_namespace = 'my_namespace'

    my_task = MyTask(my_param='hello')
    print(my_task)                      # --> my_namespace.MyTask(my_param=hello)

    print(my_task.get_task_namespace()) # --> my_namespace
    print(my_task.get_task_family())    # --> my_namespace.MyTask
    print(my_task.task_id)              # --> my_namespace.MyTask_hello_890907e7ce

    print(MyTask.get_task_namespace())  # --> my_namespace
    print(MyTask.get_task_family())     # --> my_namespace.MyTask
    print(MyTask.task_id)               # --> Error!

The full documentation for this machinery exists in the :py:mod:`~luigi.task` module.

Instance caching
~~~~~~~~~~~~~~~~

In addition to the stuff mentioned above,
Luigi also does some metaclass logic so that
if e.g. ``DailyReport(datetime.date(2012, 5, 10))`` is instantiated twice in the code,
it will in fact result in the same object.
See :ref:`Parameter-instance-caching` for more info


================================================
FILE: doc/workflows.rst
================================================
Building workflows
------------------

There are two fundamental building blocks of Luigi -
the :class:`~luigi.task.Task` class and the :class:`~luigi.target.Target` class.
Both are abstract classes and expect a few methods to be implemented.
In addition to those two concepts,
the :class:`~luigi.parameter.Parameter` class is an important concept that governs how a Task is run.

Target
~~~~~~

The :py:class:`~luigi.target.Target` class corresponds to a file on a disk,
a file on HDFS or some kind of a checkpoint, like an entry in a database.
Actually, the only method that Targets have to implement is the *exists*
method which returns True if and only if the Target exists.

In practice, implementing Target subclasses is rarely needed.
Luigi comes with a toolbox of several useful Targets.
In particular, :class:`~luigi.file.LocalTarget` and :class:`~luigi.contrib.hdfs.target.HdfsTarget`,
but there is also support for other file systems:
:class:`luigi.contrib.s3.S3Target`,
:class:`luigi.contrib.ssh.RemoteTarget`,
:class:`luigi.contrib.ftp.RemoteTarget`,
:class:`luigi.contrib.mysqldb.MySqlTarget`,
:class:`luigi.contrib.redshift.RedshiftTarget`, and several more.

Most of these targets, are file system-like.
For instance, :class:`~luigi.file.LocalTarget` and :class:`~luigi.contrib.hdfs.target.HdfsTarget` map to a file on the local drive or a file in HDFS.
In addition these also wrap the underlying operations to make them atomic.
They both implement the :func:`~luigi.file.LocalTarget.open` method which returns a stream object that
could be read (``mode='r'``) from or written to (``mode='w'``).

Luigi comes with Gzip support by providing ``format=format.Gzip``.
Adding support for other formats is pretty simple.

Task
~~~~

The :class:`~luigi.task.Task` class is a bit more conceptually interesting because this is
where computation is done.
There are a few methods that can be implemented to alter its behavior,
most notably :func:`~luigi.task.Task.run`, :func:`~luigi.task.Task.output` and :func:`~luigi.task.Task.requires`.

Tasks consume Targets that were created by some other task. They usually also output targets:

    .. figure:: task_with_targets.png
       :alt: Task and targets

You can define dependencies between *Tasks* using the :py:meth:`~luigi.task.Task.requires` method. See :doc:`/tasks` for more info.

    .. figure:: tasks_with_dependencies.png
       :alt: Tasks and dependencies

Each task defines its outputs using the :py:meth:`~luigi.task.Task.output` method.
Additionally, there is a helper method :py:meth:`~luigi.task.Task.input` that returns the corresponding Target classes for each Task dependency.

    .. figure:: tasks_input_output_requires.png
       :alt: Tasks and methods

.. _Parameter:

Parameter
~~~~~~~~~

The Task class corresponds to some type of job that is run, but in
general you want to allow some form of parameterization of it.
For instance, if your Task class runs a Hadoop job to create a report every night,
you probably want to make the date a parameter of the class.
See :doc:`/parameters` for more info.

    .. figure:: task_parameters.png
       :alt: Tasks with parameters

Dependencies
~~~~~~~~~~~~

Using tasks, targets, and parameters, Luigi lets you express arbitrary dependencies in *code*, rather than using some kind of awkward config DSL.
This is really useful because in the real world, dependencies are often very messy.
For instance, some examples of the dependencies you might encounter:

    .. figure:: parameters_date_algebra.png
       :alt: Dependencies with date algebra

    .. figure:: parameters_recursion.png
       :alt: Dependencies with recursion

    .. figure:: parameters_enum.png
       :alt: Dependencies with enums

(These diagrams are from a `Luigi presentation in late 2014 at NYC Data Science meetup <http://www.slideshare.net/erikbern/luigi-presentation-nyc-data-science>`_)


================================================
FILE: examples/__init__.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#


================================================
FILE: examples/config.toml
================================================

[hdfs]
client = "hadoopcli"
namenode_host = "localhost"
namenode_port = 50030

# LOGGING

[logging]
version = 1
disable_existing_loggers = false

# logs format
[logging.formatters.simple]
format = "{levelname:8} {asctime} {module}:{lineno} {message}"
style = "{"
datefmt = "%Y-%m-%d %H:%M:%S"

# write logs to console
[logging.handlers.console]
level = "DEBUG"
class = "logging.StreamHandler"
formatter = "simple"

# luigi worker logging
[logging.loggers.luigi-interface]
handlers = ["console"]
level = "INFO"
disabled = false
propagate = false

# luigid logging
[logging.loggers.luigi]
handlers = ["console"]
level = "INFO"
disabled = false
propagate = false

# luigid builded on tornado
[logging.loggers.tornado]
handlers = ["console"]
level = "INFO"
disabled = false
propagate = false

# custom logger for "project"
[logging.loggers.project]
handlers = ["console"]
level = "DEBUG"
disabled = false
propagate = false


================================================
FILE: examples/dynamic_requirements.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import os
import random as rnd
import time

import luigi


class Configuration(luigi.Task):
    seed = luigi.IntParameter()

    def output(self):
        """
        Returns the target output for this task.
        In this case, a successful execution of this task will create a file on the local filesystem.

        :return: the target output for this task.
        :rtype: object (:py:class:`luigi.target.Target`)
        """
        return luigi.LocalTarget("/tmp/Config_%d.txt" % self.seed)

    def run(self):
        time.sleep(5)
        rnd.seed(self.seed)

        result = ",".join([str(x) for x in rnd.sample(list(range(300)), rnd.randint(7, 25))])
        with self.output().open("w") as f:
            f.write(result)


class Data(luigi.Task):
    magic_number = luigi.IntParameter()

    def output(self):
        """
        Returns the target output for this task.
        In this case, a successful execution of this task will create a file on the local filesystem.

        :return: the target output for this task.
        :rtype: object (:py:class:`luigi.target.Target`)
        """
        return luigi.LocalTarget("/tmp/Data_%d.txt" % self.magic_number)

    def run(self):
        time.sleep(1)
        with self.output().open("w") as f:
            f.write("%s" % self.magic_number)


class Dynamic(luigi.Task):
    seed = luigi.IntParameter(default=1)

    def output(self):
        """
        Returns the target output for this task.
        In this case, a successful execution of this task will create a file on the local filesystem.

        :return: the target output for this task.
        :rtype: object (:py:class:`luigi.target.Target`)
        """
        return luigi.LocalTarget("/tmp/Dynamic_%d.txt" % self.seed)

    def run(self):
        # This could be done using regular requires method
        config = self.clone(Configuration)
        yield config

        with config.output().open() as f:
            data = [int(x) for x in f.read().split(",")]

        # ... but not this
        data_dependent_deps = [Data(magic_number=x) for x in data]
        yield data_dependent_deps

        with self.output().open("w") as f:
            f.write("Tada!")

        # and in case data is rather long, consider wrapping the requirements
        # in DynamicRequirements and optionally define a custom complete method
        def custom_complete(complete_fn):
            # example: Data() stores all outputs in the same directory, so avoid doing len(data) fs
            # calls but rather check only the first, and compare basenames for the rest
            # (complete_fn defaults to "lambda task: task.complete()" but can also include caching)
            if not complete_fn(data_dependent_deps[0]):
                return False
            paths = [task.output().path for task in data_dependent_deps]
            basenames = os.listdir(os.path.dirname(paths[0]))  # a single fs call
            return all(os.path.basename(path) in basenames for path in paths)

        yield luigi.DynamicRequirements(data_dependent_deps, custom_complete)


if __name__ == "__main__":
    luigi.run()


================================================
FILE: examples/elasticsearch_index.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import datetime
import json

import luigi
from luigi.contrib.esindex import CopyToIndex


class FakeDocuments(luigi.Task):
    """
    Generates a local file containing 5 elements of data in JSON format.
    """

    #: the date parameter.
    date = luigi.DateParameter(default=datetime.date.today())

    def run(self):
        """
        Writes data in JSON format into the task's output target.

        The data objects have the following attributes:

        * `_id` is the default Elasticsearch id field,
        * `text`: the text,
        * `date`: the day when the data was created.

        """
        today = datetime.date.today()
        with self.output().open("w") as output:
            for i in range(5):
                output.write(json.dumps({"_id": i, "text": "Hi %s" % i, "date": str(today)}))
                output.write("\n")

    def output(self):
        """
        Returns the target output for this task.
        In this case, a successful execution of this task will create a file on the local filesystem.

        :return: the target output for this task.
        :rtype: object (:py:class:`luigi.target.Target`)
        """
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.date)


class IndexDocuments(CopyToIndex):
    """
    This task loads JSON data contained in a :py:class:`luigi.target.Target` into an ElasticSearch index.

    This task's input will the target returned by :py:meth:`~.FakeDocuments.output`.

    This class uses :py:meth:`luigi.contrib.esindex.CopyToIndex.run`.

    After running this task you can run:

    .. code-block:: console

        $ curl "localhost:9200/example_index/_search?pretty"

    to see the indexed documents.

    To see the update log, run

    .. code-block:: console

        $ curl "localhost:9200/update_log/_search?q=target_index:example_index&pretty"

    To cleanup both indexes run:

    .. code-block:: console

        $ curl -XDELETE "localhost:9200/example_index"
        $ curl -XDELETE "localhost:9200/update_log/_query?q=target_index:example_index"

    """

    #: date task parameter (default = today)
    date = luigi.DateParameter(default=datetime.date.today())

    #: the name of the index in ElasticSearch to be updated.
    index = "example_index"
    #: the name of the document type.
    doc_type = "greetings"
    #: the host running the ElasticSearch service.
    host = "localhost"
    #: the port used by the ElasticSearch service.
    port = 9200

    def requires(self):
        """
        This task's dependencies:

        * :py:class:`~.FakeDocuments`

        :return: object (:py:class:`luigi.task.Task`)
        """
        return FakeDocuments()


if __name__ == "__main__":
    luigi.run(["IndexDocuments", "--local-scheduler"])


================================================
FILE: examples/execution_summary_example.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2015-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
You can run this example like this:

    .. code:: console

            $ luigi --module examples.execution_summary_example examples.EntryPoint --local-scheduler
            ...
            ... lots of spammy output
            ...
            INFO: There are 11 pending tasks unique to this worker
            INFO: Worker Worker(salt=843361665, workers=1, host=arash-spotify-T440s, username=arash, pid=18534) was stopped. Shutting down Keep-Alive thread
            INFO:
            ===== Luigi Execution Summary =====

            Scheduled 218 tasks of which:
            * 195 complete ones were encountered:
                - 195 examples.Bar(num=5...199)
            * 1 ran successfully:
                - 1 examples.Boom(...)
            * 22 were left pending, among these:
                * 1 were missing external dependencies:
                    - 1 MyExternal()
                * 21 had missing dependencies:
                    - 1 examples.EntryPoint()
                    - examples.Foo(num=100, num2=16) and 9 other examples.Foo
                    - 10 examples.DateTask(date=1998-03-23...1998-04-01, num=5)

            This progress looks :| because there were missing external dependencies

            ===== Luigi Execution Summary =====
"""

import datetime

import luigi


class MyExternal(luigi.ExternalTask):
    def complete(self):
        return False


class Boom(luigi.Task):
    task_namespace = "examples"
    this_is_a_really_long_I_mean_way_too_long_and_annoying_parameter = luigi.IntParameter()

    def run(self):
        print("Running Boom")

    def requires(self):
        for i in range(5, 200):
            yield Bar(i)


class Foo(luigi.Task):
    task_namespace = "examples"
    num = luigi.IntParameter()
    num2 = luigi.IntParameter()

    def run(self):
        print("Running Foo")

    def requires(self):
        yield MyExternal()
        yield Boom(0)


class Bar(luigi.Task):
    task_namespace = "examples"
    num = luigi.IntParameter()

    def run(self):
        self.output().open("w").close()

    def output(self):
        return luigi.LocalTarget("/tmp/bar/%d" % self.num)


class DateTask(luigi.Task):
    task_namespace = "examples"
    date = luigi.DateParameter()
    num = luigi.IntParameter()

    def run(self):
        print("Running DateTask")

    def requires(self):
        yield MyExternal()
        yield Boom(0)


class EntryPoint(luigi.Task):
    task_namespace = "examples"

    def run(self):
        print("Running EntryPoint")

    def requires(self):
        for i in range(10):
            yield Foo(100, 2 * i)
        for i in range(10):
            yield DateTask(datetime.date(1998, 3, 23) + datetime.timedelta(days=i), 5)


================================================
FILE: examples/foo.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
You can run this example like this:

    .. code:: console

            $ rm -rf '/tmp/bar'
            $ luigi --module examples.foo examples.Foo --workers 2 --local-scheduler

"""

import time

import luigi


class Foo(luigi.WrapperTask):
    task_namespace = "examples"

    def run(self):
        print("Running Foo")

    def requires(self):
        for i in range(10):
            yield Bar(i)


class Bar(luigi.Task):
    task_namespace = "examples"
    num = luigi.IntParameter()

    def run(self):
        time.sleep(1)
        self.output().open("w").close()

    def output(self):
        """
        Returns the target output for this task.

        :return: the target output for this task.
        :rtype: object (:py:class:`~luigi.target.Target`)
        """
        time.sleep(1)
        return luigi.LocalTarget("/tmp/bar/%d" % self.num)


================================================
FILE: examples/foo_complex.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
You can run this example like this:

    .. code:: console

            $ rm -rf '/tmp/bar'
            $ luigi --module examples.foo_complex examples.Foo --workers 2 --local-scheduler

"""

import random
import time

import luigi

max_depth = 10
max_total_nodes = 50
current_nodes = 0


class Foo(luigi.Task):
    task_namespace = "examples"

    def run(self):
        print("Running Foo")

    def requires(self):
        global current_nodes
        for i in range(30 // max_depth):
            current_nodes += 1
            yield Bar(i)


class Bar(luigi.Task):
    task_namespace = "examples"

    num = luigi.IntParameter()

    def run(self):
        time.sleep(1)
        self.output().open("w").close()

    def requires(self):
        global current_nodes

        if max_total_nodes > current_nodes:
            valor = int(random.uniform(1, 30))
            for i in range(valor // max_depth):
                current_nodes += 1
                yield Bar(current_nodes)

    def output(self):
        """
        Returns the target output for this task.

        :return: the target output for this task.
        :rtype: object (:py:class:`~luigi.target.Target`)
        """
        time.sleep(1)
        return luigi.LocalTarget("/tmp/bar/%d" % self.num)


================================================
FILE: examples/ftp_experiment_outputs.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import luigi
from luigi.contrib.ftp import RemoteTarget

#: the FTP server
HOST = "some_host"
#: the username
USER = "user"
#: the password
PWD = "some_password"


class ExperimentTask(luigi.ExternalTask):
    """
    This class represents something that was created elsewhere by an external process,
    so all we want to do is to implement the output method.
    """

    def output(self):
        """
        Returns the target output for this task.
        In this case, a successful execution of this task will create a file that will be created in a FTP server.

        :return: the target output for this task.
        :rtype: object (:py:class:`~luigi.target.Target`)
        """
        return RemoteTarget("/experiment/output1.txt", HOST, username=USER, password=PWD)

    def run(self):
        """
        The execution of this task will write 4 lines of data on this task's target output.
        """
        with self.output().open("w") as outfile:
            print("data 0 200 10 50 60", file=outfile)
            print("data 1 190 9 52 60", file=outfile)
            print("data 2 200 10 52 60", file=outfile)
            print("data 3 195 1 52 60", file=outfile)


class ProcessingTask(luigi.Task):
    """
    This class represents something that was created elsewhere by an external process,
    so all we want to do is to implement the output method.
    """

    def requires(self):
        """
        This task's dependencies:

        * :py:class:`~.ExperimentTask`

        :return: object (:py:class:`luigi.task.Task`)
        """
        return ExperimentTask()

    def output(self):
        """
        Returns the target output for this task.
        In this case, a successful execution of this task will create a file on the local filesystem.

        :return: the target output for this task.
        :rtype: object (:py:class:`~luigi.target.Target`)
        """
        return luigi.LocalTarget("/tmp/processeddata.txt")

    def run(self):
        avg = 0.0
        elements = 0
        sumval = 0.0

        # Target objects are a file system/format abstraction and this will return a file stream object
        # NOTE: self.input() actually returns the ExperimentTask.output() target
        for line in self.input().open("r"):
            values = line.split(" ")
            avg += float(values[2])
            sumval += float(values[3])
            elements = elements + 1

        # average
        avg = avg / elements

        # save calculated values
        with self.output().open("w") as outfile:
            print(avg, sumval, file=outfile)


if __name__ == "__main__":
    luigi.run()


================================================
FILE: examples/hello_world.py
================================================
"""
You can run this example like this:

    .. code:: console

            $ luigi --module examples.hello_world examples.HelloWorldTask --local-scheduler

If that does not work, see :ref:`CommandLine`.
"""

import luigi


class HelloWorldTask(luigi.Task):
    task_namespace = "examples"

    def run(self):
        print("{task} says: Hello world!".format(task=self.__class__.__name__))


if __name__ == "__main__":
    luigi.run(["examples.HelloWorldTask", "--workers", "1", "--local-scheduler"])


================================================
FILE: examples/kubernetes.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2015 Outlier Bio, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""
Example Kubernetes Job Task.

Requires:

- pykube: ``pip install pykube-ng``
- A local minikube custer up and running: http://kubernetes.io/docs/getting-started-guides/minikube/

**WARNING**: For Python versions < 3.5 the kubeconfig file must point to a Kubernetes API
hostname, and NOT to an IP address.

You can run this code example like this:

    .. code:: console
        $ luigi --module examples.kubernetes_job PerlPi --local-scheduler

Running this code will create a pi-luigi-uuid kubernetes job within the cluster
pointed to by the default context in "~/.kube/config".

If running within a kubernetes cluster, set auth_method = "service-account" to
access the local cluster.
"""

# import os
# import luigi
from luigi.contrib.kubernetes import KubernetesJobTask


class PerlPi(KubernetesJobTask):
    name = "pi"
    max_retrials = 3
    spec_schema = {"containers": [{"name": "pi", "image": "perl", "command": ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]}]}

    # defining the two functions below allows for dependency checking,
    # but isn't a requirement
    # def signal_complete(self):
    #     with self.output().open('w') as output:
    #         output.write('')
    #
    # def output(self):
    #     target = os.path.join("/tmp", "PerlPi")
    #     return luigi.LocalTarget(target)


================================================
FILE: examples/per_task_retry_policy.py
================================================
# -*- coding: utf-8 -*-

"""
You can run this example like this:

    .. code:: console

            $ luigi --module examples.per_task_retry_policy examples.PerTaskRetryPolicy --worker-keep-alive \
            --local-scheduler --scheduler-retry-delay 5  --logging-conf-file test/testconfig/logging.cfg

            ...
            ... lots of spammy output
            ...
            DEBUG: ErrorTask1__99914b932b task num failures is 1 and limit is 5
            DEBUG: ErrorTask2__99914b932b task num failures is 1 and limit is 2
            DEBUG: DynamicErrorTask1__99914b932b task num failures is 1 and limit is 3
            DEBUG: ErrorTask1__99914b932b task num failures is 2 and limit is 5
            DEBUG: ErrorTask2__99914b932b task num failures is 2 and limit is 2
            DEBUG: ErrorTask2__99914b932b task num failures limit(2) is exceeded
            DEBUG: DynamicErrorTask1__99914b932b task num failures is 2 and limit is 3
            DEBUG: ErrorTask1__99914b932b task num failures is 3 and limit is 5
            DEBUG: DynamicErrorTask1__99914b932b task num failures is 3 and limit is 3
            DEBUG: DynamicErrorTask1__99914b932b task num failures limit(3) is exceeded
            DEBUG: ErrorTask1__99914b932b task num failures is 4 and limit is 5
            DEBUG: ErrorTask1__99914b932b task num failures is 5 and limit is 5
            DEBUG: ErrorTask1__99914b932b task num failures limit(5) is exceeded
            INFO:
            ===== Luigi Execution Summary =====

            Scheduled 8 tasks of which:
            * 2 ran successfully:
                - 1 SuccessSubTask1()
                - 1 SuccessTask1()
            * 3 failed:
                - 1 DynamicErrorTask1()
                - 1 ErrorTask1()
                - 1 ErrorTask2()
            * 3 were left pending, among these:
                * 1 were missing external dependencies:
                    - 1 DynamicErrorTaskSubmitter()
                * 1 had failed dependencies:
                    - 1 examples.PerTaskRetryPolicy()
                * 1 had missing dependencies:
                    - 1 examples.PerTaskRetryPolicy()
                * 1 was not granted run permission by the scheduler:
                    - 1 DynamicErrorTaskSubmitter()

            This progress looks :( because there were failed tasks

            ===== Luigi Execution Summary =====
"""

import luigi


class PerTaskRetryPolicy(luigi.Task):
    """
    Wrapper class for some error and success tasks. Worker won't be shutdown unless there is
    pending tasks or failed tasks which will be retried. While keep-alive is active, workers
    are not shutdown while there is/are some pending task(s).

    """

    task_namespace = "examples"

    def requires(self):
        return [ErrorTask1(), ErrorTask2(), SuccessTask1(), DynamicErrorTaskSubmitter()]

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


class ErrorTask1(luigi.Task):
    """
    This error class raises error to retry the task. retry-count for this task is 5. It can be seen on
    """

    retry = 0

    retry_count = 5

    def run(self):
        self.retry += 1
        raise Exception("Test Exception. Retry Index %s for %s" % (self.retry, self.task_family))

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


class ErrorTask2(luigi.Task):
    """
    This error class raises error to retry the task. retry-count for this task is 2
    """

    retry = 0

    retry_count = 2

    def run(self):
        self.retry += 1
        raise Exception("Test Exception. Retry Index %s for %s" % (self.retry, self.task_family))

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


class DynamicErrorTaskSubmitter(luigi.Task):
    target = None

    def run(self):
        target = yield DynamicErrorTask1()

        if target.exists():
            with self.output().open("w") as output:
                output.write("SUCCESS DynamicErrorTaskSubmitter\n")

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


class DynamicErrorTask1(luigi.Task):
    """
    This dynamic error task raises error to retry the task. retry-count for this task is 3
    """

    retry = 0

    retry_count = 3

    def run(self):
        self.retry += 1
        raise Exception("Test Exception. Retry Index %s for %s" % (self.retry, self.task_family))

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


class SuccessTask1(luigi.Task):
    def requires(self):
        return [SuccessSubTask1()]

    def run(self):
        with self.output().open("w") as output:
            output.write("SUCCESS Test Task 4\n")

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


class SuccessSubTask1(luigi.Task):
    """
    This success task sleeps for a while and then it is completed successfully.
    """

    def run(self):
        with self.output().open("w") as output:
            output.write("SUCCESS Test Task 4.1\n")

    def output(self):
        return luigi.LocalTarget(path="/tmp/_docs-%s.ldj" % self.task_id)


================================================
FILE: examples/pyspark_wc.py
================================================
# -*- coding: utf-8 -*-
#
# Copyright 2012-2015 Spotify AB
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import luigi
from luigi.contrib.s3 import S3Target
from luigi.contrib.spark import PySparkTask, SparkSubmitTask


class InlinePySparkWordCount(PySparkTask):
    """
    This task runs a :py:class:`luigi.contrib.spark.PySparkTask` task
    over the target data in :py:meth:`wordcount.input` (a file in S3) and
    writes the result into its :py:meth:`wordcount.output` target (a file in S3).

    This class uses :py:meth:`luigi.contrib.spark.PySparkTask.main`.

    Example luigi configuration::

        [spark]
        spark-submit: /usr/local/spark/bin/spark-submit
        master: spark://spark.example.org:7077
        # py-packages: numpy, pandas

    """

    driver_memory = "2g"
    executor_memory = "3g"

    def input(self):
        return S3Target("s3n://bucket.example.org/wordcount.input")

    def output(self):
        return S3Target("s3n://bucket.example.org/wordcount.output")

    def main(self, sc, *args):
        sc.textFile(self.input().path).flatMap(lambda line: line.split()).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b).saveAsTextFile(
            self.output().path
Download .txt
gitextract_i5ice6bg/

├── .coveragerc
├── .github/
│   ├── CODEOWNERS
│   ├── ISSUE_TEMPLATE.md
│   ├── PULL_REQUEST_TEMPLATE.md
│   ├── stale.yml
│   └── workflows/
│       ├── codeql.yml
│       └── pythonbuild.yml
├── .gitignore
├── .readthedocs.yaml
├── CONTRIBUTING.rst
├── LICENSE
├── README.rst
├── RELEASE-PROCESS.rst
├── SECURITY.md
├── bin/
│   ├── luigi
│   └── luigid
├── catalog-info.yaml
├── codecov.yml
├── doc/
│   ├── .gitignore
│   ├── Makefile
│   ├── central_scheduler.rst
│   ├── conf.py
│   ├── configuration.rst
│   ├── design_and_limitations.rst
│   ├── example_top_artists.rst
│   ├── execution_model.rst
│   ├── index.rst
│   ├── logging.rst
│   ├── luigi_patterns.rst
│   ├── mypy.rst
│   ├── parameters.rst
│   ├── running_luigi.rst
│   ├── tasks.rst
│   └── workflows.rst
├── examples/
│   ├── __init__.py
│   ├── config.toml
│   ├── dynamic_requirements.py
│   ├── elasticsearch_index.py
│   ├── execution_summary_example.py
│   ├── foo.py
│   ├── foo_complex.py
│   ├── ftp_experiment_outputs.py
│   ├── hello_world.py
│   ├── kubernetes.py
│   ├── per_task_retry_policy.py
│   ├── pyspark_wc.py
│   ├── spark_als.py
│   ├── ssh_remote_execution.py
│   ├── terasort.py
│   ├── top_artists.py
│   ├── top_artists_spark.py
│   ├── wordcount.py
│   └── wordcount_hadoop.py
├── luigi/
│   ├── __init__.py
│   ├── __main__.py
│   ├── __version__.py
│   ├── batch_notifier.py
│   ├── cmdline.py
│   ├── cmdline_parser.py
│   ├── configuration/
│   │   ├── __init__.py
│   │   ├── base_parser.py
│   │   ├── cfg_parser.py
│   │   ├── core.py
│   │   └── toml_parser.py
│   ├── contrib/
│   │   ├── __init__.py
│   │   ├── azureblob.py
│   │   ├── batch.py
│   │   ├── beam_dataflow.py
│   │   ├── bigquery.py
│   │   ├── bigquery_avro.py
│   │   ├── datadog_metric.py
│   │   ├── dataproc.py
│   │   ├── docker_runner.py
│   │   ├── dropbox.py
│   │   ├── ecs.py
│   │   ├── esindex.py
│   │   ├── external_daily_snapshot.py
│   │   ├── external_program.py
│   │   ├── ftp.py
│   │   ├── gcp.py
│   │   ├── gcs.py
│   │   ├── hadoop.py
│   │   ├── hadoop_jar.py
│   │   ├── hdfs/
│   │   │   ├── __init__.py
│   │   │   ├── abstract_client.py
│   │   │   ├── clients.py
│   │   │   ├── config.py
│   │   │   ├── error.py
│   │   │   ├── format.py
│   │   │   ├── hadoopcli_clients.py
│   │   │   ├── target.py
│   │   │   └── webhdfs_client.py
│   │   ├── hive.py
│   │   ├── kubernetes.py
│   │   ├── lsf.py
│   │   ├── lsf_runner.py
│   │   ├── mongodb.py
│   │   ├── mssqldb.py
│   │   ├── mysqldb.py
│   │   ├── opener.py
│   │   ├── pai.py
│   │   ├── pig.py
│   │   ├── postgres.py
│   │   ├── presto.py
│   │   ├── prometheus_metric.py
│   │   ├── pyspark_runner.py
│   │   ├── rdbms.py
│   │   ├── redis_store.py
│   │   ├── redshift.py
│   │   ├── s3.py
│   │   ├── salesforce.py
│   │   ├── scalding.py
│   │   ├── sge.py
│   │   ├── sge_runner.py
│   │   ├── simulate.py
│   │   ├── spark.py
│   │   ├── sparkey.py
│   │   ├── sqla.py
│   │   ├── ssh.py
│   │   ├── target.py
│   │   └── webhdfs.py
│   ├── date_interval.py
│   ├── db_task_history.py
│   ├── event.py
│   ├── execution_summary.py
│   ├── format.py
│   ├── freezing.py
│   ├── interface.py
│   ├── local_target.py
│   ├── lock.py
│   ├── metrics.py
│   ├── mock.py
│   ├── mypy.py
│   ├── notifications.py
│   ├── parameter.py
│   ├── process.py
│   ├── py.typed
│   ├── retcodes.py
│   ├── rpc.py
│   ├── safe_extractor.py
│   ├── scheduler.py
│   ├── server.py
│   ├── setup_logging.py
│   ├── static/
│   │   └── visualiser/
│   │       ├── css/
│   │       │   ├── luigi.css
│   │       │   └── tipsy.css
│   │       ├── fonts/
│   │       │   └── FontAwesome.otf
│   │       ├── index.html
│   │       ├── js/
│   │       │   ├── graph.js
│   │       │   ├── luigi.js
│   │       │   ├── test/
│   │       │   │   └── graph_test.js
│   │       │   ├── tipsy.js
│   │       │   ├── util.js
│   │       │   └── visualiserApp.js
│   │       ├── lib/
│   │       │   ├── URI/
│   │       │   │   └── 1.18.2/
│   │       │   │       └── URI.js
│   │       │   ├── datatables/
│   │       │   │   └── images/
│   │       │   │       └── Sorting icons.psd
│   │       │   └── mustache.js
│   │       ├── mockdata/
│   │       │   ├── dep_graph
│   │       │   ├── fetch_error
│   │       │   └── task_list
│   │       └── test.html
│   ├── target.py
│   ├── task.py
│   ├── task_history.py
│   ├── task_register.py
│   ├── task_status.py
│   ├── templates/
│   │   ├── history.html
│   │   ├── layout.html
│   │   ├── menu.html
│   │   ├── recent.html
│   │   └── show.html
│   ├── tools/
│   │   ├── __init__.py
│   │   ├── deps.py
│   │   ├── deps_tree.py
│   │   ├── luigi_grep.py
│   │   └── range.py
│   ├── util.py
│   └── worker.py
├── pyproject.toml
├── scripts/
│   └── ci/
│       ├── conditional_tox.sh
│       ├── install_start_azurite.sh
│       ├── setup_hadoop_env.sh
│       └── stop_azurite.sh
├── test/
│   ├── _mysqldb_test.py
│   ├── _test_ftp.py
│   ├── auto_namespace_test/
│   │   ├── __init__.py
│   │   └── my_namespace_test.py
│   ├── batch_notifier_test.py
│   ├── choice_parameter_test.py
│   ├── clone_test.py
│   ├── cmdline_test.py
│   ├── config_env_test.py
│   ├── config_toml_test.py
│   ├── conftest.py
│   ├── contrib/
│   │   ├── __init__.py
│   │   ├── _webhdfs_test.py
│   │   ├── azureblob_test.py
│   │   ├── batch_test.py
│   │   ├── beam_dataflow_test.py
│   │   ├── bigquery_avro_test.py
│   │   ├── bigquery_gcloud_test.py
│   │   ├── bigquery_test.py
│   │   ├── cascading_test.py
│   │   ├── datadog_metric_test.py
│   │   ├── dataproc_test.py
│   │   ├── docker_runner_test.py
│   │   ├── dropbox_test.py
│   │   ├── ecs_test.py
│   │   ├── esindex_test.py
│   │   ├── external_daily_snapshot_test.py
│   │   ├── external_program_test.py
│   │   ├── gcs_test.py
│   │   ├── hadoop_jar_test.py
│   │   ├── hdfs/
│   │   │   └── webhdfs_client_test.py
│   │   ├── hdfs_test.py
│   │   ├── hive_test.py
│   │   ├── kubernetes_test.py
│   │   ├── lsf_test.py
│   │   ├── mongo_test.py
│   │   ├── mysqldb_test.py
│   │   ├── opener_test.py
│   │   ├── pai_test.py
│   │   ├── pig_test.py
│   │   ├── postgres_test.py
│   │   ├── postgres_with_server_test.py
│   │   ├── presto_test.py
│   │   ├── prometheus_metric_test.py
│   │   ├── rdbms_test.py
│   │   ├── redis_test.py
│   │   ├── redshift_test.py
│   │   ├── s3_test.py
│   │   ├── salesforce_test.py
│   │   ├── scalding_test.py
│   │   ├── sge_test.py
│   │   ├── spark_test.py
│   │   ├── sqla_test.py
│   │   ├── streaming_test.py
│   │   └── test_ssh.py
│   ├── create_packages_archive_root/
│   │   ├── module.py
│   │   └── package/
│   │       ├── __init__.py
│   │       ├── submodule.py
│   │       ├── submodule_with_absolute_import.py
│   │       ├── submodule_without_imports.py
│   │       └── subpackage/
│   │           ├── __init__.py
│   │           └── submodule.py
│   ├── custom_metrics_test.py
│   ├── customized_run_test.py
│   ├── date_interval_test.py
│   ├── date_parameter_test.py
│   ├── db_task_history_test.py
│   ├── decorator_test.py
│   ├── dict_parameter_test.py
│   ├── dynamic_import_test.py
│   ├── event_callbacks_test.py
│   ├── execution_summary_test.py
│   ├── factorial_test.py
│   ├── fib_test.py
│   ├── gcloud-credentials.json.enc
│   ├── hdfs_client_test.py
│   ├── helpers.py
│   ├── helpers_test.py
│   ├── import_test.py
│   ├── instance_test.py
│   ├── instance_wrap_test.py
│   ├── interface_test.py
│   ├── list_parameter_test.py
│   ├── local_target_test.py
│   ├── lock_test.py
│   ├── metrics_test.py
│   ├── mock_test.py
│   ├── most_common_test.py
│   ├── mypy_test.py
│   ├── notifications_test.py
│   ├── numerical_parameter_test.py
│   ├── optional_parameter_test.py
│   ├── other_module.py
│   ├── parameter_test.py
│   ├── priority_test.py
│   ├── range_test.py
│   ├── recursion_test.py
│   ├── remote_scheduler_test.py
│   ├── retcodes_test.py
│   ├── rpc_test.py
│   ├── runtests.py
│   ├── safe_extractor_test.py
│   ├── scheduler_api_test.py
│   ├── scheduler_message_test.py
│   ├── scheduler_parameter_visibilities_test.py
│   ├── scheduler_test.py
│   ├── scheduler_visualisation_test.py
│   ├── server_test.py
│   ├── set_task_name_test.py
│   ├── setup_logging_test.py
│   ├── simulate_test.py
│   ├── subtask_test.py
│   ├── target_test.py
│   ├── task_bulk_complete_test.py
│   ├── task_forwarded_attributes_test.py
│   ├── task_history_test.py
│   ├── task_progress_percentage_test.py
│   ├── task_register_test.py
│   ├── task_running_resources_test.py
│   ├── task_serialize_test.py
│   ├── task_status_message_test.py
│   ├── task_test.py
│   ├── test_sigpipe.py
│   ├── test_ssh.py
│   ├── testconfig/
│   │   ├── core-site.xml
│   │   ├── log4j.properties
│   │   ├── logging.cfg
│   │   ├── luigi.toml
│   │   ├── luigi_local.toml
│   │   ├── luigi_logging.toml
│   │   └── pyproject.toml
│   ├── util_previous_test.py
│   ├── util_test.py
│   ├── visible_parameters_test.py
│   ├── visualiser/
│   │   ├── __init__.py
│   │   ├── phantomjs_test.js
│   │   └── visualiser_test.py
│   ├── worker_external_task_test.py
│   ├── worker_keep_alive_test.py
│   ├── worker_multiprocess_test.py
│   ├── worker_parallel_scheduling_test.py
│   ├── worker_scheduler_com_test.py
│   ├── worker_task_process_test.py
│   ├── worker_task_test.py
│   ├── worker_test.py
│   └── wrap_test.py
└── tox.ini
Download .txt
Showing preview only (413K chars total). Download the full file or copy to clipboard to get everything.
SYMBOL INDEX (5364 symbols across 245 files)

FILE: doc/conf.py
  function parameter_repr (line 25) | def parameter_repr(self):
  function assertIn (line 39) | def assertIn(needle, haystack):

FILE: examples/dynamic_requirements.py
  class Configuration (line 25) | class Configuration(luigi.Task):
    method output (line 28) | def output(self):
    method run (line 38) | def run(self):
  class Data (line 47) | class Data(luigi.Task):
    method output (line 50) | def output(self):
    method run (line 60) | def run(self):
  class Dynamic (line 66) | class Dynamic(luigi.Task):
    method output (line 69) | def output(self):
    method run (line 79) | def run(self):

FILE: examples/elasticsearch_index.py
  class FakeDocuments (line 24) | class FakeDocuments(luigi.Task):
    method run (line 32) | def run(self):
    method output (line 49) | def output(self):
  class IndexDocuments (line 60) | class IndexDocuments(CopyToIndex):
    method requires (line 103) | def requires(self):

FILE: examples/execution_summary_example.py
  class MyExternal (line 53) | class MyExternal(luigi.ExternalTask):
    method complete (line 54) | def complete(self):
  class Boom (line 58) | class Boom(luigi.Task):
    method run (line 62) | def run(self):
    method requires (line 65) | def requires(self):
  class Foo (line 70) | class Foo(luigi.Task):
    method run (line 75) | def run(self):
    method requires (line 78) | def requires(self):
  class Bar (line 83) | class Bar(luigi.Task):
    method run (line 87) | def run(self):
    method output (line 90) | def output(self):
  class DateTask (line 94) | class DateTask(luigi.Task):
    method run (line 99) | def run(self):
    method requires (line 102) | def requires(self):
  class EntryPoint (line 107) | class EntryPoint(luigi.Task):
    method run (line 110) | def run(self):
    method requires (line 113) | def requires(self):

FILE: examples/foo.py
  class Foo (line 32) | class Foo(luigi.WrapperTask):
    method run (line 35) | def run(self):
    method requires (line 38) | def requires(self):
  class Bar (line 43) | class Bar(luigi.Task):
    method run (line 47) | def run(self):
    method output (line 51) | def output(self):

FILE: examples/foo_complex.py
  class Foo (line 37) | class Foo(luigi.Task):
    method run (line 40) | def run(self):
    method requires (line 43) | def requires(self):
  class Bar (line 50) | class Bar(luigi.Task):
    method run (line 55) | def run(self):
    method requires (line 59) | def requires(self):
    method output (line 68) | def output(self):

FILE: examples/ftp_experiment_outputs.py
  class ExperimentTask (line 28) | class ExperimentTask(luigi.ExternalTask):
    method output (line 34) | def output(self):
    method run (line 44) | def run(self):
  class ProcessingTask (line 55) | class ProcessingTask(luigi.Task):
    method requires (line 61) | def requires(self):
    method output (line 71) | def output(self):
    method run (line 81) | def run(self):

FILE: examples/hello_world.py
  class HelloWorldTask (line 14) | class HelloWorldTask(luigi.Task):
    method run (line 17) | def run(self):

FILE: examples/kubernetes.py
  class PerlPi (line 45) | class PerlPi(KubernetesJobTask):

FILE: examples/per_task_retry_policy.py
  class PerTaskRetryPolicy (line 56) | class PerTaskRetryPolicy(luigi.Task):
    method requires (line 66) | def requires(self):
    method output (line 69) | def output(self):
  class ErrorTask1 (line 73) | class ErrorTask1(luigi.Task):
    method run (line 82) | def run(self):
    method output (line 86) | def output(self):
  class ErrorTask2 (line 90) | class ErrorTask2(luigi.Task):
    method run (line 99) | def run(self):
    method output (line 103) | def output(self):
  class DynamicErrorTaskSubmitter (line 107) | class DynamicErrorTaskSubmitter(luigi.Task):
    method run (line 110) | def run(self):
    method output (line 117) | def output(self):
  class DynamicErrorTask1 (line 121) | class DynamicErrorTask1(luigi.Task):
    method run (line 130) | def run(self):
    method output (line 134) | def output(self):
  class SuccessTask1 (line 138) | class SuccessTask1(luigi.Task):
    method requires (line 139) | def requires(self):
    method run (line 142) | def run(self):
    method output (line 146) | def output(self):
  class SuccessSubTask1 (line 150) | class SuccessSubTask1(luigi.Task):
    method run (line 155) | def run(self):
    method output (line 159) | def output(self):

FILE: examples/pyspark_wc.py
  class InlinePySparkWordCount (line 22) | class InlinePySparkWordCount(PySparkTask):
    method input (line 42) | def input(self):
    method output (line 45) | def output(self):
    method main (line 48) | def main(self, sc, *args):
  class PySparkWordCount (line 54) | class PySparkWordCount(SparkSubmitTask):
    method app_options (line 81) | def app_options(self):
    method input (line 85) | def input(self):
    method output (line 88) | def output(self):

FILE: examples/spark_als.py
  class UserItemMatrix (line 26) | class UserItemMatrix(luigi.Task):
    method run (line 30) | def run(self):
    method output (line 48) | def output(self):
  class SparkALS (line 59) | class SparkALS(SparkSubmitTask):
    method app_options (line 84) | def app_options(self):
    method requires (line 88) | def requires(self):
    method output (line 98) | def output(self):

FILE: examples/ssh_remote_execution.py
  class CreateRemoteData (line 27) | class CreateRemoteData(luigi.Task):
    method output (line 33) | def output(self):
    method run (line 43) | def run(self):
  class ProcessRemoteData (line 48) | class ProcessRemoteData(luigi.Task):
    method requires (line 55) | def requires(self):
    method run (line 65) | def run(self):
    method output (line 78) | def output(self):

FILE: examples/terasort.py
  function hadoop_examples_jar (line 28) | def hadoop_examples_jar():
  class TeraGen (line 44) | class TeraGen(luigi.contrib.hadoop_jar.HadoopJarJobTask):
    method output (line 52) | def output(self):
    method jar (line 62) | def jar(self):
    method main (line 65) | def main(self):
    method args (line 68) | def args(self):
  class TeraSort (line 73) | class TeraSort(luigi.contrib.hadoop_jar.HadoopJarJobTask):
    method requires (line 81) | def requires(self):
    method output (line 91) | def output(self):
    method jar (line 101) | def jar(self):
    method main (line 104) | def main(self):
    method args (line 107) | def args(self):

FILE: examples/top_artists.py
  class ExternalStreams (line 28) | class ExternalStreams(luigi.ExternalTask):
    method output (line 38) | def output(self):
  class Streams (line 49) | class Streams(luigi.Task):
    method run (line 56) | def run(self):
    method output (line 64) | def output(self):
  class StreamsHdfs (line 75) | class StreamsHdfs(Streams):
    method output (line 83) | def output(self):
  class AggregateArtists (line 94) | class AggregateArtists(luigi.Task):
    method output (line 102) | def output(self):
    method requires (line 112) | def requires(self):
    method run (line 122) | def run(self):
  class AggregateArtistsSpark (line 136) | class AggregateArtistsSpark(luigi.contrib.spark.SparkSubmitTask):
    method output (line 158) | def output(self):
    method requires (line 168) | def requires(self):
    method app_options (line 178) | def app_options(self):
  class Top10Artists (line 184) | class Top10Artists(luigi.Task):
    method requires (line 194) | def requires(self):
    method output (line 208) | def output(self):
    method run (line 218) | def run(self):
    method _input_iterator (line 225) | def _input_iterator(self):
  class ArtistToplistToDatabase (line 232) | class ArtistToplistToDatabase(luigi.contrib.postgres.CopyToTable):
    method requires (line 254) | def requires(self):

FILE: examples/top_artists_spark.py
  function main (line 9) | def main(argv):

FILE: examples/wordcount.py
  class InputText (line 20) | class InputText(luigi.ExternalTask):
    method output (line 28) | def output(self):
  class WordCount (line 39) | class WordCount(luigi.Task):
    method requires (line 42) | def requires(self):
    method output (line 52) | def output(self):
    method run (line 62) | def run(self):

FILE: examples/wordcount_hadoop.py
  class InputText (line 28) | class InputText(luigi.ExternalTask):
    method output (line 37) | def output(self):
  class WordCount (line 48) | class WordCount(luigi.contrib.hadoop.JobTask):
    method requires (line 59) | def requires(self):
    method output (line 69) | def output(self):
    method mapper (line 79) | def mapper(self, line):
    method reducer (line 83) | def reducer(self, key, values):

FILE: luigi/batch_notifier.py
  class batch_email (line 16) | class batch_email(luigi.task.Config):
  class ExplQueue (line 35) | class ExplQueue(collections.OrderedDict):
    method __init__ (line 36) | def __init__(self, num_items):
    method enqueue (line 40) | def enqueue(self, item):
  function _fail_queue (line 47) | def _fail_queue(num_messages):
  function _plural_format (line 51) | def _plural_format(template, number, plural="s"):
  class BatchNotifier (line 57) | class BatchNotifier:
    method __init__ (line 58) | def __init__(self, **kwargs):
    method _update_next_send (line 72) | def _update_next_send(self):
    method _key (line 75) | def _key(self, task_name, family, unbatched_args):
    method _format_expl (line 86) | def _format_expl(self, expl):
    method _expl_body (line 93) | def _expl_body(self, expls):
    method _format_task (line 99) | def _format_task(self, task_tuple):
    method _format_tasks (line 109) | def _format_tasks(self, tasks):
    method _owners (line 116) | def _owners(self, owners):
    method add_failure (line 119) | def add_failure(self, task_name, family, unbatched_args, expl, owners):
    method add_disable (line 125) | def add_disable(self, task_name, family, unbatched_args, owners):
    method add_scheduling_fail (line 131) | def add_scheduling_fail(self, task_name, family, unbatched_args, expl,...
    method _task_expl_groups (line 138) | def _task_expl_groups(self, expls):
    method _expls_key (line 147) | def _expls_key(self, expls_tuple):
    method _expl_key (line 154) | def _expl_key(self, expl):
    method _email_body (line 157) | def _email_body(self, fail_counts, disable_counts, scheduling_counts, ...
    method _send_email (line 172) | def _send_email(self, fail_counts, disable_counts, scheduling_counts, ...
    method send_email (line 188) | def send_email(self):
    method update (line 205) | def update(self):

FILE: luigi/cmdline.py
  function luigi_run (line 8) | def luigi_run(argv=sys.argv[1:]):
  function luigid (line 12) | def luigid(argv=sys.argv[1:]):

FILE: luigi/cmdline_parser.py
  class CmdlineParser (line 29) | class CmdlineParser:
    method get_instance (line 40) | def get_instance(cls):
    method global_instance (line 46) | def global_instance(cls, cmdline_args, allow_override=False):
    method __init__ (line 61) | def __init__(self, cmdline_args):
    method _build_parser (line 82) | def _build_parser(root_task=None, help_all=False):
    method get_task_obj (line 106) | def get_task_obj(self):
    method _get_task_cls (line 112) | def _get_task_cls(self):
    method _get_task_kwargs (line 118) | def _get_task_kwargs(self):
    method _attempt_load_module (line 132) | def _attempt_load_module(known_args):
    method _possibly_exit_with_help (line 141) | def _possibly_exit_with_help(parser, known_args):

FILE: luigi/configuration/base_parser.py
  class BaseParser (line 23) | class BaseParser:
    method instance (line 25) | def instance(cls, *args, **kwargs):
    method add_config_path (line 35) | def add_config_path(cls, path):
    method reload (line 40) | def reload(cls):

FILE: luigi/configuration/cfg_parser.py
  class InterpolationMissingEnvvarError (line 40) | class InterpolationMissingEnvvarError(InterpolationError):
    method __init__ (line 45) | def __init__(self, option, section, value, envvar):
  class EnvironmentInterpolation (line 50) | class EnvironmentInterpolation(Interpolation):
    method before_get (line 58) | def before_get(self, parser, section, option, value, defaults):
    method _interpolate_env (line 61) | def _interpolate_env(self, option, section, value):
  class CombinedInterpolation (line 81) | class CombinedInterpolation(Interpolation):
    method __init__ (line 88) | def __init__(self, interpolations):
    method before_get (line 91) | def before_get(self, parser, section, option, value, defaults):
    method before_read (line 96) | def before_read(self, parser, section, option, value):
    method before_set (line 101) | def before_set(self, parser, section, option, value):
    method before_write (line 106) | def before_write(self, parser, section, option, value):
  class LuigiConfigParser (line 112) | class LuigiConfigParser(BaseParser, ConfigParser):
    method reload (line 126) | def reload(cls):
    method _get_with_default (line 137) | def _get_with_default(self, method, section, option, default, expected...
    method has_option (line 164) | def has_option(self, section, option):
    method get (line 185) | def get(self, section, option, default=NO_DEFAULT, **kwargs):
    method getboolean (line 188) | def getboolean(self, section, option, default=NO_DEFAULT):
    method getint (line 191) | def getint(self, section, option, default=NO_DEFAULT):
    method getfloat (line 194) | def getfloat(self, section, option, default=NO_DEFAULT):
    method getintdict (line 197) | def getintdict(self, section):
    method set (line 204) | def set(self, section, option, value=None):

FILE: luigi/configuration/core.py
  function _get_default_parser (line 37) | def _get_default_parser():
  function _check_parser (line 45) | def _check_parser(parser_class, parser):
  function get_config (line 51) | def get_config(parser=None):
  function add_config_path (line 60) | def add_config_path(path):

FILE: luigi/configuration/toml_parser.py
  class LuigiTomlParser (line 32) | class LuigiTomlParser(BaseParser, ConfigParser):
    method _update_data (line 43) | def _update_data(data, new_data):
    method read (line 54) | def read(self, config_paths):
    method get (line 68) | def get(self, section, option, default=NO_DEFAULT, **kwargs):
    method getboolean (line 76) | def getboolean(self, section, option, default=NO_DEFAULT):
    method getint (line 79) | def getint(self, section, option, default=NO_DEFAULT):
    method getfloat (line 82) | def getfloat(self, section, option, default=NO_DEFAULT):
    method getintdict (line 85) | def getintdict(self, section):
    method set (line 88) | def set(self, section, option, value=None):
    method has_option (line 93) | def has_option(self, section, option):
    method __getitem__ (line 96) | def __getitem__(self, name):

FILE: luigi/contrib/azureblob.py
  class AzureBlobClient (line 31) | class AzureBlobClient(FileSystem):
    method __init__ (line 45) | def __init__(self, account_name=None, account_key=None, sas_token=None...
    method connection (line 79) | def connection(self):
    method container_client (line 87) | def container_client(self, container_name):
    method blob_client (line 90) | def blob_client(self, container_name, blob_name):
    method upload (line 94) | def upload(self, tmp_path, container, blob, **kwargs):
    method download_as_bytes (line 108) | def download_as_bytes(self, container, blob, bytes_to_read=None):
    method download_as_file (line 114) | def download_as_file(self, container, blob, location):
    method create_container (line 122) | def create_container(self, container_name):
    method delete_container (line 126) | def delete_container(self, container_name):
    method exists (line 131) | def exists(self, path):
    method remove (line 138) | def remove(self, path, recursive=True, skip_trash=True):
    method mkdir (line 148) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method isdir (line 153) | def isdir(self, path):
    method move (line 161) | def move(self, path, dest):
    method copy (line 168) | def copy(self, path, dest):
    method rename_dont_move (line 189) | def rename_dont_move(self, path, dest):
    method splitfilepath (line 193) | def splitfilepath(filepath):
  class ReadableAzureBlobFile (line 201) | class ReadableAzureBlobFile:
    method __init__ (line 202) | def __init__(self, container, blob, client, download_when_reading, **k...
    method read (line 212) | def read(self, n=None):
    method __enter__ (line 215) | def __enter__(self):
    method __exit__ (line 223) | def __exit__(self, exc_type, exc, traceback):
    method __del__ (line 226) | def __del__(self):
    method close (line 231) | def close(self):
    method readable (line 237) | def readable(self):
    method writable (line 240) | def writable(self):
    method seekable (line 243) | def seekable(self):
    method seek (line 246) | def seek(self, offset, whence=None):
  class AtomicAzureBlobFile (line 250) | class AtomicAzureBlobFile(AtomicLocalFile):
    method __init__ (line 251) | def __init__(self, container, blob, client, **kwargs):
    method move_to_final_destination (line 258) | def move_to_final_destination(self):
  class AzureBlobTarget (line 262) | class AzureBlobTarget(FileSystemTarget):
    method __init__ (line 267) | def __init__(self, container, blob, client=None, format=None, download...
    method fs (line 297) | def fs(self):
    method open (line 303) | def open(self, mode):

FILE: luigi/contrib/batch.py
  class BatchJobException (line 78) | class BatchJobException(Exception):
  function _random_id (line 85) | def _random_id():
  class BatchClient (line 89) | class BatchClient:
    method __init__ (line 90) | def __init__(self, poll_time=POLL_TIME):
    method get_active_queue (line 96) | def get_active_queue(self):
    method get_job_id_from_name (line 107) | def get_job_id_from_name(self, job_name):
    method get_job_status (line 114) | def get_job_status(self, job_id):
    method get_logs (line 131) | def get_logs(self, log_stream_name, get_last=50):
    method submit_job (line 137) | def submit_job(self, job_definition, parameters, job_name=None, queue=...
    method wait_on_job (line 144) | def wait_on_job(self, job_id):
    method register_job_definition (line 165) | def register_job_definition(self, json_fpath):
  class BatchTask (line 177) | class BatchTask(luigi.Task):
    method run (line 196) | def run(self):
    method parameters (line 202) | def parameters(self):

FILE: luigi/contrib/beam_dataflow.py
  class DataflowParamKeys (line 33) | class DataflowParamKeys(metaclass=abc.ABCMeta):
    method runner (line 43) | def runner(self):
    method project (line 48) | def project(self):
    method zone (line 53) | def zone(self):
    method region (line 58) | def region(self):
    method staging_location (line 63) | def staging_location(self):
    method temp_location (line 68) | def temp_location(self):
    method gcp_temp_location (line 73) | def gcp_temp_location(self):
    method num_workers (line 78) | def num_workers(self):
    method autoscaling_algorithm (line 83) | def autoscaling_algorithm(self):
    method max_num_workers (line 88) | def max_num_workers(self):
    method disk_size_gb (line 93) | def disk_size_gb(self):
    method worker_machine_type (line 98) | def worker_machine_type(self):
    method worker_disk_type (line 103) | def worker_disk_type(self):
    method job_name (line 108) | def job_name(self):
    method service_account (line 113) | def service_account(self):
    method network (line 118) | def network(self):
    method subnetwork (line 123) | def subnetwork(self):
    method labels (line 128) | def labels(self):
  class _CmdLineRunner (line 132) | class _CmdLineRunner:
    method run (line 141) | def run(cmd, task=None):
  class BeamDataflowJobTask (line 158) | class BeamDataflowJobTask(MixinNaiveBulkComplete, luigi.Task, metaclass=...
    method __init__ (line 232) | def __init__(self):
    method dataflow_executable (line 238) | def dataflow_executable(self):
    method args (line 247) | def args(self):
    method before_run (line 256) | def before_run(self):
    method on_successful_run (line 263) | def on_successful_run(self):
    method validate_output (line 270) | def validate_output(self):
    method file_pattern (line 278) | def file_pattern(self):
    method on_successful_output_validation (line 289) | def on_successful_output_validation(self):
    method cleanup_on_error (line 296) | def cleanup_on_error(self, error):
    method run (line 303) | def run(self):
    method _mk_cmd_line (line 325) | def _mk_cmd_line(self):
    method _get_runner (line 334) | def _get_runner(self):
    method _get_dataflow_args (line 343) | def _get_dataflow_args(self):
    method _format_input_args (line 388) | def _format_input_args(self):
    method _format_output_args (line 452) | def _format_output_args(self):
    method get_target_path (line 478) | def get_target_path(target):

FILE: luigi/contrib/bigquery.py
  function is_error_5xx (line 42) | def is_error_5xx(err):
  class CreateDisposition (line 55) | class CreateDisposition:
  class WriteDisposition (line 60) | class WriteDisposition:
  class QueryMode (line 66) | class QueryMode:
  class SourceFormat (line 71) | class SourceFormat:
  class FieldDelimiter (line 79) | class FieldDelimiter:
  class PrintHeader (line 96) | class PrintHeader:
  class DestinationFormat (line 101) | class DestinationFormat:
  class Compression (line 107) | class Compression:
  class Encoding (line 112) | class Encoding:
  class BQTable (line 126) | class BQTable(collections.namedtuple("BQTable", "project_id dataset_id t...
    method dataset (line 128) | def dataset(self):
    method uri (line 132) | def uri(self):
  class BigQueryClient (line 136) | class BigQueryClient:
    method __init__ (line 144) | def __init__(self, oauth_credentials=None, descriptor="", http_=None):
    method _initialise_client (line 153) | def _initialise_client(self):
    method dataset_exists (line 162) | def dataset_exists(self, dataset):
    method table_exists (line 189) | def table_exists(self, table):
    method make_dataset (line 207) | def make_dataset(self, dataset, raise_if_exists=False, body=None):
    method delete_dataset (line 233) | def delete_dataset(self, dataset, delete_nonempty=True):
    method delete_table (line 246) | def delete_table(self, table):
    method list_datasets (line 258) | def list_datasets(self, project_id):
    method list_tables (line 278) | def list_tables(self, dataset):
    method get_view (line 298) | def get_view(self, table):
    method update_view (line 316) | def update_view(self, table, view):
    method run_job (line 335) | def run_job(self, project_id, body, dataset=None):
    method copy (line 364) | def copy(self, source_table, dest_table, create_disposition=CreateDisp...
  class BigQueryTarget (line 399) | class BigQueryTarget(luigi.target.Target):
    method __init__ (line 400) | def __init__(self, project_id, dataset_id, table_id, client=None, loca...
    method from_bqtable (line 405) | def from_bqtable(cls, table, client=None):
    method exists (line 413) | def exists(self):
    method __str__ (line 416) | def __str__(self):
  class MixinBigQueryBulkComplete (line 420) | class MixinBigQueryBulkComplete:
    method bulk_complete (line 430) | def bulk_complete(cls, parameter_tuples):
  class BigQueryLoadTask (line 452) | class BigQueryLoadTask(MixinBigQueryBulkComplete, luigi.Task):
    method source_format (line 456) | def source_format(self):
    method encoding (line 461) | def encoding(self):
    method write_disposition (line 466) | def write_disposition(self):
    method schema (line 473) | def schema(self):
    method max_bad_records (line 480) | def max_bad_records(self):
    method field_delimiter (line 487) | def field_delimiter(self):
    method source_uris (line 491) | def source_uris(self):
    method skip_leading_rows (line 498) | def skip_leading_rows(self):
    method allow_jagged_rows (line 505) | def allow_jagged_rows(self):
    method ignore_unknown_values (line 514) | def ignore_unknown_values(self):
    method allow_quoted_new_lines (line 527) | def allow_quoted_new_lines(self):
    method configure_job (line 531) | def configure_job(self, configuration):
    method run (line 541) | def run(self):
  class BigQueryRunQueryTask (line 584) | class BigQueryRunQueryTask(MixinBigQueryBulkComplete, luigi.Task):
    method write_disposition (line 586) | def write_disposition(self):
    method create_disposition (line 593) | def create_disposition(self):
    method flatten_results (line 598) | def flatten_results(self):
    method query (line 604) | def query(self):
    method query_mode (line 609) | def query_mode(self):
    method udf_resource_uris (line 614) | def udf_resource_uris(self):
    method use_legacy_sql (line 619) | def use_legacy_sql(self):
    method configure_job (line 623) | def configure_job(self, configuration):
    method run (line 633) | def run(self):
  class BigQueryCreateViewTask (line 671) | class BigQueryCreateViewTask(luigi.Task):
    method view (line 682) | def view(self):
    method complete (line 686) | def complete(self):
    method run (line 696) | def run(self):
  class ExternalBigQueryTask (line 710) | class ExternalBigQueryTask(MixinBigQueryBulkComplete, luigi.ExternalTask):
  class BigQueryExtractTask (line 718) | class BigQueryExtractTask(luigi.Task):
    method destination_uris (line 728) | def destination_uris(self):
    method print_header (line 742) | def print_header(self):
    method field_delimiter (line 747) | def field_delimiter(self):
    method destination_format (line 755) | def destination_format(self):
    method compression (line 762) | def compression(self):
    method configure_job (line 766) | def configure_job(self, configuration):
    method run (line 776) | def run(self):
  class BigQueryExecutionError (line 821) | class BigQueryExecutionError(Exception):
    method __init__ (line 822) | def __init__(self, job_id, error_message) -> None:

FILE: luigi/contrib/bigquery_avro.py
  class BigQueryLoadAvro (line 18) | class BigQueryLoadAvro(BigQueryLoadTask):
    method _avro_uri (line 33) | def _avro_uri(self, target):
    method source_uris (line 37) | def source_uris(self):
    method _get_input_schema (line 40) | def _get_input_schema(self):
    method _get_writer_schema (line 78) | def _get_writer_schema(datum_reader):
    method _set_output_doc (line 89) | def _set_output_doc(self, avro_schema):
    method run (line 99) | def run(self):

FILE: luigi/contrib/datadog_metric.py
  class datadog (line 15) | class datadog(Config):
  class DatadogMetricsCollector (line 25) | class DatadogMetricsCollector(MetricsCollector):
    method __init__ (line 26) | def __init__(self, *args, **kwargs):
    method handle_task_started (line 31) | def handle_task_started(self, task):
    method handle_task_failed (line 41) | def handle_task_failed(self, task):
    method handle_task_disabled (line 51) | def handle_task_disabled(self, task, config):
    method handle_task_done (line 68) | def handle_task_done(self, task):
    method _send_event (line 85) | def _send_event(self, **params):
    method _send_gauge (line 90) | def _send_gauge(self, metric_name, value, tags=[]):
    method _send_increment (line 96) | def _send_increment(self, metric_name, value=1, tags=[]):
    method _format_task_params_to_tags (line 102) | def _format_task_params_to_tags(self, task):
    method default_tags (line 110) | def default_tags(self):

FILE: luigi/contrib/dataproc.py
  function get_dataproc_client (line 29) | def get_dataproc_client():
  function set_dataproc_client (line 33) | def set_dataproc_client(client):
  class _DataprocBaseTask (line 38) | class _DataprocBaseTask(luigi.Task):
  class DataprocBaseTask (line 46) | class DataprocBaseTask(_DataprocBaseTask):
    method submit_job (line 56) | def submit_job(self, job_config):
    method submit_spark_job (line 63) | def submit_spark_job(self, jars, main_class, job_args=None):
    method submit_pyspark_job (line 76) | def submit_pyspark_job(self, job_file, extra_files=list(), job_args=No...
    method wait_for_job (line 92) | def wait_for_job(self):
  class DataprocSparkTask (line 112) | class DataprocSparkTask(DataprocBaseTask):
    method run (line 121) | def run(self):
  class DataprocPysparkTask (line 128) | class DataprocPysparkTask(DataprocBaseTask):
    method run (line 137) | def run(self):
  class CreateDataprocClusterTask (line 146) | class CreateDataprocClusterTask(_DataprocBaseTask):
    method _get_cluster_status (line 160) | def _get_cluster_status(self):
    method complete (line 169) | def complete(self):
    method run (line 179) | def run(self):
  class DeleteDataprocClusterTask (line 221) | class DeleteDataprocClusterTask(_DataprocBaseTask):
    method _get_cluster_status (line 230) | def _get_cluster_status(self):
    method complete (line 245) | def complete(self):
    method run (line 248) | def run(self):

FILE: luigi/contrib/docker_runner.py
  class DockerTask (line 67) | class DockerTask(luigi.Task):
    method image (line 69) | def image(self):
    method command (line 73) | def command(self):
    method name (line 77) | def name(self):
    method host_config_options (line 81) | def host_config_options(self):
    method container_options (line 91) | def container_options(self):
    method environment (line 101) | def environment(self):
    method container_tmp_dir (line 105) | def container_tmp_dir(self):
    method binds (line 109) | def binds(self):
    method network_mode (line 118) | def network_mode(self):
    method docker_url (line 122) | def docker_url(self):
    method auto_remove (line 126) | def auto_remove(self):
    method force_pull (line 130) | def force_pull(self):
    method mount_tmp (line 134) | def mount_tmp(self):
    method __init__ (line 137) | def __init__(self, *args, **kwargs):
    method run (line 183) | def run(self):

FILE: luigi/contrib/dropbox.py
  function accept_trailing_slash_in_existing_dirpaths (line 42) | def accept_trailing_slash_in_existing_dirpaths(func):
  function accept_trailing_slash (line 57) | def accept_trailing_slash(func):
  class DropboxClient (line 67) | class DropboxClient(FileSystem):
    method __init__ (line 72) | def __init__(self, token, user_agent="Luigi", root_namespace_id=None):
    method exists (line 92) | def exists(self, path):
    method remove (line 109) | def remove(self, path, recursive=True, skip_trash=True):
    method mkdir (line 116) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method isdir (line 128) | def isdir(self, path):
    method listdir (line 141) | def listdir(self, path, **kwargs):
    method move (line 151) | def move(self, path, dest):
    method copy (line 155) | def copy(self, path, dest):
    method download_as_bytes (line 158) | def download_as_bytes(self, path):
    method upload (line 162) | def upload(self, tmp_path, dest_path):
    method _exists_and_is_dir (line 182) | def _exists_and_is_dir(self, path):
  class ReadableDropboxFile (line 197) | class ReadableDropboxFile:
    method __init__ (line 198) | def __init__(self, path, client):
    method read (line 212) | def read(self):
    method __enter__ (line 215) | def __enter__(self):
    method __exit__ (line 218) | def __exit__(self, exc_type, exc, traceback):
    method __del__ (line 221) | def __del__(self):
    method close (line 226) | def close(self):
    method readable (line 229) | def readable(self):
    method writable (line 232) | def writable(self):
    method seekable (line 235) | def seekable(self):
  class AtomicWritableDropboxFile (line 239) | class AtomicWritableDropboxFile(AtomicLocalFile):
    method __init__ (line 240) | def __init__(self, path, client):
    method move_to_final_destination (line 251) | def move_to_final_destination(self):
  class DropboxTarget (line 258) | class DropboxTarget(FileSystemTarget):
    method __init__ (line 263) | def __init__(self, path, token, format=None, user_agent="Luigi", root_...
    method __str__ (line 305) | def __str__(self):
    method fs (line 309) | def fs(self):
    method temporary_path (line 313) | def temporary_path(self):
    method open (line 322) | def open(self, mode):

FILE: luigi/contrib/ecs.py
  function _get_task_statuses (line 71) | def _get_task_statuses(task_ids, cluster):
  function _track_tasks (line 90) | def _track_tasks(task_ids, cluster):
  class ECSTask (line 101) | class ECSTask(luigi.Task):
    method ecs_task_ids (line 142) | def ecs_task_ids(self):
    method command (line 148) | def command(self):
    method update_container_overrides_command (line 170) | def update_container_overrides_command(container_overrides, command):
    method combined_overrides (line 186) | def combined_overrides(self):
    method run_task_kwargs (line 204) | def run_task_kwargs(self):
    method run (line 243) | def run(self):

FILE: luigi/contrib/esindex.py
  class ElasticsearchTarget (line 107) | class ElasticsearchTarget(luigi.Target):
    method __init__ (line 113) | def __init__(self, host, port, index, doc_type, update_id, marker_inde...
    method marker_index_document_id (line 154) | def marker_index_document_id(self):
    method touch (line 161) | def touch(self):
    method exists (line 179) | def exists(self):
    method create_marker_index (line 192) | def create_marker_index(self):
    method ensure_hist_size (line 199) | def ensure_hist_size(self):
  class CopyToIndex (line 217) | class CopyToIndex(luigi.Task):
    method host (line 246) | def host(self):
    method port (line 253) | def port(self):
    method http_auth (line 260) | def http_auth(self):
    method index (line 269) | def index(self):
    method doc_type (line 278) | def doc_type(self):
    method mapping (line 285) | def mapping(self):
    method settings (line 292) | def settings(self):
    method chunk_size (line 299) | def chunk_size(self):
    method raise_on_error (line 306) | def raise_on_error(self):
    method purge_existing_index (line 313) | def purge_existing_index(self):
    method marker_index_hist_size (line 320) | def marker_index_hist_size(self):
    method timeout (line 327) | def timeout(self):
    method extra_elasticsearch_args (line 334) | def extra_elasticsearch_args(self):
    method docs (line 340) | def docs(self):
    method _docs (line 352) | def _docs(self):
    method _init_connection (line 375) | def _init_connection(self):
    method create_index (line 385) | def create_index(self):
    method delete_index (line 395) | def delete_index(self):
    method update_id (line 403) | def update_id(self):
    method output (line 409) | def output(self):
    method run (line 427) | def run(self):

FILE: luigi/contrib/external_daily_snapshot.py
  class ExternalDailySnapshot (line 29) | class ExternalDailySnapshot(luigi.ExternalTask):
    method latest (line 55) | def latest(cls, *args, **kwargs):
    method __latest (line 69) | def __latest(cls, date, lookback, args, kwargs):

FILE: luigi/contrib/external_program.py
  class ExternalProgramTask (line 49) | class ExternalProgramTask(luigi.Task):
    method program_args (line 100) | def program_args(self):
    method program_environment (line 108) | def program_environment(self):
    method always_log_stderr (line 118) | def always_log_stderr(self):
    method _clean_output_file (line 126) | def _clean_output_file(self, file_object):
    method build_tracking_url (line 130) | def build_tracking_url(self, logs_output):
    method run (line 138) | def run(self):
    method _proc_with_tracking_url_context (line 179) | def _proc_with_tracking_url_context(self, proc_args, proc_kwargs):
  class ExternalProgramRunContext (line 216) | class ExternalProgramRunContext:
    method __init__ (line 217) | def __init__(self, proc):
    method __enter__ (line 220) | def __enter__(self):
    method __exit__ (line 225) | def __exit__(self, exc_type, exc_val, exc_tb):
    method kill_job (line 230) | def kill_job(self, captured_signal=None, stack_frame=None):
  class ExternalProgramRunError (line 237) | class ExternalProgramRunError(RuntimeError):
    method __init__ (line 238) | def __init__(self, message, args, env=None, stdout=None, stderr=None):
    method __str__ (line 246) | def __str__(self):
  class ExternalPythonProgramTask (line 260) | class ExternalPythonProgramTask(ExternalProgramTask):
    method program_environment (line 280) | def program_environment(self):

FILE: luigi/contrib/ftp.py
  class RemoteFileSystem (line 45) | class RemoteFileSystem(luigi.target.FileSystem):
    method __init__ (line 46) | def __init__(self, host, username=None, password=None, port=None, tls=...
    method _connect (line 63) | def _connect(self):
    method _sftp_connect (line 72) | def _sftp_connect(self):
    method _ftp_connect (line 80) | def _ftp_connect(self):
    method _close (line 90) | def _close(self):
    method _sftp_close (line 99) | def _sftp_close(self):
    method _ftp_close (line 102) | def _ftp_close(self):
    method exists (line 105) | def exists(self, path, mtime=None):
    method _sftp_exists (line 124) | def _sftp_exists(self, path, mtime):
    method _ftp_exists (line 132) | def _ftp_exists(self, path, mtime):
    method remove (line 147) | def remove(self, path, recursive=True):
    method _sftp_remove (line 166) | def _sftp_remove(self, path, recursive):
    method _ftp_remove (line 182) | def _ftp_remove(self, path, recursive):
    method _rm_recursive (line 193) | def _rm_recursive(self, ftp, path):
    method put (line 234) | def put(self, local_path, path, atomic=True):
    method _sftp_put (line 247) | def _sftp_put(self, local_path, path, atomic):
    method _ftp_put (line 262) | def _ftp_put(self, local_path, path, atomic):
    method get (line 287) | def get(self, path, local_path):
    method _sftp_get (line 310) | def _sftp_get(self, path, tmp_local_path):
    method _ftp_get (line 313) | def _ftp_get(self, path, tmp_local_path):
    method listdir (line 316) | def listdir(self, path="."):
    method _sftp_listdir (line 331) | def _sftp_listdir(self, path):
    method _ftp_listdir (line 334) | def _ftp_listdir(self, path):
  class AtomicFtpFile (line 338) | class AtomicFtpFile(luigi.target.AtomicLocalFile):
    method __init__ (line 345) | def __init__(self, fs, path):
    method move_to_final_destination (line 355) | def move_to_final_destination(self):
    method fs (line 359) | def fs(self):
  class RemoteTarget (line 363) | class RemoteTarget(luigi.target.FileSystemTarget):
    method __init__ (line 371) | def __init__(
    method fs (line 386) | def fs(self):
    method open (line 389) | def open(self, mode):
    method exists (line 423) | def exists(self):
    method put (line 426) | def put(self, local_path, atomic=True):
    method get (line 429) | def get(self, local_path):

FILE: luigi/contrib/gcp.py
  function get_authenticate_kwargs (line 19) | def get_authenticate_kwargs(oauth_credentials=None, http_=None):

FILE: luigi/contrib/gcs.py
  function is_error_5xx (line 68) | def is_error_5xx(err):
  function _wait_for_consistency (line 81) | def _wait_for_consistency(checker):
  class InvalidDeleteException (line 96) | class InvalidDeleteException(luigi.target.FileSystemException):
  class GCSClient (line 100) | class GCSClient(luigi.target.FileSystem):
    method __init__ (line 124) | def __init__(self, oauth_credentials=None, descriptor="", http_=None, ...
    method _path_to_bucket_and_key (line 137) | def _path_to_bucket_and_key(self, path):
    method _is_root (line 143) | def _is_root(self, key):
    method _add_path_delimiter (line 146) | def _add_path_delimiter(self, key):
    method _obj_exists (line 150) | def _obj_exists(self, bucket, obj):
    method _list_iter (line 160) | def _list_iter(self, bucket, prefix):
    method _do_put (line 175) | def _do_put(self, media, dest_path):
    method exists (line 191) | def exists(self, path):
    method isdir (line 198) | def isdir(self, path):
    method remove (line 217) | def remove(self, path, recursive=True):
    method put (line 242) | def put(self, filename, dest_path, mimetype=None, chunksize=None):
    method _forward_args_to_put (line 251) | def _forward_args_to_put(self, kwargs):
    method put_multiple (line 254) | def put_multiple(self, filepaths, remote_directory, mimetype=None, chu...
    method put_string (line 278) | def put_string(self, contents, dest_path, mimetype=None):
    method mkdir (line 286) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method copy (line 297) | def copy(self, source_path, destination_path):
    method rename (line 322) | def rename(self, *args, **kwargs):
    method move (line 328) | def move(self, source_path, destination_path):
    method listdir (line 335) | def listdir(self, path):
    method list_wildcard (line 350) | def list_wildcard(self, wildcard_path):
    method download (line 372) | def download(self, path, chunksize=None, chunk_callback=lambda _: False):
  class _DeleteOnCloseFile (line 401) | class _DeleteOnCloseFile(io.FileIO):
    method close (line 402) | def close(self):
    method readable (line 411) | def readable(self):
    method writable (line 414) | def writable(self):
    method seekable (line 417) | def seekable(self):
  class AtomicGCSFile (line 421) | class AtomicGCSFile(luigi.target.AtomicLocalFile):
    method __init__ (line 426) | def __init__(self, path, gcs_client):
    method move_to_final_destination (line 430) | def move_to_final_destination(self):
  class GCSTarget (line 434) | class GCSTarget(luigi.target.FileSystemTarget):
    method __init__ (line 437) | def __init__(self, path, format=None, client=None):
    method open (line 445) | def open(self, mode="r"):
  class GCSFlagTarget (line 454) | class GCSFlagTarget(GCSTarget):
    method __init__ (line 476) | def __init__(self, path, format=None, client=None, flag="_SUCCESS"):
    method exists (line 497) | def exists(self):

FILE: luigi/contrib/hadoop.py
  class hadoop (line 65) | class hadoop(Config):
  function attach (line 72) | def attach(*packages):
  function dereference (line 80) | def dereference(f):
  function get_extra_files (line 88) | def get_extra_files(extra_files):
  function create_packages_archive (line 112) | def create_packages_archive(packages, filename):
  function flatten (line 181) | def flatten(sequence):
  class HadoopRunContext (line 200) | class HadoopRunContext:
    method __init__ (line 201) | def __init__(self):
    method __enter__ (line 205) | def __enter__(self):
    method kill_job (line 210) | def kill_job(self, captured_signal=None, stack_frame=None):
    method __exit__ (line 221) | def __exit__(self, exc_type, exc_val, exc_tb):
  class HadoopJobError (line 227) | class HadoopJobError(RuntimeError):
    method __init__ (line 228) | def __init__(self, message, out=None, err=None):
    method __str__ (line 234) | def __str__(self):
  function run_and_track_hadoop_job (line 238) | def run_and_track_hadoop_job(arglist, tracking_url_callback=None, env=No...
  function fetch_task_failures (line 340) | def fetch_task_failures(tracking_url):
  class JobRunner (line 379) | class JobRunner:
  class HadoopJobRunner (line 383) | class HadoopJobRunner(JobRunner):
    method __init__ (line 390) | def __init__(
    method run_job (line 418) | def run_job(self, job, tracking_url_callback=None):
    method finish (line 571) | def finish(self):
    method __del__ (line 577) | def __del__(self):
  class DefaultHadoopJobRunner (line 581) | class DefaultHadoopJobRunner(HadoopJobRunner):
    method __init__ (line 586) | def __init__(self):
  class LocalJobRunner (line 593) | class LocalJobRunner(JobRunner):
    method __init__ (line 602) | def __init__(self, samplelines=None):
    method sample (line 605) | def sample(self, input_stream, n, output):
    method group (line 611) | def group(self, input_stream):
    method run_job (line 623) | def run_job(self, job):
  class BaseHadoopJobTask (line 657) | class BaseHadoopJobTask(luigi.Task):
    method _get_pool (line 672) | def _get_pool(self):
    method job_runner (line 680) | def job_runner(self):
    method jobconfs (line 683) | def jobconfs(self):
    method init_local (line 698) | def init_local(self):
    method init_hadoop (line 708) | def init_hadoop(self):
    method run (line 714) | def run(self):
    method requires_local (line 726) | def requires_local(self):
    method requires_hadoop (line 732) | def requires_hadoop(self):
    method input_local (line 735) | def input_local(self):
    method input_hadoop (line 738) | def input_hadoop(self):
    method deps (line 741) | def deps(self):
    method on_failure (line 745) | def on_failure(self, exception):
  class JobTask (line 766) | class JobTask(BaseHadoopJobTask):
    method jobconfs (line 771) | def jobconfs(self):
    method init_mapper (line 781) | def init_mapper(self):
    method init_combiner (line 784) | def init_combiner(self):
    method init_reducer (line 787) | def init_reducer(self):
    method _setup_remote (line 790) | def _setup_remote(self):
    method job_runner (line 793) | def job_runner(self):
    method reader (line 810) | def reader(self, input_stream):
    method writer (line 818) | def writer(self, outputs, stdout, stderr=sys.stderr):
    method mapper (line 839) | def mapper(self, item):
    method incr_counter (line 849) | def incr_counter(self, *args, **kwargs):
    method _flush_batch_incr_counter (line 872) | def _flush_batch_incr_counter(self):
    method _incr_counter (line 883) | def _incr_counter(self, *args):
    method extra_modules (line 899) | def extra_modules(self):
    method extra_files (line 902) | def extra_files(self):
    method extra_streaming_arguments (line 915) | def extra_streaming_arguments(self):
    method extra_archives (line 922) | def extra_archives(self):
    method add_link (line 926) | def add_link(self, src, dst):
    method _setup_links (line 931) | def _setup_links(self):
    method dump (line 951) | def dump(self, directory=""):
    method _map_input (line 966) | def _map_input(self, input_stream):
    method _reduce_input (line 983) | def _reduce_input(self, inputs, reducer, final=NotImplemented):
    method run_mapper (line 995) | def run_mapper(self, stdin=sys.stdin, stdout=sys.stdout):
    method run_reducer (line 1007) | def run_reducer(self, stdin=sys.stdin, stdout=sys.stdout):
    method run_combiner (line 1016) | def run_combiner(self, stdin=sys.stdin, stdout=sys.stdout):
    method internal_reader (line 1022) | def internal_reader(self, input_stream):
    method internal_writer (line 1030) | def internal_writer(self, outputs, stdout):

FILE: luigi/contrib/hadoop_jar.py
  function fix_paths (line 33) | def fix_paths(job):
  class HadoopJarJobError (line 65) | class HadoopJarJobError(Exception):
  class HadoopJarJobRunner (line 69) | class HadoopJarJobRunner(luigi.contrib.hadoop.JobRunner):
    method __init__ (line 74) | def __init__(self):
    method run_job (line 77) | def run_job(self, job, tracking_url_callback=None):
  class HadoopJarJobTask (line 124) | class HadoopJarJobTask(luigi.contrib.hadoop.BaseHadoopJobTask):
    method jar (line 129) | def jar(self):
    method main (line 135) | def main(self):
    method job_runner (line 141) | def job_runner(self):
    method atomic_output (line 145) | def atomic_output(self):
    method ssh (line 152) | def ssh(self):
    method args (line 159) | def args(self):

FILE: luigi/contrib/hdfs/abstract_client.py
  class HdfsFileSystem (line 27) | class HdfsFileSystem(luigi.target.FileSystem, metaclass=abc.ABCMeta):
    method rename (line 32) | def rename(self, path, dest):
    method rename_dont_move (line 43) | def rename_dont_move(self, path, dest):
    method remove (line 57) | def remove(self, path, recursive=True, skip_trash=False):
    method chmod (line 61) | def chmod(self, path, permissions, recursive=False):
    method chown (line 65) | def chown(self, path, owner, group, recursive=False):
    method count (line 69) | def count(self, path):
    method copy (line 76) | def copy(self, path, destination):
    method put (line 80) | def put(self, local_path, destination):
    method get (line 84) | def get(self, path, local_destination):
    method mkdir (line 88) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method listdir (line 92) | def listdir(self, path, ignore_directories=False, ignore_files=False, ...
    method touchz (line 96) | def touchz(self, path):

FILE: luigi/contrib/hdfs/clients.py
  function get_autoconfig_client (line 34) | def get_autoconfig_client(client_cache=_AUTOCONFIG_CLIENT):
  function _with_ac (line 51) | def _with_ac(method_name):

FILE: luigi/contrib/hdfs/config.py
  class hdfs (line 32) | class hdfs(luigi.Config):
  class hadoopcli (line 43) | class hadoopcli(luigi.Config):
  function load_hadoop_cmd (line 52) | def load_hadoop_cmd():
  function get_configured_hadoop_version (line 56) | def get_configured_hadoop_version():
  function get_configured_hdfs_client (line 68) | def get_configured_hdfs_client():
  function tmppath (line 77) | def tmppath(path=None, include_unix_username=True):

FILE: luigi/contrib/hdfs/error.py
  class HDFSCliError (line 23) | class HDFSCliError(Exception):
    method __init__ (line 24) | def __init__(self, command, returncode, stdout, stderr):

FILE: luigi/contrib/hdfs/format.py
  class HdfsAtomicWriteError (line 13) | class HdfsAtomicWriteError(IOError):
  class HdfsReadPipe (line 17) | class HdfsReadPipe(luigi.format.InputPipeProcessWrapper):
    method __init__ (line 18) | def __init__(self, path):
  class HdfsAtomicWritePipe (line 22) | class HdfsAtomicWritePipe(luigi.format.OutputPipeProcessWrapper):
    method __init__ (line 35) | def __init__(self, path):
    method abort (line 42) | def abort(self):
    method close (line 47) | def close(self):
  class HdfsAtomicWriteDirPipe (line 61) | class HdfsAtomicWriteDirPipe(luigi.format.OutputPipeProcessWrapper):
    method __init__ (line 66) | def __init__(self, path, data_extension=""):
    method abort (line 72) | def abort(self):
    method close (line 77) | def close(self):
  class PlainFormat (line 97) | class PlainFormat(luigi.format.Format):
    method hdfs_writer (line 101) | def hdfs_writer(self, path):
    method hdfs_reader (line 104) | def hdfs_reader(self, path):
    method pipe_reader (line 107) | def pipe_reader(self, path):
    method pipe_writer (line 110) | def pipe_writer(self, output_pipe):
  class PlainDirFormat (line 114) | class PlainDirFormat(luigi.format.Format):
    method hdfs_writer (line 118) | def hdfs_writer(self, path):
    method hdfs_reader (line 121) | def hdfs_reader(self, path):
    method pipe_reader (line 124) | def pipe_reader(self, path):
    method pipe_writer (line 128) | def pipe_writer(self, path):
  class CompatibleHdfsFormat (line 136) | class CompatibleHdfsFormat(luigi.format.Format):
    method __init__ (line 139) | def __init__(self, writer, reader, input=None):
    method pipe_writer (line 146) | def pipe_writer(self, output):
    method pipe_reader (line 149) | def pipe_reader(self, input):
    method hdfs_writer (line 152) | def hdfs_writer(self, output):
    method hdfs_reader (line 155) | def hdfs_reader(self, input):
    method __getstate__ (line 162) | def __getstate__(self):
    method __setstate__ (line 173) | def __setstate__(self, d):

FILE: luigi/contrib/hdfs/hadoopcli_clients.py
  function create_hadoopcli_client (line 38) | def create_hadoopcli_client():
  class HdfsClient (line 54) | class HdfsClient(hdfs_abstract_client.HdfsFileSystem):
    method call_check (line 62) | def call_check(command):
    method exists (line 69) | def exists(self, path):
    method move (line 88) | def move(self, path, dest):
    method remove (line 98) | def remove(self, path, recursive=True, skip_trash=False):
    method chmod (line 110) | def chmod(self, path, permissions, recursive=False):
    method chown (line 117) | def chown(self, path, owner, group, recursive=False):
    method count (line 129) | def count(self, path):
    method copy (line 141) | def copy(self, path, destination):
    method put (line 144) | def put(self, local_path, destination):
    method get (line 147) | def get(self, path, local_destination):
    method getmerge (line 150) | def getmerge(self, path, local_destination, new_line=False):
    method mkdir (line 157) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method listdir (line 170) | def listdir(self, path, ignore_directories=False, ignore_files=False, ...
    method touchz (line 210) | def touchz(self, path):
  class HdfsClientCdh3 (line 214) | class HdfsClientCdh3(HdfsClient):
    method mkdir (line 219) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method remove (line 232) | def remove(self, path, recursive=True, skip_trash=False):
  class HdfsClientApache1 (line 245) | class HdfsClientApache1(HdfsClientCdh3):
    method exists (line 253) | def exists(self, path):

FILE: luigi/contrib/hdfs/target.py
  class HdfsTarget (line 32) | class HdfsTarget(FileSystemTarget):
    method __init__ (line 33) | def __init__(self, path=None, format=None, is_tmp=False, fs=None):
    method __del__ (line 86) | def __del__(self):
    method fs (line 92) | def fs(self):
    method glob_exists (line 95) | def glob_exists(self, expected_files):
    method open (line 101) | def open(self, mode="r"):
    method remove (line 110) | def remove(self, skip_trash=False):
    method rename (line 113) | def rename(self, path, raise_if_exists=False):
    method move (line 126) | def move(self, path, raise_if_exists=False):
    method move_dir (line 132) | def move_dir(self, path):
    method copy (line 144) | def copy(self, dst_dir):
    method is_writable (line 150) | def is_writable(self):
    method _is_writable (line 172) | def _is_writable(self, path):
  class HdfsFlagTarget (line 182) | class HdfsFlagTarget(HdfsTarget):
    method __init__ (line 196) | def __init__(self, path, format=None, client=None, flag="_SUCCESS"):
    method exists (line 212) | def exists(self):

FILE: luigi/contrib/hdfs/webhdfs_client.py
  class webhdfs (line 38) | class webhdfs(luigi.Config):
  class WebHdfsClient (line 44) | class WebHdfsClient(hdfs_abstract_client.HdfsFileSystem):
    method __init__ (line 52) | def __init__(self, host=None, port=None, user=None, client_type=None):
    method url (line 59) | def url(self):
    method client (line 67) | def client(self):
    method walk (line 82) | def walk(self, path, depth=1):
    method exists (line 85) | def exists(self, path):
    method upload (line 100) | def upload(self, hdfs_path, local_path, overwrite=False):
    method download (line 103) | def download(self, hdfs_path, local_path, overwrite=False, n_threads=-1):
    method remove (line 106) | def remove(self, hdfs_path, recursive=True, skip_trash=False):
    method read (line 110) | def read(self, hdfs_path, offset=0, length=None, buffer_size=None, chu...
    method move (line 113) | def move(self, path, dest):
    method mkdir (line 121) | def mkdir(self, path, parents=True, mode=0o755, raise_if_exists=False):
    method chmod (line 130) | def chmod(self, path, permissions, recursive=False):
    method chown (line 136) | def chown(self, path, owner, group, recursive=False):
    method count (line 142) | def count(self, path):
    method copy (line 148) | def copy(self, path, destination):
    method put (line 154) | def put(self, local_path, destination):
    method get (line 160) | def get(self, path, local_destination):
    method listdir (line 166) | def listdir(self, path, ignore_directories=False, ignore_files=False, ...
    method touchz (line 170) | def touchz(self, path):

FILE: luigi/contrib/hive.py
  class HiveCommandError (line 37) | class HiveCommandError(RuntimeError):
    method __init__ (line 38) | def __init__(self, message, out=None, err=None):
  function load_hive_cmd (line 45) | def load_hive_cmd():
  function get_hive_syntax (line 49) | def get_hive_syntax():
  function get_hive_warehouse_location (line 53) | def get_hive_warehouse_location():
  function get_ignored_file_masks (line 57) | def get_ignored_file_masks():
  function run_hive (line 61) | def run_hive(args, check_return_code=True):
  function run_hive_cmd (line 78) | def run_hive_cmd(hivecmd, check_return_code=True):
  function run_hive_script (line 85) | def run_hive_script(script):
  function _is_ordered_dict (line 94) | def _is_ordered_dict(dikt):
  function _validate_partition (line 98) | def _validate_partition(partition):
  class HiveClient (line 107) | class HiveClient(metaclass=abc.ABCMeta):
    method table_location (line 109) | def table_location(self, table, database="default", partition=None):
    method table_schema (line 117) | def table_schema(self, table, database="default"):
    method table_exists (line 124) | def table_exists(self, table, database="default", partition=None):
    method partition_spec (line 132) | def partition_spec(self, partition):
  class HiveCommandClient (line 137) | class HiveCommandClient(HiveClient):
    method table_location (line 142) | def table_location(self, table, database="default", partition=None):
    method table_exists (line 153) | def table_exists(self, table, database="default", partition=None):
    method table_schema (line 170) | def table_schema(self, table, database="default"):
    method partition_spec (line 176) | def partition_spec(self, partition):
  class ApacheHiveCommandClient (line 183) | class ApacheHiveCommandClient(HiveCommandClient):
    method table_schema (line 189) | def table_schema(self, table, database="default"):
  class MetastoreClient (line 196) | class MetastoreClient(HiveClient):
    method table_location (line 197) | def table_location(self, table, database="default", partition=None):
    method table_exists (line 211) | def table_exists(self, table, database="default", partition=None):
    method _existing_partitions (line 218) | def _existing_partitions(self, table, database, client):
    method table_schema (line 230) | def table_schema(self, table, database="default"):
    method partition_spec (line 234) | def partition_spec(self, partition):
  class HiveThriftContext (line 238) | class HiveThriftContext:
    method __enter__ (line 243) | def __enter__(self):
    method __exit__ (line 264) | def __exit__(self, exc_type, exc_val, exc_tb):
  class WarehouseHiveClient (line 268) | class WarehouseHiveClient(HiveClient):
    method __init__ (line 273) | def __init__(self, hdfs_client=None, warehouse_location=None):
    method table_schema (line 277) | def table_schema(self, table, database="default"):
    method table_location (line 280) | def table_location(self, table, database="default", partition=None):
    method table_exists (line 283) | def table_exists(self, table, database="default", partition=None):
    method partition_spec (line 302) | def partition_spec(self, partition):
  function get_default_client (line 307) | def get_default_client():
  class HiveQueryTask (line 322) | class HiveQueryTask(luigi.contrib.hadoop.BaseHadoopJobTask):
    method query (line 333) | def query(self):
    method hiverc (line 337) | def hiverc(self):
    method hivevars (line 347) | def hivevars(self):
    method hiveconfs (line 356) | def hiveconfs(self):
    method job_runner (line 384) | def job_runner(self):
  class HiveQueryRunner (line 388) | class HiveQueryRunner(luigi.contrib.hadoop.JobRunner):
    method prepare_outputs (line 393) | def prepare_outputs(self, job):
    method get_arglist (line 412) | def get_arglist(self, f_name, job):
    method run_job (line 431) | def run_job(self, job, tracking_url_callback=None):
  class HivePartitionTarget (line 446) | class HivePartitionTarget(luigi.Target):
    method __init__ (line 451) | def __init__(self, table, partition, database="default", fail_missing_...
    method __str__ (line 468) | def __str__(self):
    method exists (line 471) | def exists(self):
    method path (line 491) | def path(self):
  class HiveTableTarget (line 501) | class HiveTableTarget(HivePartitionTarget):
    method __init__ (line 506) | def __init__(self, table, database="default", client=None):
  class ExternalHiveTask (line 516) | class ExternalHiveTask(luigi.ExternalTask):
    method output (line 527) | def output(self):

FILE: luigi/contrib/kubernetes.py
  class kubernetes (line 53) | class kubernetes(luigi.Config):
  class KubernetesJobTask (line 60) | class KubernetesJobTask(luigi.Task):
    method _init_kubernetes (line 65) | def _init_kubernetes(self):
    method auth_method (line 79) | def auth_method(self):
    method kubeconfig_path (line 92) | def kubeconfig_path(self):
    method kubernetes_namespace (line 108) | def kubernetes_namespace(self):
    method name (line 119) | def name(self):
    method labels (line 127) | def labels(self):
    method spec_schema (line 136) | def spec_schema(self):
    method max_retrials (line 163) | def max_retrials(self):
    method backoff_limit (line 170) | def backoff_limit(self):
    method delete_on_success (line 178) | def delete_on_success(self):
    method print_pod_logs_on_exit (line 185) | def print_pod_logs_on_exit(self):
    method active_deadline_seconds (line 192) | def active_deadline_seconds(self):
    method kubernetes_config (line 200) | def kubernetes_config(self):
    method poll_interval (line 206) | def poll_interval(self):
    method pod_creation_wait_interal (line 211) | def pod_creation_wait_interal(self):
    method __track_job (line 215) | def __track_job(self):
    method signal_complete (line 234) | def signal_complete(self):
    method __get_pods (line 245) | def __get_pods(self):
    method __get_job (line 249) | def __get_job(self):
    method __print_pod_logs (line 254) | def __print_pod_logs(self):
    method __print_kubectl_hints (line 262) | def __print_kubectl_hints(self):
    method __verify_job_has_started (line 267) | def __verify_job_has_started(self):
    method __get_job_status (line 299) | def __get_job_status(self):
    method __delete_job_cascade (line 322) | def __delete_job_cascade(self, job):
    method run (line 328) | def run(self):
    method output (line 356) | def output(self):

FILE: luigi/contrib/lsf.py
  function track_job (line 75) | def track_job(job_id):
  function kill_job (line 91) | def kill_job(job_id):
  class LSFJobTask (line 98) | class LSFJobTask(luigi.Task):
    method fetch_task_failures (line 117) | def fetch_task_failures(self):
    method fetch_task_output (line 129) | def fetch_task_output(self):
    method _init_local (line 141) | def _init_local(self):
    method init_local (line 171) | def init_local(self):
    method run (line 179) | def run(self):
    method work (line 192) | def work(self):
    method _dump (line 203) | def _dump(self, out_dir=""):
    method _run_job (line 217) | def _run_job(self):
    method _track_job (line 276) | def _track_job(self):
    method _finish (line 325) | def _finish(self):
    method __del__ (line 331) | def __del__(self):
  class LocalLSFJobTask (line 336) | class LocalLSFJobTask(LSFJobTask):
    method run (line 341) | def run(self):

FILE: luigi/contrib/lsf_runner.py
  function do_work_on_compute_node (line 36) | def do_work_on_compute_node(work_dir):
  function extract_packages_archive (line 49) | def extract_packages_archive(work_dir):
  function main (line 65) | def main(args=sys.argv):

FILE: luigi/contrib/mongodb.py
  class MongoTarget (line 21) | class MongoTarget(Target):
    method __init__ (line 24) | def __init__(self, mongo_client, index, collection):
    method __str__ (line 37) | def __str__(self):
    method get_collection (line 40) | def get_collection(self):
    method get_index (line 47) | def get_index(self):
  class MongoCellTarget (line 54) | class MongoCellTarget(MongoTarget):
    method __init__ (line 57) | def __init__(self, mongo_client, index, collection, document_id, path):
    method exists (line 69) | def exists(self):
    method read (line 76) | def read(self):
    method write (line 89) | def write(self, value):
  class MongoRangeTarget (line 96) | class MongoRangeTarget(MongoTarget):
    method __init__ (line 99) | def __init__(self, mongo_client, index, collection, document_ids, field):
    method exists (line 111) | def exists(self):
    method read (line 118) | def read(self):
    method write (line 126) | def write(self, values):
    method get_empty_ids (line 143) | def get_empty_ids(self):
  class MongoCollectionTarget (line 152) | class MongoCollectionTarget(MongoTarget):
    method __init__ (line 155) | def __init__(self, mongo_client, index, collection):
    method exists (line 158) | def exists(self):
    method read (line 165) | def read(self):
  class MongoCountTarget (line 172) | class MongoCountTarget(MongoTarget):
    method __init__ (line 175) | def __init__(self, mongo_client, index, collection, target_count):
    method exists (line 184) | def exists(self):
    method read (line 191) | def read(self):

FILE: luigi/contrib/mssqldb.py
  class MSSqlTarget (line 33) | class MSSqlTarget(luigi.Target):
    method __init__ (line 44) | def __init__(self, host, database, user, password, table, update_id):
    method __str__ (line 71) | def __str__(self):
    method touch (line 74) | def touch(self, connection=None):
    method exists (line 106) | def exists(self, connection=None):
    method connect (line 125) | def connect(self):
    method create_marker_table (line 132) | def create_marker_table(self):

FILE: luigi/contrib/mysqldb.py
  class MySqlTarget (line 35) | class MySqlTarget(luigi.Target):
    method __init__ (line 42) | def __init__(self, host, database, user, password, table, update_id, *...
    method __str__ (line 72) | def __str__(self):
    method touch (line 75) | def touch(self, connection=None):
    method exists (line 100) | def exists(self, connection=None):
    method connect (line 120) | def connect(self, autocommit=False):
    method create_marker_table (line 126) | def create_marker_table(self):
  class CopyToTable (line 154) | class CopyToTable(rdbms.CopyToTable):
    method rows (line 166) | def rows(self):
    method output (line 176) | def output(self):
    method copy (line 184) | def copy(self, cursor, file=None):
    method run (line 199) | def run(self):
    method bulk_size (line 241) | def bulk_size(self):

FILE: luigi/contrib/opener.py
  class OpenerError (line 45) | class OpenerError(FileSystemException):
  class NoOpenerError (line 51) | class NoOpenerError(OpenerError):
  class InvalidQuery (line 57) | class InvalidQuery(OpenerError):
  class OpenerRegistry (line 63) | class OpenerRegistry:
    method __init__ (line 64) | def __init__(self, openers=None):
    method get_opener (line 81) | def get_opener(self, name):
    method add (line 94) | def add(self, opener):
    method open (line 107) | def open(self, target_uri, **kwargs):
  class Opener (line 127) | class Opener:
    method conform_query (line 136) | def conform_query(cls, query):
    method get_target (line 165) | def get_target(cls, scheme, path, fragment, username, password, hostna...
  class MockOpener (line 173) | class MockOpener(Opener):
    method get_target (line 190) | def get_target(cls, scheme, path, fragment, username, password, hostna...
  class LocalOpener (line 196) | class LocalOpener(Opener):
    method get_target (line 214) | def get_target(cls, scheme, path, fragment, username, password, hostna...
  class S3Opener (line 220) | class S3Opener(Opener):
    method get_target (line 237) | def get_target(cls, scheme, path, fragment, username, password, hostna...

FILE: luigi/contrib/pai.py
  function slot_to_dict (line 51) | def slot_to_dict(o):
  class PaiJob (line 61) | class PaiJob:
    method __init__ (line 103) | def __init__(self, jobName, image, tasks):
  class Port (line 119) | class Port:
    method __init__ (line 122) | def __init__(self, label, begin_at=0, port_number=1):
  class TaskRole (line 135) | class TaskRole:
    method __init__ (line 138) | def __init__(self, name, command, taskNumber=1, cpuNumber=1, memoryMB=...
  class OpenPai (line 161) | class OpenPai(luigi.Config):
  class PaiTask (line 168) | class PaiTask(luigi.Task):
    method name (line 173) | def name(self):
    method image (line 179) | def image(self):
    method tasks (line 185) | def tasks(self):
    method auth_file_path (line 190) | def auth_file_path(self):
    method data_dir (line 195) | def data_dir(self):
    method code_dir (line 200) | def code_dir(self):
    method output_dir (line 205) | def output_dir(self):
    method virtual_cluster (line 210) | def virtual_cluster(self):
    method gpu_type (line 215) | def gpu_type(self):
    method retry_count (line 220) | def retry_count(self):
    method __init_token (line 224) | def __init_token(self):
    method __init__ (line 238) | def __init__(self, *args, **kwargs):
    method __check_job_status (line 246) | def __check_job_status(self):
    method run (line 269) | def run(self):
    method output (line 294) | def output(self):
    method complete (line 297) | def complete(self):

FILE: luigi/contrib/pig.py
  class PigJobTask (line 41) | class PigJobTask(luigi.Task):
    method pig_home (line 42) | def pig_home(self):
    method pig_command_path (line 45) | def pig_command_path(self):
    method pig_env_vars (line 48) | def pig_env_vars(self):
    method pig_properties (line 57) | def pig_properties(self):
    method pig_parameters (line 67) | def pig_parameters(self):
    method pig_options (line 77) | def pig_options(self):
    method output (line 87) | def output(self):
    method pig_script_path (line 90) | def pig_script_path(self):
    method _build_pig_cmd (line 97) | def _build_pig_cmd(self):
    method run (line 123) | def run(self):
    method track_and_progress (line 127) | def track_and_progress(self, cmd):
  class PigRunContext (line 168) | class PigRunContext:
    method __init__ (line 169) | def __init__(self):
    method __enter__ (line 172) | def __enter__(self):
    method kill_job (line 177) | def kill_job(self, captured_signal=None, stack_frame=None):
    method __exit__ (line 185) | def __exit__(self, exc_type, exc_val, exc_tb):
  class PigJobError (line 191) | class PigJobError(RuntimeError):
    method __init__ (line 192) | def __init__(self, message, out=None, err=None):
    method __str__ (line 198) | def __str__(self):

FILE: luigi/contrib/postgres.py
  function update_error_codes (line 45) | def update_error_codes():
  function _is_pg8000_error (line 75) | def _is_pg8000_error(exception):
  function _pg8000_connection_reset (line 87) | def _pg8000_connection_reset(connection):
  function db_error_code (line 97) | def db_error_code(exception):
  class MultiReplacer (line 111) | class MultiReplacer:
    method __init__ (line 138) | def __init__(self, replace_pairs):
    method _replacer (line 150) | def _replacer(self, match_object):
    method __call__ (line 154) | def __call__(self, search_string):
  class PostgresTarget (line 164) | class PostgresTarget(luigi.Target):
    method __init__ (line 179) | def __init__(self, host, database, user, password, table, update_id, p...
    method __str__ (line 201) | def __str__(self):
    method touch (line 204) | def touch(self, connection=None):
    method exists (line 234) | def exists(self, connection=None):
    method connect (line 254) | def connect(self):
    method create_marker_table (line 262) | def create_marker_table(self):
    method open (line 293) | def open(self, mode):
  class CopyToTable (line 297) | class CopyToTable(rdbms.CopyToTable):
    method rows (line 309) | def rows(self):
    method map_column (line 317) | def map_column(self, value):
    method output (line 330) | def output(self):
    method copy (line 340) | def copy(self, cursor, file):
    method run (line 357) | def run(self):
  class PostgresQuery (line 420) | class PostgresQuery(rdbms.Query):
    method run (line 435) | def run(self):
    method output (line 451) | def output(self):

FILE: luigi/contrib/presto.py
  class presto (line 22) | class presto(luigi.Config):  # NOQA
  class PrestoClient (line 32) | class PrestoClient:
    method __init__ (line 38) | def __init__(self, connection, sleep_time=1):
    method percentage_progress (line 44) | def percentage_progress(self):
    method info_uri (line 51) | def info_uri(self):
    method execute (line 57) | def execute(self, query, parameters=None, mode=None):
  class WithPrestoClient (line 88) | class WithPrestoClient(Register):
    method __new__ (line 95) | def __new__(cls, name, bases, attrs):
  class PrestoTarget (line 118) | class PrestoTarget(luigi.Target):
    method __init__ (line 123) | def __init__(self, client, catalog, database, table, partition=None):
    method __str__ (line 131) | def __str__(self):
    method _count_query (line 135) | def _count_query(self):
    method _table_doesnot_exist (line 148) | def _table_doesnot_exist(self, exception):
    method count (line 157) | def count(self):
    method exists (line 167) | def exists(self):
  class PrestoTask (line 182) | class PrestoTask(rdbms.Query, metaclass=WithPrestoClient):
    method host (line 191) | def host(self):
    method port (line 195) | def port(self):
    method user (line 199) | def user(self):
    method username (line 203) | def username(self):
    method schema (line 207) | def schema(self):
    method password (line 211) | def password(self):
    method catalog (line 215) | def catalog(self):
    method poll_interval (line 219) | def poll_interval(self):
    method source (line 223) | def source(self):
    method partition (line 227) | def partition(self):
    method protocol (line 231) | def protocol(self):
    method session_props (line 235) | def session_props(self):
    method requests_session (line 239) | def requests_session(self):
    method requests_kwargs (line 243) | def requests_kwargs(self):
    method _maybe_set_tracking_url (line 248) | def _maybe_set_tracking_url(self):
    method _set_progress (line 253) | def _set_progress(self):
    method run (line 256) | def run(self):
    method output (line 261) | def output(self):

FILE: luigi/contrib/prometheus_metric.py
  class prometheus (line 8) | class prometheus(Config):
  class PrometheusMetricsCollector (line 13) | class PrometheusMetricsCollector(MetricsCollector):
    method _generate_task_labels (line 14) | def _generate_task_labels(self, task):
    method __init__ (line 17) | def __init__(self, *args, **kwargs):
    method generate_latest (line 32) | def generate_latest(self):
    method handle_task_started (line 35) | def handle_task_started(self, task):
    method handle_task_failed (line 39) | def handle_task_failed(self, task):
    method handle_task_disabled (line 43) | def handle_task_disabled(self, task, config):
    method handle_task_done (line 47) | def handle_task_done(self, task):
    method configure_http_handler (line 53) | def configure_http_handler(self, http_handler):

FILE: luigi/contrib/pyspark_runner.py
  class _SparkEntryPoint (line 41) | class _SparkEntryPoint(metaclass=abc.ABCMeta):
    method __init__ (line 42) | def __init__(self, conf):
    method __enter__ (line 46) | def __enter__(self):
    method __exit__ (line 50) | def __exit__(self, exc_type, exc_val, exc_tb):
  class SparkContextEntryPoint (line 54) | class SparkContextEntryPoint(_SparkEntryPoint):
    method __enter__ (line 57) | def __enter__(self):
    method __exit__ (line 63) | def __exit__(self, exc_type, exc_val, exc_tb):
  class SparkSessionEntryPoint (line 67) | class SparkSessionEntryPoint(_SparkEntryPoint):
    method _check_major_spark_version (line 70) | def _check_major_spark_version(self):
    method __enter__ (line 82) | def __enter__(self):
    method __exit__ (line 90) | def __exit__(self, exc_type, exc_val, exc_tb):
  class AbstractPySparkRunner (line 94) | class AbstractPySparkRunner(object):
    method __init__ (line 97) | def __init__(self, job, *args):
    method run (line 105) | def run(self):
  function _pyspark_runner_with (line 115) | def _pyspark_runner_with(name, entry_point_class):
  function _use_spark_session (line 123) | def _use_spark_session():
  function _get_runner_class (line 127) | def _get_runner_class():

FILE: luigi/contrib/rdbms.py
  class _MetadataColumnsMixin (line 33) | class _MetadataColumnsMixin:
    method metadata_columns (line 91) | def metadata_columns(self):
    method metadata_queries (line 99) | def metadata_queries(self):
    method enable_metadata_columns (line 103) | def enable_metadata_columns(self):
    method _add_metadata_columns (line 106) | def _add_metadata_columns(self, connection):
    method _column_exists (line 120) | def _column_exists(self, cursor, column_name):
    method _add_column_to_table (line 137) | def _add_column_to_table(self, cursor, column):
    method post_copy_metacolumns (line 149) | def post_copy_metacolumns(self, cursor):
  class CopyToTable (line 155) | class CopyToTable(luigi.task.MixinNaiveBulkComplete, _MetadataColumnsMix...
    method host (line 174) | def host(self):
    method database (line 179) | def database(self):
    method user (line 184) | def user(self):
    method password (line 189) | def password(self):
    method table (line 194) | def table(self):
    method port (line 198) | def port(self):
    method create_table (line 213) | def create_table(self, connection):
    method update_id (line 232) | def update_id(self):
    method output (line 239) | def output(self):
    method init_copy (line 242) | def init_copy(self, connection):
    method post_copy (line 259) | def post_copy(self, connection):
    method copy (line 269) | def copy(self, cursor, file):
  class Query (line 273) | class Query(luigi.task.MixinNaiveBulkComplete, luigi.Task):
    method host (line 302) | def host(self):
    method port (line 310) | def port(self):
    method database (line 318) | def database(self):
    method user (line 323) | def user(self):
    method password (line 328) | def password(self):
    method table (line 333) | def table(self):
    method query (line 338) | def query(self):
    method autocommit (line 342) | def autocommit(self):
    method update_id (line 346) | def update_id(self):
    method run (line 353) | def run(self):
    method output (line 357) | def output(self):

FILE: luigi/contrib/redis_store.py
  class RedisTarget (line 33) | class RedisTarget(Target):
    method __init__ (line 38) | def __init__(self, host, port, db, update_id, password=None, socket_ti...
    method __str__ (line 72) | def __str__(self):
    method marker_key (line 75) | def marker_key(self):
    method touch (line 81) | def touch(self):
    method exists (line 94) | def exists(self):

FILE: luigi/contrib/redshift.py
  class _CredentialsMixin (line 38) | class _CredentialsMixin:
    method configuration_section (line 46) | def configuration_section(self):
    method aws_access_key_id (line 54) | def aws_access_key_id(self):
    method aws_secret_access_key (line 61) | def aws_secret_access_key(self):
    method aws_account_id (line 68) | def aws_account_id(self):
    method aws_arn_role_name (line 75) | def aws_arn_role_name(self):
    method aws_session_token (line 82) | def aws_session_token(self):
    method _get_configuration_attribute (line 88) | def _get_configuration_attribute(self, attribute):
    method _credentials (line 98) | def _credentials(self):
  class RedshiftTarget (line 121) | class RedshiftTarget(postgres.PostgresTarget):
  class S3CopyToTable (line 137) | class S3CopyToTable(rdbms.CopyToTable, _CredentialsMixin):
    method s3_load_path (line 159) | def s3_load_path(self):
    method copy_options (line 167) | def copy_options(self):
    method prune_table (line 180) | def prune_table(self):
    method prune_column (line 189) | def prune_column(self):
    method prune_date (line 198) | def prune_date(self):
    method table_attributes (line 207) | def table_attributes(self):
    method table_constraints (line 218) | def table_constraints(self):
    method do_truncate_table (line 228) | def do_truncate_table(self):
    method do_prune (line 234) | def do_prune(self):
    method table_type (line 248) | def table_type(self):
    method queries (line 255) | def queries(self):
    method truncate_table (line 261) | def truncate_table(self, connection):
    method prune (line 269) | def prune(self, connection):
    method create_schema (line 277) | def create_schema(self, connection):
    method create_table (line 287) | def create_table(self, connection):
    method run (line 331) | def run(self):
    method copy (line 358) | def copy(self, cursor, f):
    method output (line 378) | def output(self):
    method does_schema_exist (line 386) | def does_schema_exist(self, connection):
    method does_table_exist (line 405) | def does_table_exist(self, connection):
    method init_copy (line 422) | def init_copy(self, connection):
    method post_copy (line 445) | def post_copy(self, cursor):
    method post_copy_metacolums (line 453) | def post_copy_metacolums(self, cursor):
  class S3CopyJSONToTable (line 462) | class S3CopyJSONToTable(S3CopyToTable, _CredentialsMixin):
    method jsonpath (line 487) | def jsonpath(self):
    method copy_json_options (line 495) | def copy_json_options(self):
    method copy (line 504) | def copy(self, cursor, f):
  class RedshiftManifestTask (line 521) | class RedshiftManifestTask(S3PathTask):
    method run (line 548) | def run(self):
  class KillOpenRedshiftSessions (line 564) | class KillOpenRedshiftSessions(luigi.Task):
    method host (line 584) | def host(self):
    method database (line 589) | def database(self):
    method user (line 594) | def user(self):
    method password (line 599) | def password(self):
    method update_id (line 603) | def update_id(self):
    method output (line 610) | def output(self):
    method run (line 621) | def run(self):
  class RedshiftQuery (line 657) | class RedshiftQuery(postgres.PostgresQuery):
    method output (line 671) | def output(self):
  class RedshiftUnloadTask (line 680) | class RedshiftUnloadTask(postgres.PostgresQuery, _CredentialsMixin):
    method s3_unload_path (line 694) | def s3_unload_path(self):
    method unload_options (line 701) | def unload_options(self):
    method unload_query (line 708) | def unload_query(self):
    method run (line 714) | def run(self):
    method output (line 734) | def output(self):

FILE: luigi/contrib/s3.py
  class InvalidDeleteException (line 53) | class InvalidDeleteException(FileSystemException):
  class FileNotFoundException (line 57) | class FileNotFoundException(FileSystemException):
  class DeprecatedBotoClientException (line 61) | class DeprecatedBotoClientException(Exception):
  class S3Client (line 65) | class S3Client(FileSystem):
    method __init__ (line 74) | def __init__(self, aws_access_key_id=None, aws_secret_access_key=None,...
    method s3 (line 87) | def s3(self):
    method s3 (line 139) | def s3(self, value):
    method exists (line 142) | def exists(self, path):
    method remove (line 162) | def remove(self, path, recursive=True):
    method move (line 202) | def move(self, source_path, destination_path, **kwargs):
    method get_key (line 212) | def get_key(self, path):
    method put (line 221) | def put(self, local_path, destination_s3_path, **kwargs):
    method put_string (line 233) | def put_string(self, content, destination_s3_path, **kwargs):
    method put_multipart (line 246) | def put_multipart(self, local_path, destination_s3_path, part_size=DEF...
    method copy (line 267) | def copy(self, source_path, destination_path, threads=DEFAULT_THREADS,...
    method _copy_file (line 291) | def _copy_file(self, source_path, destination_path, threads=DEFAULT_TH...
    method _copy_dir (line 302) | def _copy_dir(self, source_path, destination_path, threads=DEFAULT_THR...
    method get (line 335) | def get(self, s3_path, destination_local_path):
    method get_as_bytes (line 343) | def get_as_bytes(self, s3_path):
    method get_as_string (line 355) | def get_as_string(self, s3_path, encoding="utf-8"):
    method isdir (line 366) | def isdir(self, path):
    method mkdir (line 397) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method listdir (line 413) | def listdir(self, path, start_time=None, end_time=None, return_key=Fal...
    method list (line 448) | def list(self, path, start_time=None, end_time=None, return_key=False)...
    method _get_s3_config (line 466) | def _get_s3_config(key=None):
    method _path_to_bucket_and_key (line 485) | def _path_to_bucket_and_key(path):
    method _is_root (line 492) | def _is_root(key):
    method _add_path_delimiter (line 496) | def _add_path_delimiter(key):
    method _check_deprecated_argument (line 500) | def _check_deprecated_argument(**kwargs):
    method _exists (line 515) | def _exists(self, bucket, key):
  class AtomicS3File (line 527) | class AtomicS3File(AtomicLocalFile):
    method __init__ (line 534) | def __init__(self, path, s3_client, **kwargs):
    method move_to_final_destination (line 539) | def move_to_final_destination(self):
  class ReadableS3File (line 543) | class ReadableS3File:
    method __init__ (line 544) | def __init__(self, s3_key):
    method read (line 550) | def read(self, size=None):
    method close (line 554) | def close(self):
    method __del__ (line 558) | def __del__(self):
    method __exit__ (line 561) | def __exit__(self, exc_type, exc, traceback):
    method __enter__ (line 564) | def __enter__(self):
    method _add_to_buffer (line 567) | def _add_to_buffer(self, line):
    method _flush_buffer (line 570) | def _flush_buffer(self):
    method readable (line 575) | def readable(self):
    method writable (line 578) | def writable(self):
    method seekable (line 581) | def seekable(self):
    method __iter__ (line 584) | def __iter__(self):
  class S3Target (line 614) | class S3Target(FileSystemTarget):
    method __init__ (line 623) | def __init__(self, path, format=None, client=None, **kwargs):
    method open (line 633) | def open(self, mode="r"):
  class S3FlagTarget (line 648) | class S3FlagTarget(S3Target):
    method __init__ (line 670) | def __init__(self, path, format=None, client=None, flag="_SUCCESS"):
    method exists (line 689) | def exists(self):
  class S3EmrTarget (line 694) | class S3EmrTarget(S3FlagTarget):
    method __init__ (line 699) | def __init__(self, *args, **kwargs):
  class S3PathTask (line 704) | class S3PathTask(ExternalTask):
    method output (line 711) | def output(self):
  class S3EmrTask (line 715) | class S3EmrTask(ExternalTask):
    method output (line 722) | def output(self):
  class S3FlagTask (line 726) | class S3FlagTask(ExternalTask):
    method output (line 734) | def output(self):

FILE: luigi/contrib/salesforce.py
  function get_soql_fields (line 39) | def get_soql_fields(soql):
  function ensure_utf (line 51) | def ensure_utf(value):
  function parse_results (line 55) | def parse_results(fields, data):
  function _traverse_results (line 76) | def _traverse_results(value, fields, row, path):
  class salesforce (line 93) | class salesforce(luigi.Config):
  class QuerySalesforce (line 108) | class QuerySalesforce(Task):
    method object_name (line 111) | def object_name(self):
    method use_sandbox (line 119) | def use_sandbox(self):
    method sandbox_name (line 127) | def sandbox_name(self):
    method soql (line 133) | def soql(self):
    method is_soql_file (line 138) | def is_soql_file(self):
    method content_type (line 143) | def content_type(self):
    method run (line 149) | def run(self):
    method merge_batch_results (line 207) | def merge_batch_results(self, result_ids):
  class SalesforceAPI (line 227) | class SalesforceAPI:
    method __init__ (line 237) | def __init__(self, username, password, security_token, sb_token=None, ...
    method start_session (line 251) | def start_session(self):
    method has_active_session (line 276) | def has_active_session(self):
    method query (line 279) | def query(self, query, **kwargs):
    method query_more (line 292) | def query_more(self, next_records_identifier, identifier_is_url=False,...
    method query_all (line 318) | def query_all(self, query, **kwargs):
    method restful (line 364) | def restful(self, path, params):
    method create_operation_job (line 383) | def create_operation_job(self, operation, obj, external_id_field_name=...
    method get_job_details (line 405) | def get_job_details(self, job_id):
    method abort_job (line 418) | def abort_job(self, job_id):
    method close_job (line 431) | def close_job(self, job_id):
    method create_batch (line 446) | def create_batch(self, job_id, data, file_type):
    method block_on_batch (line 472) | def block_on_batch(self, job_id, batch_id, sleep_time_seconds=5, max_w...
    method get_batch_results (line 497) | def get_batch_results(self, job_id, batch_id):
    method get_batch_result_ids (line 504) | def get_batch_result_ids(self, job_id, batch_id):
    method get_batch_result (line 520) | def get_batch_result(self, job_id, batch_id, result_id):
    method _get_batch_info (line 533) | def _get_batch_info(self, job_id, batch_id):
    method _get_login_url (line 548) | def _get_login_url(self):
    method _get_base_url (line 552) | def _get_base_url(self):
    method _get_bulk_base_url (line 555) | def _get_bulk_base_url(self):
    method _get_norm_base_url (line 559) | def _get_norm_base_url(self):
    method _get_norm_query_url (line 563) | def _get_norm_query_url(self):
    method _get_create_job_url (line 567) | def _get_create_job_url(self):
    method _get_job_id_url (line 571) | def _get_job_id_url(self, job_id):
    method _get_job_details_url (line 575) | def _get_job_details_url(self, job_id):
    method _get_abort_job_url (line 579) | def _get_abort_job_url(self, job_id):
    method _get_close_job_url (line 583) | def _get_close_job_url(self, job_id):
    method _get_create_batch_url (line 587) | def _get_create_batch_url(self, job_id):
    method _get_batch_info_url (line 591) | def _get_batch_info_url(self, job_id, batch_id):
    method _get_batch_results_url (line 595) | def _get_batch_results_url(self, job_id, batch_id):
    method _get_batch_result_url (line 599) | def _get_batch_result_url(self, job_id, batch_id, result_id):
    method _get_login_headers (line 603) | def _get_login_headers(self):
    method _get_session_headers (line 607) | def _get_session_headers(self):
    method _get_norm_session_headers (line 611) | def _get_norm_session_headers(self):
    method _get_rest_headers (line 615) | def _get_rest_headers(self):
    method _get_job_headers (line 620) | def _get_job_headers(self):
    method _get_create_job_headers (line 625) | def _get_create_job_headers(self):
    method _get_abort_job_headers (line 628) | def _get_abort_job_headers(self):
    method _get_close_job_headers (line 631) | def _get_close_job_headers(self):
    method _get_create_batch_content_headers (line 634) | def _get_create_batch_content_headers(self, content_type):
    method _get_batch_info_headers (line 640) | def _get_batch_info_headers(self):
    method _get_login_xml (line 643) | def _get_login_xml(self):
    method _get_create_job_xml (line 657) | def _get_create_job_xml(self, operation, obj, external_id_field_name, ...
    method _get_abort_job_xml (line 671) | def _get_abort_job_xml(self):
    method _get_close_job_xml (line 678) | def _get_close_job_xml(self):

FILE: luigi/contrib/scalding.py
  class ScaldingJobRunner (line 61) | class ScaldingJobRunner(luigi.contrib.hadoop.JobRunner):
    method __init__ (line 66) | def __init__(self):
    method _get_jars (line 79) | def _get_jars(self, path):
    method get_scala_jars (line 82) | def get_scala_jars(self, include_compiler=False):
    method get_scalding_jars (line 96) | def get_scalding_jars(self):
    method get_scalding_core (line 100) | def get_scalding_core(self):
    method get_provided_jars (line 109) | def get_provided_jars(self):
    method get_libjars (line 112) | def get_libjars(self):
    method get_tmp_job_jar (line 115) | def get_tmp_job_jar(self, source):
    method get_build_dir (line 119) | def get_build_dir(self, source):
    method get_job_class (line 123) | def get_job_class(self, source):
    method build_job_jar (line 146) | def build_job_jar(self, job):
    method run_job (line 186) | def run_job(self, job, tracking_url_callback=None):
  class ScaldingJobTask (line 217) | class ScaldingJobTask(luigi.contrib.hadoop.BaseHadoopJobTask):
    method relpath (line 232) | def relpath(self, current_file, rel_path):
    method source (line 240) | def source(self):
    method jar (line 248) | def jar(self):
    method extra_jars (line 256) | def extra_jars(self):
    method job_class (line 262) | def job_class(self):
    method job_runner (line 268) | def job_runner(self):
    method atomic_output (line 271) | def atomic_output(self):
    method requires (line 278) | def requires(self):
    method job_args (line 281) | def job_args(self):
    method args (line 287) | def args(self):

FILE: luigi/contrib/sge.py
  function _parse_qstat_state (line 110) | def _parse_qstat_state(qstat_out, job_id):
  function _parse_qsub_job_id (line 131) | def _parse_qsub_job_id(qsub_out):
  function _build_qsub_command (line 142) | def _build_qsub_command(cmd, job_name, outfile, errfile, pe, n_cpu):
  class SGEJobTask (line 148) | class SGEJobTask(luigi.Task):
    method __init__ (line 190) | def __init__(self, *args, **kwargs):
    method _fetch_task_failures (line 202) | def _fetch_task_failures(self):
    method _init_local (line 214) | def _init_local(self):
    method run (line 238) | def run(self):
    method work (line 252) | def work(self):
    method _dump (line 256) | def _dump(self, out_dir=""):
    method _run_job (line 270) | def _run_job(self):
    method _track_job (line 298) | def _track_job(self):
  class LocalSGEJobTask (line 328) | class LocalSGEJobTask(SGEJobTask):
    method run (line 336) | def run(self):

FILE: luigi/contrib/sge_runner.py
  function _do_work_on_compute_node (line 43) | def _do_work_on_compute_node(work_dir, tarball=True):
  function _extract_packages_archive (line 60) | def _extract_packages_archive(work_dir):
  function main (line 76) | def main(args=sys.argv):

FILE: luigi/contrib/simulate.py
  class RunAnywayTarget (line 32) | class RunAnywayTarget(luigi.Target):
    method __init__ (line 63) | def __init__(self, task_obj):
    method __str__ (line 83) | def __str__(self):
    method get_path (line 86) | def get_path(self):
    method exists (line 95) | def exists(self):
    method done (line 101) | def done(self):

FILE: luigi/contrib/spark.py
  class SparkSubmitTask (line 36) | class SparkSubmitTask(ExternalProgramTask):
    method tracking_url_pattern (line 59) | def tracking_url_pattern(self):
    method app_options (line 66) | def app_options(self):
    method pyspark_python (line 74) | def pyspark_python(self):
    method pyspark_driver_python (line 78) | def pyspark_driver_python(self):
    method hadoop_user_name (line 82) | def hadoop_user_name(self):
    method spark_version (line 86) | def spark_version(self):
    method spark_submit (line 90) | def spark_submit(self):
    method master (line 94) | def master(self):
    method deploy_mode (line 98) | def deploy_mode(self):
    method jars (line 102) | def jars(self):
    method packages (line 106) | def packages(self):
    method py_files (line 110) | def py_files(self):
    method files (line 114) | def files(self):
    method _conf (line 118) | def _conf(self):
    method conf (line 127) | def conf(self):
    method properties_file (line 131) | def properties_file(self):
    method driver_memory (line 135) | def driver_memory(self):
    method driver_java_options (line 139) | def driver_java_options(self):
    method driver_library_path (line 143) | def driver_library_path(self):
    method driver_class_path (line 147) | def driver_class_path(self):
    method executor_memory (line 151) | def executor_memory(self):
    method driver_cores (line 155) | def driver_cores(self):
    method supervise (line 159) | def supervise(self):
    method total_executor_cores (line 163) | def total_executor_cores(self):
    method executor_cores (line 167) | def executor_cores(self):
    method queue (line 171) | def queue(self):
    method num_executors (line 175) | def num_executors(self):
    method archives (line 179) | def archives(self):
    method hadoop_conf_dir (line 183) | def hadoop_conf_dir(self):
    method get_environment (line 186) | def get_environment(self):
    method program_environment (line 194) | def program_environment(self):
    method program_args (line 197) | def program_args(self):
    method spark_command (line 200) | def spark_command(self):
    method app_command (line 226) | def app_command(self):
    method _list_config (line 231) | def _list_config(self, config):
    method _dict_config (line 235) | def _dict_config(self, config):
    method _text_arg (line 239) | def _text_arg(self, name, value):
    method _list_arg (line 244) | def _list_arg(self, name, value):
    method _dict_arg (line 249) | def _dict_arg(self, name, value):
    method _flag_arg (line 256) | def _flag_arg(self, name, value):
  class PySparkTask (line 262) | class PySparkTask(SparkSubmitTask):
    method name (line 277) | def name(self):
    method py_packages (line 281) | def py_packages(self):
    method files (line 287) | def files(self):
    method pickle_protocol (line 292) | def pickle_protocol(self):
    method setup (line 295) | def setup(self, conf):
    method setup_remote (line 302) | def setup_remote(self, sc):
    method main (line 305) | def main(self, sc, *args):
    method app_command (line 314) | def app_command(self):
    method run (line 321) | def run(self):
    method _dump (line 335) | def _dump(self, fd):
    method _setup_packages (line 345) | def _setup_packages(self, sc):

FILE: luigi/contrib/sparkey.py
  class SparkeyExportTask (line 21) | class SparkeyExportTask(luigi.Task):
    method __init__ (line 38) | def __init__(self, *args, **kwargs):
    method run (line 41) | def run(self):
    method _write_sparkey_file (line 44) | def _write_sparkey_file(self):

FILE: luigi/contrib/sqla.py
  class SQLAlchemyTarget (line 154) | class SQLAlchemyTarget(luigi.Target):
    method __init__ (line 168) | def __init__(self, connection_string, target_table, update_id, echo=Fa...
    method __str__ (line 194) | def __str__(self):
    method engine (line 198) | def engine(self):
    method touch (line 213) | def touch(self):
    method exists (line 234) | def exists(self):
    method create_marker_table (line 244) | def create_marker_table(self):
    method open (line 270) | def open(self, mode):
  class CopyToTable (line 274) | class CopyToTable(luigi.Task):
    method connection_string (line 292) | def connection_string(self):
    method table (line 297) | def table(self):
    method create_table (line 323) | def create_table(self, engine):
    method update_id (line 364) | def update_id(self):
    method output (line 370) | def output(self):
    method rows (line 375) | def rows(self):
    method run (line 385) | def run(self):
    method copy (line 400) | def copy(self, conn, ins_rows, table_bound):

FILE: luigi/contrib/ssh.py
  class RemoteCalledProcessError (line 54) | class RemoteCalledProcessError(subprocess.CalledProcessError):
    method __init__ (line 55) | def __init__(self, returncode, command, host, output=None):
    method __str__ (line 59) | def __str__(self):
  class RemoteContext (line 63) | class RemoteContext:
    method __init__ (line 64) | def __init__(self, host, **kwargs):
    method __repr__ (line 74) | def __repr__(self):
    method __eq__ (line 77) | def __eq__(self, other):
    method __hash__ (line 80) | def __hash__(self):
    method _host_ref (line 83) | def _host_ref(self):
    method _prepare_cmd (line 89) | def _prepare_cmd(self, cmd):
    method Popen (line 111) | def Popen(self, cmd, **kwargs):
    method check_output (line 118) | def check_output(self, cmd):
    method tunnel (line 131) | def tunnel(self, local_port, remote_port=None, remote_host="localhost"):
  class RemoteFileSystem (line 153) | class RemoteFileSystem(luigi.target.FileSystem):
    method __init__ (line 154) | def __init__(self, host, **kwargs):
    method exists (line 157) | def exists(self, path):
    method listdir (line 170) | def listdir(self, path):
    method isdir (line 178) | def isdir(self, path):
    method remove (line 191) | def remove(self, path, recursive=True):
    method mkdir (line 202) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method _scp (line 223) | def _scp(self, src, dest):
    method put (line 243) | def put(self, local_path, path):
    method get (line 254) | def get(self, path, local_path):
  class AtomicRemoteFileWriter (line 269) | class AtomicRemoteFileWriter(luigi.format.OutputPipeProcessWrapper):
    method __init__ (line 270) | def __init__(self, fs, path):
    method __del__ (line 283) | def __del__(self):
    method close (line 293) | def close(self):
    method tmp_path (line 298) | def tmp_path(self):
    method fs (line 302) | def fs(self):
  class RemoteTarget (line 306) | class RemoteTarget(luigi.target.FileSystemTarget):
    method __init__ (line 313) | def __init__(self, path, host, format=None, **kwargs):
    method fs (line 321) | def fs(self):
    method open (line 324) | def open(self, mode="r"):
    method put (line 340) | def put(self, local_path):
    method get (line 343) | def get(self, local_path):

FILE: luigi/contrib/target.py
  class CascadingClient (line 26) | class CascadingClient:
    method __init__ (line 55) | def __init__(self, clients, method_names=None):
    method _make_method (line 66) | def _make_method(cls, method_name):
    method _chained_call (line 72) | def _chained_call(self, method_name, *args, **kwargs):

FILE: luigi/contrib/webhdfs.py
  class WebHdfsTarget (line 35) | class WebHdfsTarget(FileSystemTarget):
    method __init__ (line 38) | def __init__(self, path, client=None, format=None):
    method open (line 47) | def open(self, mode="r"):
  class ReadableWebHdfsFile (line 57) | class ReadableWebHdfsFile:
    method __init__ (line 58) | def __init__(self, path, client):
    method read (line 63) | def read(self):
    method readlines (line 68) | def readlines(self, char="\n"):
    method __enter__ (line 72) | def __enter__(self):
    method __exit__ (line 75) | def __exit__(self, exc_type, exc, traceback):
    method __iter__ (line 78) | def __iter__(self):
    method close (line 83) | def close(self):
  class AtomicWebHdfsFile (line 87) | class AtomicWebHdfsFile(AtomicLocalFile):
    method __init__ (line 92) | def __init__(self, path, client):
    method move_to_final_destination (line 96) | def move_to_final_destination(self):

FILE: luigi/date_interval.py
  class DateInterval (line 38) | class DateInterval:
    method __init__ (line 56) | def __init__(self, date_a, date_b):
    method dates (line 60) | def dates(self):
    method hours (line 70) | def hours(self):
    method __str__ (line 76) | def __str__(self):
    method __repr__ (line 79) | def __repr__(self):
    method prev (line 82) | def prev(self):
    method next (line 86) | def next(self):
    method to_string (line 90) | def to_string(self):
    method from_date (line 94) | def from_date(cls, d):
    method parse (line 101) | def parse(cls, s):
    method __contains__ (line 107) | def __contains__(self, date):
    method __iter__ (line 110) | def __iter__(self):
    method __hash__ (line 114) | def __hash__(self):
    method __cmp__ (line 117) | def __cmp__(self, other):
    method __lt__ (line 124) | def __lt__(self, other):
    method __le__ (line 129) | def __le__(self, other):
    method __gt__ (line 134) | def __gt__(self, other):
    method __ge__ (line 139) | def __ge__(self, other):
    method __eq__ (line 144) | def __eq__(self, other):
    method __ne__ (line 152) | def __ne__(self, other):
  class Date (line 156) | class Date(DateInterval):
    method __init__ (line 159) | def __init__(self, y, m, d):
    method to_string (line 164) | def to_string(self):
    method from_date (line 168) | def from_date(cls, d):
    method parse (line 172) | def parse(cls, s):
  class Week (line 177) | class Week(DateInterval):
    method __init__ (line 183) | def __init__(self, y, w):
    method to_string (line 195) | def to_string(self):
    method from_date (line 199) | def from_date(cls, d):
    method parse (line 203) | def parse(cls, s):
  class Month (line 209) | class Month(DateInterval):
    method __init__ (line 210) | def __init__(self, y, m):
    method to_string (line 215) | def to_string(self):
    method from_date (line 219) | def from_date(cls, d):
    method parse (line 223) | def parse(cls, s):
  class Year (line 229) | class Year(DateInterval):
    method __init__ (line 230) | def __init__(self, y):
    method to_string (line 235) | def to_string(self):
    method from_date (line 239) | def from_date(cls, d):
    method parse (line 243) | def parse(cls, s):
  class Custom (line 248) | class Custom(DateInterval):
    method to_string (line 256) | def to_string(self):
    method parse (line 260) | def parse(cls, s):

FILE: luigi/db_task_history.py
  class DbTaskHistory (line 56) | class DbTaskHistory(task_history.TaskHistory):
    method _session (line 65) | def _session(self, session=None):
    method __init__ (line 78) | def __init__(self):
    method task_scheduled (line 88) | def task_scheduled(self, task):
    method task_finished (line 92) | def task_finished(self, task, successful):
    method task_started (line 97) | def task_started(self, task, worker_host):
    method _get_task (line 101) | def _get_task(self, task, status, host=None):
    method _add_task_event (line 111) | def _add_task_event(self, task, event):
    method _find_or_create_task (line 115) | def _find_or_create_task(self, task):
    method find_all_by_parameters (line 133) | def find_all_by_parameters(self, task_name, session=None, **task_params):
    method find_all_by_name (line 150) | def find_all_by_name(self, task_name, session=None):
    method find_latest_runs (line 156) | def find_latest_runs(self, session=None):
    method find_all_runs (line 171) | def find_all_runs(self, session=None):
    method find_all_events (line 178) | def find_all_events(self, session=None):
    method find_task_by_id (line 185) | def find_task_by_id(self, id, session=None):
    method find_task_by_task_id (line 192) | def find_task_by_task_id(self, task_id, session=None):
  class TaskParameter (line 200) | class TaskParameter(Base):  # type: ignore
    method __repr__ (line 210) | def __repr__(self):
  class TaskEvent (line 214) | class TaskEvent(Base):  # type: ignore
    method __repr__ (line 225) | def __repr__(self):
  class TaskRecord (line 229) | class TaskRecord(Base):  # type: ignore
    method __repr__ (line 246) | def __repr__(self):
  function _upgrade_schema (line 250) | def _upgrade_schema(engine):

FILE: luigi/event.py
  class Event (line 21) | class Event:

FILE: luigi/execution_summary.py
  class execution_summary (line 33) | class execution_summary(Config):
  class LuigiStatusCode (line 37) | class LuigiStatusCode(enum.Enum):
  class LuigiRunResult (line 66) | class LuigiRunResult:
    method __init__ (line 79) | def __init__(self, worker, worker_add_run_status=True):
    method __str__ (line 87) | def __str__(self):
    method __repr__ (line 90) | def __repr__(self):
  function _partition_tasks (line 94) | def _partition_tasks(worker):
  function _root_task (line 128) | def _root_task(worker):
  function _populate_unknown_statuses (line 135) | def _populate_unknown_statuses(set_tasks):
  function _depth_first_search (line 144) | def _depth_first_search(set_tasks, current_task, visited):
  function _get_str (line 179) | def _get_str(task_dict, extra_indent):
  function _get_len_of_params (line 242) | def _get_len_of_params(task):
  function _get_str_ranging_multiple_parameters (line 246) | def _get_str_ranging_multiple_parameters(first, last, tasks, unique_param):
  function _get_set_of_params (line 261) | def _get_set_of_params(tasks):
  function _get_unique_param_keys (line 268) | def _get_unique_param_keys(params):
  function _ranging_attributes (line 274) | def _ranging_attributes(attributes, param_class):
  function _get_str_one_parameter (line 288) | def _get_str_one_parameter(tasks):
  function _serialize_first_param (line 303) | def _serialize_first_param(task):
  function _get_number_of_tasks_for (line 307) | def _get_number_of_tasks_for(status, group_tasks):
  function _get_number_of_tasks (line 313) | def _get_number_of_tasks(task_dict):
  function _get_comments (line 317) | def _get_comments(group_tasks):
  function _get_run_by_other_worker (line 363) | def _get_run_by_other_worker(worker):
  function _get_external_workers (line 371) | def _get_external_workers(worker):
  function _group_tasks_by_name_and_status (line 389) | def _group_tasks_by_name_and_status(task_dict):
  function _summary_dict (line 403) | def _summary_dict(worker):
  function _summary_format (line 410) | def _summary_format(set_tasks, worker):
  function _create_one_line_summary (line 460) | def _create_one_line_summary(status_code):
  function _tasks_status (line 467) | def _tasks_status(set_tasks):
  function _summary_wrap (line 488) | def _summary_wrap(str_output):
  function summary (line 498) | def summary(worker):

FILE: luigi/format.py
  class FileWrapper (line 28) | class FileWrapper:
    method __init__ (line 33) | def __init__(self, file_object):
    method __getattr__ (line 36) | def __getattr__(self, name):
    method __enter__ (line 40) | def __enter__(self, *args, **kwargs):
    method __exit__ (line 47) | def __exit__(self, *args, **kwargs):
    method __iter__ (line 50) | def __iter__(self):
  class InputPipeProcessWrapper (line 54) | class InputPipeProcessWrapper:
    method __init__ (line 55) | def __init__(self, command, input_pipe=None):
    method create_subprocess (line 90) | def create_subprocess(self, command):
    method _finish (line 102) | def _finish(self):
    method close (line 117) | def close(self):
    method __del__ (line 120) | def __del__(self):
    method __enter__ (line 123) | def __enter__(self):
    method _abort (line 126) | def _abort(self):
    method __exit__ (line 137) | def __exit__(self, type, value, traceback):
    method __getattr__ (line 143) | def __getattr__(self, name):
    method __iter__ (line 151) | def __iter__(self):
    method readable (line 156) | def readable(self):
    method writable (line 159) | def writable(self):
    method seekable (line 162) | def seekable(self):
  class OutputPipeProcessWrapper (line 166) | class OutputPipeProcessWrapper:
    method __init__ (line 169) | def __init__(self, command, output_pipe=None):
    method write (line 176) | def write(self, *args, **kwargs):
    method writeLine (line 183) | def writeLine(self, line):
    method _finish (line 187) | def _finish(self):
    method __del__ (line 197) | def __del__(self):
    method __exit__ (line 201) | def __exit__(self, type, value, traceback):
    method __enter__ (line 207) | def __enter__(self):
    method close (line 210) | def close(self):
    method abort (line 218) | def abort(self):
    method __getattr__ (line 221) | def __getattr__(self, name):
    method readable (line 229) | def readable(self):
    method writable (line 232) | def writable(self):
    method seekable (line 235) | def seekable(self):
  class BaseWrapper (line 239) | class BaseWrapper:
    method __init__ (line 240) | def __init__(self, stream, *args, **kwargs):
    method __getattr__ (line 247) | def __getattr__(self, name):
    method __enter__ (line 252) | def __enter__(self):
    method __exit__ (line 256) | def __exit__(self, *args):
    method __iter__ (line 259) | def __iter__(self):
  class NewlineWrapper (line 267) | class NewlineWrapper(BaseWrapper):
    method __init__ (line 268) | def __init__(self, stream, newline=None):
    method read (line 278) | def read(self, n=-1):
    method writelines (line 289) | def writelines(self, lines):
    method write (line 297) | def write(self, b):
  class MixedUnicodeBytesWrapper (line 306) | class MixedUnicodeBytesWrapper(BaseWrapper):
    method __init__ (line 309) | def __init__(self, stream, encoding=None):
    method write (line 315) | def write(self, b):
    method writelines (line 318) | def writelines(self, lines):
    method _convert (line 321) | def _convert(self, b):
  class Format (line 328) | class Format:
    method pipe_reader (line 334) | def pipe_reader(cls, input_pipe):
    method pipe_writer (line 338) | def pipe_writer(cls, output_pipe):
    method __rshift__ (line 341) | def __rshift__(a, b):
  class ChainFormat (line 345) | class ChainFormat(Format):
    method __init__ (line 346) | def __init__(self, *args, **kwargs):
    method pipe_reader (line 374) | def pipe_reader(self, input_pipe):
    method pipe_writer (line 379) | def pipe_writer(self, output_pipe):
  class TextWrapper (line 385) | class TextWrapper(io.TextIOWrapper):
    method __exit__ (line 386) | def __exit__(self, *args):
    method __del__ (line 393) | def __del__(self, *args):
    method __init__ (line 403) | def __init__(self, stream, *args, **kwargs):
    method __getattr__ (line 410) | def __getattr__(self, name):
    method __enter__ (line 415) | def __enter__(self):
  class NopFormat (line 420) | class NopFormat(Format):
    method pipe_reader (line 421) | def pipe_reader(self, input_pipe):
    method pipe_writer (line 424) | def pipe_writer(self, output_pipe):
  class WrappedFormat (line 428) | class WrappedFormat(Format):
    method __init__ (line 429) | def __init__(self, *args, **kwargs):
    method pipe_reader (line 433) | def pipe_reader(self, input_pipe):
    method pipe_writer (line 436) | def pipe_writer(self, output_pipe):
  class TextFormat (line 440) | class TextFormat(WrappedFormat):
  class MixedUnicodeBytesFormat (line 446) | class MixedUnicodeBytesFormat(WrappedFormat):
  class NewlineFormat (line 451) | class NewlineFormat(WrappedFormat):
  class GzipFormat (line 457) | class GzipFormat(Format):
    method __init__ (line 461) | def __init__(self, compression_level=None):
    method pipe_reader (line 464) | def pipe_reader(self, input_pipe):
    method pipe_writer (line 467) | def pipe_writer(self, output_pipe):
  class Bzip2Format (line 474) | class Bzip2Format(Format):
    method pipe_reader (line 478) | def pipe_reader(self, input_pipe):
    method pipe_writer (line 481) | def pipe_writer(self, output_pipe):
  function get_default_format (line 494) | def get_default_format():

FILE: luigi/freezing.py
  class FrozenOrderedDict (line 16) | class FrozenOrderedDict(Mapping):
    method __init__ (line 22) | def __init__(self, *args, **kwargs):
    method __getitem__ (line 26) | def __getitem__(self, key):
    method __iter__ (line 29) | def __iter__(self):
    method __len__ (line 32) | def __len__(self):
    method __repr__ (line 35) | def __repr__(self):
    method __hash__ (line 39) | def __hash__(self):
    method get_wrapped (line 46) | def get_wrapped(self):
  function recursively_freeze (line 50) | def recursively_freeze(value):
  function recursively_unfreeze (line 61) | def recursively_unfreeze(value):

FILE: luigi/interface.py
  class core (line 37) | class core(task.Config):
  class _WorkerSchedulerFactory (line 85) | class _WorkerSchedulerFactory:
    method create_local_scheduler (line 86) | def create_local_scheduler(self):
    method create_remote_scheduler (line 89) | def create_remote_scheduler(self, url):
    method create_worker (line 92) | def create_worker(self, scheduler, worker_processes, assistant=False):
  function _schedule_and_run (line 96) | def _schedule_and_run(tasks, worker_scheduler_factory=None, override_def...
  class PidLockAlreadyTakenExit (line 146) | class PidLockAlreadyTakenExit(SystemExit):
  function run (line 154) | def run(*args, **kwargs):
  function _run (line 166) | def _run(cmdline_args=None, main_task_cls=None, worker_scheduler_factory...
  function build (line 180) | def build(tasks, worker_scheduler_factory=None, detailed_summary=False, ...

FILE: luigi/local_target.py
  class atomic_file (line 34) | class atomic_file(AtomicLocalFile):
    method move_to_final_destination (line 39) | def move_to_final_destination(self):
    method generate_tmp_path (line 42) | def generate_tmp_path(self, path):
  class LocalFileSystem (line 46) | class LocalFileSystem(FileSystem):
    method copy (line 53) | def copy(self, old_path, new_path, raise_if_exists=False):
    method exists (line 61) | def exists(self, path):
    method mkdir (line 64) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method isdir (line 85) | def isdir(self, path):
    method listdir (line 88) | def listdir(self, path):
    method remove (line 94) | def remove(self, path, recursive=True):
    method move (line 100) | def move(self, old_path, new_path, raise_if_exists=False):
    method rename_dont_move (line 122) | def rename_dont_move(self, path, dest):
  class LocalTarget (line 131) | class LocalTarget(FileSystemTarget):
    method __init__ (line 134) | def __init__(self, path=None, format=None, is_tmp=False):
    method makedirs (line 146) | def makedirs(self):
    method open (line 158) | def open(self, mode="r"):
    method move (line 171) | def move(self, new_path, raise_if_exists=False):
    method move_dir (line 174) | def move_dir(self, new_path):
    method remove (line 177) | def remove(self):
    method copy (line 180) | def copy(self, new_path, raise_if_exists=False):
    method fn (line 184) | def fn(self):
    method __del__ (line 188) | def __del__(self):

FILE: luigi/lock.py
  function getpcmd (line 30) | def getpcmd(pid):
  function get_info (line 76) | def get_info(pid_dir, my_pid=None):
  function acquire_for (line 88) | def acquire_for(pid_dir, num_available=1, kill_signal=None):
  function _read_pids_file (line 133) | def _read_pids_file(pid_file):
  function _write_pids_file (line 150) | def _write_pids_file(pid_file, pids_set):

FILE: luigi/metrics.py
  class MetricsCollectors (line 6) | class MetricsCollectors(Enum):
    method get (line 14) | def get(cls, which, custom_import=None):
  class MetricsCollector (line 45) | class MetricsCollector(metaclass=abc.ABCMeta):
    method __init__ (line 51) | def __init__(self):
    method handle_task_started (line 55) | def handle_task_started(self, task):
    method handle_task_failed (line 59) | def handle_task_failed(self, task):
    method handle_task_disabled (line 63) | def handle_task_disabled(self, task, config):
    method handle_task_done (line 67) | def handle_task_done(self, task):
    method handle_task_statistics (line 70) | def handle_task_statistics(self, task, statistics):
    method generate_latest (line 73) | def generate_latest(self):
    method configure_http_handler (line 76) | def configure_http_handler(self, http_handler):
  class NoMetricsCollector (line 80) | class NoMetricsCollector(MetricsCollector):
    method __init__ (line 83) | def __init__(self):
    method handle_task_started (line 86) | def handle_task_started(self, task):
    method handle_task_failed (line 89) | def handle_task_failed(self, task):
    method handle_task_disabled (line 92) | def handle_task_disabled(self, task, config):
    method handle_task_done (line 95) | def handle_task_done(self, task):

FILE: luigi/mock.py
  class MockFileSystem (line 31) | class MockFileSystem(target.FileSystem):
    method copy (line 38) | def copy(self, path, dest, raise_if_exists=False):
    method get_all_data (line 47) | def get_all_data(self):
    method get_data (line 53) | def get_data(self, fn):
    method exists (line 56) | def exists(self, path):
    method remove (line 59) | def remove(self, path, recursive=True, skip_trash=True):
    method move (line 73) | def move(self, path, dest, raise_if_exists=False):
    method listdir (line 82) | def listdir(self, path):
    method isdir (line 88) | def isdir(self, path):
    method mkdir (line 91) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method clear (line 97) | def clear(self):
  class MockTarget (line 101) | class MockTarget(target.FileSystemTarget):
    method __init__ (line 104) | def __init__(self, fn, is_tmp=None, mirror_on_stderr=False, format=None):
    method exists (line 109) | def exists(
    method move (line 114) | def move(self, path, raise_if_exists=False):
    method rename (line 120) | def rename(self, *args, **kwargs):
    method open (line 126) | def open(self, mode="r"):

FILE: luigi/mypy.py
  class TaskPlugin (line 63) | class TaskPlugin(Plugin):
    method get_base_class_hook (line 64) | def get_base_class_hook(self, fullname: str) -> Callable[[ClassDefCont...
    method get_function_hook (line 71) | def get_function_hook(self, fullname: str) -> Callable[[FunctionContex...
    method check_parameter (line 77) | def check_parameter(self, fullname):
    method _task_class_maker_callback (line 82) | def _task_class_maker_callback(self, ctx: ClassDefContext) -> None:
    method _infer_choice_enum_element_type (line 86) | def _infer_choice_enum_element_type(self, ctx: FunctionContext, defaul...
    method _task_parameter_field_callback (line 100) | def _task_parameter_field_callback(self, ctx: FunctionContext) -> Type:
  class TaskAttribute (line 162) | class TaskAttribute:
    method __init__ (line 163) | def __init__(
    method to_argument (line 181) | def to_argument(self, current_info: TypeInfo, *, of: Literal["__init__...
    method expand_type (line 193) | def expand_type(self, current_info: TypeInfo) -> Type | None:
    method to_var (line 203) | def to_var(self, current_info: TypeInfo) -> Var:
    method serialize (line 206) | def serialize(self) -> JsonDict:
    method deserialize (line 217) | def deserialize(cls, info: TypeInfo, data: JsonDict, api: SemanticAnal...
    method expand_typevar_from_subtype (line 222) | def expand_typevar_from_subtype(self, sub_type: TypeInfo) -> None:
  class TaskTransformer (line 230) | class TaskTransformer:
    method __init__ (line 233) | def __init__(
    method transform (line 245) | def transform(self) -> bool:
    method _get_assignment_statements_from_if_statement (line 269) | def _get_assignment_statements_from_if_statement(self, stmt: IfStmt) -...
    method _get_assignment_statements_from_block (line 276) | def _get_assignment_statements_from_block(self, block: Block) -> Itera...
    method collect_attributes (line 283) | def collect_attributes(self) -> Optional[List[TaskAttribute]]:
    method _collect_parameter_args (line 393) | def _collect_parameter_args(self, expr: Expression) -> tuple[bool, Dic...
    method _infer_task_attr_init_type (line 413) | def _infer_task_attr_init_type(self, sym: SymbolTableNode, context: Co...
    method is_parameter_call (line 464) | def is_parameter_call(self, expr: Expression) -> bool:
    method _resolve_parameter_type (line 485) | def _resolve_parameter_type(self, t: Instance) -> Type:
  function plugin (line 504) | def plugin(version: str) -> type[Plugin]:

FILE: luigi/notifications.py
  class TestNotificationsTask (line 42) | class TestNotificationsTask(Task):
    method run (line 57) | def run(self):
    method complete (line 60) | def complete(self):
  class email (line 66) | class email(Config):
  class smtp (line 82) | class smtp(Config):
  class sendgrid (line 103) | class sendgrid(Config):
  function generate_email (line 107) | def generate_email(sender, subject, message, recipients, image_png):
  function wrap_traceback (line 129) | def wrap_traceback(traceback):
  function send_email_smtp (line 154) | def send_email_smtp(sender, subject, message, recipients, image_png):
  function send_email_ses (line 181) | def send_email_ses(sender, subject, message, recipients, image_png):
  function send_email_sendgrid (line 205) | def send_email_sendgrid(sender, subject, message, recipients, image_png):
  function _email_disabled_reason (line 223) | def _email_disabled_reason():
  function send_email_sns (line 234) | def send_email_sns(sender, subject, message, topic_ARN, image_png):
  function send_email (line 262) | def send_email(subject, message, sender, recipients, image_png=None):
  function _email_recipients (line 300) | def _email_recipients(additional_recipients=None):
  function send_error_email (line 311) | def send_error_email(subject, message, additional_recipients=None):
  function _prefix (line 320) | def _prefix(subject):
  function format_task_error (line 331) | def format_task_error(headline, task, command, formatted_exception=None):

FILE: luigi/parameter.py
  class _NoValueType (line 65) | class _NoValueType:
    method __new__ (line 70) | def __new__(cls) -> "_NoValueType":
    method __repr__ (line 75) | def __repr__(self) -> str:
  class ParameterVisibility (line 82) | class ParameterVisibility(IntEnum):
    method has_value (line 93) | def has_value(cls, value):
    method serialize (line 96) | def serialize(self):
  class ParameterException (line 100) | class ParameterException(Exception):
  class MissingParameterException (line 108) | class MissingParameterException(ParameterException):
  class UnknownParameterException (line 116) | class UnknownParameterException(ParameterException):
  class DuplicateParameterException (line 124) | class DuplicateParameterException(ParameterException):
  class OptionalParameterTypeWarning (line 132) | class OptionalParameterTypeWarning(UserWarning):
  class UnconsumedParameterWarning (line 140) | class UnconsumedParameterWarning(UserWarning):
  class ConfigPath (line 148) | class ConfigPath(TypedDict):
  class _ParameterKwargs (line 153) | class _ParameterKwargs(TypedDict, total=False):
  class Parameter (line 164) | class Parameter(Generic[T]):
    method __init__ (line 205) | def __init__(
    method __get__ (line 268) | def __get__(self, instance: None, owner: Any) -> "Parameter[T]": ...
    method __get__ (line 271) | def __get__(self, instance: Any, owner: Any) -> T: ...
    method __get__ (line 273) | def __get__(self, instance: Any, owner: Any) -> Any:
    method __set_name__ (line 278) | def __set_name__(self, owner, name):
    method __set__ (line 281) | def __set__(self, instance: Any, value: T):
    method _get_value_from_config (line 286) | def _get_value_from_config(self, section, name):
    method _get_value (line 298) | def _get_value(self, task_name, param_name):
    method _value_iterator (line 306) | def _value_iterator(self, task_name, param_name):
    method has_task_value (line 327) | def has_task_value(self, task_name, param_name):
    method task_value (line 330) | def task_value(self, task_name, param_name):
    method _is_batchable (line 337) | def _is_batchable(self):
    method parse (line 340) | def parse(self, x):
    method _parse_list (line 352) | def _parse_list(self, xs):
    method serialize (line 369) | def serialize(self, x):
    method _warn_on_wrong_param_type (line 379) | def _warn_on_wrong_param_type(self, param_name, param_value):
    method normalize (line 385) | def normalize(self, x):
    method next_in_enumeration (line 399) | def next_in_enumeration(self, value):
    method _parse_or_no_value (line 412) | def _parse_or_no_value(self, x):
    method _parser_global_dest (line 419) | def _parser_global_dest(param_name, task_name):
    method _parser_kwargs (line 423) | def _parser_kwargs(cls, param_name, task_name=None):
  class OptionalParameterMixin (line 430) | class OptionalParameterMixin(Generic[_OptT]):
    method __init__ (line 437) | def __init__(
    method __get__ (line 445) | def __get__(self, instance: None, owner: Any) -> "Parameter[Optional[_...
    method __get__ (line 448) | def __get__(self, instance: Any, owner: Any) -> Optional[_OptT]: ...
    method __get__ (line 450) | def __get__(self, instance: Any, owner: Any) -> Any:
    method __set__ (line 453) | def __set__(self, instance: Any, value: Optional[_OptT]):
    method serialize (line 456) | def serialize(self, x):
    method parse (line 465) | def parse(self, x):
    method normalize (line 476) | def normalize(self, x):
    method _warn_on_wrong_param_type (line 484) | def _warn_on_wrong_param_type(self, param_name, param_value):
    method next_in_enumeration (line 495) | def next_in_enumeration(self, value):
  class OptionalParameter (line 499) | class OptionalParameter(OptionalParameterMixin[str], Parameter[Optional[...
  class OptionalStrParameter (line 505) | class OptionalStrParameter(OptionalParameterMixin[str], Parameter[Option...
  class _DateParameterBase (line 514) | class _DateParameterBase(Parameter[datetime.date]):
    method __init__ (line 519) | def __init__(
    method date_format (line 532) | def date_format(self):
    method parse (line 538) | def parse(self, s):
    method serialize (line 544) | def serialize(self, dt):
  class DateParameter (line 553) | class DateParameter(_DateParameterBase):
    method next_in_enumeration (line 586) | def next_in_enumeration(self, value):
    method normalize (line 589) | def normalize(self, x):
  class MonthParameter (line 600) | class MonthParameter(DateParameter):
    method _add_months (line 612) | def _add_months(self, date, months):
    method next_in_enumeration (line 624) | def next_in_enumeration(self, value):
    method normalize (line 627) | def normalize(self, x):
  class YearParameter (line 640) | class YearParameter(DateParameter):
    method next_in_enumeration (line 651) | def next_in_enumeration(self, value):
    method normalize (line 654) | def normalize(self, x):
  class _DatetimeParameterBase (line 665) | class _DatetimeParameterBase(Parameter[datetime.datetime]):
    method __init__ (line 670) | def __init__(
    method date_format (line 683) | def date_format(self):
    method _timedelta (line 691) | def _timedelta(self):
    method parse (line 697) | def parse(self, s):
    method serialize (line 703) | def serialize(self, dt):
    method _convert_to_dt (line 712) | def _convert_to_dt(dt):
    method normalize (line 717) | def normalize(self, dt):
    method next_in_enumeration (line 732) | def next_in_enumeration(self, value):
  class DateHourParameter (line 736) | class DateHourParameter(_DatetimeParameterBase):
  class DateMinuteParameter (line 749) | class DateMinuteParameter(_DatetimeParameterBase):
    method parse (line 764) | def parse(self, x):
  class DateSecondParameter (line 773) | class DateSecondParameter(_DatetimeParameterBase):
  class StrParameter (line 788) | class StrParameter(Parameter[str]):
    method parse (line 793) | def parse(self, x):
  class IntParameter (line 797) | class IntParameter(Parameter[int]):
    method parse (line 802) | def parse(self, x):
    method next_in_enumeration (line 808) | def next_in_enumeration(self, value):
  class OptionalIntParameter (line 812) | class OptionalIntParameter(OptionalParameterMixin[int], IntParameter):  ...
  class FloatParameter (line 818) | class FloatParameter(Parameter[float]):
    method parse (line 823) | def parse(self, x):
  class OptionalFloatParameter (line 830) | class OptionalFloatParameter(OptionalParameterMixin[float], FloatParamet...
  class BoolParameter (line 836) | class BoolParameter(Parameter[bool]):
    method __init__ (line 870) | def __init__(
    method parse (line 881) | def parse(self, x):
    method normalize (line 893) | def normalize(self, x):
    method _parser_kwargs (line 899) | def _parser_kwargs(self, *args, **kwargs):
  class OptionalBoolParameter (line 911) | class OptionalBoolParameter(OptionalParameterMixin[bool], BoolParameter)...
  class DateIntervalParameter (line 917) | class DateIntervalParameter(Parameter[date_interval.DateInterval]):
    method parse (line 927) | def parse(self, x):
  class TimeDeltaParameter (line 946) | class TimeDeltaParameter(Parameter[datetime.timedelta]):
    method _apply_regex (line 959) | def _apply_regex(self, regex, input):
    method _parseIso8601 (line 974) | def _parseIso8601(self, input):
    method _parseSimple (line 985) | def _parseSimple(self, input):
    method parse (line 992) | def parse(self, x):
    method serialize (line 1010) | def serialize(self, x):
    method _warn_on_wrong_param_type (line 1024) | def _warn_on_wrong_param_type(self, param_name, param_value):
  class TaskParameter (line 1034) | class TaskParameter(Parameter[Type[TaskType]]):
    method parse (line 1053) | def parse(self, x):
    method serialize (line 1059) | def serialize(self, x):
  class EnumParameter (line 1069) | class EnumParameter(Parameter[EnumParameterType]):
    method __init__ (line 1092) | def __init__(
    method parse (line 1104) | def parse(self, x):
    method serialize (line 1110) | def serialize(self, x):
  class EnumListParameter (line 1114) | class EnumListParameter(Parameter[Tuple[EnumParameterType, ...]]):
    method __init__ (line 1141) | def __init__(
    method parse (line 1153) | def parse(self, x):
    method serialize (line 1164) | def serialize(self, x):
  class _DictParamEncoder (line 1168) | class _DictParamEncoder(JSONEncoder):
    method default (line 1173) | def default(self, obj):
  class DictParameter (line 1182) | class DictParameter(Parameter[DictT]):
    method __init__ (line 1266) | def __init__(
    method normalize (line 1280) | def normalize(self, x):
    method parse (line 1293) | def parse(self, x):
    method serialize (line 1309) | def serialize(self, x):
  class OptionalDictParameter (line 1313) | class OptionalDictParameter(OptionalParameterMixin[FrozenOrderedDict], D...
  class ListParameter (line 1322) | class ListParameter(Parameter[ListT]):
    method __init__ (line 1415) | def __init__(
    method normalize (line 1429) | def normalize(self, x):
    method parse (line 1445) | def parse(self, x):
    method serialize (line 1457) | def serialize(self, x):
  class OptionalListParameter (line 1468) | class OptionalListParameter(OptionalParameterMixin[ListT], ListParameter...
  class TupleParameter (line 1474) | class TupleParameter(ListParameter[ListT]):
    method parse (line 1503) | def parse(self, x):
    method _convert_iterable_to_tuple (line 1533) | def _convert_iterable_to_tuple(self, x):
  class OptionalTupleParameter (line 1539) | class OptionalTupleParameter(OptionalParameterMixin[ListT], TupleParamet...
  class NumericalParameter (line 1548) | class NumericalParameter(Parameter[NumericalType]):
    method __init__ (line 1570) | def __init__(
    method parse (line 1625) | def parse(self, x):
  class OptionalNumericalParameter (line 1633) | class OptionalNumericalParameter(OptionalParameterMixin[NumericalType], ...
    method __init__ (line 1636) | def __init__(
  class ChoiceParameter (line 1648) | class ChoiceParameter(Parameter[ChoiceType]):
    method __init__ (line 1673) | def __init__(
    method parse (line 1700) | def parse(self, x):
    method normalize (line 1704) | def normalize(self, x):
  class ChoiceListParameter (line 1711) | class ChoiceListParameter(ChoiceParameter[ChoiceType]):
    method __get__ (line 1741) | def __get__(self, instance: None, owner: Any) -> "Parameter[Tuple[Choi...
    method __get__ (line 1744) | def __get__(self, instance: Any, owner: Any) -> Tuple[ChoiceType, ...]...
    method __get__ (line 1746) | def __get__(self, instance: Any, owner: Any) -> Any:
    method __init__ (line 1749) | def __init__(
    method parse (line 1758) | def parse(self, x):
    method normalize (line 1762) | def normalize(self, x):
    method serialize (line 1768) | def serialize(self, x):
  class OptionalChoiceParameter (line 1772) | class OptionalChoiceParameter(OptionalParameterMixin[ChoiceType], Choice...
    method __init__ (line 1775) | def __init__(
  class PathParameter (line 1786) | class PathParameter(Parameter[Path]):
    method __init__ (line 1816) | def __init__(
    method normalize (line 1833) | def normalize(self, x):
  class OptionalPathParameter (line 1845) | class OptionalPathParameter(OptionalParameter, PathParameter):  # type: ...

FILE: luigi/process.py
  function check_pid (line 30) | def check_pid(pidfile):
  function write_pid (line 41) | def write_pid(pidfile):
  function get_log_format (line 54) | def get_log_format():
  function get_spool_handler (line 58) | def get_spool_handler(filename):
  function _server_already_running (line 70) | def _server_already_running(pidfile):
  function daemonize (line 77) | def daemonize(cmd, pidfile=None, logdir=None, api_port=8082, address=Non...

FILE: luigi/retcodes.py
  class retcode (line 31) | class retcode(luigi.Config):
  function run_with_retcodes (line 66) | def run_with_retcodes(argv):

FILE: luigi/rpc.py
  function _urljoin (line 55) | def _urljoin(base, url):
  class RPCError (line 66) | class RPCError(Exception):
    method __init__ (line 67) | def __init__(self, message, sub_exception=None):
  class _FetcherInterface (line 72) | class _FetcherInterface(metaclass=abc.ABCMeta):
    method fetch (line 74) | def fetch(self, full_url, body, timeout):
    method close (line 78) | def close(self):
  class URLLibFetcher (line 82) | class URLLibFetcher(_FetcherInterface):
    method _create_request (line 85) | def _create_request(self, full_url, body=None):
    method fetch (line 105) | def fetch(self, full_url, body, timeout):
    method close (line 109) | def close(self):
  class RequestsFetcher (line 113) | class RequestsFetcher(_FetcherInterface):
    method __init__ (line 114) | def __init__(self):
    method check_pid (line 121) | def check_pid(self):
    method fetch (line 128) | def fetch(self, full_url, body, timeout):
    method close (line 134) | def close(self):
  class RemoteScheduler (line 138) | class RemoteScheduler:
    method __init__ (line 143) | def __init__(self, url="http://localhost:8082/", connect_timeout=None):
    method close (line 162) | def close(self):
    method _get_retryer (line 165) | def _get_retryer(self):
    method _fetch (line 174) | def _fetch(self, url_suffix, body):
    method _request (line 184) | def _request(self, url, data, attempts=3, allow_null=True):

FILE: luigi/safe_extractor.py
  class SafeExtractor (line 38) | class SafeExtractor:
    method __init__ (line 54) | def __init__(self, path="."):
    method _is_within_directory (line 64) | def _is_within_directory(directory, target):
    method safe_extract (line 80) | def safe_extract(self, tar_path, members=None, *, numeric_owner=False):

FILE: luigi/scheduler.py
  function _get_empty_retry_policy (line 84) | def _get_empty_retry_policy():
  function rpc_method (line 88) | def rpc_method(**request_args):
  class scheduler (line 115) | class scheduler(Config):
    method _get_retry_policy (line 148) | def _get_retry_policy(self):
  function _get_default (line 152) | def _get_default(x, default):
  class OrderedSet (line 159) | class OrderedSet(MutableSet):
    method __init__ (line 166) | def __init__(self, iterable=None):
    method __len__ (line 173) | def __len__(self):
    method __contains__ (line 176) | def __contains__(self, key):
    method add (line 179) | def add(self, key):
    method discard (line 185) | def discard(self, key):
    method __iter__ (line 191) | def __iter__(self):
    method __reversed__ (line 198) | def __reversed__(self):
    method peek (line 205) | def peek(self, last=True):
    method pop (line 211) | def pop(self, last=True):
    method __repr__ (line 216) | def __repr__(self):
    method __eq__ (line 221) | def __eq__(self, other):
  class Task (line 227) | class Task:
    method __init__ (line 228) | def __init__(
    method __repr__ (line 282) | def __repr__(self):
    method set_params (line 285) | def set_params(self, params):
    method is_batchable (line 296) | def is_batchable(self):
    method add_failure (line 302) | def add_failure(self):
    method num_failures (line 313) | def num_failures(self):
    method has_excessive_failures (line 324) | def has_excessive_failures(self):
    method clear_failures (line 336) | def clear_failures(self):
    method pretty_id (line 344) | def pretty_id(self):
  class Worker (line 349) | class Worker:
    method __init__ (line 354) | def __init__(self, worker_id, last_active=None):
    method add_info (line 365) | def add_info(self, info):
    method update (line 368) | def update(self, worker_reference, get_work=False):
    method prune (line 375) | def prune(self, config):
    method get_tasks (line 380) | def get_tasks(self, state, *statuses):
    method is_trivial_worker (line 388) | def is_trivial_worker(self, state):
    method assistant (line 400) | def assistant(self):
    method enabled (line 404) | def enabled(self):
    method state (line 408) | def state(self):
    method add_rpc_message (line 414) | def add_rpc_message(self, name, **kwargs):
    method fetch_rpc_messages (line 418) | def fetch_rpc_messages(self):
    method __str__ (line 423) | def __str__(self):
  class SimpleTaskState (line 427) | class SimpleTaskState:
    method __init__ (line 436) | def __init__(self, state_path):
    method get_state (line 444) | def get_state(self):
    method set_state (line 447) | def set_state(self, state):
    method dump (line 452) | def dump(self):
    method load (line 462) | def load(self):
    method get_active_tasks (line 479) | def get_active_tasks(self):
    method get_active_tasks_by_status (line 482) | def get_active_tasks_by_status(self, *statuses):
    method get_active_task_count_for_status (line 485) | def get_active_task_count_for_status(self, status):
    method get_batch_running_tasks (line 491) | def get_batch_running_tasks(self, batch_id):
    method set_batcher (line 495) | def set_batcher(self, worker_id, family, batcher_args, max_batch_size):
    method get_batcher (line 499) | def get_batcher(self, worker_id, family):
    method num_pending_tasks (line 502) | def num_pending_tasks(self):
    method get_task (line 508) | def get_task(self, task_id, default=None, setdefault=None):
    method has_task (line 516) | def has_task(self, task_id):
    method re_enable (line 519) | def re_enable(self, task, config=None):
    method set_batch_running (line 526) | def set_batch_running(self, task, batch_id, worker_id):
    method set_status (line 533) | def set_status(self, task, new_status, config=None):
    method fail_dead_worker_task (line 586) | def fail_dead_worker_task(self, task, config, assistants):
    method update_status (line 599) | def update_status(self, task, config):
    method may_prune (line 616) | def may_prune(self, task):
    method inactivate_tasks (line 619) | def inactivate_tasks(self, delete_tasks):
    method get_active_workers (line 627) | def get_active_workers(self, last_active_lt=None, last_get_work_gt=None):
    method get_assistants (line 636) | def get_assistants(self, last_active_lt=None):
    method get_worker_ids (line 639) | def get_worker_ids(self):
    method get_worker (line 642) | def get_worker(self, worker_id):
    method inactivate_workers (line 645) | def inactivate_workers(self, delete_workers):
    method _remove_workers_from_tasks (line 651) | def _remove_workers_from_tasks(self, workers, remove_stakeholders=True):
    method disable_workers (line 657) | def disable_workers(self, worker_ids):
    method update_metrics (line 664) | def update_metrics(self, task, config):
  class Scheduler (line 673) | class Scheduler:
    method __init__ (line 680) | def __init__(self, config=None, resources=None, task_history_impl=None...
    method load (line 708) | def load(self):
    method dump (line 711) | def dump(self):
    method prune (line 717) | def prune(self):
    method _prune_workers (line 724) | def _prune_workers(self):
    method _prune_tasks (line 733) | def _prune_tasks(self):
    method _prune_emails (line 746) | def _prune_emails(self):
    method _update_worker (line 750) | def _update_worker(self, worker_id, worker_reference=None, get_work=Fa...
    method _update_priority (line 757) | def _update_priority(self, task, prio, worker):
    method add_task_batcher (line 771) | def add_task_batcher(self, worker, task_family, batched_args, max_batc...
    method forgive_failures (line 775) | def forgive_failures(self, task_id=None):
    method mark_as_done (line 789) | def mark_as_done(self, task_id=None):
    method add_task (line 802) | def add_task(
    method announce_scheduling_failure (line 966) | def announce_scheduling_failure(self, task_name, family, params, expl,...
    method add_worker (line 978) | def add_worker(self, worker, info, **kwargs):
    method disable_worker (line 982) | def disable_worker(self, worker):
    method set_worker_processes (line 986) | def set_worker_processes(self, worker, n):
    method send_scheduler_message (line 990) | def send_scheduler_message(self, worker, task, content):
    method add_scheduler_message_response (line 1000) | def add_scheduler_message_response(self, task_id, message_id, response):
    method get_scheduler_message_response (line 1006) | def get_scheduler_message_response(self, task_id, message_id):
    method has_task_history (line 1014) | def has_task_history(self):
    method is_pause_enabled (line 1018) | def is_pause_enabled(self):
    method is_paused (line 1022) | def is_paused(self):
    method pause (line 1026) | def pause(self):
    method unpause (line 1031) | def unpause(self):
    method update_resources (line 1036) | def update_resources(self, **resources):
    method update_resource (line 1042) | def update_resource(self, resource, amount):
    method _generate_retry_policy (line 1048) | def _generate_retry_policy(self, task_retry_policy_dict):
    method _has_resources (line 1053) | def _has_resources(self, needed_resources, used_resources):
    method _used_resources (line 1063) | def _used_resources(self):
    method _rank (line 1073) | def _rank(self, task):
    method _schedulable (line 1082) | def _schedulable(self, task):
    method _reset_orphaned_batch_running_tasks (line 1091) | def _reset_orphaned_batch_running_tasks(self, worker_id):
    method count_pending (line 1102) | def count_pending(self, worker):
    method get_work (line 1136) | def get_work(self, host=None, assistant=False, current_tasks=None, wor...
    method ping (line 1283) | def ping(self, **kwargs):
    method _upstream_status (line 1288) | def _upstream_status(self, task_id, upstream_status_table):
    method _serialize_task (line 1314) | def _serialize_task(self, task_id, include_deps=True, deps=None):
    method graph (line 1343) | def graph(self, **kwargs):
    method _filter_done (line 1351) | def _filter_done(self, task_ids):
    method _traverse_graph (line 1357) | def _traverse_graph(self, root_task_id, seen=None, dep_func=None, incl...
    method dep_graph (line 1420) | def dep_graph(self, task_id, include_done=True, **kwargs):
    method inverse_dep_graph (line 1427) | def inverse_dep_graph(self, task_id, include_done=True, **kwargs):
    method task_list (line 1438) | def task_list(self, status="", upstream_status="", limit=True, search=...
    method _first_task_display_name (line 1470) | def _first_task_display_name(self, worker):
    method worker_list (line 1478) | def worker_list(self, include_running=True, **kwargs):
    method resource_list (line 1516) | def resource_list(self):
    method resources (line 1534) | def resources(self):
    method task_search (line 1547) | def task_search(self, task_str, **kwargs):
    method re_enable_task (line 1563) | def re_enable_task(self, task_id):
    method fetch_error (line 1572) | def fetch_error(self, task_id, **kwargs):
    method set_task_status_message (line 1587) | def set_task_status_message(self, task_id, status_message):
    method get_task_status_message (line 1596) | def get_task_status_message(self, task_id):
    method set_task_progress_percentage (line 1604) | def set_task_progress_percentage(self, task_id, progress_percentage):
    method get_task_progress_percentage (line 1613) | def get_task_progress_percentage(self, task_id):
    method decrease_running_task_resources (line 1621) | def decrease_running_task_resources(self, task_id, decrease_resources):
    method get_running_task_resources (line 1638) | def get_running_task_resources(self, task_id):
    method _update_task_history (line 1645) | def _update_task_history(self, task, status, host=None):
    method task_history (line 1658) | def task_history(self):
    method update_metrics_task_started (line 1663) | def update_metrics_task_started(self, task):
    method report_task_statistics (line 1667) | def report_task_statistics(self, task_id, statistics):

FILE: luigi/server.py
  class cors (line 59) | class cors(Config):
    method __init__ (line 70) | def __init__(self, *args, **kwargs):
  class RPCHandler (line 75) | class RPCHandler(tornado.web.RequestHandler):
    method __init__ (line 80) | def __init__(self, *args, **kwargs):
    method initialize (line 84) | def initialize(self, scheduler):
    method options (line 87) | def options(self, *args):
    method get (line 94) | def get(self, method):
    method _handle_cors_preflight (line 113) | def _handle_cors_preflight(self):
    method _handle_cors (line 130) | def _handle_cors(self):
    method _set_other_cors_headers (line 145) | def _set_other_cors_headers(self):
  class BaseTaskHistoryHandler (line 155) | class BaseTaskHistoryHandler(tornado.web.RequestHandler):
    method initialize (line 156) | def initialize(self, scheduler):
    method get_template_path (line 159) | def get_template_path(self):
  class AllRunHandler (line 163) | class AllRunHandler(BaseTaskHistoryHandler):
    method get (line 164) | def get(self):
  class SelectedRunHandler (line 173) | class SelectedRunHandler(BaseTaskHistoryHandler):
    method get (line 174) | def get(self, name):
  function from_utc (line 213) | def from_utc(utcTime, fmt=None):
  class RecentRunHandler (line 232) | class RecentRunHandler(BaseTaskHistoryHandler):
    method get (line 233) | def get(self):
  class ByNameHandler (line 239) | class ByNameHandler(BaseTaskHistoryHandler):
    method get (line 240) | def get(self, name):
  class ByIdHandler (line 246) | class ByIdHandler(BaseTaskHistoryHandler):
    method get (line 247) | def get(self, id):
  class ByTaskIdHandler (line 253) | class ByTaskIdHandler(BaseTaskHistoryHandler):
    method get (line 254) | def get(self, task_id):
  class ByParamsHandler (line 260) | class ByParamsHandler(BaseTaskHistoryHandler):
    method get (line 261) | def get(self, name):
  class RootPathHandler (line 269) | class RootPathHandler(BaseTaskHistoryHandler):
    method get (line 270) | def get(self):
    method head (line 279) | def head(self):
  class MetricsHandler (line 285) | class MetricsHandler(tornado.web.RequestHandler):
    method initialize (line 286) | def initialize(self, scheduler):
    method get (line 289) | def get(self):
  function app (line 297) | def app(scheduler):
  function _init_api (line 319) | def _init_api(scheduler, api_port=None, address=None, unix_socket=None):
  function run (line 332) | def run(api_port=8082, address=None, unix_socket=None, scheduler=None):
  function stop (line 375) | def stop():

FILE: luigi/setup_logging.py
  class BaseLogging (line 31) | class BaseLogging:
    method _section (line 35) | def _section(cls, opts):
    method setup (line 47) | def setup(cls, opts=type("opts", (), {"background": None, "logdir": No...
  class DaemonLogging (line 85) | class DaemonLogging(BaseLogging):
    method _cli (line 92) | def _cli(cls, opts):
    method _conf (line 112) | def _conf(cls, opts):
    method _default (line 127) | def _default(cls, opts):
  class InterfaceLogging (line 134) | class InterfaceLogging(BaseLogging):
    method _cli (line 140) | def _cli(cls, opts):
    method _conf (line 144) | def _conf(cls, opts):
    method _default (line 158) | def _default(cls, opts):

FILE: luigi/static/visualiser/js/graph.js
  function nodeFromTask (line 32) | function nodeFromTask(task) {
  function uniqueIndexByProperty (line 48) | function uniqueIndexByProperty(data, propertyName) {
  function createDependencyEdges (line 57) | function createDependencyEdges(nodes, nodeIndex) {
  function computeDepth (line 72) | function computeDepth(nodes, nodeIndex) {
  function groupTasks (line 93) | function groupTasks(nodes) {
  function computeRows (line 131) | function computeRows(nodes, nodeIndex) {
  function layoutNodes (line 165) | function layoutNodes(nodes, rowSizes) {
  function createGraph (line 190) | function createGraph(tasks, hashBase) {
  function findBounds (line 213) | function findBounds(nodes) {
  function DependencyGraph (line 228) | function DependencyGraph(containerElement) {
  function svgElement (line 235) | function svgElement(name) {
  function svgLink (line 239) | function svgLink(url) {

FILE: luigi/static/visualiser/js/luigi.js
  function LuigiAPI (line 2) | function LuigiAPI (urlRoot) {
  function flatten (line 6) | function flatten(response, rootId) {
  function flatten_running (line 22) | function flatten_running(response) {
  function jsonRPC (line 29) | function jsonRPC(url, paramObject, callback) {
  function searchTerm (line 38) | function searchTerm() {

FILE: luigi/static/visualiser/js/tipsy.js
  function maybeCall (line 8) | function maybeCall(thing, ctx) {
  function Tipsy (line 12) | function Tipsy(element, options) {
  function get (line 180) | function get(ele) {
  function enter (line 189) | function enter() {
  function leave (line 200) | function leave() {

FILE: luigi/static/visualiser/js/util.js
  function escapeHtml (line 1) | function escapeHtml(unsafe) {

FILE: luigi/static/visualiser/js/visualiserApp.js
  function visualiserApp (line 1) | function visualiserApp(luigi) {

FILE: luigi/static/visualiser/lib/URI/1.18.2/URI.js
  function URI (line 35) | function URI(url, base) {
  function escapeRegEx (line 79) | function escapeRegEx(string) {
  function getType (line 84) | function getType(value) {
  function isArray (line 93) | function isArray(obj) {
  function filterArrayValues (line 97) | function filterArrayValues(data, value) {
  function arrayContains (line 126) | function arrayContains(list, value) {
  function arraysEqual (line 155) | function arraysEqual(one, two) {
  function trimSlashes (line 177) | function trimSlashes(text) {
  function escapeForDumbFirefox36 (line 271) | function escapeForDumbFirefox36(value) {
  function strictEncodeURIComponent (line 277) | function strictEncodeURIComponent(string) {
  function generateSimpleAccessor (line 1048) | function generateSimpleAccessor(_part){
  function generatePrefixAccessor (line 1060) | function generatePrefixAccessor(_part, _key){

FILE: luigi/static/visualiser/lib/mustache.js
  function testRegExp (line 32) | function testRegExp(re, string) {
  function isWhitespace (line 36) | function isWhitespace(string) {
  function escapeRegExp (line 45) | function escapeRegExp(string) {
  function escapeHtml (line 58) | function escapeHtml(string) {
  function Scanner (line 64) | function Scanner(string) {
  function Context (line 118) | function Context(view, parent) {
  function Writer (line 166) | function Writer() {
  function renderTokens (line 227) | function renderTokens(tokens, writer, context, template) {
  function nestTokens (line 295) | function nestTokens(tokens) {
  function squashTokens (line 327) | function squashTokens(tokens) {
  function escapeTags (line 347) | function escapeTags(tags) {
  function parseTemplate (line 360) | function parseTemplate(template, tags) {

FILE: luigi/target.py
  class Target (line 34) | class Target(metaclass=abc.ABCMeta):
    method exists (line 47) | def exists(self):
  class FileSystemException (line 54) | class FileSystemException(Exception):
  class FileAlreadyExists (line 62) | class FileAlreadyExists(FileSystemException):
  class MissingParentDirectory (line 71) | class MissingParentDirectory(FileSystemException):
  class NotADirectory (line 80) | class NotADirectory(FileSystemException):
  class FileSystem (line 89) | class FileSystem(metaclass=abc.ABCMeta):
    method exists (line 102) | def exists(self, path):
    method remove (line 111) | def remove(self, path, recursive=True, skip_trash=True):
    method mkdir (line 120) | def mkdir(self, path, parents=True, raise_if_exists=False):
    method isdir (line 136) | def isdir(self, path):
    method listdir (line 146) | def listdir(self, path):
    method move (line 158) | def move(self, path, dest):
    method rename_dont_move (line 164) | def rename_dont_move(self, path, dest):
    method rename (line 178) | def rename(self, *args, **kwargs):
    method copy (line 184) | def copy(self, path, dest):
  class FileSystemTarget (line 193) | class FileSystemTarget(Target):
    method __init__ (line 212) | def __init__(self, path):
    method __str__ (line 221) | def __str__(self):
    method fs (line 226) | def fs(self):
    method open (line 233) | def open(self, mode):
    method exists (line 247) | def exists(self):
    method remove (line 258) | def remove(self):
    method temporary_path (line 267) | def temporary_path(self):
    method _touchz (line 304) | def _touchz(self):
    method _trailing_slash (line 308) | def _trailing_slash(self):
  class AtomicLocalFile (line 314) | class AtomicLocalFile(io.BufferedWriter):
    method __init__ (line 323) | def __init__(self, path):
    method close (line 328) | def close(self):
    method generate_tmp_path (line 332) | def generate_tmp_path(self, path):
    method move_to_final_destination (line 335) | def move_to_final_destination(self):
    method __del__ (line 338) | def __del__(self):
    method tmp_path (line 343) | def tmp_path(self):
    method __exit__ (line 346) | def __exit__(self, exc_type, exc, traceback):

FILE: luigi/task.py
  function namespace (line 53) | def namespace(namespace=None, scope=""):
  function auto_namespace (line 88) | def auto_namespace(scope=""):
  function task_id_str (line 118) | def task_id_str(task_family, params):
  class BulkCompleteNotImplementedError (line 137) | class BulkCompleteNotImplementedError(NotImplementedError):
  class Task (line 149) | class Task(metaclass=Register):
    method batchable (line 196) | def batchable(self):
    method retry_count (line 204) | def retry_count(self):
    method disable_hard_timeout (line 212) | def disable_hard_timeout(self):
    method disable_window (line 220) | def disable_window(self):
    method disable_window_seconds (line 228) | def disable_window_seconds(self):
    method owner_email (line 233) | def owner_email(self):
    method _owner_list (line 241) | def _owner_list(self):
    method use_cmdline_section (line 254) | def use_cmdline_section(self):
    method event_handler (line 260) | def event_handler(cls, event):
    method remove_event_handler (line 272) | def remove_event_handler(cls, event, callback):
    method trigger_event (line 278) | def trigger_event(self, event, *args, **kwargs):
    method accepts_messages (line 295) | def accepts_messages(self):
    method task_module (line 303) | def task_module(self):
    method get_task_namespace (line 328) | def get_task_namespace(cls):
    method task_family (line 341) | def task_family(self):
    method get_task_family (line 352) | def get_task_family(cls):
    method get_params (line 367) | def get_params(cls):
    method batch_param_names (line 385) | def batch_param_names(cls):
    method get_param_names (line 389) | def get_param_names(cls, include_significant=False):
    method get_param_values (line 393) | def get_param_values(cls, params, args, kwargs):
    method __init__ (line 465) | def __init__(self, *args, **kwargs):
    method param_args (line 485) | def param_args(self):
    method initialized (line 489) | def initialized(self):
    method _warn_on_wrong_param_types (line 495) | def _warn_on_wrong_param_types(self):
    method from_str_params (line 501) | def from_str_params(cls, params_str):
    method to_str_params (line 518) | def to_str_params(self, only_significant=False, only_public=False):
    method _get_param_visibilities (line 534) | def _get_param_visibilities(self):
    method clone (line 543) | def clone(self, cls=None, **kwargs):
    method __hash__ (line 568) | def __hash__(self):
    method __repr__ (line 571) | def __repr__(self):
    method __eq__ (line 589) | def __eq__(self, other):
    method complete (line 592) | def complete(self):
    method bulk_complete (line 607) | def bulk_complete(cls, parameter_tuples):
    method output (line 616) | def output(self):
    method requires (line 634) | def requires(self):
    method _requires (line 648) | def _requires(self):
    method process_resources (line 660) | def process_resources(self):
    method input (line 668) | def input(self):
    method deps (line 679) | def deps(self):
    method run (line 688) | def run(self):
    method on_failure (line 696) | def on_failure(self, exception):
    method on_success (line 711) | def on_success(self):
    method no_unpicklable_properties (line 723) | def no_unpicklable_properties(self):
  class MixinNaiveBulkComplete (line 754) | class MixinNaiveBulkComplete:
    method bulk_complete (line 764) | def bulk_complete(cls, parameter_tuples):
  class DynamicRequirements (line 779) | class DynamicRequirements(object):
    method __init__ (line 815) | def __init__(self, requirements, custom_complete=None):
    method flat_requirements (line 827) | def flat_requirements(self):
    method paths (line 833) | def paths(self):
    method complete (line 838) | def complete(self, complete_fn=None):
  class ExternalTask (line 853) | class ExternalTask(Task):
  function externalize (line 865) | def externalize(taskclass_or_taskobject):
  class WrapperTask (line 926) | class WrapperTask(Task):
    method complete (line 931) | def complete(self):
  class Config (line 935) | class Config(Task):
  function getpaths (line 945) | def getpaths(struct):
  function flatten (line 963) | def flatten(struct):
  function flatten_output (line 999) | def flatten_output(task):
  function _task_wraps (line 1017) | def _task_wraps(task_class):

FILE: luigi/task_history.py
  class StoredTask (line 28) | class StoredTask:
    method __init__ (line 36) | def __init__(self, task, status, host=None):
    method task_family (line 43) | def task_family(self):
    method parameters (line 47) | def parameters(self):
  class TaskHistory (line 51) | class TaskHistory(metaclass=abc.ABCMeta):
    method task_scheduled (line 57) | def task_scheduled(self, task):
    method task_finished (line 61) | def task_finished(self, task, successful):
    method task_started (line 65) | def task_started(self, task, worker_host):
  class NopHistory (line 71) | class NopHistory(TaskHistory):
    method task_scheduled (line 72) | def task_scheduled(self, task):
    method task_finished (line 75) | def task_finished(self, task, successful):
    method task_started (line 78) | def task_started(self, task, worker_host):

FILE: luigi/task_register.py
  class TaskClassException (line 28) | class TaskClassException(Exception):
  class TaskClassNotFoundException (line 32) | class TaskClassNotFoundException(TaskClassException):
  class TaskClassAmbigiousException (line 36) | class TaskClassAmbigiousException(TaskClassException):
  class Register (line 40) | class Register(abc.ABCMeta):
    method __new__ (line 59) | def __new__(metacls, classname, bases, classdict, **kwargs):
    method __call__ (line 73) | def __call__(cls, *args, **kwargs):
    method clear_instance_cache (line 106) | def clear_instance_cache(cls):
    method disable_instance_cache (line 113) | def disable_instance_cache(cls):
    method task_family (line 120) | def task_family(cls):
    method _get_reg (line 131) | def _get_reg(cls):
    method _set_reg (line 157) | def _set_reg(cls, reg):
    method task_names (line 162) | def task_names(cls):
    method tasks_str (line 169) | def tasks_str(cls):
    method get_task_cls (line 176) | def get_task_cls(cls, name):
    method get_all_params (line 189) | def get_all_params(cls):
    method _editdistance (line 202) | def _editdistance(a, b):
    method _missing_task_msg (line 219) | def _missing_task_msg(cls, task_name):
    method _get_namespace (line 229) | def _get_namespace(mcs, module_name):
    method _module_parents (line 237) | def _module_parents(module_name):
  function load_task (line 249) | def load_task(module, task_name, params_str):

FILE: luigi/tools/deps.py
  function get_task_requires (line 52) | def get_task_requires(task):
  function dfs_paths (line 56) | def dfs_paths(start_task, goal_task_family, path=None):
  class upstream (line 67) | class upstream(luigi.task.Config):
  function find_deps (line 75) | def find_deps(task, upstream_task_family):
  function find_deps_cli (line 85) | def find_deps_cli():
  function get_task_output_description (line 94) | def get_task_output_description(task_output):
  function main (line 114) | def main():

FILE: luigi/tools/deps_tree.py
  class bcolors (line 33) | class bcolors:
  function print_tree (line 43) | def print_tree(task, indent="", last=True):
  function main (line 68) | def main():

FILE: luigi/tools/luigi_grep.py
  class LuigiGrep (line 9) | class LuigiGrep:
    method __init__ (line 10) | def __init__(self, host, port):
    method graph_url (line 15) | def graph_url(self):
    method _fetch_json (line 18) | def _fetch_json(self):
    method _build_results (line 24) | def _build_results(self, jobs, job):
    method prefix_search (line 35) | def prefix_search(self, job_name_prefix):
    method status_search (line 43) | def status_search(self, status):
  function main (line 53) | def main():

FILE: luigi/tools/range.py
  class RangeEvent (line 47) | class RangeEvent(luigi.Event):  # Not sure if subclassing currently serv...
  class RangeBase (line 71) | class RangeBase(luigi.WrapperTask):
    method of_cls (line 114) | def of_cls(self):
    method datetime_to_parameter (line 124) | def datetime_to_parameter(self, dt):
    method parameter_to_datetime (line 127) | def parameter_to_datetime(self, p):
    method datetime_to_parameters (line 130) | def datetime_to_parameters(self, dt):
    method parameters_to_datetime (line 136) | def parameters_to_datetime(self, p):
    method moving_start (line 142) | def moving_start(self, now):
    method moving_stop (line 149) | def moving_stop(self, now):
    method finite_datetimes (line 156) | def finite_datetimes(self, finite_start, finite_stop):
    method _emit_metrics (line 163) | def _emit_metrics(self, missing_datetimes, finite_start, finite_stop):
    method _format_datetime (line 183) | def _format_datetime(self, dt):
    method _format_range (line 186) | def _format_range(self, datetimes):
    method _instantiate_task_cls (line 191) | def _instantiate_task_cls(self, param):
    method _param_name (line 195) | def _param_name(self):
    method _task_parameters (line 201) | def _task_parameters(self, param):
    method requires (line 206) | def requires(self):
    method missing_datetimes (line 254) | def missing_datetimes(self, finite_datetimes):
    method _missing_datetimes (line 266) | def _missing_datetimes(self, finite_datetimes):
  class RangeDailyBase (line 280) | class RangeDailyBase(RangeBase):
    method datetime_to_parameter (line 304) | def datetime_to_parameter(self, dt):
    method parameter_to_datetime (line 307) | def parameter_to_datetime(self, p):
    method datetime_to_parameters (line 310) | def datetime_to_parameters(self, dt):
    method parameters_to_datetime (line 316) | def parameters_to_datetime(self, p):
    method moving_start (line 323) | def moving_start(self, now):
    method moving_stop (line 326) | def moving_stop(self, now):
    method finite_datetimes (line 329) | def finite_datetimes(self, finite_start, finite_stop):
  class RangeHourlyBase (line 343) | class RangeHourlyBase(RangeBase):
    method datetime_to_parameter (line 368) | def datetime_to_parameter(self, dt):
    method parameter_to_datetime (line 371) | def parameter_to_datetime(self, p):
    method datetime_to_parameters (line 374) | def datetime_to_parameters(self, dt):
    method parameters_to_datetime (line 380) | def parameters_to_datetime(self, p):
    method moving_start (line 386) | def moving_start(self, now):
    method moving_stop (line 389) | def moving_stop(self, now):
    method finite_datetimes (line 392) | def finite_datetimes(self, finite_start, finite_stop):
    method _format_datetime (line 405) | def _format_datetime(self, dt):
  class RangeByMinutesBase (line 409) | class RangeByMinutesBase(RangeBase):
    method datetime_to_parameter (line 437) | def datetime_to_parameter(self, dt):
    method parameter_to_datetime (line 440) | def parameter_to_datetime(self, p):
    method datetime_to_parameters (line 443) | def datetime_to_parameters(self, dt):
    method parameters_to_datetime (line 449) | def parameters_to_datetime(self, p):
    method moving_start (line 456) | def moving_start(self, now):
    method moving_stop (line 459) | def moving_stop(self, now):
    method finite_datetimes (line 462) | def finite_datetimes(self, finite_start, finite_stop):
    method _format_datetime (line 482) | def _format_datetime(self, dt):
  function _constrain_glob (line 486) | def _constrain_glob(glob, paths, limit=5):
  function most_common (line 526) | def most_common(items):
  function _get_per_location_glob (line 531) | def _get_per_location_glob(tasks, outputs, regexes):
  function _get_filesystems_and_globs (line 562) | def _get_filesystems_and_globs(datetime_to_task, datetime_to_re):
  function _list_existing (line 592) | def _list_existing(filesystem, glob, paths):
  function infer_bulk_complete_from_fs (line 610) | def infer_bulk_complete_from_fs(datetimes, datetime_to_task, datetime_to...
  class RangeMonthly (line 638) | class RangeMonthly(RangeBase):
    method datetime_to_parameter (line 666) | def datetime_to_parameter(self, dt):
    method parameter_to_datetime (line 669) | def parameter_to_datetime(self, p):
    method datetime_to_parameters (line 672) | def datetime_to_parameters(self, dt):
    method parameters_to_datetime (line 678) | def parameters_to_datetime(self, p):
    method _format_datetime (line 685) | def _format_datetime(self, dt):
    method moving_start (line 688) | def moving_start(self, now):
    method moving_stop (line 691) | def moving_stop(self, now):
    method _align (line 694) | def _align(self, dt):
    method finite_datetimes (line 697) | def finite_datetimes(self, finite_start, finite_stop):
  class RangeDaily (line 712) | class RangeDaily(RangeDailyBase):
    method missing_datetimes (line 726) | def missing_datetimes(self, finite_datetimes):
  class RangeHourly (line 737) | class RangeHourly(RangeHourlyBase):
    method missing_datetimes (line 753) | def missing_datetimes(self, finite_datetimes):
  class RangeByMinutes (line 765) | class RangeByMinutes(RangeByMinutesBase):
    method missing_datetimes (line 781) | def missing_datetimes(self, finite_datetimes):

FILE: luigi/util.py
  function common_params (line 228) | def common_params(task_instance, task_cls):
  class inherits (line 245) | class inherits:
    method __init__ (line 280) | def __init__(self, *tasks_to_inherit, **kw_tasks_to_inherit):
    method __call__ (line 289) | def __call__(self, task_that_inherits):
  class requires (line 323) | class requires:
    method __init__ (line 331) | def __init__(self, *tasks_to_require, **kw_tasks_to_require):
    method __call__ (line 337) | def __call__(self, task_that_requires):
  class copies (line 351) | class copies:
    method __init__ (line 365) | def __init__(self, task_to_copy):
    method __call__ (line 369) | def __call__(self, task_that_copies):
  function delegates (line 385) | def delegates(task_that_delegates):
  function previous (line 426) | def previous(task):
  function get_previous_completed (line 464) | def get_previous_completed(task, max_steps=10):

FILE: luigi/worker.py
  function _is_external (line 77) | def _is_external(task):
  function _get_retry_policy_dict (line 81) | def _get_retry_policy_dict(task):
  class TaskException (line 85) | class TaskException(Exception):
  class TaskProcess (line 102) | class TaskProcess(multiprocessing.Process):
    method __init__ (line 117) | def __init__(
    method _run_get_new_deps (line 144) | def _run_get_new_deps(self):
    method run (line 172) | def run(self):
    method _handle_run_exception (line 246) | def _handle_run_exception(self, ex):
    method _recursive_terminate (line 251) | def _recursive_terminate(self):
    method terminate (line 271) | def terminate(self):
    method _forward_attributes (line 280) | def _forward_attributes(self):
  class ContextManagedTaskProcess (line 294) | class ContextManagedTaskProcess(TaskProcess):
    method __init__ (line 295) | def __init__(self, context, *args, **kwargs):
    method run (line 299) | def run(self):
  class TaskStatusReporter (line 312) | class TaskStatusReporter:
    method __init__ (line 320) | def __init__(self, scheduler, task_id, worker_id, scheduler_messages):
    method update_tracking_url (line 326) | def update_tracking_url(self, tracking_url):
    method update_status_message (line 329) | def update_status_message(self, message):
    method update_progress_percentage (line 332) | def update_progress_percentage(self, percentage):
    method decrease_running_resources (line 335) | def decrease_running_resources(self, decrease_resources):
    method report_task_statistics (line 338) | def report_task_statistics(self, statistics):
  class SchedulerMessage (line 342) | class SchedulerMessage:
    method __init__ (line 348) | def __init__(self, scheduler, task_id, message_id, content, **payload):
    method __str__ (line 358) | def __str__(self):
    method __eq__ (line 361) | def __eq__(self, other):
    method respond (line 364) | def respond(self, response):
  class SingleProcessPool (line 368) | class SingleProcessPool:
    method apply_async (line 375) | def apply_async(self, function, args):
    method close (line 378) | def close(self):
    method join (line 381) | def join(self):
  class DequeQueue (line 385) | class DequeQueue(collections.deque):
    method put (line 390) | def put(self, obj, block=None, timeout=None):
    method get (line 393) | def get(self, block=None, timeout=None):
  class AsyncCompletionException (line 400) | class AsyncCompletionException(Exception):
    method __init__ (line 405) | def __init__(self, trace):
  class TracebackWrapper (line 409) | class TracebackWrapper:
    method __init__ (line 414) | def __init__(self, trace):
  function check_complete_cached (line 418) | def check_complete_cached(task, completion_cache=None):
  function check_complete (line 434) | def check_complete(task, out_queue, completion_cache=None):
  class worker (line 446) | class worker(Config):
  class KeepAliveThread (line 498) | class KeepAliveThread(threading.Thread):
    method __init__ (line 503) | def __init__(self, scheduler, worker_id, ping_interval, rpc_message_ca...
    method stop (line 511) | def stop(self):
    method run (line 514) | def run(self):
  function rpc_message_callback (line 533) | def rpc_message_callback(fn):
  class Worker (line 538) | class Worker:
    method __init__ (line 548) | def __init__(self, scheduler=None, worker_id=None, worker_processes=1,...
    method _add_task (line 603) | def _add_task(self, *args, **kwargs):
    method __enter__ (line 627) | def __enter__(self):
    method __exit__ (line 636) | def __exit__(self, type, value, traceback):
    method _generate_worker_info (line 648) | def _generate_worker_info(self):
    method _generate_worker_id (line 672) | def _generate_worker_id(self, worker_info):
    method _validate_task (line 676) | def _validate_task(self, task):
    method _log_complete_error (line 684) | def _log_complete_error(self, task, tb):
    method _log_dependency_error (line 688) | def _log_dependency_error(self, task, tb):
    method _log_unexpected_error (line 692) | def _log_unexpected_error(self, task):
    method _announce_scheduling_failure (line 695) | def _announce_scheduling_failure(self, task, expl):
    method _email_complete_error (line 710) | def _email_complete_error(self, task, formatted_traceback):
    method _email_dependency_error (line 720) | def _email_dependency_error(self, task, formatted_traceback):
    method _email_unexpected_error (line 730) | def _email_unexpected_error(self, task, formatted_traceback):
    method _email_task_failure (line 741) | def _email_task_failure(self, task, formatted_traceback):
    method _email_error (line 750) | def _email_error(self, task, formatted_traceback, subject, headline):
    method _handle_task_load_error (line 757) | def _handle_task_load_error(self, exception, task_ids):
    method add (line 772) | def add(self, task, multiprocess=False, processes=0):
    method _add_task_batcher (line 818) | def _add_task_batcher(self, task):
    method _add (line 832) | def _add(self, task, is_complete):
    method _validate_dependency (line 917) | def _validate_dependency(self, dependency):
    method _check_complete_value (line 923) | def _check_complete_value(self, is_complete):
    method _add_worker (line 929) | def _add_worker(self):
    method _log_remote_tasks (line 933) | def _log_remote_tasks(self, get_work_response):
    method _get_work_task_id (line 946) | def _get_work_task_id(self, get_work_response):
    method _get_work (line 974) | def _get_work(self):
    method _run_task (line 1026) | def _run_task(self, task_id):
    method _create_task_process (line 1045) | def _create_task_process(self, task):
    method _purge_children (line 1062) | def _purge_children(self):
    method _handle_next_task (line 1082) | def _handle_next_task(self):
    method _sleeper (line 1150) | def _sleeper(self):
    method _keep_alive (line 1159) | def _keep_alive(self, get_work_response):
    method handle_interrupt (line 1189) | def handle_interrupt(self, signum, _):
    method _start_phasing_out (line 1196) | def _start_phasing_out(self):
    method run (line 1204) | def run(self):
    method _handle_rpc_message (line 1249) | def _handle_rpc_message(self, message):
    method set_worker_processes (line 1269) | def set_worker_processes(self, n):
    method dispatch_scheduler_message (line 1277) | def dispatch_scheduler_message(self, task_id, message_id, content, **k...

FILE: test/_mysqldb_test.py
  function _create_test_database (line 31) | def _create_test_database():
  class MySqlTargetTest (line 40) | class MySqlTargetTest(unittest.TestCase):
    method test_touch_and_exists (line 41) | def test_touch_and_exists(self):
  function drop (line 48) | def drop():

FILE: test/_test_ftp.py
  class TestFTPFilesystem (line 46) | class TestFTPFilesystem(unittest.TestCase):
    method setUp (line 47) | def setUp(self):
    method test_file_remove (line 73) | def test_file_remove(self):
    method test_recursive_remove (line 87) | def test_recursive_remove(self):
  class TestFTPFilesystemUpload (line 98) | class TestFTPFilesystemUpload(unittest.TestCase):
    method test_single (line 99) | def test_single(self):
  class TestRemoteTarget (line 127) | class TestRemoteTarget(unittest.TestCase):
    method test_put (line 128) | def test_put(self):
    method test_get (line 155) | def test_get(self):
  function _run_ftp_server (line 209) | def _run_ftp_server():

FILE: test/auto_namespace_test/my_namespace_test.py
  class MyNamespaceTest (line 6) | class MyNamespaceTest(LuigiTestCase):
    method test_auto_namespace_scope (line 7) | def test_auto_namespace_scope(self):

FILE: test/batch_notifier_test.py
  class BatchNotifier (line 16) | class BatchNotifier(luigi.batch_notifier.BatchNotifier):
    method __init__ (line 19) | def __init__(self, **kwargs):
  class BatchNotifierTest (line 25) | class BatchNotifierTest(unittest.TestCase):
    method setUp (line 26) | def setUp(self):
    method tearDown (line 39) | def tearDown(self):
    method incr_time (line 44) | def incr_time(self, minutes):
    method check_email_send (line 47) | def check_email_send(self, subject, message, receiver="r@test.com", se...
    method test_send_single_failure (line 50) | def test_send_single_failure(self):
    method test_do_not_send_single_failure_without_receiver (line 56) | def test_do_not_send_single_failure_without_receiver(self):
    method test_send_single_failure_to_owner_only (line 63) | def test_send_single_failure_to_owner_only(self):
    method test_send_single_disable (line 74) | def test_send_single_disable(self):
    method test_send_multiple_disables (line 82) | def test_send_multiple_disables(self):
    method test_send_single_scheduling_fail (line 92) | def test_send_single_scheduling_fail(self):
    method test_multiple_failures_of_same_job (line 101) | def test_multiple_failures_of_same_job(self):
    method test_multiple_failures_of_multiple_jobs (line 109) | def test_multiple_failures_of_multiple_jobs(self):
    method test_group_on_family (line 117) | def test_group_on_family(self):
    method test_group_on_unbatched_params (line 126) | def test_group_on_unbatched_params(self):
    method test_include_one_expl_includes_latest (line 144) | def test_include_one_expl_includes_latest(self):
    method test_include_two_expls (line 153) | def test_include_two_expls(self):
    method test_limit_expl_length (line 164) | def test_limit_expl_length(self):
    method test_expl_varies_by_owner (line 170) | def test_expl_varies_by_owner(self):
    method test_include_two_expls_html_format (line 197) | def test_include_two_expls_html_format(self):
    method test_limit_expl_length_html_format (line 210) | def test_limit_expl_length_html_format(self):
    method test_send_clears_backlog (line 217) | def test_send_clears_backlog(self):
    method test_email_gets_cleared_on_failure (line 228) | def test_email_gets_cleared_on_failure(self):
    method test_send_clears_all_old_data (line 239) | def test_send_clears_all_old_data(self):
    method test_auto_send_on_update_after_time_period (line 253) | def test_auto_send_on_update_after_time_period(self):
    method test_auto_send_on_update_after_time_period_with_disable_only (line 265) | def test_auto_send_on_update_after_time_period_with_disable_only(self):
    method test_no_auto_send_until_end_of_interval_with_error (line 277) | def test_no_auto_send_until_end_of_interval_with_error(self):
    method test_no_auto_send_for_interval_after_exception (line 294) | def test_no_auto_send_for_interval_after_exception(self):
    method test_send_batch_failure_emails_to_owners (line 313) | def test_send_batch_failure_emails_to_owners(self):
    method test_send_batch_disable_email_to_owners (line 342) | def test_send_batch_disable_email_to_owners(self):
    method test_batch_identical_expls (line 363) | def test_batch_identical_expls(self):
    method test_batch_identical_expls_html (line 376) | def test_batch_identical_expls_html(self):
    method test_unicode_error_message (line 397) | def test_unicode_error_message(self):
    method test_unicode_error_message_html (line 403) | def test_unicode_error_message_html(self):
    method test_unicode_param_value (line 410) | def test_unicode_param_value(self):
    method test_unicode_param_value_html (line 418) | def test_unicode_param_value_html(self):
    method test_unicode_param_name (line 427) | def test_unicode_param_name(self):
    method test_unicode_param_name_html (line 435) | def test_unicode_param_name_html(self):
    method test_unicode_class_name (line 444) | def test_unicode_class_name(self):
    method test_unicode_class_name_html (line 450) | def test_unicode_class_name_html(self):

FILE: test/choice_parameter_test.py
  class ChoiceParameterTest (line 23) | class ChoiceParameterTest(unittest.TestCase):
    method test_parse_str (line 24) | def test_parse_str(self):
    method test_parse_int (line 28) | def test_parse_int(self):
    method test_parse_int_conv (line 32) | def test_parse_int_conv(self):
    method test_invalid_choice (line 36) | def test_invalid_choice(self):
    method test_invalid_choice_type (line 40) | def test_invalid_choice_type(self):
    method test_choices_parameter_exception (line 43) | def test_choices_parameter_exception(self):
    method test_hash_str (line 46) | def test_hash_str(self):
    method test_serialize_parse (line 53) | def test_serialize_parse(self):
    method test_invalid_choice_task (line 58) | def test_invalid_choice_task(self):

FILE: test/clone_test.py
  class LinearSum (line 26) | class LinearSum(luigi.Task):
    method requires (line 30) | def requires(self):
    method run (line 34) | def run(self):
    method complete (line 41) | def complete(self):
    method f (line 44) | def f(self, x):
  class PowerSum (line 48) | class PowerSum(LinearSum):
    method f (line 51) | def f(self, x):
  class CloneTest (line 55) | class CloneTest(unittest.TestCase):
    method test_args (line 56) | def test_args(self):
    method test_recursion (line 61) | def test_recursion(self):
    method test_inheritance (line 66) | def test_inheritance(self):
    method test_inheritance_from_non_parameter (line 71) | def test_inheritance_from_non_parameter(self):

FILE: test/cmdline_test.py
  class SomeTask (line 31) | class SomeTask(luigi.Task):
    method output (line 34) | def output(self):
    method run (line 37) | def run(self):
  class AmbiguousClass (line 43) | class AmbiguousClass(luigi.Task):
  class AmbiguousClass (line 47) | class AmbiguousClass(luigi.Task):  # NOQA
  class TaskWithSameName (line 51) | class TaskWithSameName(luigi.Task):
    method run (line 52) | def run(self):
    method run (line 59) | def run(self):
  class TaskWithSameName (line 56) | class TaskWithSameName(luigi.Task):  # NOQA
    method run (line 52) | def run(self):
    method run (line 59) | def run(self):
  class WriteToFile (line 63) | class WriteToFile(luigi.Task):
    method output (line 66) | def output(self):
    method run (line 69) | def run(self):
  class FooBaseClass (line 75) | class FooBaseClass(luigi.Task):
  class FooSubClass (line 79) | class FooSubClass(FooBaseClass):
  class ATaskThatFails (line 83) | class ATaskThatFails(luigi.Task):
    method run (line 84) | def run(self):
  class RequiredConfig (line 88) | class RequiredConfig(luigi.Config):
  class TaskThatRequiresConfig (line 92) | class TaskThatRequiresConfig(luigi.WrapperTask):
    method requires (line 93) | def requires(self):
  class SubTaskThatFails (line 98) | class SubTaskThatFails(luigi.Task):
    method complete (line 99) | def complete(self):
    method run (line 102) | def run(self):
  class CmdlineTest (line 106) | class CmdlineTest(unittest.TestCase):
    method setUp (line 107) | def setUp(self):
    method tearDown (line 111) | def tearDown(self):
    method _clean_config (line 116) | def _clean_config(self):
    method _restore_config (line 120) | def _restore_config(self):
    method test_cmdline_main_task_cls (line 124) | def test_cmdline_main_task_cls(self, logger):
    method test_cmdline_local_scheduler (line 129) | def test_cmdline_local_scheduler(self, logger):
    method test_cmdline_other_task (line 134) | def test_cmdline_other_task(self, logger):
    method test_cmdline_ambiguous_class (line 139) | def test_cmdline_ambiguous_class(self, logger):
    method test_setup_interface_logging (line 144) | def test_setup_interface_logging(self, handler, logger):
    method test_non_existent_class (line 167) | def test_non_existent_class(self, print_usage):
    method test_no_task (line 171) | def test_no_task(self, print_usage):
    method test_luigid_logging_conf (line 174) | def test_luigid_logging_conf(self):
    method test_luigid_no_logging_conf (line 181) | def test_luigid_no_logging_conf(self):
    method test_luigid_missing_logging_conf (line 194) | def test_luigid_missing_logging_conf(self):
  class InvokeOverCmdlineTest (line 208) | class InvokeOverCmdlineTest(unittest.TestCase):
    method _run_cmdline (line 209) | def _run_cmdline(self, args):
    method test_bin_luigi (line 217) | def test_bin_luigi(self):
    method test_direct_python (line 223) | def test_direct_python(self):
    method test_python_module (line 229) | def test_python_module(self):
    method test_direct_python_help (line 235) | def test_direct_python_help(self):
    method test_direct_python_help_class (line 240) | def test_direct_python_help_class(self):
    method test_bin_luigi_help (line 245) | def test_bin_luigi_help(self):
    method test_python_module_luigi_help (line 250) | def test_python_module_luigi_help(self):
    method test_bin_luigi_help_no_module (line 255) | def test_bin_luigi_help_no_module(self):
    method test_bin_luigi_help_not_spammy (line 259) | def test_bin_luigi_help_not_spammy(self):
    method test_bin_luigi_all_help_spammy (line 266) | def test_bin_luigi_all_help_spammy(self):
    method test_error_mesage_on_misspelled_task (line 276) | def test_error_mesage_on_misspelled_task(self):
    method test_bin_luigi_no_parameters (line 280) | def test_bin_luigi_no_parameters(self):
    method test_python_module_luigi_no_parameters (line 284) | def test_python_module_luigi_no_parameters(self):
    method test_bin_luigi_help_class (line 288) | def test_bin_luigi_help_class(self):
    method test_python_module_help_class (line 293) | def test_python_module_help_class(self):
    method test_bin_luigi_options_before_task (line 298) | def test_bin_luigi_options_before_task(self):
    method test_bin_fail_on_unrecognized_args (line 303) | def test_bin_fail_on_unrecognized_args(self):
    method test_deps_py_script (line 307) | def test_deps_py_script(self):
    method test_deps_tree_py_script (line 317) | def test_deps_tree_py_script(self):
    method test_bin_mentions_misspelled_task (line 327) | def test_bin_mentions_misspelled_task(self):
    method test_stack_trace_has_no_inner (line 338) | def test_stack_trace_has_no_inner(self):
    method test_cmd_line_params_are_available_for_execution_summary (line 352) | def test_cmd_line_params_are_available_for_execution_summary(self):

FILE: test/config_env_test.py
  class ConfigParserTest (line 25) | class ConfigParserTest(LuigiTestCase):
    method setUp (line 30) | def setUp(self):
    method tearDown (line 37) | def tearDown(self):
    method test_basic_interpolation (line 54) | def test_basic_interpolation(self):
    method test_env_interpolation (line 71) | def test_env_interpolation(self):
    method test_underscore_vs_dash_style (line 93) | def test_underscore_vs_dash_style(self):
    method test_underscore_vs_dash_style_priority (line 108) | def test_underscore_vs_dash_style_priority(self):
    method test_default_parser (line 113) | def test_default_parser(self):

FILE: test/config_toml_test.py
  class TomlConfigParserTest (line 22) | class TomlConfigParserTest(LuigiTestCase):
    method setUpClass (line 24) | def setUpClass(cls):
    method setUp (line 28) | def setUp(self):
    method test_get_config (line 32) | def test_get_config(self):
    method test_file_reading (line 36) | def test_file_reading(self):
    method test_get (line 40) | def test_get(self):
    method test_set (line 57) | def test_set(self):
    method test_has_option (line 66) | def test_has_option(self):
  class HelpersTest (line 73) | class HelpersTest(LuigiTestCase):
    method test_add_without_install (line 74) | def test_add_without_install(self):
    method test_get_without_install (line 81) | def test_get_without_install(self):

FILE: test/conftest.py
  function reset_luigi_registry (line 9) | def reset_luigi_registry():
  function pytest_collection_modifyitems (line 25) | def pytest_collection_modifyitems(items: List[pytest.Item]) -> None:

FILE: test/contrib/_webhdfs_test.py
  class TestWebHdfsTarget (line 27) | class TestWebHdfsTarget(unittest.TestCase):
    method setUp (line 34) | def setUp(self):
    method tearDown (line 40) | def tearDown(self):
    method test_write (line 44) | def test_write(self):
    method test_read (line 52) | def test_read(self):
    method test_read_lines (line 59) | def test_read_lines(self):

FILE: test/contrib/azureblob_test.py
  class AzureBlobClientTest (line 40) | class AzureBlobClientTest(unittest.TestCase):
    method setUp (line 41) | def setUp(self):
    method tearDown (line 44) | def tearDown(self):
    method test_splitfilepath_blob_none (line 47) | def test_splitfilepath_blob_none(self):
    method test_splitfilepath_blob_toplevel (line 52) | def test_splitfilepath_blob_toplevel(self):
    method test_splitfilepath_blob_nested (line 57) | def test_splitfilepath_blob_nested(self):
    method test_create_delete_container (line 62) | def test_create_delete_container(self):
    method test_upload_copy_move_remove_blob (line 76) | def test_upload_copy_move_remove_blob(self):
  class MovieScriptTask (line 129) | class MovieScriptTask(luigi.Task):
    method output (line 130) | def output(self):
    method run (line 133) | def run(self):
  class AzureJsonDumpTask (line 143) | class AzureJsonDumpTask(luigi.Task):
    method output (line 144) | def output(self):
    method run (line 147) | def run(self):
  class FinalTask (line 152) | class FinalTask(luigi.Task):
    method requires (line 153) | def requires(self):
    method run (line 156) | def run(self):
    method output (line 166) | def output(self):
  class AzureBlobTargetTest (line 171) | class AzureBlobTargetTest(unittest.TestCase):
    method setUp (line 172) | def setUp(self):
    method tearDown (line 175) | def tearDown(self):
    method test_AzureBlobTarget (line 178) | def test_AzureBlobTarget(self):

FILE: test/contrib/batch_test.py
  class MockBotoBatchClient (line 31) | class MockBotoBatchClient:
    method describe_job_queues (line 32) | def describe_job_queues(self):
    method list_jobs (line 35) | def list_jobs(self, jobQueue="", jobStatus=""):
    method describe_jobs (line 38) | def describe_jobs(self, jobs=[]):
    method submit_job (line 44) | def submit_job(self, jobDefinition="", jobName="", jobQueue="", parame...
    method register_job_definition (line 47) | def register_job_definition(self, **kwargs):
  class MockBotoLogsClient (line 51) | class MockBotoLogsClient:
    method get_log_events (line 52) | def get_log_events(self, logGroupName="", logStreamName="", startFromH...
  class BatchClientTest (line 58) | class BatchClientTest(unittest.TestCase):
    method setUp (line 59) | def setUp(self):
    method test_get_active_queue (line 64) | def test_get_active_queue(self):
    method test_get_job_id_from_name (line 67) | def test_get_job_id_from_name(self):
    method test_get_job_status (line 70) | def test_get_job_status(self):
    method test_get_logs (line 73) | def test_get_logs(self):
    method test_submit_job (line 77) | def test_submit_job(self):
    method test_submit_job_specific_queue (line 81) | def test_submit_job_specific_queue(self):
    method test_submit_job_non_existant_queue (line 85) | def test_submit_job_non_existant_queue(self):
    method test_wait_on_job (line 89) | def test_wait_on_job(self):
    method test_wait_on_job_failed (line 93) | def test_wait_on_job_failed(self):
  class BatchTaskTest (line 103) | class BatchTaskTest(unittest.TestCase):
    method setUp (line 104) | def setUp(self):

FILE: test/contrib/beam_dataflow_test.py
  class TestDataflowParamKeys (line 30) | class TestDataflowParamKeys(beam_dataflow.DataflowParamKeys):
  class TestRequires (line 51) | class TestRequires(luigi.ExternalTask):
    method output (line 52) | def output(self):
  class SimpleTestTask (line 56) | class SimpleTestTask(beam_dataflow.BeamDataflowJobTask):
    method requires (line 59) | def requires(self):
    method output (line 62) | def output(self):
    method dataflow_executable (line 65) | def dataflow_executable(self):
  class FullTestTask (line 69) | class FullTestTask(beam_dataflow.BeamDataflowJobTask):
    method requires (line 91) | def requires(self):
    method output (line 94) | def output(self):
    method args (line 97) | def args(self):
    method dataflow_executable (line 100) | def dataflow_executable(self):
  class FilePatternsTestTask (line 104) | class FilePatternsTestTask(beam_dataflow.BeamDataflowJobTask):
    method requires (line 107) | def requires(self):
    method file_pattern (line 110) | def file_pattern(self):
    method output (line 113) | def output(self):
    method dataflow_executable (line 116) | def dataflow_executable(self):
  class DummyCmdLineTestTask (line 120) | class DummyCmdLineTestTask(beam_dataflow.BeamDataflowJobTask):
    method dataflow_executable (line 123) | def dataflow_executable(self):
    method requires (line 126) | def requires(self):
    method output (line 129) | def output(self):
    method _mk_cmd_line (line 132) | def _mk_cmd_line(self):
  class BeamDataflowTest (line 137) | class BeamDataflowTest(unittest.TestCase):
    method test_dataflow_simple_cmd_line_args (line 138) | def test_dataflow_simple_cmd_line_args(self):
    method test_dataflow_full_cmd_line_args (line 146) | def test_dataflow_full_cmd_line_args(self):
    method test_dataflow_with_file_patterns (line 179) | def test_dataflow_with_file_patterns(self):
    method test_dataflow_with_invalid_file_patterns (line 185) | def test_dataflow_with_invalid_file_patterns(self):
    method test_dataflow_input_arg_formatting (line 191) | def test_dataflow_input_arg_formatting(self):
    method test_task_output_arg_completion (line 236) | def test_task_output_arg_completion(self):
    method test_get_target_path (line 263) | def test_get_target_path(self):
    method test_dataflow_runner_resolution (line 273) | def test_dataflow_runner_resolution(self):
    method test_dataflow_successful_run_callbacks (line 285) | def test_dataflow_successful_run_callbacks(self):
    method test_dataflow_successful_run_invalid_output_callbacks (line 302) | def test_dataflow_successful_run_invalid_output_callbacks(self):
    method test_dataflow_failed_run_callbacks (line 322) | def test_dataflow_failed_run_callbacks(self, popen, os_exit):

FILE: test/contrib/bigquery_avro_test.py
  class BigQueryAvroTest (line 32) | class BigQueryAvroTest(unittest.TestCase):
    method test_writer_schema_method_existence (line 33) | def test_writer_schema_method_existence(self):

FILE: test/contrib/bigquery_gcloud_test.py
  function bucket_url (line 59) | def bucket_url(suffix):
  class TestLoadTask (line 67) | class TestLoadTask(bigquery.BigQueryLoadTask):
    method schema (line 74) | def schema(self):
    method source_uris (line 80) | def source_uris(self):
    method output (line 83) | def output(self):
  class TestRunQueryTask (line 88) | class TestRunQueryTask(bigquery.BigQueryRunQueryTask):
    method output (line 93) | def output(self):
  class TestExtractTask (line 98) | class TestExtractTask(bigquery.BigQueryExtractTask):
    method output (line 109) | def output(self):
    method requires (line 112) | def requires(self):
  class BigQueryGcloudTest (line 117) | class BigQueryGcloudTest(unittest.TestCase):
    method setUp (line 118) | def setUp(self):
    method test_extract_to_gcs_csv (line 150) | def test_extract_to_gcs_csv(self):
    method test_extract_to_gcs_csv_alternate (line 165) | def test_extract_to_gcs_csv_alternate(self):
    method test_extract_to_gcs_json (line 182) | def test_extract_to_gcs_json(self):
    method test_extract_to_gcs_avro (line 197) | def test_extract_to_gcs_avro(self):
    method test_load_eu_to_undefined (line 212) | def test_load_eu_to_undefined(self):
    method test_load_us_to_eu (line 216) | def test_load_us_to_eu(self):
    method test_load_eu_to_eu (line 220) | def test_load_e
Condensed preview — 329 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (2,757K chars).
[
  {
    "path": ".coveragerc",
    "chars": 284,
    "preview": "[report]\nomit =\n    luigi/mrrunner.py\n    test/_test_time_generated_module*.py\n    */python?.?/*\n    */site-packages/nos"
  },
  {
    "path": ".github/CODEOWNERS",
    "chars": 434,
    "preview": "# The following patterns are used to auto-assign review requests\n# to specific individuals. Order is important; the last"
  },
  {
    "path": ".github/ISSUE_TEMPLATE.md",
    "chars": 811,
    "preview": "<!---\nWe use GitHub issues mainly for tracking bugs and feature requests.\nQuestions for how to use luigi can be sent to "
  },
  {
    "path": ".github/PULL_REQUEST_TEMPLATE.md",
    "chars": 685,
    "preview": "<!--- This template is optional. Please use it as a starting point to help guide PRs -->\n\n<!--- Provide a general summar"
  },
  {
    "path": ".github/stale.yml",
    "chars": 800,
    "preview": "# Number of days of inactivity before an issue becomes stale\ndaysUntilStale: 120\n# Number of days of inactivity before a"
  },
  {
    "path": ".github/workflows/codeql.yml",
    "chars": 2518,
    "preview": "name: \"CodeQL\"\n\non:\n  push:\n    branches: [ 'master' ]\n  pull_request:\n    # The branches below must be a subset of the "
  },
  {
    "path": ".github/workflows/pythonbuild.yml",
    "chars": 6252,
    "preview": "name: Build\n\non:\n  push:\n    branches:\n      - master\n  pull_request:\n\njobs:\n  core:\n    runs-on: ubuntu-22.04\n\n    stra"
  },
  {
    "path": ".gitignore",
    "chars": 2347,
    "preview": ".coverage.*\ndoc/api/*.rst\ntest/gcloud-credentials.json\n.hypothesis/\n\n.nicesetup\n\nclient.cfg\nluigi.cfg\n\nhadoop_test.py\nmi"
  },
  {
    "path": ".readthedocs.yaml",
    "chars": 435,
    "preview": "version: 2\n\nbuild:\n  os: ubuntu-24.04\n  tools:\n    python: \"3.13\"\n  jobs:\n    pre_create_environment:\n      - asdf plugi"
  },
  {
    "path": "CONTRIBUTING.rst",
    "chars": 2280,
    "preview": "Code of conduct\n---------------\n\nThis project adheres to the `Open Code of Conduct \n<https://github.com/spotify/code-of-"
  },
  {
    "path": "LICENSE",
    "chars": 11345,
    "preview": "                                 Apache License\n                           Version 2.0, January 2004\n                   "
  },
  {
    "path": "README.rst",
    "chars": 12986,
    "preview": ".. figure:: https://raw.githubusercontent.com/spotify/luigi/master/doc/luigi.png\n   :alt: Luigi Logo\n   :align: center\n\n"
  },
  {
    "path": "RELEASE-PROCESS.rst",
    "chars": 1147,
    "preview": "For maintainers of Luigi, who have push access to pypi. Here's how you upload\nLuigi to pypi.\n\n#. Make sure [uv](https://"
  },
  {
    "path": "SECURITY.md",
    "chars": 265,
    "preview": "# Security Policy\n\n## Reporting a Vulnerability\n\nPlease report sensitive security issues via Spotify's [bug-bounty progr"
  },
  {
    "path": "bin/luigi",
    "chars": 264,
    "preview": "#!/usr/bin/env python\n\nimport sys\nimport warnings\nimport luigi.cmdline\n\n\ndef main(argv):\n    warnings.warn(\"'bin/luigi' "
  },
  {
    "path": "bin/luigid",
    "chars": 263,
    "preview": "#!/usr/bin/env python\n\nimport sys\nimport warnings\nimport luigi.cmdline\n\n\ndef main(argv):\n    warnings.warn(\"'bin/luigid'"
  },
  {
    "path": "catalog-info.yaml",
    "chars": 112,
    "preview": "apiVersion: backstage.io/v1alpha1\nkind: Component\nmetadata:\n  name: luigi\nspec:\n  type: library\n  owner: dataex\n"
  },
  {
    "path": "codecov.yml",
    "chars": 1130,
    "preview": "codecov:\n  require_ci_to_pass: true\n  notify:\n    wait_for_ci: true\n\ncoverage:\n  precision: 2\n  round: down\n  range: \"50"
  },
  {
    "path": "doc/.gitignore",
    "chars": 26,
    "preview": "_static\n_build\n_templates\n"
  },
  {
    "path": "doc/Makefile",
    "chars": 6758,
    "preview": "# Makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS    =\nSPHINXBUILD "
  },
  {
    "path": "doc/central_scheduler.rst",
    "chars": 4141,
    "preview": "Using the Central Scheduler\n---------------------------\n\nWhile the ``--local-scheduler`` flag is useful for development "
  },
  {
    "path": "doc/conf.py",
    "chars": 10317,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Luigi documentation build configuration file, created by\n# sphinx-quickstart on Sat Feb  8 0"
  },
  {
    "path": "doc/configuration.rst",
    "chars": 34930,
    "preview": "Configuration\n=============\n\nAll configuration can be done by adding configuration files.\n\nSupported config parsers:\n\n* "
  },
  {
    "path": "doc/design_and_limitations.rst",
    "chars": 2230,
    "preview": "Design and limitations\n----------------------\n\nLuigi is the successor to a couple of attempts that we weren't fully happ"
  },
  {
    "path": "doc/example_top_artists.rst",
    "chars": 11014,
    "preview": "Example – Top Artists\n---------------------\n\nThis is a very simplified case of something we do at Spotify a lot.\nAll use"
  },
  {
    "path": "doc/execution_model.rst",
    "chars": 4282,
    "preview": "Execution Model\n---------------\n\nLuigi has a quite simple model for execution and triggering.\n\nWorkers and task executio"
  },
  {
    "path": "doc/index.rst",
    "chars": 737,
    "preview": ".. Luigi documentation master file, created by\n   sphinx-quickstart on Sat Feb  8 00:56:43 2014.\n   You can adapt this f"
  },
  {
    "path": "doc/logging.rst",
    "chars": 1626,
    "preview": "Configure logging\n-----------------\n\n\nConfig options:\n~~~~~~~~~~~~~~~\n\nSome config options for config [core] section\n\nlo"
  },
  {
    "path": "doc/luigi_patterns.rst",
    "chars": 18117,
    "preview": "Luigi Patterns\n--------------\n\nCode Reuse\n~~~~~~~~~~\n\nOne nice thing about Luigi is that it's super easy to depend on ta"
  },
  {
    "path": "doc/mypy.rst",
    "chars": 784,
    "preview": "Mypy plugin\n--------------\n\nMypy plugin provides type checking for ``luigi.Task`` using Mypy.\n\nRequire Python 3.8 or lat"
  },
  {
    "path": "doc/parameters.rst",
    "chars": 5536,
    "preview": "Parameters\n----------\n\nParameters is the Luigi equivalent of creating a constructor for each Task.\nLuigi requires you to"
  },
  {
    "path": "doc/running_luigi.rst",
    "chars": 4627,
    "preview": "Running Luigi\n-------------\n\nRunning from the Command Line\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThe preferred way to run Luigi"
  },
  {
    "path": "doc/tasks.rst",
    "chars": 12900,
    "preview": "Tasks\n-----\n\nTasks are where the execution takes place.\nTasks depend on each other and output targets.\n\nAn outline of ho"
  },
  {
    "path": "doc/workflows.rst",
    "chars": 3892,
    "preview": "Building workflows\n------------------\n\nThere are two fundamental building blocks of Luigi -\nthe :class:`~luigi.task.Task"
  },
  {
    "path": "examples/__init__.py",
    "chars": 603,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/config.toml",
    "chars": 920,
    "preview": "\n[hdfs]\nclient = \"hadoopcli\"\nnamenode_host = \"localhost\"\nnamenode_port = 50030\n\n# LOGGING\n\n[logging]\nversion = 1\ndisable"
  },
  {
    "path": "examples/dynamic_requirements.py",
    "chars": 3733,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/elasticsearch_index.py",
    "chars": 3370,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/execution_summary_example.py",
    "chars": 3319,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/foo.py",
    "chars": 1463,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/foo_complex.py",
    "chars": 1877,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/ftp_experiment_outputs.py",
    "chars": 3239,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/hello_world.py",
    "chars": 501,
    "preview": "\"\"\"\nYou can run this example like this:\n\n    .. code:: console\n\n            $ luigi --module examples.hello_world exampl"
  },
  {
    "path": "examples/kubernetes.py",
    "chars": 1924,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Outlier Bio, LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lic"
  },
  {
    "path": "examples/per_task_retry_policy.py",
    "chars": 5236,
    "preview": "# -*- coding: utf-8 -*-\n\n\"\"\"\nYou can run this example like this:\n\n    .. code:: console\n\n            $ luigi --module ex"
  },
  {
    "path": "examples/pyspark_wc.py",
    "chars": 3360,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/spark_als.py",
    "chars": 4363,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/ssh_remote_execution.py",
    "chars": 2721,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/terasort.py",
    "chars": 3267,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/top_artists.py",
    "chars": 8851,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/top_artists_spark.py",
    "chars": 623,
    "preview": "# -*- coding: utf-8 -*-\n\nimport operator\nimport sys\n\nfrom pyspark.sql import SparkSession\n\n\ndef main(argv):\n    input_pa"
  },
  {
    "path": "examples/wordcount.py",
    "chars": 2905,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "examples/wordcount_hadoop.py",
    "chars": 2728,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/__init__.py",
    "chars": 3993,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/__main__.py",
    "chars": 683,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2016 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/__version__.py",
    "chars": 35,
    "preview": "# coding: utf-8\n\nVERSION = \"3.8.0\"\n"
  },
  {
    "path": "luigi/batch_notifier.py",
    "chars": 8797,
    "preview": "\"\"\"\nLibrary for sending batch notifications from the Luigi scheduler. This module\nis internal to Luigi and not designed "
  },
  {
    "path": "luigi/cmdline.py",
    "chars": 1370,
    "preview": "import argparse\nimport sys\n\nfrom luigi.retcodes import run_with_retcodes\nfrom luigi.setup_logging import DaemonLogging\n\n"
  },
  {
    "path": "luigi/cmdline_parser.py",
    "chars": 5159,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\")"
  },
  {
    "path": "luigi/configuration/__init__.py",
    "chars": 836,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/configuration/base_parser.py",
    "chars": 1305,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/configuration/cfg_parser.py",
    "chars": 8274,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/configuration/core.py",
    "chars": 2641,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/configuration/toml_parser.py",
    "chars": 2917,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Vote Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");"
  },
  {
    "path": "luigi/contrib/__init__.py",
    "chars": 661,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/azureblob.py",
    "chars": 13928,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright (c) 2018 Microsoft Corporation\n#\n# Licensed under the Apache License, Version 2.0 "
  },
  {
    "path": "luigi/contrib/batch.py",
    "chars": 7790,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Outlier Bio, LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lic"
  },
  {
    "path": "luigi/contrib/beam_dataflow.py",
    "chars": 16331,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2019 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\")"
  },
  {
    "path": "luigi/contrib/bigquery.py",
    "chars": 29480,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Twitter Inc\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\""
  },
  {
    "path": "luigi/contrib/bigquery_avro.py",
    "chars": 4227,
    "preview": "\"\"\"Specialized tasks for handling Avro data in BigQuery from GCS.\"\"\"\n\nimport logging\n\nfrom luigi.contrib.bigquery import"
  },
  {
    "path": "luigi/contrib/datadog_metric.py",
    "chars": 5430,
    "preview": "import logging\n\nfrom luigi import parameter\nfrom luigi.metrics import MetricsCollector\nfrom luigi.task import Config\n\nlo"
  },
  {
    "path": "luigi/contrib/dataproc.py",
    "chars": 9976,
    "preview": "\"\"\"luigi bindings for Google Dataproc on Google Cloud\"\"\"\n\nimport logging\nimport os\nimport time\n\nimport luigi\nfrom luigi."
  },
  {
    "path": "luigi/contrib/docker_runner.py",
    "chars": 8902,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2017 Open Targets\n#\n# Licensed under the Apache License, Version 2.0 (the \"License"
  },
  {
    "path": "luigi/contrib/dropbox.py",
    "chars": 11570,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright (c) 2019 Jose-Ignacio Riaño Chico\n#\n# Licensed under the Apache License, Version 2"
  },
  {
    "path": "luigi/contrib/ecs.py",
    "chars": 9535,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Outlier Bio, LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lic"
  },
  {
    "path": "luigi/contrib/esindex.py",
    "chars": 13672,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/external_daily_snapshot.py",
    "chars": 2380,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2017 Spotify AB.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\""
  },
  {
    "path": "luigi/contrib/external_program.py",
    "chars": 11159,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2016 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/ftp.py",
    "chars": 13140,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/gcp.py",
    "chars": 1410,
    "preview": "\"\"\"\nCommon code for GCP (google cloud services) integration\n\"\"\"\n\nimport logging\n\nlogger = logging.getLogger(\"luigi-inter"
  },
  {
    "path": "luigi/contrib/gcs.py",
    "chars": 18052,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Twitter Inc\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\""
  },
  {
    "path": "luigi/contrib/hadoop.py",
    "chars": 37054,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hadoop_jar.py",
    "chars": 5415,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/__init__.py",
    "chars": 2900,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/abstract_client.py",
    "chars": 2824,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/clients.py",
    "chars": 1924,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/config.py",
    "chars": 4005,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/error.py",
    "chars": 1029,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/format.py",
    "chars": 5975,
    "preview": "import logging\nimport os\n\nimport luigi.format\nfrom luigi.contrib.hdfs import config as hdfs_config\nfrom luigi.contrib.hd"
  },
  {
    "path": "luigi/contrib/hdfs/hadoopcli_clients.py",
    "chars": 9424,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hdfs/target.py",
    "chars": 7196,
    "preview": "#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use"
  },
  {
    "path": "luigi/contrib/hdfs/webhdfs_client.py",
    "chars": 6482,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 VNG Corporation\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/hive.py",
    "chars": 19129,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/kubernetes.py",
    "chars": 13674,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Outlier Bio, LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lic"
  },
  {
    "path": "luigi/contrib/lsf.py",
    "chars": 11980,
    "preview": "# -*- coding: utf-8 -*-\n\n\"\"\"\n.. Copyright 2012-2015 Spotify AB\n   Copyright 2018\n   Copyright 2018 EMBL-European Bioinfo"
  },
  {
    "path": "luigi/contrib/lsf_runner.py",
    "chars": 2301,
    "preview": "# -*- coding: utf-8 -*-\n\n\"\"\"\n.. Copyright 2012-2015 Spotify AB\n   Copyright 2018\n   Copyright 2018 EMBL-European Bioinfo"
  },
  {
    "path": "luigi/contrib/mongodb.py",
    "chars": 6400,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2017 Big Datext Inc\n#\n# Licensed under the Apache License, Version 2.0 (the \"Licen"
  },
  {
    "path": "luigi/contrib/mssqldb.py",
    "chars": 5295,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/mysqldb.py",
    "chars": 8325,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/opener.py",
    "chars": 7047,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/pai.py",
    "chars": 10923,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2017 Open Targets\n#\n# Licensed under the Apache License, Version 2.0 (the \"License"
  },
  {
    "path": "luigi/contrib/pig.py",
    "chars": 6517,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/postgres.py",
    "chars": 15874,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/presto.py",
    "chars": 7620,
    "preview": "import inspect\nimport logging\nimport re\nfrom collections import OrderedDict\nfrom contextlib import closing\nfrom enum imp"
  },
  {
    "path": "luigi/contrib/prometheus_metric.py",
    "chars": 3039,
    "preview": "from prometheus_client import CONTENT_TYPE_LATEST, CollectorRegistry, Counter, Gauge, generate_latest\n\nfrom luigi import"
  },
  {
    "path": "luigi/contrib/pyspark_runner.py",
    "chars": 3841,
    "preview": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2020 Spotify AB\n#\n# Licensed under the Apache License, "
  },
  {
    "path": "luigi/contrib/rdbms.py",
    "chars": 11125,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/redis_store.py",
    "chars": 3014,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/redshift.py",
    "chars": 24400,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/s3.py",
    "chars": 27135,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/salesforce.py",
    "chars": 26833,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/scalding.py",
    "chars": 10409,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/sge.py",
    "chars": 13385,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/sge_runner.py",
    "chars": 2833,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/simulate.py",
    "chars": 3407,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/spark.py",
    "chars": 12273,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/sparkey.py",
    "chars": 1970,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/sqla.py",
    "chars": 16184,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright (c) 2015 Gouthaman Balaraman\n#\n# Licensed under the Apache License, Version 2.0 (t"
  },
  {
    "path": "luigi/contrib/ssh.py",
    "chars": 12242,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/target.py",
    "chars": 2812,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/contrib/webhdfs.py",
    "chars": 2921,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/date_interval.py",
    "chars": 8608,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/db_task_history.py",
    "chars": 11397,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/event.py",
    "chars": 1757,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/execution_summary.py",
    "chars": 20201,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/format.py",
    "chars": 14427,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/freezing.py",
    "chars": 2138,
    "preview": "\"\"\"Internal-only module with immutable data structures.\n\nPlease, do not use it outside of Luigi codebase itself.\n\"\"\"\n\nfr"
  },
  {
    "path": "luigi/interface.py",
    "chars": 8540,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/local_target.py",
    "chars": 6185,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/lock.py",
    "chars": 5366,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/metrics.py",
    "chars": 2582,
    "preview": "import abc\nimport importlib\nfrom enum import Enum\n\n\nclass MetricsCollectors(Enum):\n    custom = -1\n    default = 1\n    n"
  },
  {
    "path": "luigi/mock.py",
    "chars": 5679,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/mypy.py",
    "chars": 20313,
    "preview": "\"\"\"Plugin that provides support for luigi.Task\n\nThis Code reuses the code from mypy.plugins.dataclasses\nhttps://github.c"
  },
  {
    "path": "luigi/notifications.py",
    "chars": 13833,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/parameter.py",
    "chars": 61330,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/process.py",
    "chars": 3446,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/py.typed",
    "chars": 0,
    "preview": ""
  },
  {
    "path": "luigi/retcodes.py",
    "chars": 4347,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/rpc.py",
    "chars": 6736,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/safe_extractor.py",
    "chars": 3567,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/scheduler.py",
    "chars": 65815,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/server.py",
    "chars": 14515,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/setup_logging.py",
    "chars": 5734,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Vote Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");"
  },
  {
    "path": "luigi/static/visualiser/css/luigi.css",
    "chars": 3413,
    "preview": ".nodeCircle {\n  stroke: #fff;\n  stroke-width: 1.5px;\n}\ntext {\n    font-size:8pt;\n}\n\n.link {\n  stroke: #999;\n  stroke-opa"
  },
  {
    "path": "luigi/static/visualiser/css/tipsy.css",
    "chars": 2162,
    "preview": ".tipsy { font-size: 10px; position: absolute; padding: 5px; z-index: 100000; }\n  .tipsy-inner { background-color: #000; "
  },
  {
    "path": "luigi/static/visualiser/index.html",
    "chars": 33256,
    "preview": "<!DOCTYPE html>\n<html lang=\"en\">\n    <head>\n        <title>Luigi Task Visualiser</title>\n        <link href=\"css/luigi.c"
  },
  {
    "path": "luigi/static/visualiser/js/graph.js",
    "chars": 12101,
    "preview": "Graph = (function() {\n    var statusColors = {\n        \"FAILED\":\"#DD0000\",\n        \"RUNNING\":\"#0044DD\",\n        \"BATCH_R"
  },
  {
    "path": "luigi/static/visualiser/js/luigi.js",
    "chars": 8076,
    "preview": "var LuigiAPI = (function() {\n    function LuigiAPI (urlRoot) {\n        this.urlRoot = urlRoot;\n    }\n\n    function flatt"
  },
  {
    "path": "luigi/static/visualiser/js/test/graph_test.js",
    "chars": 3969,
    "preview": "module(\"graph.js\");\n\ntest(\"nodeFromTask\", function() {\n    var task = {\n        deps: [\"B1\",\"C1\"],\n        taskId: \"A1\","
  },
  {
    "path": "luigi/static/visualiser/js/tipsy.js",
    "chars": 11097,
    "preview": "// tipsy, facebook style tooltips for jquery\n// version 1.0.0a\n// (c) 2008-2010 jason frame [jason@onehackoranother.com]"
  },
  {
    "path": "luigi/static/visualiser/js/util.js",
    "chars": 189,
    "preview": "function escapeHtml(unsafe) {\n  return unsafe\n    .replace(/&/g, \"&amp;\")\n    .replace(/</g, \"&lt;\")\n    .replace(/>/g, "
  },
  {
    "path": "luigi/static/visualiser/js/visualiserApp.js",
    "chars": 58572,
    "preview": "function visualiserApp(luigi) {\n    var templates = {};\n    var typingTimer = 0;\n    var dt; // DataTable instantiated i"
  },
  {
    "path": "luigi/static/visualiser/lib/URI/1.18.2/URI.js",
    "chars": 62764,
    "preview": "/*!\n * URI.js - Mutating URLs\n *\n * Version: 1.18.2\n *\n * Author: Rodney Rehm\n * Web: http://medialize.github.io/URI.js/"
  },
  {
    "path": "luigi/static/visualiser/lib/mustache.js",
    "chars": 14853,
    "preview": "/*!\n * mustache.js - Logic-less {{mustache}} templates with JavaScript\n * http://github.com/janl/mustache.js\n */\n\n/*glob"
  },
  {
    "path": "luigi/static/visualiser/mockdata/dep_graph",
    "chars": 1097,
    "preview": "{\n    \"response\": {\n        \"FactorTask(product=12)\": {\n            \"deps\": [\n                \"FactorTask(product=2)\",\n "
  },
  {
    "path": "luigi/static/visualiser/mockdata/fetch_error",
    "chars": 400,
    "preview": "{\n    \"response\": {\n        \"taskId\": \"FactorTask(product=2)\",\n        \"error\": \"Runtime error:\\nTraceback (most recent "
  },
  {
    "path": "luigi/static/visualiser/mockdata/task_list",
    "chars": 1097,
    "preview": "{\n    \"response\": {\n        \"FactorTask(product=12)\": {\n            \"deps\": [\n                \"FactorTask(product=2)\",\n "
  },
  {
    "path": "luigi/static/visualiser/test.html",
    "chars": 510,
    "preview": "<!DOCTYPE html>\n<html>\n    <head>\n        <title>Luigi Visualiser Tests</title>\n        <link rel=\"stylesheet\" href=\"htt"
  },
  {
    "path": "luigi/target.py",
    "chars": 12155,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/task.py",
    "chars": 36318,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/task_history.py",
    "chars": 1989,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/task_register.py",
    "chars": 8071,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/task_status.py",
    "chars": 892,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/templates/history.html",
    "chars": 4270,
    "preview": "<!--\n@Copyright 2015 Naver Corp.\n@Author Yeseul Park (yeseul.park@navercorp.com)\n-->\n\n<!-- Extend our site layout -->\n{%"
  },
  {
    "path": "luigi/templates/layout.html",
    "chars": 3033,
    "preview": "<!--\n@Copyright 2015 Naver Corp.\n@Author Yeseul Park (yeseul.park@navercorp.com)\n-->\n\n<!DOCTYPE html>\n<html lang=\"en\">\n "
  },
  {
    "path": "luigi/templates/menu.html",
    "chars": 553,
    "preview": "<!--\n@Copyright 2015 Naver Corp.\n@Author Yeseul Park (yeseul.park@navercorp.com)\n-->\n\n<!-- Extend our site layout -->\n{%"
  },
  {
    "path": "luigi/templates/recent.html",
    "chars": 730,
    "preview": "{% extends \"layout.html\" %}\n{% block content %}\n<h2>Luigi Task History</h2>\n<table class=\"table table-striped table-bord"
  },
  {
    "path": "luigi/templates/show.html",
    "chars": 1282,
    "preview": "{% extends \"layout.html\" %}\n{% block content %}\n<div class=\"row\">\n  <div class=\"span6\">\n    <h3>Info</h3>\n    <table cla"
  },
  {
    "path": "luigi/tools/__init__.py",
    "chars": 751,
    "preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2014 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License"
  },
  {
    "path": "luigi/tools/deps.py",
    "chars": 4689,
    "preview": "#!/usr/bin/env python\n\n\n# Finds all tasks and task outputs on the dependency paths from the given downstream task T\n# up"
  },
  {
    "path": "luigi/tools/deps_tree.py",
    "chars": 2532,
    "preview": "# -*- coding: utf-8 -*-\n\"\"\"\nThis module parses commands exactly the same as the luigi task runner. You must specify the "
  },
  {
    "path": "luigi/tools/luigi_grep.py",
    "chars": 2779,
    "preview": "#!/usr/bin/env python\n\nimport argparse\nimport json\nfrom collections import defaultdict\nfrom urllib.request import urlope"
  },
  {
    "path": "luigi/tools/range.py",
    "chars": 33506,
    "preview": "# -*- coding: utf-8 -*-\n# Copyright (c) 2014 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License"
  },
  {
    "path": "luigi/util.py",
    "chars": 16371,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "luigi/worker.py",
    "chars": 51163,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "pyproject.toml",
    "chars": 5090,
    "preview": "[build-system]\nrequires = ['hatchling', 'hatch-fancy-pypi-readme']\nbuild-backend = 'hatchling.build'\n\n[project]\nname = \""
  },
  {
    "path": "scripts/ci/conditional_tox.sh",
    "chars": 167,
    "preview": "#!/usr/bin/env bash\n\nset -ex\n\nENDENV=$(echo $TOXENV | tail -c 7)\nif [[ $ENDENV == gcloud ]]\nthen\n  [[ $DIDNT_CREATE_GCP_"
  },
  {
    "path": "scripts/ci/install_start_azurite.sh",
    "chars": 385,
    "preview": "#!/usr/bin/env bash\n\necho \"$DOCKERHUB_TOKEN\" | docker login -u spotifyci --password-stdin\n\ndocker pull mcr.microsoft.com"
  },
  {
    "path": "scripts/ci/setup_hadoop_env.sh",
    "chars": 2460,
    "preview": "#!/usr/bin/env bash\n\nHADOOP_DISTRO=${HADOOP_DISTRO:-\"hdp\"}\n\nONLY_DOWNLOAD=${ONLY_DOWNLOAD:-false}\nONLY_EXTRACT=${ONLY_EX"
  },
  {
    "path": "scripts/ci/stop_azurite.sh",
    "chars": 107,
    "preview": "#!/usr/bin/env bash\ndocker stop \"$(docker ps -q --filter ancestor=mcr.microsoft.com/azure-storage/azurite)\""
  },
  {
    "path": "test/_mysqldb_test.py",
    "chars": 1546,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "test/_test_ftp.py",
    "chars": 7293,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "test/auto_namespace_test/__init__.py",
    "chars": 51,
    "preview": "import luigi\n\nluigi.auto_namespace(scope=__name__)\n"
  },
  {
    "path": "test/auto_namespace_test/my_namespace_test.py",
    "chars": 368,
    "preview": "from helpers import LuigiTestCase\n\nimport luigi\n\n\nclass MyNamespaceTest(LuigiTestCase):\n    def test_auto_namespace_scop"
  },
  {
    "path": "test/batch_notifier_test.py",
    "chars": 19923,
    "preview": "# coding=utf-8\nimport unittest\nfrom smtplib import SMTPServerDisconnected\n\nimport mock\n\nimport luigi.batch_notifier\n\nBAT"
  },
  {
    "path": "test/choice_parameter_test.py",
    "chars": 2252,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "test/clone_test.py",
    "chars": 2277,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "test/cmdline_test.py",
    "chars": 14445,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "test/config_env_test.py",
    "chars": 3725,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Vote inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");"
  },
  {
    "path": "test/config_toml_test.py",
    "chars": 3011,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Vote inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");"
  },
  {
    "path": "test/conftest.py",
    "chars": 1339,
    "preview": "from typing import List\n\nimport pytest\n\nimport luigi.task_register\n\n\n@pytest.fixture(autouse=True)\ndef reset_luigi_regis"
  },
  {
    "path": "test/contrib/__init__.py",
    "chars": 0,
    "preview": ""
  },
  {
    "path": "test/contrib/_webhdfs_test.py",
    "chars": 2104,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  },
  {
    "path": "test/contrib/azureblob_test.py",
    "chars": 6413,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Microsoft Corporation\n#\n# Licensed under the Apache License, Version 2.0 (the"
  },
  {
    "path": "test/contrib/batch_test.py",
    "chars": 4124,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2018 Outlier Bio, LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lic"
  },
  {
    "path": "test/contrib/beam_dataflow_test.py",
    "chars": 12065,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2019 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\")"
  },
  {
    "path": "test/contrib/bigquery_avro_test.py",
    "chars": 1617,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2019 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\")"
  },
  {
    "path": "test/contrib/bigquery_gcloud_test.py",
    "chars": 20810,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2015 Twitter Inc\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\""
  },
  {
    "path": "test/contrib/bigquery_test.py",
    "chars": 6164,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2019 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\")"
  },
  {
    "path": "test/contrib/cascading_test.py",
    "chars": 2770,
    "preview": "# -*- coding: utf-8 -*-\n#\n# Copyright 2012-2015 Spotify AB\n#\n# Licensed under the Apache License, Version 2.0 (the \"Lice"
  }
]

// ... and 129 more files (download for full content)

About this extraction

This page contains the full source code of the spotify/luigi GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 329 files (2.5 MB), approximately 669.9k tokens, and a symbol index with 5364 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.

Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.

Copied to clipboard!