Repository: microsoft/qlib Branch: main Commit: 3097dcc9953b Files: 571 Total size: 3.1 MB Directory structure: gitextract_o4krgpl4/ ├── .commitlintrc.js ├── .deepsource.toml ├── .dockerignore ├── .github/ │ ├── ISSUE_TEMPLATE/ │ │ ├── bug-report.md │ │ ├── documentation.md │ │ ├── feature-request.md │ │ └── question.md │ ├── PULL_REQUEST_TEMPLATE.md │ ├── brew_install.sh │ ├── release-drafter.yml │ └── workflows/ │ ├── lint_title.yml │ ├── release.yml │ ├── stale.yml │ ├── test_qlib_from_pip.yml │ ├── test_qlib_from_source.yml │ └── test_qlib_from_source_slow.yml ├── .gitignore ├── .mypy.ini ├── .pre-commit-config.yaml ├── .pylintrc ├── .readthedocs.yaml ├── CHANGELOG.md ├── CHANGES.rst ├── CODE_OF_CONDUCT.md ├── Dockerfile ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.md ├── SECURITY.md ├── build_docker_image.sh ├── docs/ │ ├── FAQ/ │ │ └── FAQ.rst │ ├── Makefile │ ├── _static/ │ │ └── demo.sh │ ├── advanced/ │ │ ├── PIT.rst │ │ ├── alpha.rst │ │ ├── serial.rst │ │ ├── server.rst │ │ └── task_management.rst │ ├── changelog/ │ │ └── changelog.rst │ ├── component/ │ │ ├── data.rst │ │ ├── highfreq.rst │ │ ├── meta.rst │ │ ├── model.rst │ │ ├── online.rst │ │ ├── recorder.rst │ │ ├── report.rst │ │ ├── rl/ │ │ │ ├── framework.rst │ │ │ ├── guidance.rst │ │ │ ├── overall.rst │ │ │ ├── quickstart.rst │ │ │ └── toctree.rst │ │ ├── strategy.rst │ │ └── workflow.rst │ ├── conf.py │ ├── developer/ │ │ ├── code_standard_and_dev_guide.rst │ │ └── how_to_build_image.rst │ ├── hidden/ │ │ ├── client.rst │ │ ├── online.rst │ │ └── tuner.rst │ ├── index.rst │ ├── introduction/ │ │ ├── introduction.rst │ │ └── quick.rst │ ├── make.bat │ ├── reference/ │ │ └── api.rst │ ├── requirements.txt │ └── start/ │ ├── getdata.rst │ ├── initialization.rst │ ├── installation.rst │ └── integration.rst ├── examples/ │ ├── README.md │ ├── benchmarks/ │ │ ├── ADARNN/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_adarnn_Alpha360.yaml │ │ ├── ADD/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_add_Alpha360.yaml │ │ ├── ALSTM/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_alstm_Alpha158.yaml │ │ │ └── workflow_config_alstm_Alpha360.yaml │ │ ├── CatBoost/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_catboost_Alpha158.yaml │ │ │ ├── workflow_config_catboost_Alpha158_csi500.yaml │ │ │ ├── workflow_config_catboost_Alpha360.yaml │ │ │ └── workflow_config_catboost_Alpha360_csi500.yaml │ │ ├── DoubleEnsemble/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_doubleensemble_Alpha158.yaml │ │ │ ├── workflow_config_doubleensemble_Alpha158_csi500.yaml │ │ │ ├── workflow_config_doubleensemble_Alpha360.yaml │ │ │ ├── workflow_config_doubleensemble_Alpha360_csi500.yaml │ │ │ └── workflow_config_doubleensemble_early_stop_Alpha158.yaml │ │ ├── GATs/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_gats_Alpha158.yaml │ │ │ └── workflow_config_gats_Alpha360.yaml │ │ ├── GRU/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_gru_Alpha158.yaml │ │ │ └── workflow_config_gru_Alpha360.yaml │ │ ├── GeneralPtNN/ │ │ │ ├── README.md │ │ │ ├── workflow_config_gru.yaml │ │ │ ├── workflow_config_gru2mlp.yaml │ │ │ └── workflow_config_mlp.yaml │ │ ├── HIST/ │ │ │ ├── README.md │ │ │ ├── qlib_csi300_stock_index.npy │ │ │ ├── requirements.txt │ │ │ └── workflow_config_hist_Alpha360.yaml │ │ ├── IGMTF/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_igmtf_Alpha360.yaml │ │ ├── KRNN/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_krnn_Alpha360.yaml │ │ ├── LSTM/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_lstm_Alpha158.yaml │ │ │ └── workflow_config_lstm_Alpha360.yaml │ │ ├── LightGBM/ │ │ │ ├── README.md │ │ │ ├── features_resample_N.py │ │ │ ├── features_sample.py │ │ │ ├── multi_freq_handler.py │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_lightgbm_Alpha158.yaml │ │ │ ├── workflow_config_lightgbm_Alpha158_csi500.yaml │ │ │ ├── workflow_config_lightgbm_Alpha158_multi_freq.yaml │ │ │ ├── workflow_config_lightgbm_Alpha360.yaml │ │ │ ├── workflow_config_lightgbm_Alpha360_csi500.yaml │ │ │ ├── workflow_config_lightgbm_configurable_dataset.yaml │ │ │ └── workflow_config_lightgbm_multi_freq.yaml │ │ ├── Linear/ │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_linear_Alpha158.yaml │ │ │ ├── workflow_config_linear_Alpha158_csi500.yaml │ │ │ └── workflow_config_linear_Alpha158_multi_pass_bt.yaml │ │ ├── Localformer/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_localformer_Alpha158.yaml │ │ │ └── workflow_config_localformer_Alpha360.yaml │ │ ├── MLP/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_mlp_Alpha158.yaml │ │ │ ├── workflow_config_mlp_Alpha158_csi500.yaml │ │ │ ├── workflow_config_mlp_Alpha360.yaml │ │ │ └── workflow_config_mlp_Alpha360_csi500.yaml │ │ ├── README.md │ │ ├── SFM/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_sfm_Alpha360.yaml │ │ ├── Sandwich/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_sandwich_Alpha360.yaml │ │ ├── TCN/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_tcn_Alpha158.yaml │ │ │ └── workflow_config_tcn_Alpha360.yaml │ │ ├── TCTS/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ └── workflow_config_tcts_Alpha360.yaml │ │ ├── TFT/ │ │ │ ├── README.md │ │ │ ├── data_formatters/ │ │ │ │ ├── __init__.py │ │ │ │ ├── base.py │ │ │ │ └── qlib_Alpha158.py │ │ │ ├── expt_settings/ │ │ │ │ ├── __init__.py │ │ │ │ └── configs.py │ │ │ ├── libs/ │ │ │ │ ├── __init__.py │ │ │ │ ├── hyperparam_opt.py │ │ │ │ ├── tft_model.py │ │ │ │ └── utils.py │ │ │ ├── requirements.txt │ │ │ ├── tft.py │ │ │ └── workflow_config_tft_Alpha158.yaml │ │ ├── TRA/ │ │ │ ├── README.md │ │ │ ├── configs/ │ │ │ │ ├── config_alstm.yaml │ │ │ │ ├── config_alstm_tra.yaml │ │ │ │ ├── config_alstm_tra_init.yaml │ │ │ │ ├── config_transformer.yaml │ │ │ │ ├── config_transformer_tra.yaml │ │ │ │ └── config_transformer_tra_init.yaml │ │ │ ├── data/ │ │ │ │ └── README.md │ │ │ ├── example.py │ │ │ ├── requirements.txt │ │ │ ├── run.sh │ │ │ ├── src/ │ │ │ │ ├── dataset.py │ │ │ │ └── model.py │ │ │ ├── workflow_config_tra_Alpha158.yaml │ │ │ ├── workflow_config_tra_Alpha158_full.yaml │ │ │ └── workflow_config_tra_Alpha360.yaml │ │ ├── TabNet/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_TabNet_Alpha158.yaml │ │ │ └── workflow_config_TabNet_Alpha360.yaml │ │ ├── Transformer/ │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── workflow_config_transformer_Alpha158.yaml │ │ │ └── workflow_config_transformer_Alpha360.yaml │ │ └── XGBoost/ │ │ ├── README.md │ │ ├── requirements.txt │ │ ├── workflow_config_xgboost_Alpha158.yaml │ │ └── workflow_config_xgboost_Alpha360.yaml │ ├── benchmarks_dynamic/ │ │ ├── DDG-DA/ │ │ │ ├── Makefile │ │ │ ├── README.md │ │ │ ├── requirements.txt │ │ │ ├── vis_data.py │ │ │ └── workflow.py │ │ ├── README.md │ │ └── baseline/ │ │ ├── README.md │ │ ├── rolling_benchmark.py │ │ ├── workflow_config_lightgbm_Alpha158.yaml │ │ └── workflow_config_linear_Alpha158.yaml │ ├── data_demo/ │ │ ├── README.md │ │ ├── data_cache_demo.py │ │ └── data_mem_resuse_demo.py │ ├── highfreq/ │ │ ├── README.md │ │ ├── highfreq_handler.py │ │ ├── highfreq_ops.py │ │ ├── highfreq_processor.py │ │ ├── workflow.py │ │ └── workflow_config_High_Freq_Tree_Alpha158.yaml │ ├── hyperparameter/ │ │ └── LightGBM/ │ │ ├── Readme.md │ │ ├── hyperparameter_158.py │ │ ├── hyperparameter_360.py │ │ └── requirements.txt │ ├── model_interpreter/ │ │ └── feature.py │ ├── model_rolling/ │ │ ├── requirements.txt │ │ └── task_manager_rolling.py │ ├── nested_decision_execution/ │ │ ├── README.md │ │ └── workflow.py │ ├── online_srv/ │ │ ├── online_management_simulate.py │ │ ├── rolling_online_management.py │ │ └── update_online_pred.py │ ├── orderbook_data/ │ │ ├── README.md │ │ ├── create_dataset.py │ │ └── example.py │ ├── portfolio/ │ │ ├── README.md │ │ ├── config_enhanced_indexing.yaml │ │ └── prepare_riskdata.py │ ├── rl_order_execution/ │ │ ├── README.md │ │ ├── exp_configs/ │ │ │ ├── backtest_opds.yml │ │ │ ├── backtest_ppo.yml │ │ │ ├── backtest_twap.yml │ │ │ ├── train_opds.yml │ │ │ └── train_ppo.yml │ │ └── scripts/ │ │ ├── gen_pickle_data.py │ │ ├── gen_training_orders.py │ │ ├── merge_orders.py │ │ └── pickle_data_config.yml │ ├── rolling_process_data/ │ │ ├── README.md │ │ ├── rolling_handler.py │ │ └── workflow.py │ ├── run_all_model.py │ └── workflow_by_code.py ├── pyproject.toml ├── qlib/ │ ├── __init__.py │ ├── backtest/ │ │ ├── __init__.py │ │ ├── account.py │ │ ├── backtest.py │ │ ├── decision.py │ │ ├── exchange.py │ │ ├── executor.py │ │ ├── high_performance_ds.py │ │ ├── position.py │ │ ├── profit_attribution.py │ │ ├── report.py │ │ ├── signal.py │ │ └── utils.py │ ├── cli/ │ │ ├── __init__.py │ │ ├── data.py │ │ └── run.py │ ├── config.py │ ├── constant.py │ ├── contrib/ │ │ ├── __init__.py │ │ ├── data/ │ │ │ ├── __init__.py │ │ │ ├── data.py │ │ │ ├── dataset.py │ │ │ ├── handler.py │ │ │ ├── highfreq_handler.py │ │ │ ├── highfreq_processor.py │ │ │ ├── highfreq_provider.py │ │ │ ├── loader.py │ │ │ ├── processor.py │ │ │ └── utils/ │ │ │ ├── __init__.py │ │ │ └── sepdf.py │ │ ├── eva/ │ │ │ ├── __init__.py │ │ │ └── alpha.py │ │ ├── evaluate.py │ │ ├── evaluate_portfolio.py │ │ ├── meta/ │ │ │ ├── __init__.py │ │ │ └── data_selection/ │ │ │ ├── __init__.py │ │ │ ├── dataset.py │ │ │ ├── model.py │ │ │ ├── net.py │ │ │ └── utils.py │ │ ├── model/ │ │ │ ├── __init__.py │ │ │ ├── catboost_model.py │ │ │ ├── double_ensemble.py │ │ │ ├── gbdt.py │ │ │ ├── highfreq_gdbt_model.py │ │ │ ├── linear.py │ │ │ ├── pytorch_adarnn.py │ │ │ ├── pytorch_add.py │ │ │ ├── pytorch_alstm.py │ │ │ ├── pytorch_alstm_ts.py │ │ │ ├── pytorch_gats.py │ │ │ ├── pytorch_gats_ts.py │ │ │ ├── pytorch_general_nn.py │ │ │ ├── pytorch_gru.py │ │ │ ├── pytorch_gru_ts.py │ │ │ ├── pytorch_hist.py │ │ │ ├── pytorch_igmtf.py │ │ │ ├── pytorch_krnn.py │ │ │ ├── pytorch_localformer.py │ │ │ ├── pytorch_localformer_ts.py │ │ │ ├── pytorch_lstm.py │ │ │ ├── pytorch_lstm_ts.py │ │ │ ├── pytorch_nn.py │ │ │ ├── pytorch_sandwich.py │ │ │ ├── pytorch_sfm.py │ │ │ ├── pytorch_tabnet.py │ │ │ ├── pytorch_tcn.py │ │ │ ├── pytorch_tcn_ts.py │ │ │ ├── pytorch_tcts.py │ │ │ ├── pytorch_tra.py │ │ │ ├── pytorch_transformer.py │ │ │ ├── pytorch_transformer_ts.py │ │ │ ├── pytorch_utils.py │ │ │ ├── tcn.py │ │ │ └── xgboost.py │ │ ├── online/ │ │ │ ├── __init__.py │ │ │ ├── manager.py │ │ │ ├── online_model.py │ │ │ ├── operator.py │ │ │ ├── user.py │ │ │ └── utils.py │ │ ├── ops/ │ │ │ ├── __init__.py │ │ │ └── high_freq.py │ │ ├── report/ │ │ │ ├── __init__.py │ │ │ ├── analysis_model/ │ │ │ │ ├── __init__.py │ │ │ │ └── analysis_model_performance.py │ │ │ ├── analysis_position/ │ │ │ │ ├── __init__.py │ │ │ │ ├── cumulative_return.py │ │ │ │ ├── parse_position.py │ │ │ │ ├── rank_label.py │ │ │ │ ├── report.py │ │ │ │ ├── risk_analysis.py │ │ │ │ └── score_ic.py │ │ │ ├── data/ │ │ │ │ ├── __init__.py │ │ │ │ ├── ana.py │ │ │ │ └── base.py │ │ │ ├── graph.py │ │ │ └── utils.py │ │ ├── rolling/ │ │ │ ├── __init__.py │ │ │ ├── __main__.py │ │ │ ├── base.py │ │ │ └── ddgda.py │ │ ├── strategy/ │ │ │ ├── __init__.py │ │ │ ├── cost_control.py │ │ │ ├── optimizer/ │ │ │ │ ├── __init__.py │ │ │ │ ├── base.py │ │ │ │ ├── enhanced_indexing.py │ │ │ │ └── optimizer.py │ │ │ ├── order_generator.py │ │ │ ├── rule_strategy.py │ │ │ └── signal_strategy.py │ │ ├── torch.py │ │ ├── tuner/ │ │ │ ├── __init__.py │ │ │ ├── config.py │ │ │ ├── launcher.py │ │ │ ├── pipeline.py │ │ │ ├── space.py │ │ │ └── tuner.py │ │ └── workflow/ │ │ ├── __init__.py │ │ └── record_temp.py │ ├── data/ │ │ ├── __init__.py │ │ ├── _libs/ │ │ │ ├── __init__.py │ │ │ ├── expanding.pyx │ │ │ └── rolling.pyx │ │ ├── base.py │ │ ├── cache.py │ │ ├── client.py │ │ ├── data.py │ │ ├── dataset/ │ │ │ ├── __init__.py │ │ │ ├── handler.py │ │ │ ├── loader.py │ │ │ ├── processor.py │ │ │ ├── storage.py │ │ │ ├── utils.py │ │ │ └── weight.py │ │ ├── filter.py │ │ ├── inst_processor.py │ │ ├── ops.py │ │ ├── pit.py │ │ └── storage/ │ │ ├── __init__.py │ │ ├── file_storage.py │ │ └── storage.py │ ├── log.py │ ├── model/ │ │ ├── __init__.py │ │ ├── base.py │ │ ├── ens/ │ │ │ ├── __init__.py │ │ │ ├── ensemble.py │ │ │ └── group.py │ │ ├── interpret/ │ │ │ ├── __init__.py │ │ │ └── base.py │ │ ├── meta/ │ │ │ ├── __init__.py │ │ │ ├── dataset.py │ │ │ ├── model.py │ │ │ └── task.py │ │ ├── riskmodel/ │ │ │ ├── __init__.py │ │ │ ├── base.py │ │ │ ├── poet.py │ │ │ ├── shrink.py │ │ │ └── structured.py │ │ ├── trainer.py │ │ └── utils.py │ ├── rl/ │ │ ├── __init__.py │ │ ├── aux_info.py │ │ ├── contrib/ │ │ │ ├── __init__.py │ │ │ ├── backtest.py │ │ │ ├── naive_config_parser.py │ │ │ ├── train_onpolicy.py │ │ │ └── utils.py │ │ ├── data/ │ │ │ ├── __init__.py │ │ │ ├── base.py │ │ │ ├── integration.py │ │ │ ├── native.py │ │ │ └── pickle_styled.py │ │ ├── interpreter.py │ │ ├── order_execution/ │ │ │ ├── __init__.py │ │ │ ├── interpreter.py │ │ │ ├── network.py │ │ │ ├── policy.py │ │ │ ├── reward.py │ │ │ ├── simulator_qlib.py │ │ │ ├── simulator_simple.py │ │ │ ├── state.py │ │ │ ├── strategy.py │ │ │ └── utils.py │ │ ├── reward.py │ │ ├── seed.py │ │ ├── simulator.py │ │ ├── strategy/ │ │ │ ├── __init__.py │ │ │ └── single_order.py │ │ ├── trainer/ │ │ │ ├── __init__.py │ │ │ ├── api.py │ │ │ ├── callbacks.py │ │ │ ├── trainer.py │ │ │ └── vessel.py │ │ └── utils/ │ │ ├── __init__.py │ │ ├── data_queue.py │ │ ├── env_wrapper.py │ │ ├── finite_env.py │ │ └── log.py │ ├── strategy/ │ │ ├── __init__.py │ │ └── base.py │ ├── tests/ │ │ ├── __init__.py │ │ ├── config.py │ │ └── data.py │ ├── typehint.py │ ├── utils/ │ │ ├── __init__.py │ │ ├── data.py │ │ ├── exceptions.py │ │ ├── file.py │ │ ├── index_data.py │ │ ├── mod.py │ │ ├── objm.py │ │ ├── paral.py │ │ ├── pickle_utils.py │ │ ├── resam.py │ │ ├── serial.py │ │ └── time.py │ └── workflow/ │ ├── __init__.py │ ├── exp.py │ ├── expm.py │ ├── online/ │ │ ├── __init__.py │ │ ├── manager.py │ │ ├── strategy.py │ │ ├── update.py │ │ └── utils.py │ ├── record_temp.py │ ├── recorder.py │ ├── task/ │ │ ├── __init__.py │ │ ├── collect.py │ │ ├── gen.py │ │ ├── manage.py │ │ └── utils.py │ └── utils.py ├── scripts/ │ ├── README.md │ ├── check_data_health.py │ ├── check_dump_bin.py │ ├── collect_info.py │ ├── data_collector/ │ │ ├── README.md │ │ ├── baostock_5min/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirements.txt │ │ ├── base.py │ │ ├── br_index/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirements.txt │ │ ├── cn_index/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirements.txt │ │ ├── contrib/ │ │ │ ├── fill_cn_1min_data/ │ │ │ │ ├── README.md │ │ │ │ ├── fill_cn_1min_data.py │ │ │ │ └── requirements.txt │ │ │ └── future_trading_date_collector/ │ │ │ ├── README.md │ │ │ ├── future_trading_date_collector.py │ │ │ └── requirements.txt │ │ ├── crowd_source/ │ │ │ └── README.md │ │ ├── crypto/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirement.txt │ │ ├── fund/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirements.txt │ │ ├── future_calendar_collector.py │ │ ├── index.py │ │ ├── pit/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirements.txt │ │ ├── us_index/ │ │ │ ├── README.md │ │ │ ├── collector.py │ │ │ └── requirements.txt │ │ ├── utils.py │ │ └── yahoo/ │ │ ├── README.md │ │ ├── collector.py │ │ └── requirements.txt │ ├── dump_bin.py │ ├── dump_pit.py │ └── get_data.py ├── setup.py └── tests/ ├── backtest/ │ ├── test_file_strategy.py │ ├── test_high_freq_trading.py │ ├── test_soft_topk_strategy.py │ └── test_soft_topk_strategy_cold_start.py ├── conftest.py ├── data_mid_layer_tests/ │ ├── README.md │ ├── test_dataloader.py │ ├── test_dataset.py │ ├── test_handler.py │ ├── test_handler_storage.py │ └── test_processor.py ├── dataset_tests/ │ ├── README.md │ └── test_datalayer.py ├── dependency_tests/ │ ├── README.md │ └── test_mlflow.py ├── misc/ │ ├── test_get_multi_proc.py │ ├── test_index_data.py │ ├── test_sepdf.py │ └── test_utils.py ├── model/ │ └── test_general_nn.py ├── ops/ │ ├── test_elem_operator.py │ └── test_special_ops.py ├── pytest.ini ├── rl/ │ ├── test_data_queue.py │ ├── test_finite_env.py │ ├── test_logger.py │ ├── test_qlib_simulator.py │ ├── test_saoe_simple.py │ └── test_trainer.py ├── rolling_tests/ │ └── test_update_pred.py ├── storage_tests/ │ └── test_storage.py ├── test_all_pipeline.py ├── test_contrib_model.py ├── test_contrib_workflow.py ├── test_dump_data.py ├── test_get_data.py ├── test_pit.py ├── test_register_ops.py ├── test_structured_cov_estimator.py └── test_workflow.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .commitlintrc.js ================================================ module.exports = { extends: ["@commitlint/config-conventional"], rules: { // Configuration Format: [level, applicability, value] // level: Error level, usually expressed as a number: // 0 - disable rule // 1 - Warning (does not prevent commits) // 2 - Error (will block the commit) // applicability: the conditions under which the rule applies, commonly used values: // “always” - always apply the rule // “never” - never apply the rule // value: the specific value of the rule, e.g. a maximum length of 100. // Refs: https://commitlint.js.org/reference/rules-configuration.html "header-max-length": [2, "always", 100], "type-enum": [ 2, "always", ["build", "chore", "ci", "docs", "feat", "fix", "perf", "refactor", "revert", "style", "test", "Release-As"] ] } }; ================================================ FILE: .deepsource.toml ================================================ version = 1 test_patterns = ["tests/test_*.py"] exclude_patterns = ["examples/**"] [[analyzers]] name = "python" enabled = true [analyzers.meta] runtime_version = "3.x.x" ================================================ FILE: .dockerignore ================================================ __pycache__ *.pyc *.pyo *.pyd .Python .env .git ================================================ FILE: .github/ISSUE_TEMPLATE/bug-report.md ================================================ --- name: "\U0001F41B Bug Report" about: Submit a bug report to help us improve Qlib labels: bug --- ## 🐛 Bug Description ## To Reproduce Steps to reproduce the behavior: 1. 1. 1. ## Expected Behavior ## Screenshot ## Environment **Note**: User could run `cd scripts && python collect_info.py all` under project directory to get system information and paste them here directly. - Qlib version: - Python version: - OS (`Windows`, `Linux`, `MacOS`): - Commit number (optional, please provide it if you are using the dev version): ## Additional Notes ================================================ FILE: .github/ISSUE_TEMPLATE/documentation.md ================================================ --- name: "\U0001F4D6 Documentation" about: Report an issue related to documentation --- ## 📖 Documentation ================================================ FILE: .github/ISSUE_TEMPLATE/feature-request.md ================================================ --- name: "\U0001F31FFeature Request" about: Request for a new Qlib feature labels: enhancement --- ## 🌟 Feature Description ## Motivation 1. Application scenario 2. Related works (Papers, Github repos etc.): 3. Any other relevant and important information: ## Alternatives ## Additional Notes ================================================ FILE: .github/ISSUE_TEMPLATE/question.md ================================================ --- name: "❓Questions & Help" about: Have some questions? We can offer help. labels: question --- ## ❓ Questions and Help We sincerely suggest you to carefully read the [documentation](http://qlib.readthedocs.io/) of our library as well as the official [paper](https://arxiv.org/abs/2009.11189). After that, if you still feel puzzled, please describe the question clearly under this issue. ================================================ FILE: .github/PULL_REQUEST_TEMPLATE.md ================================================ ## Description ## Motivation and Context ## How Has This Been Tested? - [ ] Pass the test by running: `pytest qlib/tests/test_all_pipeline.py` under upper directory of `qlib`. - [ ] If you are adding a new feature, test on your own test scripts. ## Screenshots of Test Results (if appropriate): 1. Pipeline test: 2. Your own tests: ## Types of changes - [ ] Fix bugs - [ ] Add new feature - [ ] Update documentation ================================================ FILE: .github/brew_install.sh ================================================ #!/bin/bash set -u # First check if the OS is Linux. if [[ "$(uname)" = "Linux" ]]; then HOMEBREW_ON_LINUX=1 fi # On macOS, this script installs to /usr/local only. # On Linux, it installs to /home/linuxbrew/.linuxbrew if you have sudo access # and ~/.linuxbrew otherwise. # To install elsewhere (which is unsupported) # you can untar https://github.com/Homebrew/brew/tarball/master # anywhere you like. if [[ -z "${HOMEBREW_ON_LINUX-}" ]]; then HOMEBREW_PREFIX="/usr/local" HOMEBREW_REPOSITORY="/usr/local/Homebrew" HOMEBREW_CACHE="${HOME}/Library/Caches/Homebrew" STAT="stat -f" CHOWN="/usr/sbin/chown" CHGRP="/usr/bin/chgrp" GROUP="admin" TOUCH="/usr/bin/touch" else HOMEBREW_PREFIX_DEFAULT="/home/linuxbrew/.linuxbrew" HOMEBREW_CACHE="${HOME}/.cache/Homebrew" STAT="stat --printf" CHOWN="/bin/chown" CHGRP="/bin/chgrp" GROUP="$(id -gn)" TOUCH="/bin/touch" fi BREW_REPO="https://github.com/Homebrew/brew" # TODO: bump version when new macOS is released MACOS_LATEST_SUPPORTED="10.15" # TODO: bump version when new macOS is released MACOS_OLDEST_SUPPORTED="10.13" # For Homebrew on Linux REQUIRED_RUBY_VERSION=2.6 # https://github.com/Homebrew/brew/pull/6556 REQUIRED_GLIBC_VERSION=2.13 # https://docs.brew.sh/Homebrew-on-Linux#requirements # no analytics during installation export HOMEBREW_NO_ANALYTICS_THIS_RUN=1 export HOMEBREW_NO_ANALYTICS_MESSAGE_OUTPUT=1 # string formatters if [[ -t 1 ]]; then tty_escape() { printf "\033[%sm" "$1"; } else tty_escape() { :; } fi tty_mkbold() { tty_escape "1;$1"; } tty_underline="$(tty_escape "4;39")" tty_blue="$(tty_mkbold 34)" tty_red="$(tty_mkbold 31)" tty_bold="$(tty_mkbold 39)" tty_reset="$(tty_escape 0)" have_sudo_access() { local -a args if [[ -n "${SUDO_ASKPASS-}" ]]; then args=("-A") fi if [[ -z "${HAVE_SUDO_ACCESS-}" ]]; then if [[ -n "${args[*]-}" ]]; then /usr/bin/sudo "${args[@]}" -l mkdir &>/dev/null else /usr/bin/sudo -l mkdir &>/dev/null fi HAVE_SUDO_ACCESS="$?" fi if [[ -z "${HOMEBREW_ON_LINUX-}" ]] && [[ "$HAVE_SUDO_ACCESS" -ne 0 ]]; then abort "Need sudo access on macOS (e.g. the user $USER to be an Administrator)!" fi return "$HAVE_SUDO_ACCESS" } shell_join() { local arg printf "%s" "$1" shift for arg in "$@"; do printf " " printf "%s" "${arg// /\ }" done } chomp() { printf "%s" "${1/"$'\n'"/}" } ohai() { printf "${tty_blue}==>${tty_bold} %s${tty_reset}\n" "$(shell_join "$@")" } warn() { printf "${tty_red}Warning${tty_reset}: %s\n" "$(chomp "$1")" } abort() { printf "%s\n" "$1" exit 1 } execute() { if ! "$@"; then abort "$(printf "Failed during: %s" "$(shell_join "$@")")" fi } execute_sudo() { local -a args=("$@") if [[ -n "${SUDO_ASKPASS-}" ]]; then args=("-A" "${args[@]}") fi if have_sudo_access; then ohai "/usr/bin/sudo" "${args[@]}" execute "/usr/bin/sudo" "${args[@]}" else ohai "${args[@]}" execute "${args[@]}" fi } getc() { local save_state save_state=$(/bin/stty -g) /bin/stty raw -echo IFS= read -r -n 1 -d '' "$@" /bin/stty "$save_state" } wait_for_user() { local c echo echo "Press RETURN to continue or any other key to abort" getc c # we test for \r and \n because some stuff does \r instead if ! [[ "$c" == $'\r' || "$c" == $'\n' ]]; then exit 1 fi } major_minor() { echo "${1%%.*}.$(x="${1#*.}"; echo "${x%%.*}")" } if [[ -z "${HOMEBREW_ON_LINUX-}" ]]; then macos_version="$(major_minor "$(/usr/bin/sw_vers -productVersion)")" fi version_gt() { [[ "${1%.*}" -gt "${2%.*}" ]] || [[ "${1%.*}" -eq "${2%.*}" && "${1#*.}" -gt "${2#*.}" ]] } version_ge() { [[ "${1%.*}" -gt "${2%.*}" ]] || [[ "${1%.*}" -eq "${2%.*}" && "${1#*.}" -ge "${2#*.}" ]] } version_lt() { [[ "${1%.*}" -lt "${2%.*}" ]] || [[ "${1%.*}" -eq "${2%.*}" && "${1#*.}" -lt "${2#*.}" ]] } should_install_command_line_tools() { if [[ -n "${HOMEBREW_ON_LINUX-}" ]]; then return 1 fi if version_gt "$macos_version" "10.13"; then ! [[ -e "/Library/Developer/CommandLineTools/usr/bin/git" ]] else ! [[ -e "/Library/Developer/CommandLineTools/usr/bin/git" ]] || ! [[ -e "/usr/include/iconv.h" ]] fi } get_permission() { $STAT "%A" "$1" } user_only_chmod() { [[ -d "$1" ]] && [[ "$(get_permission "$1")" != "755" ]] } exists_but_not_writable() { [[ -e "$1" ]] && ! [[ -r "$1" && -w "$1" && -x "$1" ]] } get_owner() { $STAT "%u" "$1" } file_not_owned() { [[ "$(get_owner "$1")" != "$(id -u)" ]] } get_group() { $STAT "%g" "$1" } file_not_grpowned() { [[ " $(id -G "$USER") " != *" $(get_group "$1") "* ]] } # Please sync with 'test_ruby()' in 'Library/Homebrew/utils/ruby.sh' from Homebrew/brew repository. test_ruby () { if [[ ! -x $1 ]] then return 1 fi "$1" --enable-frozen-string-literal --disable=gems,did_you_mean,rubyopt -rrubygems -e \ "abort if Gem::Version.new(RUBY_VERSION.to_s.dup).to_s.split('.').first(2) != \ Gem::Version.new('$REQUIRED_RUBY_VERSION').to_s.split('.').first(2)" 2>/dev/null } no_usable_ruby() { local ruby_exec IFS=$'\n' # Do word splitting on new lines only for ruby_exec in $(which -a ruby); do if test_ruby "$ruby_exec"; then return 1 fi done IFS=$' \t\n' # Restore IFS to its default value return 0 } outdated_glibc() { local glibc_version glibc_version=$(ldd --version | head -n1 | grep -o '[0-9.]*$' | grep -o '^[0-9]\+\.[0-9]\+') version_lt "$glibc_version" "$REQUIRED_GLIBC_VERSION" } if [[ -n "${HOMEBREW_ON_LINUX-}" ]] && no_usable_ruby && outdated_glibc then abort "$(cat <<-EOFABORT Homebrew requires Ruby $REQUIRED_RUBY_VERSION which was not found on your system. Homebrew portable Ruby requires Glibc version $REQUIRED_GLIBC_VERSION or newer, and your Glibc version is too old. See ${tty_underline}https://docs.brew.sh/Homebrew-on-Linux#requirements${tty_reset} Install Ruby $REQUIRED_RUBY_VERSION and add its location to your PATH. EOFABORT )" fi # USER isn't always set so provide a fall back for the installer and subprocesses. if [[ -z "${USER-}" ]]; then USER="$(chomp "$(id -un)")" export USER fi # Invalidate sudo timestamp before exiting (if it wasn't active before). if ! /usr/bin/sudo -n -v 2>/dev/null; then trap '/usr/bin/sudo -k' EXIT fi # Things can fail later if `pwd` doesn't exist. # Also sudo prints a warning message for no good reason cd "/usr" || exit 1 ####################################################################### script if ! command -v git >/dev/null; then abort "$(cat </dev/null; then abort "$(cat <&1) != *"mkdir"* ]]; then ohai "Select the Homebrew installation directory" echo "- ${tty_bold}Enter your password${tty_reset} to install to ${tty_underline}${HOMEBREW_PREFIX_DEFAULT}${tty_reset} (${tty_bold}recommended${tty_reset})" echo "- ${tty_bold}Press Control-D${tty_reset} to install to ${tty_underline}$HOME/.linuxbrew${tty_reset}" echo "- ${tty_bold}Press Control-C${tty_reset} to cancel installation" fi if have_sudo_access; then HOMEBREW_PREFIX="$HOMEBREW_PREFIX_DEFAULT" else HOMEBREW_PREFIX="$HOME/.linuxbrew" fi trap - SIGINT fi HOMEBREW_REPOSITORY="${HOMEBREW_PREFIX}/Homebrew" fi if [[ "$UID" == "0" ]]; then abort "Don't run this as root!" elif [[ -d "$HOMEBREW_PREFIX" && ! -x "$HOMEBREW_PREFIX" ]]; then abort "$(cat </dev/null || return # we do it in four steps to avoid merge errors when reinstalling execute "git" "init" "-q" # "git remote add" will fail if the remote is defined in the global config execute "git" "config" "remote.origin.url" "${BREW_REPO}" execute "git" "config" "remote.origin.fetch" "+refs/heads/*:refs/remotes/origin/*" # ensure we don't munge line endings on checkout execute "git" "config" "core.autocrlf" "false" execute "git" "fetch" "origin" "--force" execute "git" "fetch" "origin" "--tags" "--force" execute "git" "reset" "--hard" "origin/master" execute "ln" "-sf" "${HOMEBREW_REPOSITORY}/bin/brew" "${HOMEBREW_PREFIX}/bin/brew" ) || exit 1 if [[ ":${PATH}:" != *":${HOMEBREW_PREFIX}/bin:"* ]]; then warn "${HOMEBREW_PREFIX}/bin is not in your PATH." fi ohai "Installation successful!" echo # Use the shell's audible bell. if [[ -t 1 ]]; then printf "\a" fi # Use an extra newline and bold to avoid this being missed. ohai "Homebrew has enabled anonymous aggregate formulae and cask analytics." echo "$(cat </dev/null || return execute "git" "config" "--replace-all" "homebrew.analyticsmessage" "true" execute "git" "config" "--replace-all" "homebrew.caskanalyticsmessage" "true" ) || exit 1 ohai "Next steps:" echo "- Run \`brew help\` to get started" echo "- Further documentation: " echo " ${tty_underline}https://docs.brew.sh${tty_reset}" if [[ -n "${HOMEBREW_ON_LINUX-}" ]]; then case "$SHELL" in */bash*) if [[ -r "$HOME/.bash_profile" ]]; then shell_profile="$HOME/.bash_profile" else shell_profile="$HOME/.profile" fi ;; */zsh*) shell_profile="$HOME/.zprofile" ;; *) shell_profile="$HOME/.profile" ;; esac echo "- Install the Homebrew dependencies if you have sudo access:" if [[ $(command -v apt-get) ]]; then echo " sudo apt-get install build-essential" elif [[ $(command -v yum) ]]; then echo " sudo yum groupinstall 'Development Tools'" elif [[ $(command -v pacman) ]]; then echo " sudo pacman -S base-devel" elif [[ $(command -v apk) ]]; then echo " sudo apk add build-base" fi cat <> ${shell_profile} eval \$(${HOMEBREW_PREFIX}/bin/brew shellenv) - We recommend that you install GCC: brew install gcc EOS fi ================================================ FILE: .github/release-drafter.yml ================================================ name-template: 'v$RESOLVED_VERSION 🌈' tag-template: 'v$RESOLVED_VERSION' categories: - title: '🌟 Features' labels: - 'feature' - 'enhancement' - title: '🐛 Bug Fixes' labels: - 'fix' - 'bugfix' - 'bug' - title: '📚 Documentation' label: - 'doc' - 'documentation' - title: '🧹 Maintenance' label: - 'maintenance' change-template: '- $TITLE @$AUTHOR (#$NUMBER)' change-title-escapes: '\<*_&' # You can add # and @ to disable mentions, and add ` to disable code blocks. version-resolver: major: labels: - 'major' minor: labels: - 'minor' patch: labels: - 'patch' default: patch template: | ## Changes $CHANGES ================================================ FILE: .github/workflows/lint_title.yml ================================================ name: Lint pull request title on: pull_request: types: - opened - synchronize - reopened - edited concurrency: cancel-in-progress: true group: ${{ github.workflow }}-${{ github.ref }} jobs: lint-title: runs-on: ubuntu-latest steps: # This step is necessary because the lint title uses the .commitlintrc.js file in the project root directory. - name: Checkout Repository uses: actions/checkout@v4 - name: Setup Node.js uses: actions/setup-node@v4 with: node-version: '16' - name: Install commitlint run: npm install --save-dev @commitlint/{config-conventional,cli} - name: Validate PR Title with commitlint env: BODY: ${{ github.event.pull_request.title }} run: | echo "$BODY" | npx commitlint --config .commitlintrc.js ================================================ FILE: .github/workflows/release.yml ================================================ name: Release on: push: branches: - main permissions: contents: read jobs: release: runs-on: ubuntu-latest outputs: release_created: ${{ steps.release_please.outputs.release_created }} steps: - name: Release please id: release_please uses: googleapis/release-please-action@v4 with: token: ${{ secrets.PAT }} release-type: simple deploy_with_manylinux: needs: release permissions: contents: write pull-requests: read runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 if: needs.release.outputs.release_created == 'true' with: fetch-depth: 0 - name: Set up Python ${{ matrix.python-version }} if: needs.release.outputs.release_created == 'true' uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Build wheel on Linux if: needs.release.outputs.release_created == 'true' uses: RalfG/python-wheels-manylinux-build@v0.7.1-manylinux2014_x86_64 with: python-versions: 'cp38-cp38 cp39-cp39 cp310-cp310 cp311-cp311 cp312-cp312' build-requirements: 'numpy cython' - name: Install dependencies if: needs.release.outputs.release_created == 'true' run: | python -m pip install twine - name: Upload to PyPi if: needs.release.outputs.release_created == 'true' env: TWINE_USERNAME: __token__ TWINE_PASSWORD: ${{ secrets.TESTPYPI }} run: | twine check dist/pyqlib-*-manylinux*.whl twine upload --repository-url https://test.pypi.org/legacy/ dist/pyqlib-*-manylinux*.whl --verbose deploy_with_bdist_wheel: needs: release runs-on: ${{ matrix.os }} strategy: matrix: # After testing, the whl files of pyqlib built by macos-14 and macos-15 in python environments of 3.8, 3.9, 3.10, 3.11, 3.12, # the filenames are exactly duplicated, which will result in the duplicated whl files not being able to be uploaded to pypi, # so we chose to just keep the latest macos-latest. macos-latest currently points to macos-15. # Also, macos-13 will stop being supported on 2025-11-14. # Refs: https://github.blog/changelog/2025-07-11-upcoming-changes-to-macos-hosted-runners-macos-latest-migration-and-xcode-support-policy-updates/ os: [windows-latest, macos-latest] python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - uses: actions/checkout@v4 if: needs.release.outputs.release_created == 'true' with: fetch-depth: 0 - name: Set up Python ${{ matrix.python-version }} if: needs.release.outputs.release_created == 'true' uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Install dependencies if: needs.release.outputs.release_created == 'true' run: | make dev - name: Build wheel on ${{ matrix.os }} if: needs.release.outputs.release_created == 'true' run: | make build - name: Upload to PyPi if: needs.release.outputs.release_created == 'true' env: TWINE_USERNAME: __token__ TWINE_PASSWORD: ${{ secrets.TESTPYPI }} run: | twine check dist/*.whl twine upload --repository-url https://test.pypi.org/legacy/ dist/*.whl --verbose ================================================ FILE: .github/workflows/stale.yml ================================================ name: Mark stale issues and pull requests on: schedule: - cron: "0 0/3 * * *" jobs: stale: runs-on: ubuntu-latest steps: - uses: actions/stale@v3 with: repo-token: ${{ secrets.GITHUB_TOKEN }} stale-issue-message: 'This issue is stale because it has been open for three months with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days' stale-pr-message: 'This PR is stale because it has been open for a year with no activity. Remove the stale label or comment on the PR otherwise this will be closed in 5 days' stale-issue-label: 'stale' stale-pr-label: 'stale' days-before-stale: 90 days-before-pr-stale: 365 days-before-close: 5 operations-per-run: 100 exempt-issue-labels: 'bug,enhancement' remove-stale-when-updated: true ================================================ FILE: .github/workflows/test_qlib_from_pip.yml ================================================ name: Test qlib from pip on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: timeout-minutes: 120 runs-on: ${{ matrix.os }} strategy: matrix: os: [windows-latest, ubuntu-24.04, ubuntu-22.04, macos-14, macos-15] # In github action, using python 3.7, pip install will not match the latest version of the package. # Also, python 3.7 is no longer supported from macos-14, and will be phased out from macos-13 in the near future. # All things considered, we have removed python 3.7. python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - name: Test qlib from pip uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Update pip to the latest version run: | python -m pip install --upgrade pip - name: Qlib installation test run: | python -m pip install pyqlib - name: Install Lightgbm for MacOS if: ${{ matrix.os == 'macos-14' || matrix.os == 'macos-15' }} run: | brew update brew install libomp || brew reinstall libomp python -m pip install --no-binary=:all: lightgbm - name: Downloads dependencies data run: | cd .. python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn cd qlib - name: Test workflow by config run: | qrun examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml ================================================ FILE: .github/workflows/test_qlib_from_source.yml ================================================ name: Test qlib from source on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: timeout-minutes: 180 # we may retry for 3 times for `Unit tests with Pytest` runs-on: ${{ matrix.os }} strategy: matrix: os: [windows-latest, ubuntu-24.04, ubuntu-22.04, macos-14, macos-15] # In github action, using python 3.7, pip install will not match the latest version of the package. # Also, python 3.7 is no longer supported from macos-14, and will be phased out from macos-13 in the near future. # All things considered, we have removed python 3.7. python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - name: Test qlib from source uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Update pip to the latest version run: | python -m pip install --upgrade pip - name: Installing pytorch for macos if: ${{ matrix.os == 'macos-14' || matrix.os == 'macos-15' }} run: | python -m pip install torch torchvision torchaudio - name: Installing pytorch for ubuntu if: ${{ matrix.os == 'ubuntu-24.04' || matrix.os == 'ubuntu-22.04' }} run: | python -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu - name: Installing pytorch for windows if: ${{ matrix.os == 'windows-latest' }} run: | python -m pip install torch torchvision torchaudio - name: Set up Python tools run: | make dev - name: Lint with Black run: | make black - name: Make html with sphinx # Since read the docs builds on ubuntu 22.04, we only need to test that the build passes on ubuntu 22.04. if: ${{ matrix.os == 'ubuntu-22.04' }} run: | make docs-gen - name: Check Qlib with pylint run: | make pylint - name: Check Qlib with flake8 run: | make flake8 - name: Check Qlib with mypy run: | make mypy # Due to issues that cannot be automatically fixed when running `nbqa black . -l 120 --check --diff` on Jupyter notebooks, # we reverted to a version of `black` earlier than 26.1.0 before performing the checks. - name: Check Qlib ipynb with nbqa run: | python -m pip install "black<26.1" make nbqa - name: Test data downloads run: | python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn python scripts/get_data.py download_data --file_name rl_data.zip --target_dir tests/.data/rl - name: Install Lightgbm for MacOS if: ${{ matrix.os == 'macos-14' || matrix.os == 'macos-15' }} run: | brew update brew install libomp || brew reinstall libomp python -m pip install --no-binary=:all: lightgbm - name: Check Qlib ipynb with nbconvert run: | make nbconvert - name: Test workflow by config (install from source) run: | python -m pip install numba python qlib/cli/run.py examples/benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml - name: Unit tests with Pytest (MacOS) if: ${{ matrix.os == 'macos-14' || matrix.os == 'macos-15' }} uses: nick-fields/retry@v2 with: timeout_minutes: 60 max_attempts: 3 command: | # Limit the number of threads in various libraries to prevent Segmentation faults caused by OpenMP multithreading conflicts under macOS. export OMP_NUM_THREADS=1 # Limit the number of OpenMP threads export MKL_NUM_THREADS=1 # Limit the number of Intel MKL threads export NUMEXPR_NUM_THREADS=1 # Limit the number of NumExpr threads export OPENBLAS_NUM_THREADS=1 # Limit the number of OpenBLAS threads export VECLIB_MAXIMUM_THREADS=1 # Limit the number of macOS Accelerate/vecLib threads cd tests python -m pytest . -m "not slow" --durations=0 - name: Unit tests with Pytest (Ubuntu and Windows) if: ${{ matrix.os != 'macos-13' && matrix.os != 'macos-14' && matrix.os != 'macos-15' }} uses: nick-fields/retry@v2 with: timeout_minutes: 60 max_attempts: 3 command: | cd tests python -m pytest . -m "not slow" --durations=0 ================================================ FILE: .github/workflows/test_qlib_from_source_slow.yml ================================================ name: Test qlib from source slow on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build: timeout-minutes: 720 # we may retry for 3 times for `Unit tests with Pytest` runs-on: ${{ matrix.os }} strategy: matrix: os: [windows-latest, ubuntu-24.04, ubuntu-22.04, macos-14, macos-15] # In github action, using python 3.7, pip install will not match the latest version of the package. # Also, python 3.7 is no longer supported from macos-14, and will be phased out from macos-13 in the near future. # All things considered, we have removed python 3.7. python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] steps: - name: Test qlib from source slow uses: actions/checkout@v4 with: fetch-depth: 0 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: python-version: ${{ matrix.python-version }} - name: Set up Python tools run: | make dev - name: Downloads dependencies data run: | python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn - name: Install Lightgbm for MacOS if: ${{ matrix.os == 'macos-14' || matrix.os == 'macos-15' }} run: | brew update brew install libomp || brew reinstall libomp python -m pip install --no-binary=:all: lightgbm - name: Unit tests with Pytest uses: nick-fields/retry@v2 with: timeout_minutes: 240 max_attempts: 3 command: | cd tests python -m pytest . -m "slow" --durations=0 ================================================ FILE: .gitignore ================================================ # https://github.com/github/gitignore/blob/master/Python.gitignore __pycache__/ *.pyc *.pyd *.so *.ipynb .ipynb_checkpoints _build build/ dist/ *.pkl *.hd5 *.csv .env .vim .nvimrc .vscode qlib/VERSION.txt qlib/data/_libs/expanding.cpp qlib/data/_libs/rolling.cpp qlib/_version.py examples/estimator/estimator_example/ examples/rl/data/ examples/rl/checkpoints/ examples/rl/outputs/ examples/rl_order_execution/data/ examples/rl_order_execution/outputs/ *.egg-info/ # test related test-output.xml .output .data # special software mlruns/ tags .pytest_cache/ .mypy_cache/ .vscode/ *.swp ./pretrain .idea/ .aider* ================================================ FILE: .mypy.ini ================================================ [mypy] exclude = (?x)( ^qlib/backtest/high_performance_ds\.py$ | ^qlib/contrib | ^qlib/data | ^qlib/model | ^qlib/strategy | ^qlib/tests | ^qlib/utils | ^qlib/workflow | ^qlib/config\.py$ | ^qlib/log\.py$ | ^qlib/__init__\.py$ ) ignore_missing_imports = true disallow_incomplete_defs = true follow_imports = skip ================================================ FILE: .pre-commit-config.yaml ================================================ repos: - repo: https://github.com/psf/black rev: 23.7.0 hooks: - id: black args: ["qlib", "-l 120"] - repo: https://github.com/PyCQA/flake8 rev: 4.0.1 hooks: - id: flake8 args: ["--ignore=E501,F541,E266,E402,W503,E731,E203"] ================================================ FILE: .pylintrc ================================================ [TYPECHECK] # https://stackoverflow.com/a/53572939 # List of members which are set dynamically and missed by Pylint inference # system, and so shouldn't trigger E1101 when accessed. generated-members=numpy.*, torch.* ================================================ FILE: .readthedocs.yaml ================================================ # .readthedocs.yml # Read the Docs configuration file # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the version of Python and other tools you might need build: os: ubuntu-22.04 tools: python: "3.8" # Build documentation in the docs/ directory with Sphinx sphinx: configuration: docs/conf.py # Build all formats formats: all # Optionally set the version of Python and requirements required to build your docs python: install: - requirements: docs/requirements.txt - method: pip path: . ================================================ FILE: CHANGELOG.md ================================================ ================================================ FILE: CHANGES.rst ================================================ Changelog ========= Here you can see the full list of changes between each QLib release. Version 0.1.0 ------------- This is the initial release of QLib library. Version 0.1.1 ------------- Performance optimize. Add more features and operators. Version 0.1.2 ------------- - Support operator syntax. Now ``High() - Low()`` is equivalent to ``Sub(High(), Low())``. - Add more technical indicators. Version 0.1.3 ------------- Bug fix and add instruments filtering mechanism. Version 0.2.0 ------------- - Redesign ``LocalProvider`` database format for performance improvement. - Support load features as string fields. - Add scripts for database construction. - More operators and technical indicators. Version 0.2.1 ------------- - Support registering user-defined ``Provider``. - Support use operators in string format, e.g. ``['Ref($close, 1)']`` is valid field format. - Support dynamic fields in ``$some_field`` format. And existing fields like ``Close()`` may be deprecated in the future. Version 0.2.2 ------------- - Add ``disk_cache`` for reusing features (enabled by default). - Add ``qlib.contrib`` for experimental model construction and evaluation. Version 0.2.3 ------------- - Add ``backtest`` module - Decoupling the Strategy, Account, Position, Exchange from the backtest module Version 0.2.4 ------------- - Add ``profit attribution`` module - Add ``rick_control`` and ``cost_control`` strategies Version 0.3.0 ------------- - Add ``estimator`` module Version 0.3.1 ------------- - Add ``filter`` module Version 0.3.2 ------------- - Add real price trading, if the ``factor`` field in the data set is incomplete, use ``adj_price`` trading - Refactor ``handler`` ``launcher`` ``trainer`` code - Support ``backtest`` configuration parameters in the configuration file - Fix bug in position ``amount`` is 0 - Fix bug of ``filter`` module Version 0.3.3 ------------- - Fix bug of ``filter`` module Version 0.3.4 ------------- - Support for ``finetune model`` - Refactor ``fetcher`` code Version 0.3.5 ------------- - Support multi-label training, you can provide multiple label in ``handler``. (But LightGBM doesn't support due to the algorithm itself) - Refactor ``handler`` code, dataset.py is no longer used, and you can deploy your own labels and features in ``feature_label_config`` - Handler only offer DataFrame. Also, ``trainer`` and model.py only receive DataFrame - Change ``split_rolling_data``, we roll the data on market calendar now, not on normal date - Move some date config from ``handler`` to ``trainer`` Version 0.4.0 ------------- - Add `data` package that holds all data-related codes - Reform the data provider structure - Create a server for data centralized management `qlib-server `_ - Add a `ClientProvider` to work with server - Add a pluggable cache mechanism - Add a recursive backtracking algorithm to inspect the furthest reference date for an expression .. note:: The ``D.instruments`` function does not support ``start_time``, ``end_time``, and ``as_list`` parameters, if you want to get the results of previous versions of ``D.instruments``, you can do this: >>> from qlib.data import D >>> instruments = D.instruments(market='csi500') >>> D.list_instruments(instruments=instruments, start_time='2015-01-01', end_time='2016-02-15', as_list=True) Version 0.4.1 ------------- - Add support Windows - Fix ``instruments`` type bug - Fix ``features`` is empty bug(It will cause failure in updating) - Fix ``cache`` lock and update bug - Fix use the same cache for the same field (the original space will add a new cache) - Change "logger handler" from config - Change model load support 0.4.0 later - The default value of the ``method`` parameter of ``risk_analysis`` function is changed from **ci** to **si** Version 0.4.2 ------------- - Refactor DataHandler - Add ``Alpha360`` DataHandler Version 0.4.3 ------------- - Implementing Online Inference and Trading Framework - Refactoring The interfaces of backtest and strategy module. Version 0.4.4 ------------- - Optimize cache generation performance - Add report module - Fix bug when using ``ServerDatasetCache`` offline. - In the previous version of ``long_short_backtest``, there is a case of ``np.nan`` in long_short. The current version ``0.4.4`` has been fixed, so ``long_short_backtest`` will be different from the previous version. - In the ``0.4.2`` version of ``risk_analysis`` function, ``N`` is ``250``, and ``N`` is ``252`` from ``0.4.3``, so ``0.4.2`` is ``0.002122`` smaller than the ``0.4.3`` the backtest result is slightly different between ``0.4.2`` and ``0.4.3``. - refactor the argument of backtest function. - **NOTE**: - The default arguments of topk margin strategy is changed. Please pass the arguments explicitly if you want to get the same backtest result as previous version. - The TopkWeightStrategy is changed slightly. It will try to sell the stocks more than ``topk``. (The backtest result of TopkAmountStrategy remains the same) - The margin ratio mechanism is supported in the Topk Margin strategies. Version 0.4.5 ------------- - Add multi-kernel implementation for both client and server. - Support a new way to load data from client which skips dataset cache. - Change the default dataset method from single kernel implementation to multi kernel implementation. - Accelerate the high frequency data reading by optimizing the relative modules. - Support a new method to write config file by using dict. Version 0.4.6 ------------- - Some bugs are fixed - The default config in `Version 0.4.5` is not friendly to daily frequency data. - Backtest error in TopkWeightStrategy when `WithInteract=True`. Version 0.5.0 ------------- - First opensource version - Refine the docs, code - Add baselines - public data crawler Version 0.8.0 ------------- - The backtest is greatly refactored. - Nested decision execution framework is supported - There are lots of changes for daily trading, it is hard to list all of them. But a few important changes could be noticed - The trading limitation is more accurate; - In `previous version `__, longing and shorting actions share the same action. - In `current version `__, the trading limitation is different between logging and shorting action. - The constant is different when calculating annualized metrics. - `Current version `_ uses more accurate constant than `previous version `__ - `A new version `__ of data is released. Due to the unstability of Yahoo data source, the data may be different after downloading data again. - Users could check out the backtesting results between `Current version `__ and `previous version `__ Other Versions -------------- Please refer to `Github release Notes `_ ================================================ FILE: CODE_OF_CONDUCT.md ================================================ # Microsoft Open Source Code of Conduct This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). Resources: - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns ================================================ FILE: Dockerfile ================================================ FROM continuumio/miniconda3:latest WORKDIR /qlib COPY . . RUN apt-get update && \ apt-get install -y build-essential RUN conda create --name qlib_env python=3.8 -y RUN echo "conda activate qlib_env" >> ~/.bashrc ENV PATH /opt/conda/envs/qlib_env/bin:$PATH RUN python -m pip install --upgrade pip RUN python -m pip install numpy==1.23.5 RUN python -m pip install pandas==1.5.3 RUN python -m pip install importlib-metadata==5.2.0 RUN python -m pip install "cloudpickle<3" RUN python -m pip install scikit-learn==1.3.2 RUN python -m pip install cython packaging tables matplotlib statsmodels RUN python -m pip install pybind11 cvxpy ARG IS_STABLE="yes" RUN if [ "$IS_STABLE" = "yes" ]; then \ python -m pip install pyqlib; \ else \ python setup.py install; \ fi ================================================ FILE: LICENSE ================================================ MIT License Copyright (c) Microsoft Corporation. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE ================================================ FILE: MANIFEST.in ================================================ exclude tests/* include qlib/* include qlib/*/* include qlib/*/*/* include qlib/*/*/*/* include qlib/*/*/*/*/* ================================================ FILE: Makefile ================================================ .PHONY: clean deepclean prerequisite dependencies lightgbm rl develop lint docs package test analysis all install dev black pylint flake8 mypy nbqa nbconvert lint build upload docs-gen #You can modify it according to your terminal SHELL := /bin/bash ######################################################################################## # Variables ######################################################################################## # Documentation target directory, will be adapted to specific folder for readthedocs. PUBLIC_DIR := $(shell [ "$$READTHEDOCS" = "True" ] && echo "$$READTHEDOCS_OUTPUT/html" || echo "public") SO_DIR := qlib/data/_libs SO_FILES := $(wildcard $(SO_DIR)/*.so) ifeq ($(OS),Windows_NT) IS_WINDOWS = true else IS_WINDOWS = false endif ######################################################################################## # Development Environment Management ######################################################################################## # Remove common intermediate files. clean: -rm -rf \ $(PUBLIC_DIR) \ qlib/data/_libs/*.cpp \ qlib/data/_libs/*.so \ mlruns \ public \ build \ .coverage \ .mypy_cache \ .pytest_cache \ .ruff_cache \ Pipfile* \ coverage.xml \ dist \ release-notes.md find . -name '*.egg-info' -print0 | xargs -0 rm -rf find . -name '*.pyc' -print0 | xargs -0 rm -f find . -name '*.swp' -print0 | xargs -0 rm -f find . -name '.DS_Store' -print0 | xargs -0 rm -f find . -name '__pycache__' -print0 | xargs -0 rm -rf # Remove pre-commit hook, virtual environment alongside itermediate files. deepclean: clean if command -v pre-commit > /dev/null 2>&1; then pre-commit uninstall --hook-type pre-push; fi if command -v pipenv >/dev/null 2>&1 && pipenv --venv >/dev/null 2>&1; then pipenv --rm; fi # Prerequisite section # What this code does is compile two Cython modules, rolling and expanding, using setuptools and Cython, # and builds them as binary expansion modules that can be imported directly into Python. # Since pyproject.toml can't do that, we compile it here. # pywinpty as a dependency of jupyter on windows, if you use pip install pywinpty installation, # will first download the tar.gz file, and then locally compiled and installed, # this will lead to some unnecessary trouble, so we choose to install the compiled whl file, to avoid trouble. prerequisite: @if [ -n "$(SO_FILES)" ]; then \ echo "Shared library files exist, skipping build."; \ else \ echo "No shared library files found, building..."; \ pip install --upgrade setuptools wheel; \ python -m pip install cython numpy; \ python -c "from setuptools import setup, Extension; from Cython.Build import cythonize; import numpy; extensions = [Extension('qlib.data._libs.rolling', ['qlib/data/_libs/rolling.pyx'], language='c++', include_dirs=[numpy.get_include()]), Extension('qlib.data._libs.expanding', ['qlib/data/_libs/expanding.pyx'], language='c++', include_dirs=[numpy.get_include()])]; setup(ext_modules=cythonize(extensions, language_level='3'), script_args=['build_ext', '--inplace'])"; \ fi @if [ "$(IS_WINDOWS)" = "true" ]; then \ python -m pip install pywinpty --only-binary=:all:; \ fi # Install the package in editable mode. dependencies: python -m pip install --no-cache-dir -e . lightgbm: python -m pip install --no-cache-dir lightgbm --prefer-binary rl: python -m pip install --no-cache-dir -e .[rl] develop: python -m pip install --no-cache-dir -e .[dev] lint: python -m pip install --no-cache-dir -e .[lint] docs: python -m pip install --no-cache-dir -e .[docs] package: python -m pip install --no-cache-dir -e .[package] test: python -m pip install --no-cache-dir -e .[test] analysis: python -m pip install --no-cache-dir -e .[analysis] client: python -m pip install --no-cache-dir -e .[client] all: python -m pip install --no-cache-dir -e .[pywinpty,dev,lint,docs,package,test,analysis,rl] install: prerequisite dependencies dev: prerequisite all ######################################################################################## # Lint and pre-commit ######################################################################################## # Check lint with black. black: black . -l 120 --check --diff --exclude qlib/_version.py # Check code folder with pylint. # TODO: These problems we will solve in the future. Important among them are: W0221, W0223, W0237, E1102 # C0103: invalid-name # C0209: consider-using-f-string # R0402: consider-using-from-import # R1705: no-else-return # R1710: inconsistent-return-statements # R1725: super-with-arguments # R1735: use-dict-literal # W0102: dangerous-default-value # W0212: protected-access # W0221: arguments-differ # W0223: abstract-method # W0231: super-init-not-called # W0237: arguments-renamed # W0612: unused-variable # W0621: redefined-outer-name # W0622: redefined-builtin # FIXME: specify exception type # W0703: broad-except # W1309: f-string-without-interpolation # E1102: not-callable # E1136: unsubscriptable-object # W4904: deprecated-class # R0917: too-many-positional-arguments # E1123: unexpected-keyword-arg # References for disable error: https://pylint.pycqa.org/en/latest/user_guide/messages/messages_overview.html # We use sys.setrecursionlimit(2000) to make the recursion depth larger to ensure that pylint works properly (the default recursion depth is 1000). # References for parameters: https://github.com/PyCQA/pylint/issues/4577#issuecomment-1000245962 pylint: pylint --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R0917,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,W4904,E0401,E1121,C0103,C0209,R0402,R1705,R1710,R1725,R1730,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0612,W0621,W0622,W0703,W1309,E1102,E1136 --const-rgx='[a-z_][a-z0-9_]{2,30}' qlib --init-hook="import astroid; astroid.context.InferenceContext.max_inferred = 500; import sys; sys.setrecursionlimit(2000)" pylint --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R0917,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,E0401,E1121,E1123,C0103,C0209,R0402,R1705,R1710,R1725,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0246,W0612,W0621,W0622,W0703,W1309,E1102,E1136 --const-rgx='[a-z_][a-z0-9_]{2,30}' scripts --init-hook="import astroid; astroid.context.InferenceContext.max_inferred = 500; import sys; sys.setrecursionlimit(2000)" # Check code with flake8. # The following flake8 error codes were ignored: # E501 line too long # Description: We have used black to limit the length of each line to 120. # F541 f-string is missing placeholders # Description: The same thing is done when using pylint for detection. # E266 too many leading '#' for block comment # Description: To make the code more readable, a lot of "#" is used. # This error code appears centrally in: # qlib/backtest/executor.py # qlib/data/ops.py # qlib/utils/__init__.py # E402 module level import not at top of file # Description: There are times when module level import is not available at the top of the file. # W503 line break before binary operator # Description: Since black formats the length of each line of code, it has to perform a line break when a line of arithmetic is too long. # E731 do not assign a lambda expression, use a def # Description: Restricts the use of lambda expressions, but at some point lambda expressions are required. # E203 whitespace before ':' # Description: If there is whitespace before ":", it cannot pass the black check. flake8: flake8 --ignore=E501,F541,E266,E402,W503,E731,E203 --per-file-ignores="__init__.py:F401,F403" qlib # Check code with mypy. # https://github.com/python/mypy/issues/10600 mypy: mypy qlib --install-types --non-interactive mypy qlib --verbose # Check ipynb with nbqa. nbqa: nbqa black . -l 120 --check --diff nbqa pylint . --disable=C0104,C0114,C0115,C0116,C0301,C0302,C0411,C0413,C1802,R0401,R0801,R0902,R0903,R0911,R0912,R0913,R0914,R0915,R1720,W0105,W0123,W0201,W0511,W0613,W1113,W1514,E0401,E1121,C0103,C0209,R0402,R1705,R1710,R1725,R1735,W0102,W0212,W0221,W0223,W0231,W0237,W0612,W0621,W0622,W0703,W1309,E1102,E1136,W0719,W0104,W0404,C0412,W0611,C0410 --const-rgx='[a-z_][a-z0-9_]{2,30}' # Check ipynb with nbconvert.(Run after data downloads) # TODO: Add more ipynb files in future nbconvert: jupyter nbconvert --to notebook --execute examples/workflow_by_code.ipynb lint: black pylint flake8 mypy nbqa ######################################################################################## # Package ######################################################################################## # Build the package. build: python -m build --wheel # Upload the package. upload: python -m twine upload dist/* ######################################################################################## # Documentation ######################################################################################## docs-gen: python -m sphinx.cmd.build -W docs $(PUBLIC_DIR) ================================================ FILE: README.md ================================================ [![Python Versions](https://img.shields.io/pypi/pyversions/pyqlib.svg?logo=python&logoColor=white)](https://pypi.org/project/pyqlib/#files) [![Platform](https://img.shields.io/badge/platform-linux%20%7C%20windows%20%7C%20macos-lightgrey)](https://pypi.org/project/pyqlib/#files) [![PypI Versions](https://img.shields.io/pypi/v/pyqlib)](https://pypi.org/project/pyqlib/#history) [![Upload Python Package](https://github.com/microsoft/qlib/workflows/Upload%20Python%20Package/badge.svg)](https://pypi.org/project/pyqlib/) [![Github Actions Test Status](https://github.com/microsoft/qlib/workflows/Test/badge.svg?branch=main)](https://github.com/microsoft/qlib/actions) [![Documentation Status](https://readthedocs.org/projects/qlib/badge/?version=latest)](https://qlib.readthedocs.io/en/latest/?badge=latest) [![License](https://img.shields.io/pypi/l/pyqlib)](LICENSE) [![Join the chat at https://gitter.im/Microsoft/qlib](https://badges.gitter.im/Microsoft/qlib.svg)](https://gitter.im/Microsoft/qlib?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) ## :newspaper: **What's NEW!**   :sparkling_heart: Recent released features ### Introducing RD_Agent: LLM-Based Autonomous Evolving Agents for Industrial Data-Driven R&D We are excited to announce the release of **RD-Agent**📢, a powerful tool that supports automated factor mining and model optimization in quant investment R&D. RD-Agent is now available on [GitHub](https://github.com/microsoft/RD-Agent), and we welcome your star🌟! To learn more, please visit our [♾️Demo page](https://rdagent.azurewebsites.net/). Here, you will find demo videos in both English and Chinese to help you better understand the scenario and usage of RD-Agent. We have prepared several demo videos for you: | Scenario | Demo video (English) | Demo video (中文) | | -- | ------ | ------ | | Quant Factor Mining | [Link](https://rdagent.azurewebsites.net/factor_loop?lang=en) | [Link](https://rdagent.azurewebsites.net/factor_loop?lang=zh) | | Quant Factor Mining from reports | [Link](https://rdagent.azurewebsites.net/report_factor?lang=en) | [Link](https://rdagent.azurewebsites.net/report_factor?lang=zh) | | Quant Model Optimization | [Link](https://rdagent.azurewebsites.net/model_loop?lang=en) | [Link](https://rdagent.azurewebsites.net/model_loop?lang=zh) | - 📃**Paper**: [R&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization](https://arxiv.org/abs/2505.15155) - 👾**Code**: https://github.com/microsoft/RD-Agent/ ```BibTeX @misc{li2025rdagentquant, title={R\&D-Agent-Quant: A Multi-Agent Framework for Data-Centric Factors and Model Joint Optimization}, author={Yuante Li and Xu Yang and Xiao Yang and Minrui Xu and Xisen Wang and Weiqing Liu and Jiang Bian}, year={2025}, eprint={2505.15155}, archivePrefix={arXiv}, primaryClass={cs.AI} } ``` ![image](https://github.com/user-attachments/assets/3198bc10-47ba-4ee0-8a8e-46d5ce44f45d) *** | Feature | Status | | -- | ------ | | [R&D-Agent-Quant](https://arxiv.org/abs/2505.15155) Published | Apply R&D-Agent to Qlib for quant trading | | BPQP for End-to-end learning | 📈Coming soon!([Under review](https://github.com/microsoft/qlib/pull/1863)) | | 🔥LLM-driven Auto Quant Factory🔥 | 🚀 Released in [♾️RD-Agent](https://github.com/microsoft/RD-Agent) on Aug 8, 2024 | | KRNN and Sandwich models | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/1414/) on May 26, 2023 | | Release Qlib v0.9.0 | :octocat: [Released](https://github.com/microsoft/qlib/releases/tag/v0.9.0) on Dec 9, 2022 | | RL Learning Framework | :hammer: :chart_with_upwards_trend: Released on Nov 10, 2022. [#1332](https://github.com/microsoft/qlib/pull/1332), [#1322](https://github.com/microsoft/qlib/pull/1322), [#1316](https://github.com/microsoft/qlib/pull/1316),[#1299](https://github.com/microsoft/qlib/pull/1299),[#1263](https://github.com/microsoft/qlib/pull/1263), [#1244](https://github.com/microsoft/qlib/pull/1244), [#1169](https://github.com/microsoft/qlib/pull/1169), [#1125](https://github.com/microsoft/qlib/pull/1125), [#1076](https://github.com/microsoft/qlib/pull/1076)| | HIST and IGMTF models | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/1040) on Apr 10, 2022 | | Qlib [notebook tutorial](https://github.com/microsoft/qlib/tree/main/examples/tutorial) | 📖 [Released](https://github.com/microsoft/qlib/pull/1037) on Apr 7, 2022 | | Ibovespa index data | :rice: [Released](https://github.com/microsoft/qlib/pull/990) on Apr 6, 2022 | | Point-in-Time database | :hammer: [Released](https://github.com/microsoft/qlib/pull/343) on Mar 10, 2022 | | Arctic Provider Backend & Orderbook data example | :hammer: [Released](https://github.com/microsoft/qlib/pull/744) on Jan 17, 2022 | | Meta-Learning-based framework & DDG-DA | :chart_with_upwards_trend: :hammer: [Released](https://github.com/microsoft/qlib/pull/743) on Jan 10, 2022 | | Planning-based portfolio optimization | :hammer: [Released](https://github.com/microsoft/qlib/pull/754) on Dec 28, 2021 | | Release Qlib v0.8.0 | :octocat: [Released](https://github.com/microsoft/qlib/releases/tag/v0.8.0) on Dec 8, 2021 | | ADD model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/704) on Nov 22, 2021 | | ADARNN model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/689) on Nov 14, 2021 | | TCN model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/668) on Nov 4, 2021 | | Nested Decision Framework | :hammer: [Released](https://github.com/microsoft/qlib/pull/438) on Oct 1, 2021. [Example](https://github.com/microsoft/qlib/blob/main/examples/nested_decision_execution/workflow.py) and [Doc](https://qlib.readthedocs.io/en/latest/component/highfreq.html) | | Temporal Routing Adaptor (TRA) | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/531) on July 30, 2021 | | Transformer & Localformer | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/508) on July 22, 2021 | | Release Qlib v0.7.0 | :octocat: [Released](https://github.com/microsoft/qlib/releases/tag/v0.7.0) on July 12, 2021 | | TCTS Model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/491) on July 1, 2021 | | Online serving and automatic model rolling | :hammer: [Released](https://github.com/microsoft/qlib/pull/290) on May 17, 2021 | | DoubleEnsemble Model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/286) on Mar 2, 2021 | | High-frequency data processing example | :hammer: [Released](https://github.com/microsoft/qlib/pull/257) on Feb 5, 2021 | | High-frequency trading example | :chart_with_upwards_trend: [Part of code released](https://github.com/microsoft/qlib/pull/227) on Jan 28, 2021 | | High-frequency data(1min) | :rice: [Released](https://github.com/microsoft/qlib/pull/221) on Jan 27, 2021 | | Tabnet Model | :chart_with_upwards_trend: [Released](https://github.com/microsoft/qlib/pull/205) on Jan 22, 2021 | Features released before 2021 are not listed here.

Qlib is an open-source, AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms, including supervised learning, market dynamics modeling, and reinforcement learning. An increasing number of SOTA Quant research works/papers in diverse paradigms are being released in Qlib to collaboratively solve key challenges in quantitative investment. For example, 1) using supervised learning to mine the market's complex non-linear patterns from rich and heterogeneous financial data, 2) modeling the dynamic nature of the financial market using adaptive concept drift technology, and 3) using reinforcement learning to model continuous investment decisions and assist investors in optimizing their trading strategies. It contains the full ML pipeline of data processing, model training, back-testing; and covers the entire chain of quantitative investment: alpha seeking, risk modeling, portfolio optimization, and order execution. For more details, please refer to our paper ["Qlib: An AI-oriented Quantitative Investment Platform"](https://arxiv.org/abs/2009.11189).
Frameworks, Tutorial, Data & DevOps Main Challenges & Solutions in Quant Research
  • Plans
  • Framework of Qlib
  • Quick Start
  • Quant Dataset Zoo
  • Learning Framework
  • More About Qlib
  • Offline Mode and Online Mode
  • Related Reports
  • Contact Us
  • Contributing
    		•
  • Main Challenges & Solutions in Quant Research
    • # Plans New features under development(order by estimated release time). Your feedbacks about the features are very important. # Framework of Qlib
    The high-level framework of Qlib can be found above(users can find the [detailed framework](https://qlib.readthedocs.io/en/latest/introduction/introduction.html#framework) of Qlib's design when getting into nitty gritty). The components are designed as loose-coupled modules, and each component could be used stand-alone. Qlib provides a strong infrastructure to support Quant research. [Data](https://qlib.readthedocs.io/en/latest/component/data.html) is always an important part. A strong learning framework is designed to support diverse learning paradigms (e.g. [reinforcement learning](https://qlib.readthedocs.io/en/latest/component/rl.html), [supervised learning](https://qlib.readthedocs.io/en/latest/component/workflow.html#model-section)) and patterns at different levels(e.g. [market dynamic modeling](https://qlib.readthedocs.io/en/latest/component/meta.html)). By modeling the market, [trading strategies](https://qlib.readthedocs.io/en/latest/component/strategy.html) will generate trade decisions that will be executed. Multiple trading strategies and executors in different levels or granularities can be [nested to be optimized and run together](https://qlib.readthedocs.io/en/latest/component/highfreq.html). At last, a comprehensive [analysis](https://qlib.readthedocs.io/en/latest/component/report.html) will be provided and the model can be [served online](https://qlib.readthedocs.io/en/latest/component/online.html) in a low cost. # Quick Start This quick start guide tries to demonstrate 1. It's very easy to build a complete Quant research workflow and try your ideas with _Qlib_. 2. Though with *public data* and *simple models*, machine learning technologies **work very well** in practical Quant investment. Here is a quick **[demo](https://terminalizer.com/view/3f24561a4470)** shows how to install ``Qlib``, and run LightGBM with ``qrun``. **But**, please make sure you have already prepared the data following the [instruction](#data-preparation). ## Installation This table demonstrates the supported Python version of `Qlib`: | | install with pip | install from source | plot | | ------------- |:---------------------:|:--------------------:|:------------------:| | Python 3.8 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Python 3.9 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Python 3.10 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Python 3.11 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | Python 3.12 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | **Note**: 1. **Conda** is suggested for managing your Python environment. In some cases, using Python outside of a `conda` environment may result in missing header files, causing the installation failure of certain packages. 2. Please pay attention that installing cython in Python 3.6 will raise some error when installing ``Qlib`` from source. If users use Python 3.6 on their machines, it is recommended to *upgrade* Python to version 3.8 or higher, or use `conda`'s Python to install ``Qlib`` from source. ### Install with pip Users can easily install ``Qlib`` by pip according to the following command. ```bash pip install pyqlib ``` **Note**: pip will install the latest stable qlib. However, the main branch of qlib is in active development. If you want to test the latest scripts or functions in the main branch. Please install qlib with the methods below. ### Install from source Also, users can install the latest dev version ``Qlib`` by the source code according to the following steps: * Before installing ``Qlib`` from source, users need to install some dependencies: ```bash pip install numpy pip install --upgrade cython ``` * Clone the repository and install ``Qlib`` as follows. ```bash git clone https://github.com/microsoft/qlib.git && cd qlib pip install . # `pip install -e .[dev]` is recommended for development. check details in docs/developer/code_standard_and_dev_guide.rst ``` **Tips**: If you fail to install `Qlib` or run the examples in your environment, comparing your steps and the [CI workflow](.github/workflows/test_qlib_from_source.yml) may help you find the problem. **Tips for Mac**: If you are using Mac with M1, you might encounter issues in building the wheel for LightGBM, which is due to missing dependencies from OpenMP. To solve the problem, install openmp first with ``brew install libomp`` and then run ``pip install .`` to build it successfully. ## Data Preparation ❗ Due to more restrict data security policy. The official dataset is disabled temporarily. You can try [this data source](https://github.com/chenditc/investment_data/releases) contributed by the community. Here is an example to download the latest data. ```bash wget https://github.com/chenditc/investment_data/releases/latest/download/qlib_bin.tar.gz mkdir -p ~/.qlib/qlib_data/cn_data tar -zxvf qlib_bin.tar.gz -C ~/.qlib/qlib_data/cn_data --strip-components=1 rm -f qlib_bin.tar.gz ``` The official dataset below will resume in short future. ---- Load and prepare data by running the following code: ### Get with module ```bash # get 1d data python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn # get 1min data python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min ``` ### Get from source ```bash # get 1d data python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn # get 1min data python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data_1min --region cn --interval 1min ``` This dataset is created by public data collected by [crawler scripts](scripts/data_collector/), which have been released in the same repository. Users could create the same dataset with it. [Description of dataset](https://github.com/microsoft/qlib/tree/main/scripts/data_collector#description-of-dataset) *Please pay **ATTENTION** that the data is collected from [Yahoo Finance](https://finance.yahoo.com/lookup), and the data might not be perfect. We recommend users to prepare their own data if they have a high-quality dataset. For more information, users can refer to the [related document](https://qlib.readthedocs.io/en/latest/component/data.html#converting-csv-format-into-qlib-format)*. ### Automatic update of daily frequency data (from yahoo finance) > This step is *Optional* if users only want to try their models and strategies on history data. > > It is recommended that users update the data manually once (--trading_date 2021-05-25) and then set it to update automatically. > > **NOTE**: Users can't incrementally update data based on the offline data provided by Qlib(some fields are removed to reduce the data size). Users should use [yahoo collector](https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#automatic-update-of-daily-frequency-datafrom-yahoo-finance) to download Yahoo data from scratch and then incrementally update it. > > For more information, please refer to: [yahoo collector](https://github.com/microsoft/qlib/tree/main/scripts/data_collector/yahoo#automatic-update-of-daily-frequency-datafrom-yahoo-finance) * Automatic update of data to the "qlib" directory each trading day(Linux) * use *crontab*: `crontab -e` * set up timed tasks: ``` * * * * 1-5 python