Repository: cea-hpc/robinhood Branch: master Commit: 806084c9d623 Files: 497 Total size: 5.5 MB Directory structure: gitextract_8lphqcmv/ ├── .commit-template ├── .gitignore ├── ChangeLog ├── LICENSE.en.txt ├── LICENSE.fr.txt ├── Makefile.am ├── README.md ├── autogen.sh ├── autotools/ │ └── m4/ │ ├── args_mgmt.m4 │ ├── as-ac-expand.m4 │ ├── ax_valgrind_check.m4 │ ├── db.m4 │ └── lustre.m4 ├── configure.ac ├── doc/ │ ├── Makefile.am │ ├── admin_guides/ │ │ ├── MOVED_TO │ │ └── rbh25-disaster_recovery.odt │ ├── doxygen/ │ │ └── doxygen.cfg.in │ └── templates/ │ ├── Makefile.am │ ├── basic.conf │ ├── example_alerts.conf │ ├── example_checksum.conf │ ├── example_cleanup.conf │ ├── example_lhsm.conf │ ├── example_modeguard.conf │ ├── example_rmdir.conf │ ├── example_shook.conf │ └── includes/ │ ├── alerts.inc │ ├── backup.inc │ ├── check.inc │ ├── lhsm.inc │ ├── modeguard.inc │ ├── rmdir.inc │ ├── rmdir_old.inc │ ├── shook.inc │ └── tmpfs.inc ├── man/ │ ├── Makefile.am │ ├── lhsmtool_cmd.1 │ ├── lhsmtool_cmd.rst │ ├── rbh-diff.1 │ ├── rbh-du.1 │ ├── rbh-find.1 │ ├── rbh-report.1 │ └── robinhood.1 ├── robinhood.spec.in ├── scripts/ │ ├── Makefile.am │ ├── bestiaire.sh │ ├── cfg_25to30.sh │ ├── check_commit.sh │ ├── check_rpm_reloc.sh │ ├── checkpatch.pl │ ├── cmd2man.sh │ ├── code_format.sh │ ├── commit-msg │ ├── demo/ │ │ ├── disk_usage.sh │ │ ├── migr_purge.sh │ │ ├── mkplot.sh │ │ ├── populate_backend.sh │ │ ├── rh.migr_purge.conf │ │ ├── trace_usage.gp │ │ └── write_data.sh │ ├── exafs.sh │ ├── fill_fs.sh │ ├── fix_man_options.sh │ ├── git_prepare_hook │ ├── indent.sh │ ├── ld.so.robinhood.conf.in │ ├── pre-commit │ ├── rbh-config │ ├── rbh_cksum.sh.in │ ├── rbhext_tool │ ├── rbhext_tool_clnt │ ├── rbhext_tool_svr │ ├── rewrite.sh │ ├── robinhood.init.in │ ├── robinhood.init.sles.in │ ├── robinhood.service.in │ ├── robinhood@.service.in │ ├── sanity-lhsm/ │ │ ├── repro_time.sh │ │ ├── rh-hsm.conf.in │ │ └── sanity.sh │ ├── sysconfig_robinhood.in │ └── type_gen.pl ├── src/ │ ├── Makefile.am │ ├── cfg_parsing/ │ │ ├── Makefile.am │ │ ├── analyze.c │ │ ├── analyze.h │ │ ├── conf_lex.c │ │ ├── conf_lex.l │ │ ├── conf_yacc.c │ │ ├── conf_yacc.h │ │ ├── conf_yacc.y │ │ ├── config_parsing.c │ │ ├── rbh_boolexpr.c │ │ ├── rbh_cfg.c │ │ └── rbh_cfg_helpers.c │ ├── chglog_reader/ │ │ ├── Makefile.am │ │ ├── chglog_reader.c │ │ └── chglog_reader_config.c │ ├── common/ │ │ ├── Makefile.am │ │ ├── RW_Lock.c │ │ ├── basename.c │ │ ├── global_config.c │ │ ├── lustre_tools.c │ │ ├── mntent_compat.c │ │ ├── mntent_compat.h │ │ ├── param_utils.c │ │ ├── queue.c │ │ ├── rbh_cmd.c │ │ ├── rbh_logs.c │ │ ├── rbh_misc.c │ │ ├── rbh_modules.c │ │ ├── rbh_params.c │ │ ├── uidgidcache.c │ │ └── update_params.c │ ├── entry_processor/ │ │ ├── Makefile.am │ │ ├── diff_pipeline.c │ │ ├── entry_proc_hash.c │ │ ├── entry_proc_impl.c │ │ ├── entry_proc_tools.c │ │ ├── entry_proc_tools.h │ │ ├── std_pipeline.c │ │ └── test_hash.c │ ├── fs_scan/ │ │ ├── Makefile.am │ │ ├── fs_scan.c │ │ ├── fs_scan.h │ │ ├── fs_scan_main.c │ │ ├── fs_scan_types.h │ │ ├── task_stack_mngmt.c │ │ ├── task_stack_mngmt.h │ │ ├── task_tree_mngmt.c │ │ └── task_tree_mngmt.h │ ├── include/ │ │ ├── Makefile.am │ │ ├── Memory.h │ │ ├── RW_Lock.h │ │ ├── chglog_reader.h │ │ ├── config_parsing.h │ │ ├── db_schema.def │ │ ├── db_types.h │ │ ├── entry_proc_hash.h │ │ ├── entry_processor.h │ │ ├── fs_scan_main.h │ │ ├── global_config.h │ │ ├── list.h │ │ ├── list_mgr.h │ │ ├── lustre/ │ │ │ └── lustre_errno.h │ │ ├── lustre_extended_types.h │ │ ├── pipeline_types.h │ │ ├── policy_rules.h │ │ ├── policy_run.h │ │ ├── queue.h │ │ ├── rbh_basename.h │ │ ├── rbh_boolexpr.h │ │ ├── rbh_cfg.h │ │ ├── rbh_cfg_helpers.h │ │ ├── rbh_const.h │ │ ├── rbh_logs.h │ │ ├── rbh_misc.h │ │ ├── rbh_modules.h │ │ ├── rbh_params.h │ │ ├── rbh_types.h │ │ ├── status_manager.h │ │ ├── uidgidcache.h │ │ ├── update_params.h │ │ └── xplatform_print.h │ ├── list_mgr/ │ │ ├── Makefile.am │ │ ├── database.h │ │ ├── listmgr_common.c │ │ ├── listmgr_common.h │ │ ├── listmgr_config.c │ │ ├── listmgr_filters.c │ │ ├── listmgr_get.c │ │ ├── listmgr_init.c │ │ ├── listmgr_insert.c │ │ ├── listmgr_internal.h │ │ ├── listmgr_iterators.c │ │ ├── listmgr_ns.c │ │ ├── listmgr_recov.c │ │ ├── listmgr_remove.c │ │ ├── listmgr_reports.c │ │ ├── listmgr_stripe.c │ │ ├── listmgr_stripe.h │ │ ├── listmgr_tags.c │ │ ├── listmgr_update.c │ │ ├── listmgr_vars.c │ │ ├── mysql_wrapper.c │ │ └── sqlite_wrapper.c │ ├── modules/ │ │ ├── Makefile.am │ │ ├── alerter.c │ │ ├── backup.c │ │ ├── backup.h │ │ ├── basic.c │ │ ├── checker.c │ │ ├── common_actions.c │ │ ├── common_sched.c │ │ ├── lhsm.c │ │ ├── mod_internal.c │ │ ├── mod_internal.h │ │ ├── modeguard.c │ │ ├── sched_ratelimit.c │ │ ├── shook.c │ │ └── test_sched.c │ ├── policies/ │ │ ├── Makefile.am │ │ ├── policy_loader.c │ │ ├── policy_matching.c │ │ ├── policy_run.c │ │ ├── policy_run_cfg.c │ │ ├── policy_sched.c │ │ ├── policy_sched.h │ │ ├── policy_triggers.c │ │ ├── run_policies.h │ │ └── status_manager.c │ ├── robinhood/ │ │ ├── Makefile.am │ │ ├── cmd_helpers.c │ │ ├── cmd_helpers.h │ │ ├── rbh_daemon.c │ │ ├── rbh_diff.c │ │ ├── rbh_du.c │ │ ├── rbh_find.c │ │ ├── rbh_find.h │ │ ├── rbh_find_printf.c │ │ ├── rbh_import.c │ │ ├── rbh_rebind.c │ │ ├── rbh_recov.c │ │ ├── rbh_report.c │ │ └── rbh_undelete.c │ ├── tests/ │ │ ├── Makefile.am │ │ ├── create_nostripe.c │ │ ├── path.sql │ │ ├── test_confparam.c │ │ ├── test_forcestripe.c │ │ ├── test_params.c │ │ ├── test_parse.c │ │ ├── test_parsing.sh │ │ ├── test_uidgidcache.c │ │ └── tst.data/ │ │ ├── bad.conf │ │ ├── inc2.inc │ │ ├── ok.conf │ │ ├── test.conf │ │ └── test.inc │ └── tools/ │ ├── Makefile.am │ ├── gen_lov_objid.c │ ├── lhsmtool_cmd.c │ ├── ost_fids_remap.c │ ├── read_lovea.c │ └── set_lovea.c ├── tests/ │ ├── Makefile.am │ ├── completion.sh │ ├── create_files.sh │ ├── fill_fs.sh │ ├── gprof-helper.c │ ├── huge_posix/ │ │ ├── 1-test_setup.sh │ │ ├── 2-run-tests.sh │ │ └── cfg/ │ │ ├── common.conf │ │ └── innodb.conf │ ├── shook_configure.sh │ ├── test_compil_switches.sh │ ├── test_rpmbuild.sh │ └── test_suite/ │ ├── 1-test_setup_lustre.sh │ ├── 1-test_setup_posix.sh │ ├── 2-run-tests.sh │ ├── 3-tests-lustre.sh │ ├── Makefile.am │ ├── README.rst │ ├── bench_rpc.sh │ ├── cfg/ │ │ ├── Generation_Report_1.conf │ │ ├── MigrationClass_ExtendedAttribut.conf │ │ ├── MigrationClass_LastAccess.conf │ │ ├── MigrationClass_LastModification.conf │ │ ├── MigrationClass_OST.conf │ │ ├── MigrationClass_Owner.conf │ │ ├── MigrationClass_Path_Name.conf │ │ ├── MigrationClass_Size.conf │ │ ├── MigrationClass_Type.conf │ │ ├── MigrationFile_ExtendedAttribut.conf │ │ ├── MigrationFile_LastAccess.conf │ │ ├── MigrationFile_LastModification.conf │ │ ├── MigrationFile_OST.conf │ │ ├── MigrationFile_Owner.conf │ │ ├── MigrationFile_Path_Name.conf │ │ ├── MigrationFile_Size.conf │ │ ├── MigrationFile_Type.conf │ │ ├── MigrationGroup.conf │ │ ├── MigrationOST.conf │ │ ├── MigrationStd_ExtendedAttribut.conf │ │ ├── MigrationStd_LastAccess.conf │ │ ├── MigrationStd_LastModification.conf │ │ ├── MigrationStd_OST.conf │ │ ├── MigrationStd_Owner.conf │ │ ├── MigrationStd_Path_Name.conf │ │ ├── MigrationStd_Size.conf │ │ ├── MigrationStd_Type.conf │ │ ├── MigrationUser.conf │ │ ├── OtherParameters_1.conf │ │ ├── OtherParameters_2.conf │ │ ├── OtherParameters_3.conf │ │ ├── OtherParameters_4.conf │ │ ├── OtherParameters_5.conf │ │ ├── PurgeClass_ExtendedAttribut.conf │ │ ├── PurgeClass_LastAccess.conf │ │ ├── PurgeClass_LastModification.conf │ │ ├── PurgeClass_OST.conf │ │ ├── PurgeClass_Owner.conf │ │ ├── PurgeClass_Path_Name.conf │ │ ├── PurgeClass_Size.conf │ │ ├── PurgeClass_Type.conf │ │ ├── PurgeOST.conf │ │ ├── PurgeStd_ExtendedAttribut.conf │ │ ├── PurgeStd_LastAccess.conf │ │ ├── PurgeStd_LastModification.conf │ │ ├── PurgeStd_OST.conf │ │ ├── PurgeStd_Owner.conf │ │ ├── PurgeStd_Path_Name.conf │ │ ├── PurgeStd_Size.conf │ │ ├── PurgeStd_Type.conf │ │ ├── RemovingDir_Dircount.conf │ │ ├── RemovingDir_ExtendedAttribute.conf │ │ ├── RemovingDir_LastAccess.conf │ │ ├── RemovingDir_LastModification.conf │ │ ├── RemovingDir_Mixed.conf │ │ ├── RemovingDir_OST.conf │ │ ├── RemovingDir_Owner.conf │ │ ├── RemovingDir_Path_Name.conf │ │ ├── RemovingEmptyDir.conf │ │ ├── TriggerPurge_GroupQuotaExceeded.conf │ │ ├── TriggerPurge_OstQuotaExceeded.conf │ │ ├── TriggerPurge_QuotaExceeded.conf │ │ ├── TriggerPurge_UserQuotaExceeded.conf │ │ ├── acct.conf │ │ ├── alert.conf │ │ ├── alert_ost.conf │ │ ├── cl_batch.conf │ │ ├── commit_update.conf │ │ ├── common.conf │ │ ├── common_noclass.conf │ │ ├── compress.conf │ │ ├── info_collect.conf │ │ ├── info_collect2.conf │ │ ├── log1.conf │ │ ├── log1b.conf │ │ ├── log2.conf │ │ ├── log2b.conf │ │ ├── log3.conf │ │ ├── log3b.conf │ │ ├── log_common.conf │ │ ├── lru_purge.conf │ │ ├── lru_sort_access.conf │ │ ├── lru_sort_archive.conf │ │ ├── lru_sort_creat_last_arch.conf │ │ ├── lru_sort_creation.conf │ │ ├── lru_sort_mod.conf │ │ ├── lru_sort_mod_2pass.conf │ │ ├── lru_sort_size_asc.conf │ │ ├── lru_sort_size_desc.conf │ │ ├── migr_fail.conf │ │ ├── migr_failer.sh │ │ ├── overflow.conf │ │ ├── rbh25.sql │ │ ├── rmdir.conf │ │ ├── test1.conf │ │ ├── test2.conf │ │ ├── test3.conf │ │ ├── test_action_params.conf │ │ ├── test_basic.conf │ │ ├── test_check_migr.conf │ │ ├── test_check_purge.conf │ │ ├── test_checker.conf │ │ ├── test_checker_invert.conf │ │ ├── test_completion.conf │ │ ├── test_copy.conf │ │ ├── test_custom_purge.conf │ │ ├── test_default_case.conf │ │ ├── test_fileclass.conf │ │ ├── test_hsm_invalidate.conf │ │ ├── test_hsm_remove_noorder.conf │ │ ├── test_hsm_remove_order.conf │ │ ├── test_iname.conf │ │ ├── test_lhsm1.conf │ │ ├── test_limits.conf │ │ ├── test_maintenance.conf │ │ ├── test_mnt_point.conf │ │ ├── test_modeguard_dir.conf │ │ ├── test_modeguard_file.conf │ │ ├── test_move.conf │ │ ├── test_multirule.conf │ │ ├── test_multirule_migr.conf │ │ ├── test_nlink.conf │ │ ├── test_path.conf │ │ ├── test_pipeline.conf │ │ ├── test_pools.conf │ │ ├── test_prepost_cmd.conf │ │ ├── test_prepost_sched.conf │ │ ├── test_purge.conf │ │ ├── test_purge2.conf │ │ ├── test_ratelim.conf │ │ ├── test_recov.conf │ │ ├── test_recov2.conf │ │ ├── test_rm1.conf │ │ ├── test_rmdir_depth.conf │ │ ├── test_run.conf │ │ ├── test_scan_only.conf │ │ ├── test_sched1.conf │ │ ├── test_trig.conf │ │ ├── test_trig2.conf │ │ ├── test_trig3.conf │ │ ├── test_trig4.conf │ │ ├── test_trig_cntpct.conf │ │ ├── test_updt.conf │ │ ├── test_uuid.conf │ │ ├── test_xattr.conf │ │ ├── tokudb1.conf │ │ ├── tokudb2.conf │ │ └── uidgidnum.conf │ ├── cleanup.sh │ ├── create-random.c │ ├── lsetup.sh │ ├── prepost_cmd.sh │ ├── rm_script │ └── valgrind.supp └── web_gui/ ├── Makefile.am ├── gui_v3/ │ ├── README.txt │ ├── api/ │ │ ├── .htaccess │ │ ├── index.php │ │ ├── rest.class.php │ │ └── robinhood.php │ ├── common.php │ ├── config.php │ ├── config_local.php.template │ ├── cron.php │ ├── css/ │ │ ├── bootstrap-datetimepicker.css │ │ ├── bootstrap-slider.css │ │ ├── bootstrap-theme.css │ │ ├── bootstrap.css │ │ ├── dashboard.css │ │ ├── dataTables.bootstrap.css │ │ ├── dataTables.bootstrap4.css │ │ ├── dataTables.foundation.css │ │ ├── dataTables.jqueryui.css │ │ ├── dataTables.material.css │ │ ├── dataTables.semanticui.css │ │ ├── dataTables.uikit.css │ │ ├── jquery.dataTables.css │ │ └── jquery.dataTables_themeroller.css │ ├── customjs/ │ │ ├── newgui.js │ │ └── param.php │ ├── images/ │ │ └── Sorting icons.psd │ ├── index.php │ ├── js/ │ │ ├── Chart.bundle.js │ │ ├── bootstrap-slider.js │ │ ├── bootstrap.js │ │ ├── dataTables.bootstrap.js │ │ ├── dataTables.bootstrap4.js │ │ ├── dataTables.foundation.js │ │ ├── dataTables.jqueryui.js │ │ ├── dataTables.material.js │ │ ├── dataTables.semanticui.js │ │ ├── dataTables.uikit.js │ │ ├── filesize.dataTables.js │ │ ├── jquery.dataTables.js │ │ ├── jquery.js │ │ ├── moment.js │ │ └── npm.js │ ├── lang/ │ │ ├── en.php │ │ ├── fr.php │ │ └── sys.php │ ├── plugin.php │ └── plugins/ │ ├── README.txt │ ├── browser/ │ │ ├── css/ │ │ │ └── bootstrap-treeview.css │ │ ├── help.html │ │ ├── js/ │ │ │ └── bootstrap-treeview.js │ │ ├── plugin.php │ │ └── script.js │ ├── colorgraph/ │ │ └── plugin.php │ ├── console/ │ │ ├── plugin.php │ │ └── script.js │ ├── customgraph/ │ │ ├── help.html │ │ ├── plugin.php │ │ └── script.js │ ├── internalstats/ │ │ ├── plugin.php │ │ └── script.js │ ├── ldapauth/ │ │ ├── plugin.php │ │ └── script.js │ ├── netauth/ │ │ ├── help.html │ │ └── plugin.php │ ├── output/ │ │ ├── help.html │ │ └── plugin.php │ ├── plugdisplay/ │ │ ├── help.html │ │ ├── plugin.php │ │ └── script.js │ ├── stackgraph/ │ │ └── plugin.php │ └── tasks/ │ ├── help.html │ ├── plugin.php │ └── script.js ├── robinhood.conf ├── scripts/ │ └── check_robinhood.py └── tests/ ├── nonreg.bash └── nonreg.php ================================================ FILE CONTENTS ================================================ ================================================ FILE: .commit-template ================================================ # Please use the following format for the commit message: # # : short description of the change. # # More detailed description about the issue # and the way it is addressed. # # Implementation details. # ================================================ FILE: .gitignore ================================================ .deps .libs .dirstamp Makefile.in Makefile *.o *.lo *.la *.diff *.orig *.rej *.gcda *.gcno *~ \#*\# TAGS cscope.* /autotools/depcomp /autotools/install-sh /autotools/ltmain.sh /autotools/missing /autotools/compile /autotools/config.guess /autotools/config.sub /autotools/ar-lib /autotools/m4/libtool.m4 /autotools/m4/ltoptions.m4 /autotools/m4/ltsugar.m4 /autotools/m4/ltversion.m4 /autotools/m4/lt~obsolete.m4 /autotools/ylwrap /autotools/test-driver /autom4te.cache /config.status /libtool /Makefile /aclocal.m4 /config.log /configure /doc/doxygen/doxygen.cfg /man/Makefile /robinhood-*.tar.gz /robinhood.spec /rpms/ /scripts/Makefile /scripts/robinhood.init /scripts/robinhood.init.sles /scripts/robinhood.service /scripts/robinhood@.service /scripts/sysconfig_robinhood /scripts/ld.so.robinhood.conf /scripts/rbh_cksum.sh /src/Makefile /src/todo /src/robinhood/todo /src/cfg_parsing/Makefile /src/chglog_reader/Makefile /src/common/Makefile /src/entry_processor/Makefile /src/fs_scan/Makefile /src/include/Makefile /src/include/config.h /src/include/config.h.in /src/include/stamp-h1 /src/list_mgr/Makefile /src/logs/Makefile /src/policy_matching/Makefile /src/policy_modules/Makefile /src/policy_modules/hsm_remove/Makefile /src/policy_modules/migration/Makefile /src/policy_modules/purge/Makefile /src/policy_modules/rmdir/Makefile /src/robinhood/Makefile /src/tests/Makefile /src/tests/test_uidgidcache /src/tests/test_params /src/tests/test_confparam /src/tests/test_parse /src/tests/create_nostripe /src/tests/test_forcestripe /src/backend_ext/Makefile /tests/Makefile /web_gui/Makefile /src/tools/Makefile /src/tools/gen_lov_objid /src/tools/read_lovea /src/tools/set_lovea /src/tools/ost_fids_remap /src/tools/lhsmtool_cmd /src/robinhood/rbh-du /src/robinhood/rbh-find /src/robinhood/rbh-report /src/robinhood/rbh-diff /src/robinhood/rbh-undelete /src/robinhood/robinhood /tests/lustre_fs/*.log /tests/posix_fs/*.log /tests/test_suite/create-random /TODO /src/include/db_schema.h /scripts/make_mans.sh /web_gui/gui_v3/config_local.php ================================================ FILE: ChangeLog ================================================ Summary of changes in versions 3.x: 3.2: - Add features for Lustre's project quota: - Retrieve project id when scanning and reading changelogs - Add report: rbh-report --project-info - New filtering options in rbh-report: --filter-project - New option --split-user-projects to split user's usage per project - Display project-info with 'rbh-find --printf %RP' - Filter project with 'rbh-find -projid num' - New 'projid' trigger target on command line - Implement policy sort order by size, e.g. lru_sort_attr = size; - Implement asc/desc modifiers for sort order, e.g. lru_sort_attr = size(desc); - Implement policy trigger thresholds as percentage of available inodes: high/low_threshold_cntpct = xx%; - policy optimization: no DB update when pre_sched_match and post_sched_match are set to "none or "cache_only". - Fix errors "Out of range value for size columns" due to DB triggers - Make top-user --by-count take all entry types into account - Adaptations for Lustre 2.15 - Adaptations for RHEL9.4 OS family 3.1.7: - Fix pool_usage triggers. - Fix partial application of policies when run on multiple OSTs. 3.1.6: - fix build on Lustre 2.12.4 - check the filesystem returns consistent statfs values - chglog_reader: reopen changelog in case of unexpected error - chglog_reader: de-duplicate HSM STATE events (keep the latest) - policy run time stat now includes initial DB request time - fix the use of archive_id parameter in lhsm - port to CentOS8 and lustre master (2.13.52) 3.1.5: - Lustre: compatibility with FPL and DoM (no full-featured support) - Fix and improvement of ratelimit scheduler - Stop retrying SQL requests on SIGTERM - Faster remove policies (disable sorting) 3.1.4: * Web gui: - new access control criteria: IP address, hostname; - tasks: scheduled requests, and possiblity to keep result history; - custom graphs. * Performance improvements: - Improved rules-to-SQL conversion engine. Enable faster policy runs. - Auto-tune hash table sizes (changelog reader and entry processor). - Pipeline tuning to reduce CPU usage and improve ingestion rate. - Default linking to jemalloc memory allocator for better performance. Note: it is advised to start MariaDB with this allocator too to achieve maximal performance. - Fix major performance bug in changelog reader (appeared in v3.1.3). * Configuration files: support %include directives in sub-blocks. * New parameter to 'common.copy': "mkdir=yes" creates target directories prior to the copy operation. * New action 'common.move' (move an entry from one path/name to another). * Command copytool 'lhsmtool_cmd': add man page. 3.1.3: * Policies: add matching modes "auto_update_attrs" and "auto_update_all" * Faster changelog reader shutdown when the process is terminated by a signal * Changelog optimization: drop CREATE/UNLINK and MKDIR/RMDIR changelog pairs * rbh-find: add "-links" criterion "-ost" accepts OST sets add "-iname" option for case-insensitive name matching support "-not" for links, size and dates * FS scan optimization (use openat() to walk through the filesystem instead of full paths). * Fix rules-to-SQL conversion 3.1.2: * Implement command rbh-rebind command to assign an archived entry to a newly created fid (e.g. used for undelete operation). * Make lhsm undelete more resilient to error cases * lhsm: add tunable to allow custom/smaller UUID * REST API: add nagios plugin add graph preview in console plugin add new filters Main changes in robinhood 3.1: * Ingest rate optimizations: up to x10 * New plugin type: action schedulers These plugins make it possible to reorder policy actions, or smooth policy execution. In this release, two of such plugins are provided: 'common.max_per_run' and 'common.rate_limit'. * Policy optimizations: - Improved pre-filtering: convert all policy rules to SQL for better pre-filtering. - Configurable attribute matching before/after scheduling (pre/post_sched_match parameters). * FS scan: make it possible to restrict scan to several directories ('scan_only' configuration parameter). - Especially useful with Lustre's static DNE partitioning. * Pre/post policy run commands * Modeguard status manager - Enforce or clear permission bits on entries of the filesystem * Fixes for Lustre 2.10 - Note: this robinhood version does not support Progressive File Layouts yet * REST API and web interface enhancements: - Plugin mechanism to add custom charts/fields/requests... - Finer access control (limit user views only to their stats) - All robinhood DB available through the REST API Main changes in robinhood 3.0: * Policies at will: - Define new custom policies at will, just by writting just a few lines of configuration. - Schedule all imaginable actions on filesystem entries. - Templates are provided to implement: old entries cleanup, directory cleanup, data corruption checks (by regularly checking file's checksum), Lustre/HSM policies. * Fully configurable policies: - All policy aspects made configurable (scope, actions, action parameters, sort order...) - Fine-grained specification of policy actions and parameters. * Plugin-based architecture: - Missing a specific feature in Robinhood core? Implement you own plugin to manage new kind of policies, maintain specific info in robinhood DB, enable interactions with specific site ecosystem (job scheduler, system load...), ... * Enabling the Robinhood community: - Share your custom policy definitions and plugins, so that other community members can benefit from them and enrich them! * All Robinhood features in a single instance: - No more distinct and incompatible robinhood "flavors" (tmpfs, lhsm, backup...) - All policies can be implemented in a single robinhood instance. * Get a better overlook of your filesystem contents than ever! - New web interface - REST interface to query filesystem stats - Tag your filesystem entries based on arbitrary criteria using the new fileclass implementation - Flexible reporting with "rbh-find -printf" * RedHat 7 integration: - systemd support. - Per filesystem service: robinhood@ * New features for Lustre/HSM: - Undelete: recover accidentally deleted files. - Implement Lustre/HSM with any backend, using the generic command copytool shipped with robinhood: 'lhsmtool_cmd'. - UUID support for Lustre/HSM copytools. * Convenient: - Automatic DB conversion after major upgrades. * Even more robust & Improved code quality. Changes between version 2.5.4 and 2.5.5: * [lustre] Support Lustre versions up to 2.7. * [DB performance] Allow both batching and parallelizing DB operations when accounting is OFF. Benchmarks show a x3~x4 speedup of DB ingest rate in this case. * [DB performance] Reduce lock contention on indexes of STRIPE_ITEMS table. * [acct] Avoid accounting stats inconsistency when this feature is enabled/ disabled/re-enabled. * [policy performance] Disable ignored class re-matching by default. * [policy performance] Disable sort by mtime by default. * [policy performance] Allow disabling sort by atime for purge policies. * [fix] Fix segfault when log rotation occurs under heavy logging. * [fix] When processing a RENAME changelog record, match the new path in filesystem instead of building it from DB (which caused 'incomplete path' errors). * [fix] Fix sendfile copy operation for backup mode. * [fix] Fix various requests about directories: --topdirs with --filter-path option, --toprmdir, ... * [fix] Fix possible duplicate insert error when updated entry attributes are only in ANNEX_INFO table (rare). * [fix] Some purge parameters may not be taken into account. * [packaging] clean permissions in distribution tarball. * [report performance] Improve performance of path matching when running directory reports with --filter-path option. Changes between version 2.5.3 and 2.5.4: * [lustre] Lustre 2.4+: detect all stripe changes and update the DB accordingly. * [scan] Prevent from dropping entries from DB when opendir or stat fail. * [DB] Optimization to batch more DB requests. * [DB] Allow using any MySQL engine (new config parameter: listmanager::mysql::engine) * [rbh-config] Avoid backup_db to lock the whole tables with innodb. * [bugfix] Improve robustness to corrupted mtimes. * [bugfix] Fix possible crash in db_exec_sql. * [bugfix] Fix possible overflow when executing a custom archive command. * [backup mode] Clean all non-printable characters in backend path. * [pkg] libattr-devel is now mandatory to build robinhood. 2.5.4 release notes: * "innodb = enabled/disabled": this parameter will be deprecated. It has to be replaced by "engine = innodb". * (Lustre only) one of the field types changed in the DB. It will be automatically converted the first time rbh 2.5.4 starts. The conversion takes about 5s per million entries. * Admin guides are now available online (http://robinhood.sf.net > "Online documentation") "odt" and "pdf" versions are no longer maintained. Changes between version 2.5.2 and 2.5.3: - custom purge_command (fix): fixed vulnerability to malicious file names. - changelog processing (fix): fixed errors 'Entry has incomplete path in DB' in some case of rm/create patterns. - migration policy (fix): don't trigger copy of files that no longer exist. - rbh-config (feature): new option 'reset_acct' to rebuild accounting info. - changelog reader (feature): new parameter 'dump_file' to dump all incoming changelog records to a file. - Port to Lustre 2.6. Changes between version 2.5.1 and 2.5.2: - rbh-du: fixed major performance regression (since v2.5.0). - rbh-find: fixed occasional crash. - HSM and backup modes: fixed a risk of removing an existing entry from the backend (in some situations of hardlink/rename+unlink). - backup mode: optimized sendfile()-based copy (Linux kernel >= 2.6.33). - logs: avoid flood of log messages in case of DB connection error. - alerts: added host name to alert mail title. - rbh-config empty_db/repair_db: also manage/fix stored procedures. - cosmetic: fix wrong display of purged blocks for count-based triggers. - cosmetic: fix migration counter display. - init script: check that 'ulimit -s' is reasonable. - fixed build dependancies on Fedora19 and Fedora20. - code sanity: fixed many 'coverity' warnings + a couple of minor memleaks. - doc: details about RPM installation locations. - doc: detail of 'backend' paramaters for backup mode. Changes between version 2.5.0 and 2.5.1: - entry processing (major fix): fixed deadlock when the pipeline is full and an entry with an unknown parent is encountered. - purge (enhancement): start purging data from the most used OSTs. - rbh-find (features): new options: -pool, -exec, -print, -nouser, -nogroup, -lsost - rbh-find (optimization): automatically switch to bulk DB request mode when command argument is filesystem root (+new option -nobulk to disable it). - logging (enhancement): new config parameters to control log header format - backup (feature): allow compressing data in archive. - backup (fix): wrong path in archive when robinhood root directory != mount point. - backup (fix): fix segfault when importing a single file with a FID-ending name. Changes in version 2.5.0: Summary: - filesystem disaster recovery features - new namespace management (new DB schema to properly handle hardlinks, renames...) - scanning and changelog processing optimizations - database optimizations (requests batching) - many other changes, improvements and code cleaning... Details: - rbh-diff: * new command to detect differences between the filesystem and the information in robinhood database. * option "--apply=fs" for disaster recovery purpose: restore the filesystem metadata from robinhood DB. * makes it possible to rebuild a Lustre MDT from scratch, or from a LVM snapshot (see "Robinhood Lustre disaster recovery guide" for more details). - database: * new namespace implementation in database with new NAMES table (Cray contribution) - fixes/improves hardlink support - fixes/improves Lustre ChangeLog hardlink/rename/unlink support - saves DB storage space * database request batching: significantly increase database ingest rate. No longer needs innodb_flush_log_at_tx_commit != 1 to speed up DB operations. * additional information in DB that can help for disaster recovery: symlink info, access rights, stripe object indexes, stripe order, nlink... * set default commit behavior to transaction (prevent from DB inconsistencies) * optimized multi-table requests * optimization: minimized attribute set in DB update operations (don't update attributes that didn't change) * Fix: deal with mysql case insensitivity for string matching * triggers and stored procedures versioning mechanism * prevent from overflows for large INSERT requests, wide stripes... * prevent from DB deadlocks - scanning: * --partial-scan option is deprecated and replaced by an optional argument to --scan (e.g. --scan=/fs/subdir). * better management of partial scans: - better detection of removed entries vs. entries moved from a directory to another. - partial scans can be used for initial DB population (even if the DB is initially empty). * garbage collection of removed entries in DB is a long operation when terminating a scan (and even more when terminating a partial scan). Added --no-gc option to skip it (recommanded for partial scans). * automatically enabling --no-gc if the DB is initially empty (eg. for initial scan). * optimization: use *at() functions (openat, fstatat) and readdir by chunk (using getdents) instead of POSIX lstat() and readdir_r(). * optimization: use NOATIME flag to access entries as much as possible * optimizations of get_stripe and get_fid operations. * new --diff option for robinhood --scan and --readlog: output detected changes in a diff-like format. - Lustre changelogs: * changelog batching (Cray contribution): to speed up changelog processing, robinhood retains changelog records in memory a short time, to aggregate similar/redundant Changelog records on the same entry before updating its database. * support multiple changelog readers (for DNE) as multiple threads (default) or as multiple processes, possibly on different hosts, by giving a MDT index to --readlog option. * resilience to filesystem umount/mount. - rbh-report: * new option --entry-info to get all the stored information about an entry * option --dump-ost can now list multiple OSTs and support ranges notation (e.g. 3,5-8,12-23). * --dump-ost output indicates if a file has data on a given OST (could be striped on the OST but have no data on it). - rbh-find: * new option -crtime to filter entries on creation time. * output ordering closer to find output * added missing info in 'rbh-find -ls' output (nlink, mode, symlink info...) - robinhood-backup: * by default, use a built-in copy function to avoid the cost of forking copy commands. * rbh-backup-rebind: tool to rebind an entry in the backend if its fid changed in the filesystem for any reason (file copied to a new one to change its stripe, etc...) * rbh-backup-recov new features and options: --list (list information about entries to be recovered) --ost to only recover entries for a given set of OSTs (support range notation): the basic use-case is OST disaster recovery. --since