Repository: Atik-Ahamed/TimeMachine Branch: main Commit: 5bf17a728349 Files: 22 Total size: 81.2 KB Directory structure: gitextract_wg48y8vq/ ├── .gitignore ├── LICENSE ├── README.md └── TimeMachine_supervised/ ├── RevIN/ │ └── RevIN.py ├── data_provider/ │ ├── data_factory.py │ └── data_loader.py ├── exp/ │ ├── exp_basic.py │ └── exp_main.py ├── models/ │ └── TimeMachine.py ├── requirements.txt ├── run_longExp.py ├── scripts/ │ └── TimeMachine/ │ ├── electricity.sh │ ├── etth1.sh │ ├── etth2.sh │ ├── ettm1.sh │ ├── ettm2.sh │ ├── traffic.sh │ └── weather.sh └── utils/ ├── masking.py ├── metrics.py ├── timefeatures.py └── tools.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ data/ TimeMachine_supervised/exp/__pycache__/ TimeMachine_supervised/data_provider/__pycache__/ TimeMachine_supervised/models/__pycache__/ TimeMachine_supervised/RevIN/__pycache__/ TimeMachine_supervised/utils/__pycache__/ TimeMachine_supervised/checkpoints/ TimeMachine_supervised/logs/LongForecasting/ TimeMachine_supervised/results/ TimeMachine_supervised/test_results/ TimeMachine_supervised/result.txt TimeMachine_supervised/csv_results/ .vscode/settings.json ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ #
TimeMachine
![Alt text](./pics/TimeMachine.PNG) ### Welcome to the official repository of: [TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting](https://arxiv.org/pdf/2403.09898.pdf). ## :triangular_flag_on_post: TimeMachine is accepted to [**ECAI**](https://www.ecai2024.eu/) ## Usage 1. Install requirements. ```pip install -r requirements.txt``` 2. Navigate through our example scripts located at ```./scripts/TimeMachine```. You'll find the core of TimeMachine in ```models/TimeMachine.py```. For example, to get the multivariate forecasting results for weather dataset, just run the following command, and you can open ```./result.txt``` to see the results once the training is completed. Moreover, the results will also be available at ```csv_results```, which can be utilized to make queries in the dataframe: ``` sh ./scripts/TimeMachine/weather.sh ``` Hyper-paramters can be tuned based upon needs (e.g. different look-back windows and prediction lengths). TimeMachine is built on the popular [PatchTST](https://github.com/yuqinie98/PatchTST) framework. ## Acknowledgement We are deeply grateful for the valuable code and efforts contributed by the following GitHub repositories. Their contributions have been immensely beneficial to our work. - Mamba (https://github.com/state-spaces/mamba) - PatchTST (https://github.com/yuqinie98/PatchTST) - iTransformer (https://github.com/thuml/iTransformer) - RevIN (https://github.com/ts-kim/RevIN) - Reformer (https://github.com/lucidrains/reformer-pytorch) - Informer (https://github.com/zhouhaoyi/Informer2020) - FlashAttention (https://github.com/shreyansh26/FlashAttention-PyTorch) - Autoformer (https://github.com/thuml/Autoformer) - Stationary (https://github.com/thuml/Nonstationary_Transformers) - Time-Series-Library (https://github.com/thuml/Time-Series-Library) ## Citation If you find this repo useful in your research, please consider citing our paper as follows: ``` @article{timemachine, title = {TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting}, author = {Ahamed, Md Atik and Cheng, Qiang}, journal = {arXiv preprint arXiv:2403.09898}, year = {2024} } ``` ================================================ FILE: TimeMachine_supervised/RevIN/RevIN.py ================================================ import torch import torch.nn as nn class RevIN(nn.Module): def __init__(self, num_features: int, eps=1e-5, affine=True): """ :param num_features: the number of features or channels :param eps: a value added for numerical stability :param affine: if True, RevIN has learnable affine parameters """ super(RevIN, self).__init__() self.num_features = num_features self.eps = eps self.affine = affine if self.affine: self._init_params() def forward(self, x, mode:str): if mode == 'norm': self._get_statistics(x) x = self._normalize(x) elif mode == 'denorm': x = self._denormalize(x) else: raise NotImplementedError return x def _init_params(self): # initialize RevIN params: (C,) self.affine_weight = nn.Parameter(torch.ones(self.num_features)) self.affine_bias = nn.Parameter(torch.zeros(self.num_features)) def _get_statistics(self, x): dim2reduce = tuple(range(1, x.ndim-1)) self.mean = torch.mean(x, dim=dim2reduce, keepdim=True).detach() self.stdev = torch.sqrt(torch.var(x, dim=dim2reduce, keepdim=True, unbiased=False) + self.eps).detach() def _normalize(self, x): x = x - self.mean x = x / self.stdev if self.affine: x = x * self.affine_weight x = x + self.affine_bias return x def _denormalize(self, x): if self.affine: x = x - self.affine_bias x = x / (self.affine_weight + self.eps*self.eps) x = x * self.stdev x = x + self.mean return x ================================================ FILE: TimeMachine_supervised/data_provider/data_factory.py ================================================ from data_provider.data_loader import Dataset_ETT_hour, Dataset_ETT_minute, Dataset_Custom, Dataset_Pred from torch.utils.data import DataLoader data_dict = { 'ETTh1': Dataset_ETT_hour, 'ETTh2': Dataset_ETT_hour, 'ETTm1': Dataset_ETT_minute, 'ETTm2': Dataset_ETT_minute, 'custom': Dataset_Custom, } def data_provider(args, flag): Data = data_dict[args.data] timeenc = 0 if args.embed != 'timeF' else 1 if flag == 'test': shuffle_flag = False drop_last = True batch_size = args.batch_size freq = args.freq elif flag == 'pred': shuffle_flag = False drop_last = False batch_size = 1 freq = args.freq Data = Dataset_Pred else: shuffle_flag = True drop_last = True batch_size = args.batch_size freq = args.freq data_set = Data( root_path=args.root_path, data_path=args.data_path, flag=flag, size=[args.seq_len, args.label_len, args.pred_len], features=args.features, target=args.target, timeenc=timeenc, freq=freq ) print(flag, len(data_set)) data_loader = DataLoader( data_set, batch_size=batch_size, shuffle=shuffle_flag, num_workers=args.num_workers, drop_last=drop_last) return data_set, data_loader ================================================ FILE: TimeMachine_supervised/data_provider/data_loader.py ================================================ import os import numpy as np import pandas as pd import os import torch from torch.utils.data import Dataset, DataLoader from sklearn.preprocessing import StandardScaler from utils.timefeatures import time_features import warnings warnings.filterwarnings('ignore') class Dataset_ETT_hour(Dataset): def __init__(self, root_path, flag='train', size=None, features='S', data_path='ETTh1.csv', target='OT', scale=True, timeenc=0, freq='h'): # size [seq_len, label_len, pred_len] # info if size == None: self.seq_len = 24 * 4 * 4 self.label_len = 24 * 4 self.pred_len = 24 * 4 else: self.seq_len = size[0] self.label_len = size[1] self.pred_len = size[2] # init assert flag in ['train', 'test', 'val'] type_map = {'train': 0, 'val': 1, 'test': 2} self.set_type = type_map[flag] self.features = features self.target = target self.scale = scale self.timeenc = timeenc self.freq = freq self.root_path = root_path self.data_path = data_path self.__read_data__() def __read_data__(self): self.scaler = StandardScaler() df_raw = pd.read_csv(os.path.join(self.root_path, self.data_path)) border1s = [0, 12 * 30 * 24 - self.seq_len, 12 * 30 * 24 + 4 * 30 * 24 - self.seq_len] border2s = [12 * 30 * 24, 12 * 30 * 24 + 4 * 30 * 24, 12 * 30 * 24 + 8 * 30 * 24] border1 = border1s[self.set_type] border2 = border2s[self.set_type] if self.features == 'M' or self.features == 'MS': cols_data = df_raw.columns[1:] df_data = df_raw[cols_data] elif self.features == 'S': df_data = df_raw[[self.target]] if self.scale: train_data = df_data[border1s[0]:border2s[0]] self.scaler.fit(train_data.values) data = self.scaler.transform(df_data.values) else: data = df_data.values df_stamp = df_raw[['date']][border1:border2] df_stamp['date'] = pd.to_datetime(df_stamp.date) if self.timeenc == 0: df_stamp['month'] = df_stamp.date.apply(lambda row: row.month, 1) df_stamp['day'] = df_stamp.date.apply(lambda row: row.day, 1) df_stamp['weekday'] = df_stamp.date.apply(lambda row: row.weekday(), 1) df_stamp['hour'] = df_stamp.date.apply(lambda row: row.hour, 1) data_stamp = df_stamp.drop(['date'], axis=1).values elif self.timeenc == 1: data_stamp = time_features(pd.to_datetime(df_stamp['date'].values), freq=self.freq) data_stamp = data_stamp.transpose(1, 0) self.data_x = data[border1:border2] self.data_y = data[border1:border2] self.data_stamp = data_stamp def __getitem__(self, index): s_begin = index s_end = s_begin + self.seq_len r_begin = s_end - self.label_len r_end = r_begin + self.label_len + self.pred_len seq_x = self.data_x[s_begin:s_end] seq_y = self.data_y[r_begin:r_end] seq_x_mark = self.data_stamp[s_begin:s_end] seq_y_mark = self.data_stamp[r_begin:r_end] return seq_x, seq_y, seq_x_mark, seq_y_mark def __len__(self): return len(self.data_x) - self.seq_len - self.pred_len + 1 def inverse_transform(self, data): return self.scaler.inverse_transform(data) class Dataset_ETT_minute(Dataset): def __init__(self, root_path, flag='train', size=None, features='S', data_path='ETTm1.csv', target='OT', scale=True, timeenc=0, freq='t'): # size [seq_len, label_len, pred_len] # info if size == None: self.seq_len = 24 * 4 * 4 self.label_len = 24 * 4 self.pred_len = 24 * 4 else: self.seq_len = size[0] self.label_len = size[1] self.pred_len = size[2] # init assert flag in ['train', 'test', 'val'] type_map = {'train': 0, 'val': 1, 'test': 2} self.set_type = type_map[flag] self.features = features self.target = target self.scale = scale self.timeenc = timeenc self.freq = freq self.root_path = root_path self.data_path = data_path self.__read_data__() def __read_data__(self): self.scaler = StandardScaler() df_raw = pd.read_csv(os.path.join(self.root_path, self.data_path)) border1s = [0, 12 * 30 * 24 * 4 - self.seq_len, 12 * 30 * 24 * 4 + 4 * 30 * 24 * 4 - self.seq_len] border2s = [12 * 30 * 24 * 4, 12 * 30 * 24 * 4 + 4 * 30 * 24 * 4, 12 * 30 * 24 * 4 + 8 * 30 * 24 * 4] border1 = border1s[self.set_type] border2 = border2s[self.set_type] if self.features == 'M' or self.features == 'MS': cols_data = df_raw.columns[1:] df_data = df_raw[cols_data] elif self.features == 'S': df_data = df_raw[[self.target]] if self.scale: train_data = df_data[border1s[0]:border2s[0]] self.scaler.fit(train_data.values) data = self.scaler.transform(df_data.values) else: data = df_data.values df_stamp = df_raw[['date']][border1:border2] df_stamp['date'] = pd.to_datetime(df_stamp.date) if self.timeenc == 0: df_stamp['month'] = df_stamp.date.apply(lambda row: row.month, 1) df_stamp['day'] = df_stamp.date.apply(lambda row: row.day, 1) df_stamp['weekday'] = df_stamp.date.apply(lambda row: row.weekday(), 1) df_stamp['hour'] = df_stamp.date.apply(lambda row: row.hour, 1) df_stamp['minute'] = df_stamp.date.apply(lambda row: row.minute, 1) df_stamp['minute'] = df_stamp.minute.map(lambda x: x // 15) data_stamp = df_stamp.drop(['date'], axis=1).values elif self.timeenc == 1: data_stamp = time_features(pd.to_datetime(df_stamp['date'].values), freq=self.freq) data_stamp = data_stamp.transpose(1, 0) self.data_x = data[border1:border2] self.data_y = data[border1:border2] self.data_stamp = data_stamp def __getitem__(self, index): s_begin = index s_end = s_begin + self.seq_len r_begin = s_end - self.label_len r_end = r_begin + self.label_len + self.pred_len seq_x = self.data_x[s_begin:s_end] seq_y = self.data_y[r_begin:r_end] seq_x_mark = self.data_stamp[s_begin:s_end] seq_y_mark = self.data_stamp[r_begin:r_end] return seq_x, seq_y, seq_x_mark, seq_y_mark def __len__(self): return len(self.data_x) - self.seq_len - self.pred_len + 1 def inverse_transform(self, data): return self.scaler.inverse_transform(data) class Dataset_Custom(Dataset): def __init__(self, root_path, flag='train', size=None, features='S', data_path='ETTh1.csv', target='OT', scale=True, timeenc=0, freq='h'): # size [seq_len, label_len, pred_len] # info if size == None: self.seq_len = 24 * 4 * 4 self.label_len = 24 * 4 self.pred_len = 24 * 4 else: self.seq_len = size[0] self.label_len = size[1] self.pred_len = size[2] # init assert flag in ['train', 'test', 'val'] type_map = {'train': 0, 'val': 1, 'test': 2} self.set_type = type_map[flag] self.features = features self.target = target self.scale = scale self.timeenc = timeenc self.freq = freq self.root_path = root_path self.data_path = data_path self.__read_data__() def __read_data__(self): self.scaler = StandardScaler() df_raw = pd.read_csv(os.path.join(self.root_path, self.data_path)) ''' df_raw.columns: ['date', ...(other features), target feature] ''' cols = list(df_raw.columns) cols.remove(self.target) cols.remove('date') df_raw = df_raw[['date'] + cols + [self.target]] # print(cols) num_train = int(len(df_raw) * 0.7) num_test = int(len(df_raw) * 0.2) num_vali = len(df_raw) - num_train - num_test border1s = [0, num_train - self.seq_len, len(df_raw) - num_test - self.seq_len] border2s = [num_train, num_train + num_vali, len(df_raw)] border1 = border1s[self.set_type] border2 = border2s[self.set_type] if self.features == 'M' or self.features == 'MS': cols_data = df_raw.columns[1:] df_data = df_raw[cols_data] elif self.features == 'S': df_data = df_raw[[self.target]] if self.scale: train_data = df_data[border1s[0]:border2s[0]] self.scaler.fit(train_data.values) # print(self.scaler.mean_) # exit() data = self.scaler.transform(df_data.values) else: data = df_data.values df_stamp = df_raw[['date']][border1:border2] df_stamp['date'] = pd.to_datetime(df_stamp.date) if self.timeenc == 0: df_stamp['month'] = df_stamp.date.apply(lambda row: row.month, 1) df_stamp['day'] = df_stamp.date.apply(lambda row: row.day, 1) df_stamp['weekday'] = df_stamp.date.apply(lambda row: row.weekday(), 1) df_stamp['hour'] = df_stamp.date.apply(lambda row: row.hour, 1) data_stamp = df_stamp.drop(['date'], axis=1).values elif self.timeenc == 1: data_stamp = time_features(pd.to_datetime(df_stamp['date'].values), freq=self.freq) data_stamp = data_stamp.transpose(1, 0) self.data_x = data[border1:border2] self.data_y = data[border1:border2] self.data_stamp = data_stamp def __getitem__(self, index): s_begin = index s_end = s_begin + self.seq_len r_begin = s_end - self.label_len r_end = r_begin + self.label_len + self.pred_len seq_x = self.data_x[s_begin:s_end] seq_y = self.data_y[r_begin:r_end] seq_x_mark = self.data_stamp[s_begin:s_end] seq_y_mark = self.data_stamp[r_begin:r_end] return seq_x, seq_y, seq_x_mark, seq_y_mark def __len__(self): return len(self.data_x) - self.seq_len - self.pred_len + 1 def inverse_transform(self, data): return self.scaler.inverse_transform(data) class Dataset_Pred(Dataset): def __init__(self, root_path, flag='pred', size=None, features='S', data_path='ETTh1.csv', target='OT', scale=True, inverse=False, timeenc=0, freq='15min', cols=None): # size [seq_len, label_len, pred_len] # info if size == None: self.seq_len = 24 * 4 * 4 self.label_len = 24 * 4 self.pred_len = 24 * 4 else: self.seq_len = size[0] self.label_len = size[1] self.pred_len = size[2] # init assert flag in ['pred'] self.features = features self.target = target self.scale = scale self.inverse = inverse self.timeenc = timeenc self.freq = freq self.cols = cols self.root_path = root_path self.data_path = data_path self.__read_data__() def __read_data__(self): self.scaler = StandardScaler() df_raw = pd.read_csv(os.path.join(self.root_path, self.data_path)) ''' df_raw.columns: ['date', ...(other features), target feature] ''' if self.cols: cols = self.cols.copy() cols.remove(self.target) else: cols = list(df_raw.columns) cols.remove(self.target) cols.remove('date') df_raw = df_raw[['date'] + cols + [self.target]] border1 = len(df_raw) - self.seq_len border2 = len(df_raw) if self.features == 'M' or self.features == 'MS': cols_data = df_raw.columns[1:] df_data = df_raw[cols_data] elif self.features == 'S': df_data = df_raw[[self.target]] if self.scale: self.scaler.fit(df_data.values) data = self.scaler.transform(df_data.values) else: data = df_data.values tmp_stamp = df_raw[['date']][border1:border2] tmp_stamp['date'] = pd.to_datetime(tmp_stamp.date) pred_dates = pd.date_range(tmp_stamp.date.values[-1], periods=self.pred_len + 1, freq=self.freq) df_stamp = pd.DataFrame(columns=['date']) df_stamp.date = list(tmp_stamp.date.values) + list(pred_dates[1:]) if self.timeenc == 0: df_stamp['month'] = df_stamp.date.apply(lambda row: row.month, 1) df_stamp['day'] = df_stamp.date.apply(lambda row: row.day, 1) df_stamp['weekday'] = df_stamp.date.apply(lambda row: row.weekday(), 1) df_stamp['hour'] = df_stamp.date.apply(lambda row: row.hour, 1) df_stamp['minute'] = df_stamp.date.apply(lambda row: row.minute, 1) df_stamp['minute'] = df_stamp.minute.map(lambda x: x // 15) data_stamp = df_stamp.drop(['date'], axis=1).values elif self.timeenc == 1: data_stamp = time_features(pd.to_datetime(df_stamp['date'].values), freq=self.freq) data_stamp = data_stamp.transpose(1, 0) self.data_x = data[border1:border2] if self.inverse: self.data_y = df_data.values[border1:border2] else: self.data_y = data[border1:border2] self.data_stamp = data_stamp def __getitem__(self, index): s_begin = index s_end = s_begin + self.seq_len r_begin = s_end - self.label_len r_end = r_begin + self.label_len + self.pred_len seq_x = self.data_x[s_begin:s_end] if self.inverse: seq_y = self.data_x[r_begin:r_begin + self.label_len] else: seq_y = self.data_y[r_begin:r_begin + self.label_len] seq_x_mark = self.data_stamp[s_begin:s_end] seq_y_mark = self.data_stamp[r_begin:r_end] return seq_x, seq_y, seq_x_mark, seq_y_mark def __len__(self): return len(self.data_x) - self.seq_len + 1 def inverse_transform(self, data): return self.scaler.inverse_transform(data) ================================================ FILE: TimeMachine_supervised/exp/exp_basic.py ================================================ import os import torch import numpy as np class Exp_Basic(object): def __init__(self, args): self.args = args self.device = self._acquire_device() self.model = self._build_model().to(self.device) def _build_model(self): raise NotImplementedError return None def _acquire_device(self): if self.args.use_gpu: os.environ["CUDA_VISIBLE_DEVICES"] = str( self.args.gpu) if not self.args.use_multi_gpu else self.args.devices device = torch.device('cuda:{}'.format(self.args.gpu)) print('Use GPU: cuda:{}'.format(self.args.gpu)) else: device = torch.device('cpu') print('Use CPU') return device def _get_data(self): pass def vali(self): pass def train(self): pass def test(self): pass ================================================ FILE: TimeMachine_supervised/exp/exp_main.py ================================================ from data_provider.data_factory import data_provider from exp.exp_basic import Exp_Basic from models import TimeMachine from utils.tools import EarlyStopping, adjust_learning_rate, visual, test_params_flop from utils.metrics import metric import numpy as np import torch import torch.nn as nn from torch import optim from torch.optim import lr_scheduler import pandas as pd import os import time import warnings import matplotlib.pyplot as plt import numpy as np warnings.filterwarnings('ignore') class Exp_Main(Exp_Basic): def __init__(self, args): super(Exp_Main, self).__init__(args) def _build_model(self): model_dict = { 'TimeMachine':TimeMachine } model = model_dict[self.args.model].Model(self.args).float() if self.args.use_multi_gpu and self.args.use_gpu: model = nn.DataParallel(model, device_ids=self.args.device_ids) return model def _get_data(self, flag): data_set, data_loader = data_provider(self.args, flag) return data_set, data_loader def _select_optimizer(self): model_optim = optim.Adam(self.model.parameters(), lr=self.args.learning_rate) return model_optim def _select_criterion(self): criterion = nn.MSELoss() return criterion def vali(self, vali_data, vali_loader, criterion): total_loss = [] self.model.eval() with torch.no_grad(): for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(vali_loader): batch_x = batch_x.float().to(self.device) batch_y = batch_y.float() batch_x_mark = batch_x_mark.float().to(self.device) batch_y_mark = batch_y_mark.float().to(self.device) # decoder input dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) # encoder - decoder if self.args.use_amp: with torch.cuda.amp.autocast(): if 'Machine' in self.args.model: outputs = self.model(batch_x) else: if 'Machine' in self.args.model: outputs = self.model(batch_x) f_dim = -1 if self.args.features == 'MS' else 0 outputs = outputs[:, -self.args.pred_len:, f_dim:] batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) pred = outputs.detach().cpu() true = batch_y.detach().cpu() loss = criterion(pred, true) total_loss.append(loss) total_loss = np.average(total_loss) self.model.train() return total_loss def train(self, setting): train_data, train_loader = self._get_data(flag='train') vali_data, vali_loader = self._get_data(flag='val') test_data, test_loader = self._get_data(flag='test') path = os.path.join(self.args.checkpoints, setting) if not os.path.exists(path): os.makedirs(path) time_now = time.time() train_steps = len(train_loader) early_stopping = EarlyStopping(patience=self.args.patience, verbose=True) model_optim = self._select_optimizer() criterion = self._select_criterion() if self.args.use_amp: scaler = torch.cuda.amp.GradScaler() scheduler = lr_scheduler.OneCycleLR(optimizer = model_optim, steps_per_epoch = train_steps, pct_start = self.args.pct_start, epochs = self.args.train_epochs, max_lr = self.args.learning_rate) for epoch in range(self.args.train_epochs): iter_count = 0 train_loss = [] self.model.train() epoch_time = time.time() for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(train_loader): iter_count += 1 model_optim.zero_grad() batch_x = batch_x.float().to(self.device) batch_y = batch_y.float().to(self.device) batch_x_mark = batch_x_mark.float().to(self.device) batch_y_mark = batch_y_mark.float().to(self.device) # decoder input dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) # encoder - decoder if self.args.use_amp: with torch.cuda.amp.autocast(): if 'Machine' in self.args.model: outputs = self.model(batch_x) f_dim = -1 if self.args.features == 'MS' else 0 outputs = outputs[:, -self.args.pred_len:, f_dim:] batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) loss = criterion(outputs, batch_y) train_loss.append(loss.item()) else: if 'Machine' in self.args.model: outputs = self.model(batch_x) # print(outputs.shape,batch_y.shape) f_dim = -1 if self.args.features == 'MS' else 0 outputs = outputs[:, -self.args.pred_len:, f_dim:] batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) loss = criterion(outputs, batch_y) train_loss.append(loss.item()) if (i + 1) % 100 == 0: print("\titers: {0}, epoch: {1} | loss: {2:.7f}".format(i + 1, epoch + 1, loss.item())) speed = (time.time() - time_now) / iter_count left_time = speed * ((self.args.train_epochs - epoch) * train_steps - i) print('\tspeed: {:.4f}s/iter; left time: {:.4f}s'.format(speed, left_time)) iter_count = 0 time_now = time.time() if self.args.use_amp: scaler.scale(loss).backward() scaler.step(model_optim) scaler.update() else: loss.backward() model_optim.step() if self.args.lradj == 'TST': adjust_learning_rate(model_optim, scheduler, epoch + 1, self.args, printout=False) scheduler.step() print("Epoch: {} cost time: {}".format(epoch + 1, time.time() - epoch_time)) train_loss = np.average(train_loss) vali_loss = self.vali(vali_data, vali_loader, criterion) test_loss = self.vali(test_data, test_loader, criterion) print("Epoch: {0}, Steps: {1} | Train Loss: {2:.7f} Vali Loss: {3:.7f} Test Loss: {4:.7f}".format( epoch + 1, train_steps, train_loss, vali_loss, test_loss)) early_stopping(vali_loss, self.model, path) if early_stopping.early_stop: print("Early stopping") break if self.args.lradj != 'TST': adjust_learning_rate(model_optim, scheduler, epoch + 1, self.args) else: print('Updating learning rate to {}'.format(scheduler.get_last_lr()[0])) best_model_path = path + '/' + 'checkpoint.pth' self.model.load_state_dict(torch.load(best_model_path)) return self.model def test(self, setting, test=0): test_data, test_loader = self._get_data(flag='test') if test: print('loading model') self.model.load_state_dict(torch.load(os.path.join('./checkpoints/' + setting, 'checkpoint.pth'))) preds = [] trues = [] inputx = [] folder_path = './test_results/' + setting + '/' if not os.path.exists(folder_path): os.makedirs(folder_path) self.model.eval() with torch.no_grad(): for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(test_loader): batch_x = batch_x.float().to(self.device) batch_y = batch_y.float().to(self.device) batch_x_mark = batch_x_mark.float().to(self.device) batch_y_mark = batch_y_mark.float().to(self.device) # decoder input dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) # encoder - decoder if self.args.use_amp: with torch.cuda.amp.autocast(): if 'Machine' in self.args.model: outputs = self.model(batch_x) else: if 'Machine' in self.args.model: outputs = self.model(batch_x) f_dim = -1 if self.args.features == 'MS' else 0 # print(outputs.shape,batch_y.shape) outputs = outputs[:, -self.args.pred_len:, f_dim:] batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) outputs = outputs.detach().cpu().numpy() batch_y = batch_y.detach().cpu().numpy() pred = outputs # outputs.detach().cpu().numpy() # .squeeze() true = batch_y # batch_y.detach().cpu().numpy() # .squeeze() preds.append(pred) trues.append(true) inputx.append(batch_x.detach().cpu().numpy()) if i % 20 == 0: input = batch_x.detach().cpu().numpy() gt = np.concatenate((input[0, :, -1], true[0, :, -1]), axis=0) prd = np.concatenate((input[0, :, -1], pred[0, :, -1]), axis=0) visual(gt, prd, os.path.join(folder_path, str(i) + '.pdf')) if self.args.test_flop: test_params_flop((batch_x.shape[1],batch_x.shape[2])) exit() preds = np.array(preds) trues = np.array(trues) inputx = np.array(inputx) preds = preds.reshape(-1, preds.shape[-2], preds.shape[-1]) trues = trues.reshape(-1, trues.shape[-2], trues.shape[-1]) inputx = inputx.reshape(-1, inputx.shape[-2], inputx.shape[-1]) # result save folder_path = './results/' + setting + '/' if not os.path.exists(folder_path): os.makedirs(folder_path) mae, mse, rmse, mape, mspe, rse, corr = metric(preds, trues) print('mse:{}, mae:{}, rse:{}'.format(mse, mae, rse)) f = open("result.txt", 'a') f.write(setting + " \n") f.write('mse:{}, mae:{}, rse:{}'.format(mse, mae, rse)) f.write('\n') f.write('\n') f.close() temp_df = pd.DataFrame() temp_df['Seed']=[self.args.random_seed] temp_df['Model']=[self.args.model] temp_df['seq_len']=[self.args.seq_len] temp_df['label_len']=[self.args.label_len] temp_df['pred_len']=[self.args.pred_len] temp_df['n1']=[self.args.n1] temp_df['n2']=[self.args.n2] temp_df['dropout']=[self.args.dropout] temp_df['train_epochs']=[self.args.train_epochs] temp_df['batch']=[self.args.batch_size] temp_df['patience']=[self.args.patience] temp_df['LR']=[self.args.learning_rate] temp_df['dropout']=[self.args.dropout] temp_df['ch_ind']=[self.args.ch_ind] temp_df['revin']=[self.args.revin] temp_df['e_fact']=[self.args.e_fact] temp_df['dconv']=[self.args.dconv] temp_df['MSE']=[mse] temp_df['MAE']=[mae] temp_df['residual']=[self.args.residual] temp_df['d_state']=[self.args.d_state] temp_df['checkpoint_path']=[setting] if not os.path.exists('./csv_results/'+'result_'+self.args.data_path): temp_df.to_csv('./csv_results/'+'result_'+self.args.data_path, index=False) else: result_df=pd.read_csv('./csv_results/'+'result_'+self.args.data_path) result_df = pd.concat([result_df,temp_df],ignore_index=True) result_df.to_csv('./csv_results/'+'result_'+self.args.data_path, index=False) # np.save(folder_path + 'metrics.npy', np.array([mae, mse, rmse, mape, mspe,rse, corr])) np.save(folder_path + 'pred.npy', preds) np.save(folder_path + 'true.npy', trues) np.save(folder_path + 'x.npy', inputx) return def predict(self, setting, load=False): pred_data, pred_loader = self._get_data(flag='pred') if load: path = os.path.join(self.args.checkpoints, setting) best_model_path = path + '/' + 'checkpoint.pth' self.model.load_state_dict(torch.load(best_model_path)) preds = [] self.model.eval() with torch.no_grad(): for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(pred_loader): batch_x = batch_x.float().to(self.device) batch_y = batch_y.float() batch_x_mark = batch_x_mark.float().to(self.device) batch_y_mark = batch_y_mark.float().to(self.device) # decoder input dec_inp = torch.zeros([batch_y.shape[0], self.args.pred_len, batch_y.shape[2]]).float().to(batch_y.device) dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) # encoder - decoder if self.args.use_amp: with torch.cuda.amp.autocast(): if 'Machine' in self.args.model: outputs = self.model(batch_x) else: if 'Machine' in self.args.model: outputs = self.model(batch_x) pred = outputs.detach().cpu().numpy() # .squeeze() preds.append(pred) preds = np.array(preds) preds = preds.reshape(-1, preds.shape[-2], preds.shape[-1]) # result save folder_path = './results/' + setting + '/' if not os.path.exists(folder_path): os.makedirs(folder_path) np.save(folder_path + 'real_prediction.npy', preds) return ================================================ FILE: TimeMachine_supervised/models/TimeMachine.py ================================================ import torch from mamba_ssm import Mamba from RevIN.RevIN import RevIN class Model(torch.nn.Module): def __init__(self,configs): super(Model, self).__init__() self.configs=configs if self.configs.revin==1: self.revin_layer = RevIN(self.configs.enc_in) self.lin1=torch.nn.Linear(self.configs.seq_len,self.configs.n1) self.dropout1=torch.nn.Dropout(self.configs.dropout) self.lin2=torch.nn.Linear(self.configs.n1,self.configs.n2) self.dropout2=torch.nn.Dropout(self.configs.dropout) if self.configs.ch_ind==1: self.d_model_param1=1 self.d_model_param2=1 else: self.d_model_param1=self.configs.n2 self.d_model_param2=self.configs.n1 self.mamba1=Mamba(d_model=self.d_model_param1,d_state=self.configs.d_state,d_conv=self.configs.dconv,expand=self.configs.e_fact) self.mamba2=Mamba(d_model=self.configs.n2,d_state=self.configs.d_state,d_conv=self.configs.dconv,expand=self.configs.e_fact) self.mamba3=Mamba(d_model=self.configs.n1,d_state=self.configs.d_state,d_conv=self.configs.dconv,expand=self.configs.e_fact) self.mamba4=Mamba(d_model=self.d_model_param2,d_state=self.configs.d_state,d_conv=self.configs.dconv,expand=self.configs.e_fact) self.lin3=torch.nn.Linear(self.configs.n2,self.configs.n1) self.lin4=torch.nn.Linear(2*self.configs.n1,self.configs.pred_len) def forward(self, x): if self.configs.revin==1: x=self.revin_layer(x,'norm') else: means = x.mean(1, keepdim=True).detach() x = x - means stdev = torch.sqrt(torch.var(x, dim=1, keepdim=True, unbiased=False) + 1e-5) x /= stdev x=torch.permute(x,(0,2,1)) if self.configs.ch_ind==1: x=torch.reshape(x,(x.shape[0]*x.shape[1],1,x.shape[2])) x=self.lin1(x) x_res1=x x=self.dropout1(x) x3=self.mamba3(x) if self.configs.ch_ind==1: x4=torch.permute(x,(0,2,1)) else: x4=x x4=self.mamba4(x4) if self.configs.ch_ind==1: x4=torch.permute(x4,(0,2,1)) x4=x4+x3 x=self.lin2(x) x_res2=x x=self.dropout2(x) if self.configs.ch_ind==1: x1=torch.permute(x,(0,2,1)) else: x1=x x1=self.mamba1(x1) if self.configs.ch_ind==1: x1=torch.permute(x1,(0,2,1)) x2=self.mamba2(x) if self.configs.residual==1: x=x1+x_res2+x2 else: x=x1+x2 x=self.lin3(x) if self.configs.residual==1: x=x+x_res1 x=torch.cat([x,x4],dim=2) x=self.lin4(x) if self.configs.ch_ind==1: x=torch.reshape(x,(-1,self.configs.enc_in,self.configs.pred_len)) x=torch.permute(x,(0,2,1)) if self.configs.revin==1: x=self.revin_layer(x,'denorm') else: x = x * (stdev[:, 0, :].unsqueeze(1).repeat(1, self.configs.pred_len, 1)) x = x + (means[:, 0, :].unsqueeze(1).repeat(1, self.configs.pred_len, 1)) return x ================================================ FILE: TimeMachine_supervised/requirements.txt ================================================ numpy matplotlib pandas scikit-learn torch mamba-ssm causal-conv1d>=1.2.0 ================================================ FILE: TimeMachine_supervised/run_longExp.py ================================================ import argparse import os import torch from exp.exp_main import Exp_Main import random import numpy as np if __name__ == '__main__': parser = argparse.ArgumentParser(description='Time Series Forecasting') # RANDOM SEED parser.add_argument('--random_seed', type=int, default=2021, help='random seed') # BASIC CONFIG parser.add_argument('--is_training', type=int, required=True, default=1, help='status') parser.add_argument('--model_id', type=str, required=True, default='test', help='model id') parser.add_argument('--model', type=str, required=True, default='Autoformer', help='model name, options: [TimeMachine]') parser.add_argument('--model_id_name', type=str, required=False, default='custom', help='model id name') # DATALOADER parser.add_argument('--data', type=str, required=True, default='ETTm1', help='dataset type') parser.add_argument('--root_path', type=str, default='./data/ETT/', help='root path of the data file') parser.add_argument('--data_path', type=str, default='ETTh1.csv', help='data file') parser.add_argument('--features', type=str, default='M', help='forecasting task, options:[M, S, MS]; M:multivariate predict multivariate, S:univariate predict univariate, MS:multivariate predict univariate') parser.add_argument('--target', type=str, default='OT', help='target feature in S or MS task') parser.add_argument('--freq', type=str, default='h', help='freq for time features encoding, options:[s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly], you can also use more detailed freq like 15min or 3h') parser.add_argument('--checkpoints', type=str, default='./checkpoints/', help='location of model checkpoints') # FORECASTING TASK parser.add_argument('--seq_len', type=int, default=96, help='input sequence length') parser.add_argument('--label_len', type=int, default=48, help='start token length') parser.add_argument('--pred_len', type=int, default=96, help='prediction sequence length') parser.add_argument('--n1',type=int,default=256,help='First Embedded representation') parser.add_argument('--n2',type=int,default=128,help='Second Embedded representation') # METHOD parser.add_argument('--revin', type=int, default=1, help='RevIN; True 1 False 0') parser.add_argument('--ch_ind', type=int, default=1, help='Channel Independence; True 1 False 0') parser.add_argument('--residual', type=int, default=1, help='Residual Connection; True 1 False 0') parser.add_argument('--d_state', type=int, default=256, help='d_state parameter of Mamba') parser.add_argument('--dconv', type=int, default=2, help='d_conv parameter of Mamba') parser.add_argument('--e_fact', type=int, default=1, help='expand factor parameter of Mamba') parser.add_argument('--enc_in', type=int, default=7, help='encoder input size') #Use this hyperparameter as the number of channels parser.add_argument('--dropout', type=float, default=0.05, help='dropout') parser.add_argument('--embed', type=str, default='timeF', help='time features encoding, options:[timeF, fixed, learned]') parser.add_argument('--do_predict', action='store_true', help='whether to predict unseen future data') # OPTIMIZATION parser.add_argument('--num_workers', type=int, default=10, help='data loader num workers') parser.add_argument('--itr', type=int, default=2, help='experiments times') parser.add_argument('--train_epochs', type=int, default=100, help='train epochs') parser.add_argument('--batch_size', type=int, default=16, help='batch size of train input data') parser.add_argument('--patience', type=int, default=100, help='early stopping patience') parser.add_argument('--learning_rate', type=float, default=0.0001, help='optimizer learning rate') parser.add_argument('--des', type=str, default='test', help='exp description') parser.add_argument('--loss', type=str, default='mse', help='loss function') parser.add_argument('--lradj', type=str, default='type3', help='adjust learning rate') parser.add_argument('--pct_start', type=float, default=0.3, help='pct_start') parser.add_argument('--use_amp', action='store_true', help='use automatic mixed precision training', default=False) # GPU parser.add_argument('--use_gpu', type=bool, default=True, help='use gpu') parser.add_argument('--gpu', type=int, default=0, help='gpu') parser.add_argument('--use_multi_gpu', action='store_true', help='use multiple gpus', default=False) parser.add_argument('--devices', type=str, default='0,1,2,3', help='device ids of multile gpus') parser.add_argument('--test_flop', action='store_true', default=False, help='See utils/tools for usage') args = parser.parse_args() # random seed fix_seed = args.random_seed random.seed(fix_seed) torch.manual_seed(fix_seed) np.random.seed(fix_seed) args.model_id_name=args.data_path[:-4] args.use_gpu = True if torch.cuda.is_available() and args.use_gpu else False if args.use_gpu and args.use_multi_gpu: args.dvices = args.devices.replace(' ', '') device_ids = args.devices.split(',') args.device_ids = [int(id_) for id_ in device_ids] args.gpu = args.device_ids[0] print('Args in experiment:') print(args) Exp = Exp_Main if args.is_training: for ii in range(args.itr): # setting record of experiments setting = '{}_{}_{}_ft{}_sl{}_ll{}_pl{}_n1{}_n2{}_dr{}_cin{}_rin{}_res{}_dst{}_dconv{}_efact{}'.format( args.model_id, args.model, args.model_id_name, args.features, args.seq_len, args.label_len, args.pred_len, args.n1, args.n2, args.dropout, args.ch_ind, args.revin, args.residual, args.d_state, args.dconv, args.e_fact) exp = Exp(args) # set experiments print('>>>>>>>start training : {}>>>>>>>>>>>>>>>>>>>>>>>>>>'.format(setting)) exp.train(setting) print('>>>>>>>testing : {}<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<'.format(setting)) exp.test(setting) if args.do_predict: print('>>>>>>>predicting : {}<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<'.format(setting)) exp.predict(setting, True) torch.cuda.empty_cache() else: ii = 0 setting = '{}_{}_{}_ft{}_sl{}_ll{}_pl{}_n1{}_n2{}_dr{}_cin{}_rin{}_res{}_dst{}_dconv{}_efact'.format( args.model_id, args.model, args.model_id_name, args.features, args.seq_len, args.label_len, args.pred_len, args.n1, args.n2, args.dropout, args.ch_ind, args.revin, args.residual, args.d_state, args.dconv, args.e_fact) exp = Exp(args) # set experiments print('>>>>>>>testing : {}<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<'.format(setting)) exp.test(setting, test=1) torch.cuda.empty_cache() ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/electricity.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/electricity data_path_name=electricity.csv model_id_name=electricity data_name=custom rin=1 random_seed=2024 one=96 two=192 three=336 four=720 residual=1 fc_drop=0.0 dstate=256 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=512 n2=128 fc_drop=0.0 fi if [ $pred_len -eq $two ] then n1=512 n2=256 fc_drop=0.0 fi if [ $pred_len -eq $three ] then n1=512 n2=256 fc_drop=0.4 fi if [ $pred_len -eq $four ] then n1=512 n2=8 fc_drop=0.0 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 321 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop\ --revin 1\ --ch_ind 0\ --residual $residual\ --dconv $dconv \ --d_state $dstate\ --e_fact $e_fact\ --des 'Exp' \ --lradj '5' \ --pct_start 0.2 \ --train_epochs 100\ --itr 1 --batch_size 16 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/etth1.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/ETT-small data_path_name=ETTh1.csv model_id_name=ETTh1 data_name=ETTh1 rin=1 random_seed=2024 one=96 two=192 three=336 four=720 residual=1 fc_drop=0.7 dstate=256 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=512 n2=32 fi if [ $pred_len -eq $two ] then n1=512 n2=64 fi if [ $pred_len -eq $three ] then n1=512 n2=128 fi if [ $pred_len -eq $four ] then n1=128 n2=16 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 7 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop\ --revin 1\ --ch_ind 1\ --residual $residual\ --dconv $dconv \ --d_state $dstate\ --e_fact $e_fact\ --des 'Exp' \ --train_epochs 100\ --itr 1 --batch_size 512 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/etth2.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/ETT-small data_path_name=ETTh2.csv model_id_name=ETTh2 data_name=ETTh2 one=96 two=192 three=336 four=720 residual=1 rin=1 fc_drop=0.7 dstate=256 random_seed=2024 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=128 n2=32 fi if [ $pred_len -eq $two ] then n1=256 n2=32 fi if [ $pred_len -eq $three ] then n1=512 n2=64 fi if [ $pred_len -eq $four ] then n1=256 n2=128 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 7 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop\ --revin 1\ --ch_ind 1\ --d_state $dstate\ --dconv $dconv \ --residual $residual\ --e_fact $e_fact\ --des 'Exp' \ --train_epochs 100\ --itr 1 --batch_size 512 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/ettm1.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/ETT-small data_path_name=ETTm1.csv model_id_name=ETTm1 data_name=ETTm1 rin=1 random_seed=2024 one=96 two=192 three=336 four=720 residual=1 fc_drop=0.6 dstate=256 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=256 n2=64 fi if [ $pred_len -eq $two ] then n1=512 n2=32 fi if [ $pred_len -eq $three ] then n1=128 n2=16 fi if [ $pred_len -eq $four ] then n1=512 n2=128 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 7 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop\ --des 'Exp' \ --train_epochs 100\ --itr 1 --batch_size 1024 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/ettm2.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/ETT-small data_path_name=ETTm2.csv model_id_name=ETTm2 data_name=ETTm2 rin=1 random_seed=2024 one=96 two=192 three=336 four=720 residual=1 fc_drop=0.6 dstate=256 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=256 n2=32 fi if [ $pred_len -eq $two ] then n1=128 n2=64 fi if [ $pred_len -eq $three ] then n1=256 n2=128 fi if [ $pred_len -eq $four ] then n1=256 n2=128 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 7 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop\ --des 'Exp' \ --train_epochs 100\ --itr 1 --batch_size 1024 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/traffic.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/traffic data_path_name=traffic.csv model_id_name=traffic data_name=custom rin=0 random_seed=2024 one=96 two=192 three=336 four=720 residual=1 fc_drop=0.3 dstate=256 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=512 n2=16 fc_drop=0.3 fi if [ $pred_len -eq $two ] then n1=512 n2=256 fc_drop=0.1 fi if [ $pred_len -eq $three ] then n1=512 n2=256 fc_drop=0.1 fi if [ $pred_len -eq $four ] then n1=512 n2=128 fc_drop=0.1 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 862 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop \ --revin 0 \ --ch_ind 0 \ --residual $residual \ --dconv $dconv \ --d_state $dstate \ --e_fact $e_fact \ --des 'Exp' \ --lradj '5' \ --pct_start 0.2 \ --train_epochs 100 \ --itr 1 --batch_size 16 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/scripts/TimeMachine/weather.sh ================================================ if [ ! -d "./logs" ]; then mkdir ./logs fi if [ ! -d "./logs/LongForecasting" ]; then mkdir ./logs/LongForecasting fi if [ ! -d "./csv_results" ]; then mkdir ./csv_results fi if [ ! -d "./results" ]; then mkdir ./results fi if [ ! -d "./test_results" ]; then mkdir ./test_results fi model_name=TimeMachine root_path_name=../data/weather data_path_name=weather.csv model_id_name=weather data_name=custom rin=1 random_seed=2024 one=96 two=192 three=336 four=720 residual=1 fc_drop=0.1 dstate=256 dconv=2 for seq_len in 96 do for pred_len in 96 192 336 720 do for e_fact in 1 do if [ $pred_len -eq $one ] then n1=128 n2=16 fc_drop=0.1 fi if [ $pred_len -eq $two ] then n1=512 n2=128 fc_drop=0.5 fi if [ $pred_len -eq $three ] then n1=128 n2=64 fc_drop=0.0 fi if [ $pred_len -eq $four ] then n1=512 n2=256 fc_drop=0.0 fi python -u run_longExp.py \ --random_seed $random_seed \ --is_training 1 \ --root_path $root_path_name \ --data_path $data_path_name \ --model_id $model_id_name_$seq_len'_'$pred_len \ --model $model_name \ --data $data_name \ --features M \ --seq_len $seq_len \ --pred_len $pred_len \ --enc_in 21 \ --n1 $n1 \ --n2 $n2 \ --dropout $fc_drop\ --revin 1\ --ch_ind 1\ --residual $residual\ --dconv $dconv \ --d_state $dstate\ --e_fact $e_fact\ --des 'Exp' \ --lradj 'constant'\ --pct_start 0.2\ --itr 1 --batch_size 512 --learning_rate 0.001 >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log done done done ================================================ FILE: TimeMachine_supervised/utils/masking.py ================================================ import torch class TriangularCausalMask(): def __init__(self, B, L, device="cpu"): mask_shape = [B, 1, L, L] with torch.no_grad(): self._mask = torch.triu(torch.ones(mask_shape, dtype=torch.bool), diagonal=1).to(device) @property def mask(self): return self._mask class ProbMask(): def __init__(self, B, H, L, index, scores, device="cpu"): _mask = torch.ones(L, scores.shape[-1], dtype=torch.bool).to(device).triu(1) _mask_ex = _mask[None, None, :].expand(B, H, L, scores.shape[-1]) indicator = _mask_ex[torch.arange(B)[:, None, None], torch.arange(H)[None, :, None], index, :].to(device) self._mask = indicator.view(scores.shape).to(device) @property def mask(self): return self._mask ================================================ FILE: TimeMachine_supervised/utils/metrics.py ================================================ import numpy as np def RSE(pred, true): return np.sqrt(np.sum((true - pred) ** 2)) / np.sqrt(np.sum((true - true.mean()) ** 2)) def CORR(pred, true): u = ((true - true.mean(0)) * (pred - pred.mean(0))).sum(0) d = np.sqrt(((true - true.mean(0)) ** 2 * (pred - pred.mean(0)) ** 2).sum(0)) d += 1e-12 return 0.01*(u / d).mean(-1) def MAE(pred, true): return np.mean(np.abs(pred - true)) def MSE(pred, true): return np.mean((pred - true) ** 2) def RMSE(pred, true): return np.sqrt(MSE(pred, true)) def MAPE(pred, true): return np.mean(np.abs((pred - true) / true)) def MSPE(pred, true): return np.mean(np.square((pred - true) / true)) def metric(pred, true): mae = MAE(pred, true) mse = MSE(pred, true) rmse = RMSE(pred, true) mape = MAPE(pred, true) mspe = MSPE(pred, true) rse = RSE(pred, true) corr = CORR(pred, true) return mae, mse, rmse, mape, mspe, rse, corr ================================================ FILE: TimeMachine_supervised/utils/timefeatures.py ================================================ from typing import List import numpy as np import pandas as pd from pandas.tseries import offsets from pandas.tseries.frequencies import to_offset class TimeFeature: def __init__(self): pass def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: pass def __repr__(self): return self.__class__.__name__ + "()" class SecondOfMinute(TimeFeature): """Minute of hour encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return index.second / 59.0 - 0.5 class MinuteOfHour(TimeFeature): """Minute of hour encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return index.minute / 59.0 - 0.5 class HourOfDay(TimeFeature): """Hour of day encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return index.hour / 23.0 - 0.5 class DayOfWeek(TimeFeature): """Hour of day encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return index.dayofweek / 6.0 - 0.5 class DayOfMonth(TimeFeature): """Day of month encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return (index.day - 1) / 30.0 - 0.5 class DayOfYear(TimeFeature): """Day of year encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return (index.dayofyear - 1) / 365.0 - 0.5 class MonthOfYear(TimeFeature): """Month of year encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return (index.month - 1) / 11.0 - 0.5 class WeekOfYear(TimeFeature): """Week of year encoded as value between [-0.5, 0.5]""" def __call__(self, index: pd.DatetimeIndex) -> np.ndarray: return (index.isocalendar().week - 1) / 52.0 - 0.5 def time_features_from_frequency_str(freq_str: str) -> List[TimeFeature]: """ Returns a list of time features that will be appropriate for the given frequency string. Parameters ---------- freq_str Frequency string of the form [multiple][granularity] such as "12H", "5min", "1D" etc. """ features_by_offsets = { offsets.YearEnd: [], offsets.QuarterEnd: [MonthOfYear], offsets.MonthEnd: [MonthOfYear], offsets.Week: [DayOfMonth, WeekOfYear], offsets.Day: [DayOfWeek, DayOfMonth, DayOfYear], offsets.BusinessDay: [DayOfWeek, DayOfMonth, DayOfYear], offsets.Hour: [HourOfDay, DayOfWeek, DayOfMonth, DayOfYear], offsets.Minute: [ MinuteOfHour, HourOfDay, DayOfWeek, DayOfMonth, DayOfYear, ], offsets.Second: [ SecondOfMinute, MinuteOfHour, HourOfDay, DayOfWeek, DayOfMonth, DayOfYear, ], } offset = to_offset(freq_str) for offset_type, feature_classes in features_by_offsets.items(): if isinstance(offset, offset_type): return [cls() for cls in feature_classes] supported_freq_msg = f""" Unsupported frequency {freq_str} The following frequencies are supported: Y - yearly alias: A M - monthly W - weekly D - daily B - business days H - hourly T - minutely alias: min S - secondly """ raise RuntimeError(supported_freq_msg) def time_features(dates, freq='h'): return np.vstack([feat(dates) for feat in time_features_from_frequency_str(freq)]) ================================================ FILE: TimeMachine_supervised/utils/tools.py ================================================ import numpy as np import torch import matplotlib.pyplot as plt import time plt.switch_backend('agg') def adjust_learning_rate(optimizer, scheduler, epoch, args, printout=True): # lr = args.learning_rate * (0.2 ** (epoch // 2)) if args.lradj == 'type1': lr_adjust = {epoch: args.learning_rate * (0.5 ** ((epoch - 1) // 1))} elif args.lradj == 'type2': lr_adjust = { 2: 5e-5, 4: 1e-5, 6: 5e-6, 8: 1e-6, 10: 5e-7, 15: 1e-7, 20: 5e-8 } elif args.lradj == 'type3': lr_adjust = {epoch: args.learning_rate if epoch < 3 else args.learning_rate * (0.9 ** ((epoch - 3) // 1))} elif args.lradj == 'constant': lr_adjust = {epoch: args.learning_rate} elif args.lradj == '3': lr_adjust = {epoch: args.learning_rate if epoch < 10 else args.learning_rate*0.1} elif args.lradj == '4': lr_adjust = {epoch: args.learning_rate if epoch < 15 else args.learning_rate*0.1} elif args.lradj == '5': lr_adjust = {epoch: args.learning_rate if epoch < 25 else args.learning_rate*0.1} elif args.lradj == '6': lr_adjust = {epoch: args.learning_rate if epoch < 5 else args.learning_rate*0.1} elif args.lradj == 'TST': lr_adjust = {epoch: scheduler.get_last_lr()[0]} if epoch in lr_adjust.keys(): lr = lr_adjust[epoch] for param_group in optimizer.param_groups: param_group['lr'] = lr if printout: print('Updating learning rate to {}'.format(lr)) class EarlyStopping: def __init__(self, patience=7, verbose=False, delta=0): self.patience = patience self.verbose = verbose self.counter = 0 self.best_score = None self.early_stop = False self.val_loss_min = np.Inf self.delta = delta def __call__(self, val_loss, model, path): score = -val_loss if self.best_score is None: self.best_score = score self.save_checkpoint(val_loss, model, path) elif score < self.best_score + self.delta: self.counter += 1 print(f'EarlyStopping counter: {self.counter} out of {self.patience}') if self.counter >= self.patience: self.early_stop = True else: self.best_score = score self.save_checkpoint(val_loss, model, path) self.counter = 0 def save_checkpoint(self, val_loss, model, path): if self.verbose: print(f'Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}). Saving model ...') torch.save(model.state_dict(), path + '/' + 'checkpoint.pth') self.val_loss_min = val_loss class dotdict(dict): """dot.notation access to dictionary attributes""" __getattr__ = dict.get __setattr__ = dict.__setitem__ __delattr__ = dict.__delitem__ class StandardScaler(): def __init__(self, mean, std): self.mean = mean self.std = std def transform(self, data): return (data - self.mean) / self.std def inverse_transform(self, data): return (data * self.std) + self.mean def visual(true, preds=None, name='./pic/test.pdf'): """ Results visualization """ plt.figure() plt.plot(true, label='GroundTruth', linewidth=2) if preds is not None: plt.plot(preds, label='Prediction', linewidth=2) plt.legend() plt.savefig(name, bbox_inches='tight') def test_params_flop(model,x_shape): """ If you want to thest former's flop, you need to give default value to inputs in model.forward(), the following code can only pass one argument to forward() """ model_params = 0 for parameter in model.parameters(): model_params += parameter.numel() print('INFO: Trainable parameter count: {:.2f}M'.format(model_params / 1000000.0)) from ptflops import get_model_complexity_info with torch.cuda.device(0): macs, params = get_model_complexity_info(model.cuda(), x_shape, as_strings=True, print_per_layer_stat=True) # print('Flops:' + flops) # print('Params:' + params) print('{:<30} {:<8}'.format('Computational complexity: ', macs)) print('{:<30} {:<8}'.format('Number of parameters: ', params))