[ { "path": ".gitignore", "content": ".vscode/\n__pycache__/\n.empty\n*.pyc\n*.egg-info\ndist/" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) [year] [fullname]\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "MANIFEST.in", "content": "include version requirements.txt" }, { "path": "README.md", "content": "# FM Synth Parameter Generator\n\nRandom machine learning experiments related to the classic Yamaha DX7\n\n## Dexed\n\nDexed is a linux open source DX7 emulator and was used heavily in the testing of this project\n\n## SYX\n\nDX7 and it's similar instruments are programmable bt\n\nformat [found here](https://github.com/asb2m10/dexed/tree/master/Documentation\n) under `sysexformat.txt`\n\n## Dataset\n\nBig thanks to Bobby Blues for collecting [these](http://bobbyblues.recup.ch/yamaha_dx7/dx7_patches.html) DX7 patches. This was the only data source\n\n# Directory structure\n```\nneuralDX7/\n├── constants.py\n├── datasets # modules to interface with preprocessed datasets \n│   ├── dx7_sysex_dataset.py\n│   └── __init__.py\n├── __init__.py\n├── models\n│   ├── attention # modules implementing transformer stack based on Attention Is All You Need\n│   │   ├── attention_encoder.py\n│   │   ├── attention_layer.py\n│   │   ├── attention.py\n│   │   ├── conditional_attention_encoder.py\n│   │   └── __init__.py\n│   ├── general\n│   │   ├── gelu_ff.py # two layer non linear layer using GeLU non-linearity \n│   │   └── __init__.py\n│   ├── __init__.py\n│   ├── stochastic_nodes # layers implementing stochastic transformations\n│   │   ├── __init__.py\n│   │   ├── normal.py\n│   │   └── triangular_sylvester.py\n│   ├── dx7_cnp.py # experimental modules\n│   ├── dx7_np.py # experimental modules\n│   ├── dx7_nsp.py # experimental modules\n│   ├── dx7_vae.py # working model used in production of thisdx7cartdoesnotexist.com\n│   └── utils.py\n├── solvers\n│   ├── dx7_np.py # experimental modules\n│   ├── dx7_nsp.py # experimental modules\n│   ├── dx7_patch_process.py # experimental modules\n│   ├── dx7_vae.py # working model used in production of thisdx7cartdoesnotexist.com\n│   ├── __init__.py\n│   └── utils.py\n└── utils.py\n```\n\n\n# thisdx7cartdoesnotexist.com\n\nIf you've found your way here through [thisdx7cartdoesnotexist.com](https://www.thisdx7cartdoesnotexist.com/) then this section will give an overview of how that site generates patches.\n\n\nThe model itself is defined under `NeuralDX7/models/dx7_vae.py`, it is a simple VAE with triangular sylvester flows implemented with attention layers over the parameters of the DX7. \n\nThe training code can be found in `NeuralDX7/solvers/dx7_vae.py` and is a fairly standard VAE+Flow optimisation setup \n\nFinally the training script itself, as well as various other scripts used to perform various functions with the trained model can be found under `projects/dx7_vae`. The scripts in here do the following:\n\n`duplicate_test.py` - samples randomly from the prior and calculates the number of identical patches. This was found to be around 99.9% unique\n\n`evaluate.py` - was a script designed to create a single cartridge and was used during development to ensure the model was outputting valid parameter configurations.\n\n`experiment.py` - contains the code to run the experiment. If you want to train your own version then start here.\n\n`features.py` - simple feature extractor for the dataset, used to calculate the model posterior.\n\n`interpolate.py` - takes two samples from the dataset and produces a cartridge that moves between the two in latent space over 32 steps\n\n`live.py` - uses jackd to provide a realtime interface to the model. This is really fun to play with as it lets you hook up a midi controller to control the latent variables of the model and update the parameters of your FM Synthesizer in real time. \n\nTo use it you will need jackd installed, add both your controller and FM synthsizer to the jack graph, update the names of the controller and fm synth in the `live.py` script and then run. Each of the first 8 midi cc controls recieved will be mapped to a latent dimension of the model.\n\n\n```\nimport torch\nimport mido\nfrom agoge import InferenceWorker\nfrom neuralDX7.utils import dx7_bulk_pack\n\n# load model (weights should download automatically)\nmodel = InferenceWorker('hasty-copper-dogfish', 'dx7-vae', with_data=False).model\n\n# sample latent from prior N(0,1)\nz = torch.randn(32, 8)\n\n# decode samples to logits\np_x = model.generate(z)\n\n# sample\nsample = p_x.logits.argmax(-1)\n\n# convert pytorch tensors to syx\nmsg = dx7_bulk_pack(sample.numpy().tolist())\n\nmido.write_syx_file('path/to/save.syx', [msg])\n```" }, { "path": "neuralDX7/__init__.py", "content": "\n\n\nfrom agoge import DEFAULTS, defaults_f\n\n\nDEFAULTS = defaults_f({\n 'ARTIFACT_ROOT': '~/agoge/artifacts'\n}, DEFAULTS)" }, { "path": "neuralDX7/constants.py", "content": "from pathlib import Path\nimport bitstruct\nimport mido\n\n\ndef take(take_from, n):\n for _ in range(n):\n yield next(take_from)\n\nN_OSC = 6\nN_VOICES = 32\n\ndef checksum(data):\n return (128-sum(data)&127)%128\n\nGLOBAL_VALID_RANGES = {\n 'PR1': range(0, 99+1),\n 'PR2': range(0, 99+1),\n 'PR3': range(0, 99+1),\n 'PR4': range(0, 99+1),\n 'PL1': range(0, 99+1),\n 'PL2': range(0, 99+1),\n 'PL3': range(0, 99+1),\n 'PL4': range(0, 99+1),\n 'ALG': range(0, 31+1),\n 'OKS': range(0, 1+1),\n 'FB': range(0, 7+1),\n 'LFS': range(0, 99+1),\n 'LFD': range(0, 99+1),\n 'LPMD': range(0, 99+1),\n 'LAMD': range(0, 99+1),\n 'LPMS': range(0, 7+1),\n 'LFW': range(0, 5+1),\n 'LKS': range(0, 1+1),\n 'TRNSP': range(0, 48+1),\n 'NAME CHAR 1': range(128),\n 'NAME CHAR 2': range(128),\n 'NAME CHAR 3': range(128),\n 'NAME CHAR 4': range(128),\n 'NAME CHAR 5': range(128),\n 'NAME CHAR 6': range(128),\n 'NAME CHAR 7': range(128),\n 'NAME CHAR 8': range(128),\n 'NAME CHAR 9': range(128),\n 'NAME CHAR 10': range(128),\n }\n\nOSCILLATOR_VALID_RANGES = {\n 'R1': range(0, 99+1),\n 'R2': range(0, 99+1),\n 'R3': range(0, 99+1),\n 'R4': range(0, 99+1),\n 'L1': range(0, 99+1),\n 'L2': range(0, 99+1),\n 'L3': range(0, 99+1),\n 'L4': range(0, 99+1),\n 'BP': range(0, 99+1),\n 'LD': range(0, 99+1),\n 'RD': range(0, 99+1),\n 'RC': range(0, 3+1),\n 'LC': range(0, 3+1),\n 'DET': range(0, 14+1),\n 'RS': range(0, 7+1),\n 'KVS': range(0, 7+1),\n 'AMS': range(0, 3+1),\n 'OL': range(0, 99+1),\n 'FC': range(0, 31+1),\n 'M': range(0, 1+1),\n 'FF': range(0, 99+1),\n}\n\nVOICE_PARAMETER_RANGES = {f'{i}_{key}': value for key, value in OSCILLATOR_VALID_RANGES.items() for i in range(N_OSC)}\nVOICE_PARAMETER_RANGES.update(GLOBAL_VALID_RANGES)\n\ndef verify(actual, ranges):\n assert set(actual.keys())==set(ranges.keys()), 'Params dont match'\n for key in actual:\n if not actual[key] in ranges[key]:\n return False\n return True\n\n\nHEADER_KEYS = [\n 'ID',\n 'Sub-status',\n 'format number',\n 'byte count',\n 'byte count',\n]\n\nGENERAL_KEYS = [\n 'PR1',\n 'PR2',\n 'PR3',\n 'PR4',\n 'PL1',\n 'PL2',\n 'PL3',\n 'PL4',\n 'ALG',\n 'OKS',\n 'FB',\n 'LFS',\n 'LFD',\n 'LPMD',\n 'LAMD',\n 'LPMS',\n 'LFW',\n 'LKS',\n 'TRNSP',\n 'NAME CHAR 1',\n 'NAME CHAR 2',\n 'NAME CHAR 3',\n 'NAME CHAR 4',\n 'NAME CHAR 5',\n 'NAME CHAR 6',\n 'NAME CHAR 7',\n 'NAME CHAR 8',\n 'NAME CHAR 9',\n 'NAME CHAR 10',\n]\n\nOSC_KEYS = [\n 'R1',\n 'R2',\n 'R3',\n 'R4',\n 'L1',\n 'L2',\n 'L3',\n 'L4',\n 'BP',\n 'LD',\n 'RD',\n 'RC',\n 'LC',\n 'DET',\n 'RS',\n 'KVS',\n 'AMS',\n 'OL',\n 'FC',\n 'M',\n 'FF',\n]\n\nFOOTER_KEYS = ['checksum']\n\n\nVOICE_KEYS = [f'{i}_{key}' for i in range(6) for key in OSC_KEYS] + \\\n GENERAL_KEYS \n\nKEYS = HEADER_KEYS + \\\n list(VOICE_KEYS * N_VOICES) + \\\n FOOTER_KEYS\n\n\n\nheader_bytes = [\n 'p1u7', # ID # (i=67; Yamaha)\n 'p1u7', # Sub-status (s=0) & channel number (n=0; ch 1)\n 'p1u7', # format number (f=9; 32 voices)\n 'p1u7', # byte count MS byte\n 'p1u7', # byte count LS byte (b=4096; 32 voices)\n]\n\n\n\n\ngeneral_parameter_bytes = [ \n 'p1u7', # PR1\n 'p1u7', # PR2\n 'p1u7', # PR3\n 'p1u7', # PR4\n 'p1u7', # PL1\n 'p1u7', # PL2\n 'p1u7', # PL3\n 'p1u7', # PL4\n 'p3u5', # ALG\n 'p4u1u3', # OKS| FB\n 'p1u7', # LFS\n 'p1u7', # LFD\n 'p1u7', # LPMD\n 'p1u7', # LAMD\n 'p1u3u3u1', # LPMS | LFW |LKS\n 'p1u7', # TRNSP\n 'p1u7', # NAME CHAR 1\n 'p1u7', # NAME CHAR 2\n 'p1u7', # NAME CHAR 3\n 'p1u7', # NAME CHAR 4\n 'p1u7', # NAME CHAR 5\n 'p1u7', # NAME CHAR 6\n 'p1u7', # NAME CHAR 7\n 'p1u7', # NAME CHAR 8\n 'p1u7', # NAME CHAR 9\n 'p1u7', # NAME CHAR 10\n]\n\nosc_parameter_bytes = [\n 'p1u7', # R1\n 'p1u7', # R2\n 'p1u7', # R3\n 'p1u7', # R4\n 'p1u7', # L1\n 'p1u7', # L2\n 'p1u7', # L3\n 'p1u7', # L4\n 'p1u7', # BP\n 'p1u7', # LD\n 'p1u7', # RD\n 'p4u2u2', # RC | LC \n 'p1u4u3', # DET | RS\n 'p3u3u2', # KVS | AMS\n 'p1u7', # OL\n 'p2u5u1', # FC | M\n 'p1u7' # FF\n]\n\nvoice_bytes = (osc_parameter_bytes * N_OSC) + general_parameter_bytes\n\ntail_bytes = [\n 'p1u7', # checksum\n]\n\n\nfull_string = ''.join(header_bytes + osc_parameter_bytes * 6 + general_parameter_bytes)\ndx7_struct = bitstruct.compile(full_string)\n\nvoice_struct = bitstruct.compile(''.join(voice_bytes), names=VOICE_KEYS)\nheader_struct = bitstruct.compile(''.join(header_bytes))\n\nN_PARAMS = len(VOICE_PARAMETER_RANGES)\nMAX_VALUE = max([max(i) for i in VOICE_PARAMETER_RANGES.values()]) + 1\n\n\n\"\"\"\nSYSEX Message: Bulk Data for 1 Voice\n------------------------------------\n bits hex description\n\n 11110000 F0 Status byte - start sysex\n 0iiiiiii 43 ID # (i=67; Yamaha)\n 0sssnnnn 00 Sub-status (s=0) & channel number (n=0; ch 1)\n 0fffffff 00 format number (f=0; 1 voice)\n 0bbbbbbb 01 byte count MS byte\n 0bbbbbbb 1B byte count LS byte (b=155; 1 voice)\n 0ddddddd ** data byte 1\n\n | | |\n\n 0ddddddd ** data byte 155\n 0eeeeeee ** checksum (masked 2's complement of sum of 155 bytes)\n 11110111 F7 Status - end sysex\n\n\n\n///////////////////////////////////////////////////////////\n\"\"\"\nclass DX7Single():\n HEADER = int('0x43', 0), int('0x00', 0), int('0x00', 0), int('0x01', 0), int('0x1B', 0)\n \n GENERAL_KEYS = [\n 'PR1',\n 'PR2',\n 'PR3',\n 'PR4',\n 'PL1',\n 'PL2',\n 'PL3',\n 'PL4',\n 'ALG',\n 'FB',\n 'OKS',\n 'LFS',\n 'LFD',\n 'LPMD',\n 'LAMD',\n 'LKS',\n 'LFW',\n 'LPMS',\n 'TRNSP',\n 'NAME CHAR 1',\n 'NAME CHAR 2',\n 'NAME CHAR 3',\n 'NAME CHAR 4',\n 'NAME CHAR 5',\n 'NAME CHAR 6',\n 'NAME CHAR 7',\n 'NAME CHAR 8',\n 'NAME CHAR 9',\n 'NAME CHAR 10',\n ]\n\n OSC_KEYS = [\n 'R1',\n 'R2',\n 'R3',\n 'R4',\n 'L1',\n 'L2',\n 'L3',\n 'L4',\n 'BP',\n 'LD',\n 'RD',\n 'LC',\n 'RC',\n 'RS',\n 'AMS',\n 'KVS',\n 'OL',\n 'M',\n 'FC',\n 'FF',\n 'DET',\n ]\n\n @staticmethod\n def keys():\n\n osc_keys = DX7Single.OSC_KEYS\n osc_params = [f'{i}_{param}' for i in range(N_OSC) for param in osc_keys]\n # print(osc_params)\n all = osc_params + DX7Single.GENERAL_KEYS\n return all\n\n @staticmethod\n def struct():\n return bitstruct.compile('p1u7'*155, names=DX7Single.keys())\n\n\n @staticmethod\n def to_syx(voices):\n\n\n assert len(voices)==1\n voice = voices[0]\n voices_bytes = bytes()\n voices_bytes = DX7Single.struct().pack(dict(zip(VOICE_KEYS, voice))) \n \n patch_checksum = [checksum(voices_bytes)]\n\n data = bytes(DX7Single.HEADER) \\\n + voices_bytes \\\n + bytes(patch_checksum)\n\n\n return mido.Message('sysex', data=data)\n \n\ndef consume_syx(path):\n\n path = Path(path).expanduser()\n try:\n preset = mido.read_syx_file(path.as_posix())[0]\n except IndexError as e:\n return None\n except ValueError as e:\n return None\n if len(preset.data) == 0:\n return None\n\n def get_voice(data):\n \n unpacked = voice_struct.unpack(data)\n\n if not verify(unpacked, VOICE_PARAMETER_RANGES):\n return None\n \n return unpacked\n\n get_header = header_struct.unpack\n sysex_iter = iter(preset.data)\n \n try:\n header = get_header(bytes(take(sysex_iter, len(header_bytes))))\n yield from (get_voice(bytes(take(sysex_iter, len(voice_bytes)))) for _ in range(N_VOICES))\n except RuntimeError:\n return None\n\nif __name__==\"__main__\":\n print(VOICE_KEYS)\n\n # print(DX7Single.to_syx(n)" }, { "path": "neuralDX7/datasets/__init__.py", "content": "from .dx7_sysex_dataset import DX7SysexDataset" }, { "path": "neuralDX7/datasets/dx7_sysex_dataset.py", "content": "from pathlib import Path\nimport numpy as np\nimport torch\nfrom neuralDX7 import DEFAULTS\n\n\n\n\nclass DX7SysexDataset():\n \"\"\"\n Pytorch Dataset module to provide access to precprocessed DX7 patch data\n \"\"\"\n \n\n def __init__(self, data_file='dx7.npy', root=DEFAULTS['ARTIFACTS_ROOT'], data_size=1.):\n \"\"\"\n data_file - the name of the prprocessed data\n root - the root directory for data\n data_size - how much of the data is used. good for development\n \"\"\"\n\n assert data_size <= 1\n self.data_size = data_size\n\n # initialise path handler\n if not isinstance(root, Path):\n root = Path(root).expanduser()\n\n # load data into memory\n self.data = np.load(root.joinpath(data_file)) \n\n def __getitem__(self, index):\n \n # turn the data item into a tensor and return\n item = torch.tensor(self.data[index].item()).long()\n\n return {'X': item}\n \n def __len__(self):\n return int(len(self.data) * self.data_size)\n\n\nif __name__ == \"__main__\":\n \n\n dataset = DX7SysexDataset()\n\n print([dataset[i] for i in np.random.randint(0, len(dataset)-1, 20)])" }, { "path": "neuralDX7/models/__init__.py", "content": "from .dx7_cnp import DX7PatchProcess\nfrom .dx7_np import DX7NeuralProcess\nfrom .dx7_nsp import DX7NeuralSylvesterProcess\nfrom .dx7_vae import DX7VAE" }, { "path": "neuralDX7/models/attention/__init__.py", "content": "from .attention import Attention\nfrom .attention_layer import AttentionLayer\nfrom .attention_encoder import ResidualAttentionEncoder\nfrom .conditional_attention_encoder import CondtionalResidualAttentionEncoder" }, { "path": "neuralDX7/models/attention/attention.py", "content": "import torch\nfrom torch import nn\n\nclass Attention(nn.Module):\n\n\n\n def __init__(self, n_features, n_hidden, n_heads=8, inf=1e9):\n \"\"\"\n n_features - number of input features\n n_hidden - hidden dim per head\n n_heads - number of heads\n \"\"\"\n\n super().__init__()\n\n self.QKV = nn.Linear(n_features, n_hidden * 3 * n_heads)\n self.n_heads = n_heads\n self._inf = inf\n\n @property\n def inf(self):\n # if self.training:\n return self._inf\n # return float('inf')\n\n def forward(self, X, A):\n\n *input_shape, _ = X.shape\n\n # calculate the query key and value vectors for all data points\n QKV = self.QKV(X).reshape(*input_shape, -1, 3, self.n_heads)\n\n # permute the heads and qkv vectors to the first dimensions\n n_dims = len(QKV.shape)\n permuter = torch.arange(n_dims).roll(2)\n Q, K, V = QKV.permute(*permuter)\n\n # calculate the attention values\n qk_t = (Q @ K.transpose(-1, -2)) / (self.n_heads**(1/2))\n qk_t_masked = qk_t.masked_fill(~A, -self.inf)\n \n # apply the attention values to the values\n Y = qk_t_masked.softmax(-1) @ V\n\n # restore heads to the final dimension and flatten (effectively concatenating them)\n n_dims = len(Y.shape)\n permuter = torch.arange(n_dims).roll(-1)\n Y = Y.permute(*permuter).flatten(-2, -1)\n return Y\n\n\n\nif __name__==\"__main__\":\n\n\n\n model = Attention(100, 20)\n X = torch.randn(3, 25, 100)\n A = torch.rand(3, 25, 25)>0.5\n Y = model(X, A)\n print(Y.shape)\n" }, { "path": "neuralDX7/models/attention/attention_encoder.py", "content": "import torch\nfrom os import environ\n\nfrom torch import nn\n\nfrom agoge import AbstractModel\nfrom neuralDX7.models.attention import AttentionLayer\nfrom neuralDX7.models.utils import position_encoding_init\n\n\n\nclass ResidualAttentionEncoder(AbstractModel):\n \"\"\"\n Residual attention stacks based on the Attention Is All You Need paper\n \"\"\"\n\n def __init__(self, features, attention_layer, max_len=200, n_layers=3):\n \"\"\"\n features - the number of features per parameter\n c_features - the number of side conditioning features per batch item\n attention_layer - a dictionary containing instantiation parameters for the AttentionLayer module\n max_len - the maximum needed size of the positional encodings\n n_layers - number of layers for the module to use\n \"\"\"\n super().__init__()\n\n # create the layers\n self.layers = nn.ModuleList(\n map(lambda x: AttentionLayer(**attention_layer), range(n_layers))\n )\n\n # pre generate the positional encodings\n positional_encoding = position_encoding_init(max_len, features)\n self.register_buffer('positional_encoding', positional_encoding)\n\n self.p2x = nn.Linear(features, features * 2)\n\n\n def forward(self, X, A):\n \"\"\"\n X - data tensor, torch.FloatTensor(batch_size, num_parameters, features)\n A - connection mask, torch.BoolTensor(batch_size, num_parameters, features)\n \"\"\"\n\n # generate FiLM parameters from positional encodings for conditioning\n gamma, beta = self.p2x(self.positional_encoding).chunk(2, -1)\n gamma, beta = torch.sigmoid(gamma), torch.tanh(beta)\n\n # Apply the data through the layers adding the positioning information in at each layer\n for layer in self.layers:\n X = layer(gamma * X + beta, A)\n\n return X\n \n\n\nif __name__=='__main__':\n\n layer_features = 100\n n_heads = 4\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': 555\n }\n\n \n max_len = 25\n \n \n model = ResidualAttentionEncoder(layer_features, attention_layer, max_len=max_len)\n A = torch.rand(3, 25, 25)>0.5\n X = torch.distributions.Categorical(torch.ones(128)).sample((3, 25))\n\n model(X, A)\n\n" }, { "path": "neuralDX7/models/attention/attention_layer.py", "content": "import torch\nfrom torch import nn\n\nfrom neuralDX7.models.attention import Attention\n\n\n\nclass AttentionLayer(nn.Module):\n \"\"\"\n Layer based on the original Attention is All You Need paper and is usable in graph network setups\n\n \"\"\"\n\n def __init__(self, features, hidden_dim, attention):\n \"\"\"\n features - the number of features the layer has at input and output\n hidden_dim - the hidden dimension of the feedforward network\n\n \"\"\"\n\n super().__init__()\n\n self.attention = Attention(**attention)\n self.feedforward = nn.Sequential(\n nn.Linear(features, hidden_dim),\n nn.GELU(),\n nn.Linear(hidden_dim, features),\n )\n\n self.attention_norm = nn.LayerNorm(features)\n self.feedforward_norm = nn.LayerNorm(features, elementwise_affine=False) # save elementwise affine for film conditioning\n\n \n\n def forward(self, X, A):\n \"\"\"\n X - data tensor, torch.FloatTensor(batch_size, num_parameters, features)\n A - connection mask, torch.BoolTensor(batch_size, num_parameters, features)\n \"\"\"\n X = self.attention_norm(self.attention(X, A) + X)\n X = self.feedforward_norm(self.feedforward(X) + X)\n\n return X\n\nif __name__==\"__main__\":\n\n attention = {\n 'n_features': 100,\n 'n_hidden': 25,\n 'n_heads': 4\n }\n \n model = AttentionLayer(100, 250, attention)\n\n X = torch.randn(3, 25, 100)\n A = torch.rand(3, 25, 25)>0.5\n Y = model(X, A)\n\n print(Y.shape)" }, { "path": "neuralDX7/models/attention/conditional_attention_encoder.py", "content": "import torch\nfrom os import environ\n\nfrom torch import nn\n\nfrom agoge import AbstractModel\nfrom neuralDX7.models.attention import AttentionLayer\nfrom neuralDX7.models.general import FeedForwardGELU\nfrom neuralDX7.models.utils import position_encoding_init\n\n\n\nclass CondtionalResidualAttentionEncoder(AbstractModel):\n \"\"\"\n Very similar to attention encoder but also allows custom side conditioning capacity\n\n \"\"\"\n def __init__(self, features, c_features, attention_layer, max_len=200, n_layers=3):\n \"\"\"\n features - the number of features per parameter\n c_features - the number of side conditioning features per batch item\n attention_layer - a dictionary containing instantiation parameters for the AttentionLayer module\n max_len - the maximum needed size of the positional encodings\n n_layers - number of layers for the module to use\n \"\"\"\n super().__init__()\n\n\n self.layers = nn.ModuleList(\n map(lambda x: AttentionLayer(**attention_layer), range(n_layers))\n )\n\n positional_encoding = position_encoding_init(max_len, features)\n self.c_layers = nn.ModuleList(\n map(lambda x: FeedForwardGELU(c_features, features*2), range(n_layers))\n )\n\n self.p2x = nn.Linear(features, features * 2)\n self.register_buffer('positional_encoding', positional_encoding)\n\n\n def forward(self, X, A, c):\n \"\"\"\n X - data tensor, torch.FloatTensor(batch_size, num_parameters, features)\n A - connection mask, torch.BoolTensor(batch_size, num_parameters, features)\n \"\"\"\n\n # generate FiLM parameters from positional encodings for conditioning\n gamma_p, beta_p = self.p2x(self.positional_encoding).chunk(2, -1)\n gamma_p, beta_p = torch.sigmoid(gamma_p), torch.tanh(beta_p)\n\n X = gamma_p * X + beta_p\n\n for layer, c_layer in zip(self.layers, self.c_layers):\n\n gamma_c, beta_c = c_layer(c).chunk(2, -1)\n gamma_c, beta_c = torch.sigmoid(gamma_c), torch.tanh(beta_c)\n\n X = layer(gamma_c * X + beta_c, A)\n\n return X\n \n\n\nif __name__=='__main__':\n\n layer_features = 100\n n_heads = 4\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': 555\n }\n\n \n max_len = 25\n \n \n model = ResidualAttentionEncoder(layer_features, attention_layer, max_len=max_len)\n A = torch.rand(3, 25, 25)>0.5\n X = torch.distributions.Categorical(torch.ones(128)).sample((3, 25))\n\n model(X, A)\n\n" }, { "path": "neuralDX7/models/dx7_cnp.py", "content": "import torch\nfrom torch import nn\n\nfrom agoge import AbstractModel\nfrom neuralDX7.models.attention import ResidualAttentionEncoder\nfrom neuralDX7.constants import MAX_VALUE, N_PARAMS\nfrom neuralDX7.utils import mask_parameters\n\n\nclass DX7PatchProcess(AbstractModel):\n \"\"\"\n EXPERIMENTAL AND UNTESTED\n\n \n \"\"\"\n\n def __init__(self, features, encoder):\n \n super().__init__()\n\n self.embedder = nn.Embedding(MAX_VALUE, features)\n self.encoder = ResidualAttentionEncoder(**encoder)\n\n self.logits = nn.Linear(features, MAX_VALUE)\n\n def forward(self, X):\n # print(X.shape, )\n\n # generate random masks\n batch_p = torch.rand(X.shape[0]) # decide p value for each item in batch\n item_logits = torch.rand(X.shape) # random value for each param\n X_a = batch_p.unsqueeze(-1) <= item_logits # active params in X\n X_a = X_a.to(self.device)\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n # 1/0\n\n X = self.embedder(X) * X_a.unsqueeze(-1).float()\n # X = self.embedder(X) \n\n X = self.encoder(X, A)\n # X_hat = mask_parameters(self.logits(X))\n X_hat = self.logits(X)\n # print(X_hat.max(), X_hat.min())\n\n return X_hat, X_a\n\n @torch.no_grad()\n def features(self, X):\n\n X_a = torch.ones_like(X).bool()\n A = X_a.unsqueeze(-1) & X_a.unsqueeze(-2)\n X = self.embedder(X)\n # X = self.embedder(X) \n\n X = self.encoder(X, A)\n\n return X\n\n @torch.no_grad()\n def generate(self, X, X_a):\n \n \n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n\n X = self.embedder(X) * X_a.unsqueeze(-1).float()\n X = self.encoder(X, A)\n X_hat = mask_parameters(self.logits(X))\n\n X_hat = torch.distributions.Categorical(logits=X_hat)\n\n return X_hat\n\n\n\nif __name__=='__main__':\n\n layer_features = 100\n n_heads = 4\n N_PARAMS = 8\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': 555\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS\n }\n \n \n model = DX7PatchProcess(layer_features, encoder=encoder)\n X = torch.distributions.Categorical(torch.ones(128)).sample((3, N_PARAMS))\n\n logits = model(X)\n print(logits.shape)\n print(logits[0])\n\n\n\n" }, { "path": "neuralDX7/models/dx7_np.py", "content": "import torch\nfrom torch import nn\nfrom torch.nn import functional as F\n\nfrom agoge import AbstractModel\nfrom neuralDX7.models.attention import ResidualAttentionEncoder, CondtionalResidualAttentionEncoder\nfrom neuralDX7.models.general import FeedForwardGELU\nfrom neuralDX7.models.stochastic_nodes import NormalNode\nfrom neuralDX7.constants import MAX_VALUE, N_PARAMS\nfrom neuralDX7.utils import mask_parameters\n\n\nclass DX7NeuralProcess(AbstractModel):\n \"\"\"\n EXPERIMENTAL AND UNTESTED\n\n \"\"\"\n\n def __init__(self, features, latent_dim, encoder, decoder, deterministic_path_drop_rate=0.5):\n \n super().__init__()\n\n self.embedder = nn.Embedding(MAX_VALUE, features)\n self.encoder = ResidualAttentionEncoder(**encoder)\n self._latent_encoder = nn.ModuleList([\n ResidualAttentionEncoder(**encoder),\n NormalNode(features, latent_dim)]\n )\n self.z_to_c = nn.Linear(latent_dim, latent_dim*155)\n self.decoder = CondtionalResidualAttentionEncoder(**decoder)\n self.logits = FeedForwardGELU(features, MAX_VALUE)\n self.drop = nn.Dropout(deterministic_path_drop_rate)\n\n def latent_encoder(self, X, A, mean=False):\n\n encoder, q_x = self._latent_encoder\n\n return q_x(encoder(X, A).mean(-2))\n\n\n def forward(self, X):\n\n # generate random masks\n batch_p = torch.rand(X.shape[0]) # decide p value for each item in batch\n item_logits = torch.rand(X.shape) # random value for each param\n X_a = batch_p.unsqueeze(-1) <= item_logits # active params in X\n X_a = X_a.to(self.device)\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n # A = A | True\n\n X_target = self.embedder(X)\n \n X_context = X_target * X_a.unsqueeze(-1).float()\n\n q_context = self.latent_encoder(X_context, A)\n q_target = self.latent_encoder(X_target, A | (~X_a.unsqueeze(-1)))\n\n # r = self.drop(self.encoder(X_context, A))\n # X_encoded = F.drop out\n z = q_target.rsample()\n # z_context = q_context.rsample()\n # mask = (torch.rand_like(z_target[...,[0]]) > 0.5).float()\n # z = (z_target * mask) + (z_context * (1-mask))\n\n c = self.z_to_c(z).view(z.shape[0], 155, -1)\n # c = z.unsqueeze(-2)\n\n X_dec = self.decoder(X_context, A, c)\n X_hat = self.logits(X_dec)\n\n return X_hat, X_a, q_context, q_target, z\n\n @torch.no_grad()\n def features(self, X, X_a):\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n\n X = self.embedder(X) * X_a.unsqueeze(-1).float()\n q = self.latent_encoder(X, A)\n\n return q\n\n @torch.no_grad()\n def generate_z(self, X, X_a, z, t=1.):\n\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n\n X = self.embedder(X)\n X = X * X_a.unsqueeze(-1).float() \n\n c = self.z_to_c(z).view(z.shape[0], 155, -1)\n X_dec = self.decoder(X, A, c)\n X_hat = mask_parameters(self.logits(X_dec))\n X_hat = torch.distributions.Categorical(logits=X_hat/t)\n\n return X_hat\n\n @torch.no_grad()\n def generate(self, X, X_a, sample=True, t=1.):\n\n q = self.features(X, X_a)\n\n \n\n z = q.sample()\n\n c_gamma, c_beta = self.z_to_c(z).chunk(2, -1)\n\n\n X_hat = mask_parameters(self.logits(c_gamma))\n\n X_hat = torch.distributions.Categorical(logits=X_hat/t)\n\n return X_hat\n\n\n\nif __name__=='__main__':\n\n layer_features = 100\n n_heads = 4\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': 555\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS\n }\n \n \n model = DX7PatchProcess(layer_features, encoder=encoder)\n X = torch.distributions.Categorical(torch.ones(128)).sample((3, N_PARAMS))\n\n logits = model(X)\n print(logits.shape)\n print(logits[0])\n" }, { "path": "neuralDX7/models/dx7_nsp.py", "content": "import torch\nfrom torch import nn\nfrom torch.nn import functional as F\n\nfrom agoge import AbstractModel\nfrom neuralDX7.models.attention import ResidualAttentionEncoder, CondtionalResidualAttentionEncoder\nfrom neuralDX7.models.general import FeedForwardGELU\nfrom neuralDX7.models.stochastic_nodes import TriangularSylvesterFlow\nfrom neuralDX7.constants import MAX_VALUE, N_PARAMS\nfrom neuralDX7.utils import mask_parameters\n\n\nclass DX7NeuralSylvesterProcess(AbstractModel):\n \"\"\"\n EXPERIMENTAL AND UNTESTED\n\n \"\"\"\n\n def __init__(self, features, latent_dim, encoder, decoder, deterministic_path_drop_rate=0.5, num_flows=3):\n \n super().__init__()\n\n self.embedder = nn.Embedding(MAX_VALUE, features)\n self.encoder = ResidualAttentionEncoder(**encoder)\n self._latent_encoder = nn.ModuleList([\n ResidualAttentionEncoder(**encoder),\n TriangularSylvesterFlow(features, latent_dim, num_flows)]\n )\n self.z_to_c = nn.Linear(latent_dim, latent_dim*155)\n self.decoder = CondtionalResidualAttentionEncoder(**decoder)\n self.logits = FeedForwardGELU(features, MAX_VALUE)\n self.drop = nn.Dropout(deterministic_path_drop_rate)\n\n def latent_encoder(self, X, A, z=None, flow=True):\n\n encoder, q_x = self._latent_encoder\n\n return q_x(encoder(X, A).mean(-2), z, flow)\n\n\n def forward(self, X):\n\n batch_size = X.shape[0]\n\n # generate random masks\n batch_p = torch.rand(batch_size) # decide p value for each item in batch\n item_logits = torch.rand(X.shape) # random value for each param\n X_a = batch_p.unsqueeze(-1) <= item_logits # active params in X\n X_a = X_a.to(self.device)\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n # A = A | True\n\n X_target = self.embedder(X)\n \n X_context = X_target * X_a.unsqueeze(-1).float()\n\n flow_target = self.latent_encoder(X_target, A | (~X_a.unsqueeze(-1)), flow=False)\n flow_context = self.latent_encoder(X_context, A)\n \n c = self.z_to_c(flow_target.z_0).view(batch_size, 155, -1)\n # c = z.unsqueeze(-2)\n\n X_dec = self.decoder(X_context, A, c)\n X_hat = self.logits(X_dec)\n\n return {\n 'X_hat': X_hat,\n 'X_a': X_a,\n 'flow_context': flow_context,\n 'flow_target': flow_target,\n }\n\n\n @torch.no_grad()\n def features(self, X, X_a):\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n\n X = self.embedder(X) * X_a.unsqueeze(-1).float()\n q = self.latent_encoder(X, A)\n\n return q\n\n @torch.no_grad()\n def generate_z(self, X, X_a, z, t=1.):\n\n\n A = (~X_a.unsqueeze(-1)) & (X_a.unsqueeze(-2))\n eye = torch.eye(A.shape[-1]).bool().to(self.device) & (~X_a.unsqueeze(-2))\n A = A | eye\n\n X = self.embedder(X)\n X = X * X_a.unsqueeze(-1).float() \n\n c = self.z_to_c(z).view(z.shape[0], 155, -1)\n X_dec = self.decoder(X, A, c)\n X_hat = mask_parameters(self.logits(X_dec))\n X_hat = torch.distributions.Categorical(logits=X_hat/t)\n\n return X_hat\n\n @torch.no_grad()\n def generate(self, X, X_a, sample=True, t=1.):\n\n q = self.features(X, X_a)\n\n z = q.sample()\n\n c_gamma, c_beta = self.z_to_c(z).chunk(2, -1)\n\n\n X_hat = mask_parameters(self.logits(c_gamma))\n\n X_hat = torch.distributions.Categorical(logits=X_hat/t)\n\n return X_hat\n\n\n\nif __name__=='__main__':\n\n layer_features = 100\n n_heads = 4\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': 555\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS\n }\n \n \n model = DX7PatchProcess(layer_features, encoder=encoder)\n X = torch.distributions.Categorical(torch.ones(128)).sample((3, N_PARAMS))\n\n logits = model(X)\n print(logits.shape)\n print(logits[0])\n" }, { "path": "neuralDX7/models/dx7_vae.py", "content": "import torch\nfrom torch import nn\nfrom torch.nn import functional as F\n\nfrom agoge import AbstractModel\nfrom neuralDX7.models.attention import ResidualAttentionEncoder, CondtionalResidualAttentionEncoder\nfrom neuralDX7.models.general import FeedForwardGELU\nfrom neuralDX7.models.stochastic_nodes import TriangularSylvesterFlow\nfrom neuralDX7.constants import MAX_VALUE, N_PARAMS\nfrom neuralDX7.utils import mask_parameters\n\n\nclass DX7VAE(AbstractModel):\n \"\"\"\n Variational Auto Encoder for a single DX7 patch. \n \n Uses a Triangular sylvester flow to transform the encoder output to decoder input\n \"\"\"\n\n def __init__(self, features, latent_dim, encoder, decoder, num_flows=3):\n \"\"\"\n features - number of features in the model\n latent_dim - the latent dimension of the model\n encoder - dictionary containing instantiation parameters for ResidualAttentionEncoder module\n decoder - dictionary containing instantiation parameters for CondtionalResidualAttentionEncoder module\n num_flows - the number of flows for the TriangularSylvesterFlow module\n \"\"\"\n \n super().__init__()\n\n self.embedder = nn.Embedding(MAX_VALUE, features)\n self.encoder = ResidualAttentionEncoder(**encoder)\n self._latent_encoder = nn.ModuleList([\n ResidualAttentionEncoder(**encoder),\n TriangularSylvesterFlow(features, latent_dim, num_flows)]\n )\n self.z_to_c = nn.Linear(latent_dim, latent_dim*155)\n self.decoder = CondtionalResidualAttentionEncoder(**decoder)\n self.logits = FeedForwardGELU(features, MAX_VALUE)\n\n self.n_features = features\n\n def latent_encoder(self, X, A, z=None, mean=False):\n \"\"\"\n Calculate the latent distribution\n\n X - data tensor, torch.FloatTensor(batch_size, 155, features)\n A - connection mask, torch.BoolTensor(batch_size, 155, features)\n z - a presampled latent, if none then the z is sampled using reparameterization technique\n mean - use the mean rather than sampling from the latent\n \"\"\"\n \n encoder, q_x = self._latent_encoder\n\n return q_x(encoder(X, A).mean(-2), z)\n\n\n def forward(self, X):\n \"\"\"\n Auto encodes the inputs variational latent layer\n\n X - the array of dx7 voices, torch.LongTensor(batch_size, 155)\n \"\"\"\n\n batch_size = X.shape[0]\n\n A = torch.ones_like(X).bool()\n A = A[...,None] | A[...,None,:]\n\n X_emb = self.embedder(X)\n \n flow = self.latent_encoder(X_emb, A)\n \n c = self.z_to_c(flow.z_k).view(batch_size, 155, -1)\n\n X_dec = self.decoder(torch.ones_like(X_emb), A, c)\n X_hat = self.logits(X_dec)\n\n return {\n 'X_hat': X_hat,\n 'flow': flow,\n }\n\n\n @torch.no_grad()\n def features(self, X):\n \"\"\"\n Get the latent distributions for a set of voices\n\n X - the array of dx7 voices, torch.LongTensor(batch_size, 155)\n\n \"\"\"\n\n A = torch.ones_like(X).bool()\n A = A[...,None] | A[...,None,:]\n\n X = self.embedder(X)\n q = self.latent_encoder(X, A)\n\n return q.q_z\n\n @torch.no_grad()\n def generate(self, z, t=1.):\n \"\"\"\n Given a sample from the latent distribution, reporojects it back to data space\n \n z - the array of dx7 voices, torch.FloatTensor(batch_size, latent_dim)\n t - the temperature of the output distribution. approaches determenistic as t->0 and approach uniforms as t->infty, requires t>0\n \"\"\"\n A = z.new(z.size(0), 155, 155).bool() | 1\n X = z.new(z.size(0), 155, self.n_features)\n X = X * 0 + 1\n\n c = self.z_to_c(z).view(z.shape[0], 155, -1)\n X_dec = self.decoder(X, A, c)\n X_hat = mask_parameters(self.logits(X_dec))\n X_hat = torch.distributions.Categorical(logits=X_hat/t)\n\n return X_hat\n\nif __name__=='__main__':\n\n layer_features = 100\n n_heads = 4\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': 555\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS\n }\n \n \n model = DX7PatchProcess(layer_features, encoder=encoder)\n X = torch.distributions.Categorical(torch.ones(128)).sample((3, N_PARAMS))\n\n logits = model(X)\n print(logits.shape)\n print(logits[0])\n" }, { "path": "neuralDX7/models/general/__init__.py", "content": "from .gelu_ff import FeedForwardGELU" }, { "path": "neuralDX7/models/general/gelu_ff.py", "content": "import torch\nfrom torch import nn\n\nclass FeedForwardGELU(nn.Module):\n \"\"\"\n Simple wrapper for two layer projection with GeLU non linearity\n\n \"\"\"\n\n def __init__(self, features, out_features=None, exapnsion_factor=3):\n \"\"\"\n features - the number of input features\n out_features - the number of output features, if None copies the input dimension\n expansion_factor - the size of the hidden dimension as a factor of the input features\n \"\"\"\n\n super().__init__()\n out_features = features if out_features is None else out_features\n\n self.net = nn.Sequential(\n nn.Linear(features, features*exapnsion_factor),\n nn.GELU(),\n nn.Linear(features*exapnsion_factor, out_features)\n )\n\n def forward(self, x):\n\n return self.net(x)" }, { "path": "neuralDX7/models/stochastic_nodes/__init__.py", "content": "from .normal import NormalNode\nfrom .triangular_sylvester import TriangularSylvesterFlow" }, { "path": "neuralDX7/models/stochastic_nodes/normal.py", "content": "from torch import nn\nfrom torch.distributions import Normal\n\n\n\nclass NormalNode(nn.Module):\n \"\"\"\n Simple module to create a normally distributed node in a ala VAE's. \n\n this node computes the function\n ```\n p(x) = N(mu(x), sigma(x)I)\n ```\n \"\"\"\n\n def __init__(self, in_features, latent_dim, hidden_dim=None):\n \"\"\"\n in_features - number of input features\n latent_dim - number of normals in the output\n hidden_dim - the inner dimension of the nonlinear feedforward network, 2x the input dimension if None\n \"\"\"\n super().__init__()\n\n if hidden_dim is None:\n\n hidden_dim = in_features * 2\n\n self.net = nn.Sequential(\n nn.Linear(in_features, hidden_dim),\n nn.GELU(),\n nn.Linear(hidden_dim, latent_dim * 2)\n )\n\n def forward(self, x, *args, **kwargs):\n \"\"\"\n x - the inpute vector, torch.FloatTensor(..., f)\n \"\"\"\n\n # calculate the parameters of the distribution\n mu, log_sigma = self.net(x).chunk(2, -1)\n\n # sqrt and ensure numerical stability in sigma\n sigma = (log_sigma*0.5).clamp(-5, 4).exp()\n\n return Normal(mu, sigma)\n\n\n\n" }, { "path": "neuralDX7/models/stochastic_nodes/triangular_sylvester.py", "content": "#%%\nfrom collections import namedtuple\nfrom itertools import count\nimport torch\nfrom torch import nn\nfrom neuralDX7.models.stochastic_nodes import NormalNode\n\n\"\"\"\nThis code modified from the reference implementation provided by the authors\nhttps://github.com/riannevdberg/sylvester-flows\n\"\"\"\n\n\n\nclass TriangularSylvester(nn.Module):\n \"\"\"\n Sylvester normalizing flow with Q=P or Q=I.\n \"\"\"\n\n def __init__(self, z_size):\n\n super(TriangularSylvester, self).__init__()\n\n self.z_size = z_size\n self.h = nn.Tanh()\n\n # diag_idx = torch.arange(0, z_size).long()\n # self.register_buffer('diag_idx', diag_idx)\n\n def der_h(self, x):\n return self.der_tanh(x)\n\n def der_tanh(self, x):\n return 1 - self.h(x) ** 2\n\n def forward(self, zk, r1, r2, b, permute_z=None, sum_ldj=True):\n \"\"\"\n All flow parameters are amortized. conditions on diagonals of R1 and R2 need to be satisfied\n outside of this function.\n Computes the following transformation:\n z' = z + QR1 h( R2Q^T z + b)\n or actually\n z'^T = z^T + h(z^T Q R2^T + b^T)R1^T Q^T\n with Q = P a permutation matrix (equal to identity matrix if permute_z=None)\n :param zk: shape: (batch_size, z_size)\n :param r1: shape: (batch_size, num_ortho_vecs, num_ortho_vecs).\n :param r2: shape: (batch_size, num_ortho_vecs, num_ortho_vecs).\n :param b: shape: (batch_size, 1, self.z_size)\n :return: z, log_det_j\n \"\"\"\n # Amortized flow parameters\n zk = zk.unsqueeze(1)\n\n # Save diagonals for log_det_j\n # diag_r1 = r1[:, self.diag_idx, self.diag_idx]\n diag_r1 = torch.diagonal(r1, 0, -1, -2)\n # diag_r2 = r2[:, self.diag_idx, self.diag_idx]\n diag_r2 = torch.diagonal(r2, 0, -1, -2)\n\n if permute_z is not None:\n # permute order of z\n z_per = zk[:, :, permute_z]\n else:\n z_per = zk\n\n r2qzb = z_per @ r2.transpose(2, 1) + b\n z = self.h(r2qzb) @ r1.transpose(2, 1)\n\n if permute_z is not None:\n # permute order of z again back again\n z = z[:, :, permute_z]\n\n z += zk\n z = z.squeeze(1)\n\n # Compute log|det J|\n # Output log_det_j in shape (batch_size) instead of (batch_size,1)\n diag_j = diag_r1 * diag_r2\n diag_j = self.der_h(r2qzb).squeeze(1) * diag_j\n diag_j += 1.\n log_diag_j = (diag_j.abs()+1e-8).log()\n\n if sum_ldj:\n log_det_j = log_diag_j.sum(-1)\n else:\n log_det_j = log_diag_j\n\n return z, log_det_j\n\nclass TriangularSylvesterFlow(nn.Module):\n \"\"\"\n Variational auto-encoder with triangular Sylvester flows in the encoder. Alternates between setting\n the orthogonal matrix equal to permutation and identity matrix for each flow.\n \"\"\"\n\n def __init__(self, in_features, latent_dim, num_flows):\n\n super().__init__()\n # Initialize log-det-jacobian to zero\n self.log_det_j = 0.\n\n # Flow parameters\n self.num_flows = num_flows\n self.latent_dim = latent_dim\n\n # permuting indices corresponding to Q=P (permutation matrix) for every other flow\n flip_idx = torch.arange(latent_dim - 1, -1, -1).long()\n self.register_buffer('flip_idx', flip_idx)\n\n # self.amor_b = nn.Linear(self.q_z_nn_output_dim, self.num_flows * latent_dim)\n self.q_z = NormalNode(in_features, latent_dim)\n self._flow_params = nn.Linear(in_features,\n self.num_flows * latent_dim * latent_dim + \\\n self.num_flows * latent_dim + \\\n self.num_flows * latent_dim + \\\n self.num_flows * latent_dim\n )\n self.flows = nn.ModuleList([\n TriangularSylvester(latent_dim) for k in range(self.num_flows)\n ])\n\n def flow_params(self, h):\n \"\"\"\n Parameterise the base distribution, sample and flow\n \"\"\"\n\n batch_size = h.size(0)\n\n params = self._flow_params(h)\n params = params.reshape(batch_size, self.num_flows, self.latent_dim, -1)\n params = params.transpose(0,1) # batch x flows x z x z -> flows x batch x z x z\n \n diag1 = torch.tanh(params[...,0])\n diag2 = torch.tanh(params[...,1])\n b = params[...,2].unsqueeze(-2)\n full_d = params[...,3:]\n\n r1 = torch.triu(full_d, diagonal=1)\n r2 = torch.triu(full_d.transpose(-1, -2), diagonal=1)\n r1 = diag1.diag_embed(0) + r1\n r2 = diag2.diag_embed(0) + r2\n\n return r1, r2, b\n\n def forward(self, h, z=None, flow=True):\n \"\"\"\n Forward pass with orthogonal flows for the transformation z_0 -> z_1 -> ... -> z_k.\n Log determinant is computed as log_det_j = N E_q_z0[\\sum_k log |det dz_k/dz_k-1| ].\n \"\"\"\n Flow = namedtuple('Flow', ('q_z', 'log_det', 'z_0', 'z_k', 'flow'))\n\n q_z = self.q_z(h)\n z_0 = z_k = q_z.rsample() if z is None else z\n\n if not flow:\n return Flow(q_z, None, z_0, None)\n\n r1, r2, b = self.flow_params(h)\n\n # Sample z_0\n def flow_f(z_k):\n log_det_j = 0.\n\n # Normalizing flows\n for k, flow_k, r1_k, r2_k, b_k in zip(count(), self.flows, r1, r2, b):\n\n if k % 2 == 1:\n # Alternate with reorderering z for triangular flow\n permute_z = self.flip_idx\n else:\n permute_z = None\n\n z_k, log_det_jacobian = flow_k(z_k, r1_k, r2_k, b_k, permute_z, sum_ldj=True)\n\n log_det_j += log_det_jacobian\n \n return z_k, log_det_j\n z_k, log_det_j = flow_f(z_0)\n return Flow(q_z, log_det_j, z_0, z_k, flow_f)\n\n\nif __name__==\"__main__\":\n \n num_ortho_vecs = z_size = 6\n batch_size = 12\n in_features = 64\n \n h = torch.randn(batch_size, in_features)\n # zk = torch.randn(batch_size, z_size)\n # r1 = torch.randn(batch_size, num_ortho_vecs, num_ortho_vecs)\n # r2 = torch.randn(batch_size, num_ortho_vecs, num_ortho_vecs)\n # b = torch.randn(batch_size, 1, z_size)\n\n f = TriangularSylvesterFlow(in_features, z_size, 3)\n\n f(h)\n\n# %%\n" }, { "path": "neuralDX7/models/utils.py", "content": "\nimport torch\nimport numpy as np\n\ndef position_encoding_init(n_position, emb_dim):\n ''' Init the sinusoid position encoding table '''\n\n # keep dim 0 for padding token position encoding zero vector\n position_enc = np.array([\n [pos / np.power(10000, 2 * (j // 2) / emb_dim) for j in range(emb_dim)]\n if pos != 0 else np.zeros(emb_dim) for pos in range(n_position)])\n \n\n position_enc[1:, 0::2] = np.sin(position_enc[1:, 0::2]) # apply sin on 0th,2nd,4th...emb_dim\n position_enc[1:, 1::2] = np.cos(position_enc[1:, 1::2]) # apply cos on 1st,3rd,5th...emb_dim\n return torch.from_numpy(position_enc).type(torch.FloatTensor)\n" }, { "path": "neuralDX7/solvers/__init__.py", "content": "from .dx7_patch_process import DX7PatchProcess\nfrom .dx7_np import DX7NeuralProcess\nfrom .dx7_nsp import DX7NeuralSylvesterProcess\nfrom .dx7_vae import DX7VAE" }, { "path": "neuralDX7/solvers/dx7_np.py", "content": "import torch\nfrom torch.nn import functional as F\nfrom importlib import import_module\nfrom torch.optim import AdamW\nfrom torch.distributions.kl import kl_divergence\n\nfrom agoge import AbstractSolver\n\nfrom .utils import sigmoidal_annealing\n\nclass DX7NeuralProcess(AbstractSolver):\n \"\"\"\n EXPERIMENTAL AND UNTESTED\n \"\"\"\n\n def __init__(self, model,\n Optim=AdamW, optim_opts=dict(lr= 1e-4),\n max_beta=0.5,\n beta_temp=1e-4,\n **kwargs):\n\n if isinstance(Optim, str):\n Optim = import_module(Optim)\n\n\n self.optim = Optim(params=model.parameters(), **optim_opts)\n self.max_beta = max_beta\n self.model = model\n\n self.iter = 0\n self.beta_temp = beta_temp\n\n def loss(self, x, x_hat, x_a, q_context, q_target, z):\n\n valid_predictions = (~x_a).nonzero().t()\n\n valid_x_hat = x_hat[(*valid_predictions,)]\n valid_x = x[(*valid_predictions,)]\n\n \n # kl = kl_divergence(q_target, q_context)#[(*valid_predictions,)]\n # kl = kl_divergence(Normal(torch.zeros_like()), q_context)#[(*valid_predictions,)]\n kl = q_target.log_prob(z) - q_context.log_prob(z)\n kl = kl.sum(-1).mean()\n entropy = q_target.entropy().mean()\n beta = sigmoidal_annealing(self.iter, self.beta_temp).item()\n\n reconstruction_loss = F.cross_entropy(valid_x_hat, valid_x)\n accuracy = (valid_x_hat.argmax(-1)==valid_x).float().mean()\n\n loss = reconstruction_loss + 0.25 * beta * kl\n\n return loss, {\n 'accuracy': accuracy,\n 'reconstruction_loss': reconstruction_loss,\n 'kl': kl,\n 'entropy': entropy,\n 'beta': beta\n }\n \n\n def solve(self, x, **kwargs):\n \n x_hat, x_a, q_context, q_target, z = self.model(x)\n loss, L = self.loss(x, x_hat, x_a, q_context, q_target, z)\n\n if loss != loss:\n raise ValueError('Nan Values detected')\n\n if self.model.training:\n self.iter += 1\n self.optim.zero_grad()\n loss.backward()\n self.optim.step()\n\n return L\n\n \n def step(self):\n\n pass\n\n\n def state_dict(self):\n \n state_dict = {\n 'optim': self.optim.state_dict(),\n 'iter': self.iter\n }\n\n return state_dict\n\n def load_state_dict(self, state_dict):\n \n self.optim.load_state_dict(state_dict['optim'])\n self.iter = state_dict['iter']" }, { "path": "neuralDX7/solvers/dx7_nsp.py", "content": "import torch\nfrom torch.nn import functional as F\nfrom importlib import import_module\nfrom torch.optim import AdamW\nfrom torch.distributions.kl import kl_divergence\n\nfrom agoge import AbstractSolver\n\nfrom .utils import sigmoidal_annealing\n\nclass DX7NeuralSylvesterProcess(AbstractSolver):\n \"\"\"\n EXPERIMENTAL AND UNTESTED\n \"\"\"\n \n def __init__(self, model,\n Optim=AdamW, optim_opts=dict(lr= 1e-4),\n max_beta=0.5,\n beta_temp=1e-4,\n **kwargs):\n\n if isinstance(Optim, str):\n Optim = import_module(Optim)\n\n self.optim = Optim(params=model.parameters(), **optim_opts)\n self.max_beta = max_beta\n self.model = model\n\n self.iter = 0\n self.beta_temp = beta_temp\n\n def loss(self, X, X_hat, X_a, flow_context, flow_target):\n\n valid_predictions = (~X_a).nonzero().t()\n\n valid_x_hat = X_hat[(*valid_predictions,)]\n valid_x = X[(*valid_predictions,)]\n\n p_z = flow_target.q_z.log_prob(flow_context.z_k).sum(-1)\n q_z = flow_context.q_z.log_prob(flow_context.z_0).sum(-1)\n kl = (q_z-p_z-flow_context.log_det).mean() / flow_context.z_k.shape[-1]\n beta = sigmoidal_annealing(self.iter, self.beta_temp).item()\n\n reconstruction_loss = F.cross_entropy(valid_x_hat, valid_x)\n accuracy = (valid_x_hat.argmax(-1)==valid_x).float().mean()\n\n loss = reconstruction_loss + self.max_beta * beta * kl\n\n return loss, {\n 'accuracy': accuracy,\n 'reconstruction_loss': reconstruction_loss,\n 'kl': kl,\n 'beta': beta,\n 'q_log_det': flow_context.log_det.mean(),\n # 'p_log_det': flow_target.log_det.mean(),\n 'q_z': q_z.mean(),\n 'p_z': p_z.mean()\n }\n \n\n def solve(self, X, **kwargs):\n \n Y = self.model(**X)\n loss, L = self.loss(**X, **Y)\n\n if loss != loss:\n raise ValueError('Nan Values detected')\n\n if self.model.training:\n self.iter += 1\n self.optim.zero_grad()\n loss.backward()\n self.optim.step()\n\n return L\n\n \n def step(self):\n\n pass\n\n\n def state_dict(self):\n \n state_dict = {\n 'optim': self.optim.state_dict(),\n 'iter': self.iter\n }\n\n return state_dict\n\n def load_state_dict(self, state_dict):\n \n self.optim.load_state_dict(state_dict['optim'])\n self.iter = state_dict['iter']" }, { "path": "neuralDX7/solvers/dx7_patch_process.py", "content": "import torch\nfrom torch.nn import functional as F\nfrom importlib import import_module\nfrom torch.optim import AdamW\n\nfrom agoge import AbstractSolver\n\n\n\nclass DX7PatchProcess(AbstractSolver):\n \"\"\"\n EXPERIMENTAL AND UNTESTED\n \"\"\"\n\n def __init__(self, model,\n Optim=AdamW, optim_opts=dict(lr= 1e-4),\n max_beta=0.5,\n **kwargs):\n\n if isinstance(Optim, str):\n Optim = import_module(Optim)\n\n\n self.optim = Optim(params=model.parameters(), **optim_opts)\n self.max_beta = max_beta\n self.model = model\n\n def loss(self, x, x_hat, x_a):\n\n valid_predictions = (~x_a).nonzero().t()\n\n valid_x_hat = x_hat[(*valid_predictions,)]\n valid_x = x[(*valid_predictions,)]\n\n reconstruction_loss = F.cross_entropy(valid_x_hat, valid_x)\n accuracy = (valid_x_hat.argmax(-1)==valid_x).float().mean()\n\n return reconstruction_loss, {\n 'accuracy': accuracy,\n 'reconstruction_loss': reconstruction_loss,\n }\n \n\n def solve(self, x, **kwargs):\n \n x_hat, x_a = self.model(x)\n loss, L = self.loss(x, x_hat, x_a)\n\n if loss != loss:\n raise ValueError('Nan Values detected')\n\n if self.model.training:\n\n self.optim.zero_grad()\n loss.backward()\n self.optim.step()\n \n return L\n\n \n def step(self):\n\n pass\n\n\n def state_dict(self):\n \n state_dict = {\n 'optim': self.optim.state_dict()\n }\n\n return state_dict\n\n def load_state_dict(self, state_dict):\n \n self.optim.load_state_dict(state_dict['optim'])" }, { "path": "neuralDX7/solvers/dx7_vae.py", "content": "import torch\nfrom torch.nn import functional as F\nfrom importlib import import_module\nfrom torch.optim import AdamW\nfrom torch.distributions.kl import kl_divergence\nfrom torch.distributions import Normal\nfrom agoge import AbstractSolver\n\nfrom .utils import sigmoidal_annealing\n\nclass DX7VAE(AbstractSolver):\n \"\"\"\n Solver used to train DX7VAE model\n \"\"\"\n\n def __init__(self, model,\n Optim=AdamW, optim_opts=dict(lr= 1e-4),\n max_beta=0.5,\n beta_temp=1e-4,\n **kwargs):\n\n if isinstance(Optim, str):\n Optim = import_module(Optim)\n\n self.optim = Optim(params=model.parameters(), **optim_opts)\n self.max_beta = max_beta\n self.model = model\n\n self.iter = 0\n self.beta_temp = beta_temp\n\n def loss(self, X, X_hat, flow):\n \"\"\"\n Computes the VAE loss objective and collects some training statistics\n\n X - data tensor, torch.LongTensor(batch_size, num_parameters=155)\n X_hat - data tensor, torch.FloatTensor(batch_size, num_parameters=155, max_value=128)\n flow - the namedtuple returned by TriangularSylvesterFlow\n \n for reference, the namedtuple is ('Flow', ('q_z', 'log_det', 'z_0', 'z_k', 'flow'))\n \"\"\"\n \n p_z_k = Normal(0,1).log_prob(flow.z_k).sum(-1)\n q_z_0 = flow.q_z.log_prob(flow.z_0).sum(-1)\n kl = (q_z_0-p_z_k-flow.log_det).mean() / flow.z_k.shape[-1]\n\n beta = sigmoidal_annealing(self.iter, self.beta_temp).item()\n\n reconstruction_loss = F.cross_entropy(X_hat.transpose(-1, -2), X)\n accuracy = (X_hat.argmax(-1)==X).float().mean()\n\n loss = reconstruction_loss + self.max_beta * beta * kl\n\n return loss, {\n 'accuracy': accuracy,\n 'reconstruction_loss': reconstruction_loss,\n 'kl': kl,\n 'beta': beta,\n 'log_det': flow.log_det.mean(),\n 'p_z_k': p_z_k.mean(),\n 'q_z_0': q_z_0.mean(),\n # 'iter': self.iter // self.\n }\n\n def solve(self, X, **kwargs):\n \"\"\"\n Take a gradient step given an input X\n\n X - data tensor, torch.LongTensor(batch_size, num_parameters=155)\n \"\"\"\n \n Y = self.model(**X)\n loss, L = self.loss(**X, **Y)\n\n if loss != loss:\n raise ValueError('Nan Values detected')\n\n if self.model.training:\n self.iter += 1\n self.optim.zero_grad()\n loss.backward()\n self.optim.step()\n\n return L\n\n def step(self):\n\n pass\n\n def state_dict(self):\n \n state_dict = {\n 'optim': self.optim.state_dict(),\n 'iter': self.iter\n }\n\n return state_dict\n\n def load_state_dict(self, state_dict):\n \n self.optim.load_state_dict(state_dict['optim'])\n self.iter = state_dict['iter']" }, { "path": "neuralDX7/solvers/utils.py", "content": "import torch\n\n\ndef sigmoidal_annealing(iter_nb, t=1e-4, s=-6):\n \"\"\"\n\n iter_nb - number of parameter updates completed\n t - step size\n s - slope of the sigmoid\n \"\"\"\n \n t, s = torch.tensor(t), torch.tensor(s).float()\n x0 = torch.sigmoid(s)\n value = (torch.sigmoid(iter_nb*t + s) - x0)/(1-x0) \n\n return value" }, { "path": "neuralDX7/utils.py", "content": "\nimport mido\nimport torch\nimport numpy as np\nfrom pathlib import Path\nfrom itertools import chain\nfrom neuralDX7.constants import VOICE_KEYS, VOICE_PARAMETER_RANGES, MAX_VALUE, checksum\nfrom neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\nimport bitstruct\n\n\ndef mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n device = x.device\n mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n mask = torch.stack(list(mapper))\n \n return torch.masked_fill(x, mask, -inf)\n\n\n# %%\n\n\ndef consume_syx(path):\n\n path = Path(path).expanduser()\n try:\n preset = mido.read_syx_file(path.as_posix())[0]\n except IndexError as e:\n return None\n except ValueError as e:\n return None\n if len(preset.data) == 0:\n return None\n\n def get_voice(data):\n \n unpacked = voice_struct.unpack(data)\n\n if not verify(unpacked, VOICE_PARAMETER_RANGES):\n return None\n \n return unpacked\n\n get_header = header_struct.unpack\n sysex_iter = iter(preset.data)\n \n try:\n header = get_header(bytes(take(sysex_iter, len(header_bytes))))\n yield from (get_voice(bytes(take(sysex_iter, len(voice_bytes)))) for _ in range(N_VOICES))\n except RuntimeError:\n return None\n\ndef dx7_bulk_pack(voices):\n\n HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n assert len(voices)==32\n voices_bytes = bytes()\n for voice in voices:\n voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n voices_bytes += voice_bytes\n \n \n patch_checksum = [checksum(voices_bytes)]\n\n data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n return mido.Message('sysex', data=data)\n\n\n\ndef generate_syx(patch_list):\n\n dx7_struct" }, { "path": "projects/dx7_np/evaluate.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/craggy-goldenrod-catfish_0_2020-04-28_02-22-57m8eftq1b/checkpoint_410/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\n\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nfor n in range(10):\n X = next(iter_X)['x']\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_d = torch.distributions.Categorical(logits=mask_parameters(torch.zeros(32, 155, 128)))\n\n X_a = torch.rand_like(X.float()) < 0.3\n X_a = torch.ones_like(X).bool()\n X_a[:,:-10] = 0\n\n X = X[[0]*32]\n X[~X_a] = X_d.sample()[~X_a]\n\n max_to_sample = max((~X_a).sum(-1))\n\n # for i in tqdm(range(max_to_sample)):\n\n logits = model.generate(X, X_a)\n samples = logits.sample()\n\n has_unsampled = ~X_a.all(-1)\n\n batch_idxs, sample_idx = (~X_a).nonzero().t()\n\n X[batch_idxs, sample_idx] = samples[batch_idxs, sample_idx]\n X_a[batch_idxs, sample_idx] = 1\n \n\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/np_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_np/experiment.py", "content": "#%%\nfrom os import environ\nenviron['MLFLOW_TRACKING_URI'] = 'http://tracking.olympus.nintorac.dev:9001/'\n\nfrom neuralDX7.constants import N_PARAMS, MAX_VALUE\nfrom agoge.utils import trial_name_creator\nfrom neuralDX7 import DEFAULTS\nfrom agoge import TrainWorker as Worker\nfrom ray import tune\nfrom neuralDX7.models import DX7NeuralProcess as Model\nfrom neuralDX7.solvers import DX7NeuralProcess as Solver\nfrom neuralDX7.datasets import DX7SysexDataset as Dataset\n\ndef config(experiment_name, trial_name, \n n_heads=8, n_features=32, \n batch_size=16, data_size=1.,\n latent_dim=8,\n **kwargs):\n \n\n\n data_handler = {\n 'Dataset': Dataset,\n 'dataset_opts': {\n 'data_size': data_size\n },\n 'loader_opts': {\n 'batch_size': batch_size,\n },\n }\n\n ### MODEL FEATURES\n layer_features = n_heads * n_features\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': layer_features * 3\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 1\n }\n \n\n model = {\n 'Model': Model,\n 'features': layer_features,\n 'latent_dim': latent_dim,\n 'encoder': encoder,\n 'decoder': {\n 'c_features': latent_dim,\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 1\n },\n 'deterministic_path_drop_rate': 0.8\n }\n\n solver = {\n 'Solver': Solver,\n 'beta_temp': 1e-4\n }\n\n tracker = {\n 'metrics': ['reconstruction_loss', 'accuracy', 'kl', 'beta', 'entropy'],\n 'experiment_name': experiment_name,\n 'trial_name': trial_name\n }\n\n return {\n 'data_handler': data_handler,\n 'model': model,\n 'solver': solver,\n 'tracker': tracker,\n }\n\nif __name__=='__main__':\n # from ray import ray\n import sys\n postfix = sys.argv[1] if len(sys.argv)==2 else ''\n # ray.init()\n # from ray.tune.utils import validate_save_restore\n # validate_save_restore(Worker)\n # client = MlflowClient(tracking_uri='localhost:5000')\n experiment_name = f'dx7-np-{postfix}'#+experiment_name_creator()\n # experiment_id = client.create_experiment(experiment_name)\n\n\n experiment_metrics = dict(metric=\"loss/accuracy\", mode=\"max\")\n\n tune.run(Worker, \n config={\n 'config_generator': config,\n 'experiment_name': experiment_name,\n 'points_per_epoch': 10\n },\n trial_name_creator=trial_name_creator,\n resources_per_trial={\n 'gpu': 1\n },\n checkpoint_freq=2,\n checkpoint_at_end=True,\n keep_checkpoints_num=1,\n # search_alg=bohb_search, \n # scheduler=bohb_hyperband,\n num_samples=1,\n verbose=0,\n local_dir=DEFAULTS['ARTIFACTS_ROOT']\n # webui_host='127.0.0.1' ## supresses an error\n # stop={'loss/loss': 0}\n )\n# points_per_epoch" }, { "path": "projects/dx7_np/features.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/bluesy-chestnut-forest_0_2020-04-30_11-11-00x02v59be/checkpoint_220/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\nfeatures_all = []\nfeatures_half = []\nfor x in map(lambda x: x['x'], tqdm(loader)):\n q = model.features(x, torch.ones_like(x.float()).bool())\n features_all += [(q.mean.numpy(), q.stddev.numpy())]\n # features_half += [model.features(x, torch.rand_like(x.float())>torch.linspace(0, 1, 32).unsqueeze(-1)).mean.numpy()]\n\n# for item in loader:\n# %%\n" }, { "path": "projects/dx7_np/interpoalte.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/craggy-goldenrod-catfish_0_2020-04-28_02-22-57m8eftq1b/checkpoint_410/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nX_og = next(iter_X)['x']\nfor n in range(n_latents):\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_l = X_og[[0]].clone()\n\n X_a = torch.ones(1,155).bool()\n X_a[:,0] = 0\n q_l = model.features(X_l, X_a)\n\n z = q_l.mean[:,[0]*155][[0]*32]\n z[:,:,n] = torch.linspace(-4, 4, 32).unsqueeze(-1)\n\n X = model.generate_z(z).sample()\n X[...,-1] = 48 + torch.arange(32)\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/np_interp_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_np/live.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport mido\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/squeaky-green-mist_0_2020-04-30_12-50-391i661hyj/checkpoint_100/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\n\n\nfrom uuid import uuid4 as uuid\nuuid = lambda: hex(uuid) \n# self._event.set()\n\n\nclient = jack.Client('DX7Parameteriser')\nport = client.midi_outports.register('output')\ninport = client.midi_inports.register('input')\nevent = threading.Event()\nfs = None # sampling rate\noffset = 0\nfrom neuralDX7.constants import DX7Single, consume_syx\nimport torch\n\nname = torch.tensor([i for i in \"horns \".encode('ascii')])\n\nX = torch.zeros(1, 155).long()\nX[:,-(len(name)):] = name\nX_a = torch.zeros(1, 155).bool()\nX_a[:,-(len(name)):] = 1\nq = model.features(X, X_a)\n\nX_a[:,-29:] = 1\n# X_a = X_a & 0\niter_X = iter(loader)\nX_a\nsyx = list(consume_syx('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/SynprezFM/SynprezFM_01.syx'))\nsyx = torch.from_numpy(np.array([list(i.values()) for i in syx]))\n# m1, m2 = q.mean[:,0]\n# syx_iter = cycle(syx)\ndef slerp(val, low, high):\n omega = np.arccos(np.clip(np.dot(low/np.linalg.norm(low), high/np.linalg.norm(high)), -1, 1))\n so = np.sin(omega)\n if so == 0:\n return (1.0-val) * low + val * high # L'Hopital's rule/LERP\n return np.sin((1.0-val)*omega) / so * low + np.sin(val*omega) / so * high\nx_iter = cycle([*torch.linspace(0, 1, 7)[1:], *torch.linspace(1, 0, 7)[1:]])\n#%%\ni=0\n\nmu, std = \\\n(np.array([ 4.1554513, 4.1125965, -1.9699959, 2.8919716, -6.056072 ,\n -2.407577 , -5.1152377, 2.811712 ], dtype=np.float32),\n np.array([0.3197436 , 0.23426053, 0.25944906, 0.17878139, 0.3019972 ,\n 0.34080952, 0.32115632, 0.34698236], dtype=np.float32))\n\nvals = torch.from_numpy(mu + np.linspace(-3, 3, 128)[:,None] * std).float()\n\ncontroller_map = {}\n\nlatent = torch.full((1, 8), 64).long()\npatch_no = 0\n\nfrom neuralDX7.utils import mask_parameters\n@client.set_process_callback\ndef process(frames):\n global offset, i\n global msg\n global syx_iter\n global controller_map, patch_no, vals, latent\n port.clear_buffer()\n needs_update = False\n X = syx[[patch_no]]\n a = X_a\n\n for offset, data in inport.incoming_midi_events():\n msg = mido.parse(bytes(data))\n\n if msg.type=='note_on':\n # print(msg.__dir__())\n patch_no = msg.note%32\n print(f\"patch set to {patch_no}\")\n needs_update = True\n\n a = X_a[[0]]\n q = model.features(X, a|1)\n vals = q.mean + torch.linspace(-4, 4, 128)[:,None] * q.stddev \n \n if msg.type!='control_change':\n continue\n\n\n if msg.control not in controller_map:\n if len(controller_map) == 8:\n continue\n print(f\"latent {len(controller_map)} set to encoder {msg.control}\")\n controller_map[msg.control] = len(controller_map)\n l_i = list(controller_map).index(msg.control)\n print(f'Latent: {latent}')\n latent[:, controller_map[msg.control]] = msg.value\n needs_update = True\n \n # print(\"{0}: 0x{1}\".format(client.last_frame_time + offset,\n # binascii.hexlify(data).decode()))\n # print(time.time()-offset)\n inport.clear_buffer()\n if (needs_update):\n offset = time.time()\n\n # X = next(iter_X)['x'][[0]]\n # X_d = torch.distributions.Categorical(logits=mask_parameters(torch.zeros(1, 155, 128)))\n\n # X_a = torch.rand_like(X.float()) < 0.3\n # X_a = torch.ones_like(X).bool()\n # X_a[:,:-10] = 0\n \n\n\n # X[~X_a] = X_d.sample()[~X_a]\n\n # max_to_sample = max((~X_a).sum(-1))\n # # X = X[[0]*1]\n\n # # for i in tqdm(range(max_to_sample)):\n\n # logits = model.generate(X, X_a)\n # samples = logits.sample()\n \n # batch_idxs, sample_idx = (~X_a).nonzero().t()\n\n # X[batch_idxs, sample_idx] = samples[batch_idxs, sample_idx]\n # X_a[batch_idxs, sample_idx] = 1\n\n # z = slerp(next(x_iter), m1, m2).unsqueeze(-2)[...,[0]*155,:].unsqueeze(0)\n # z = q.mean\n # z = torch.from_numpy(latent).float()\n # print(q.stddev)\n # z = torch.randn_like(z)\n # z = z.mean().unsqueeze\n # z = q.mean# + torch.randn(q.mean.shape) * 0.1 + q.stddev\n # val = torch.from_numpy(vals).float()\n # print(latent)\n # print(time.time())\n z = vals.gather(0, latent)\n # print(time.time())\n # print(a)\n msg = model.generate_z(X, a, z, t=0.001).sample()\n # print(time.time())\n msg[a] = X[a]\n # msg = X if i%2 else msg\n # msg[:,-(len(name)):] = name\n\n # msg = DX7Single.to_syx([list(next(syx_iter).values())])\n msg = DX7Single.to_syx(msg.numpy().tolist())\n # import mido\n # print([int(i, 2)mido.Message('control_change', control=123).bytes()])\n port.write_midi_event(0, msg.bytes())\n port.write_midi_event(1, mido.Message('control_change', control=123).bytes())\n port.write_midi_event(2, mido.Message('control_change', control=123).bytes())\n port.write_midi_event(3, mido.Message('control_change', control=123).bytes())\n port.write_midi_event(4, mido.Message('control_change', control=123).bytes())\n\n \n\n\n@client.set_samplerate_callback\ndef samplerate(samplerate):\n global fs\n fs = samplerate\n\n\n@client.set_shutdown_callback\ndef shutdown(status, reason):\n print('JACK shutdown:', reason, status)\n event.set()\n\ncapture_port = 'a2j:Arturia BeatStep [24] (capture): Arturia BeatStep MIDI 1'\nplayback_port = 'Carla:Dexed:events-in' \n\nwith client:\n # print(client.get_ports())\n offset = time.time()\n # if connect_to:\n port.connect(playback_port)\n inport.connect(capture_port)\n\n # print('Playing', repr(filename), '... press Ctrl+C to stop')\n try:\n event.wait()\n except KeyboardInterrupt:\n print('\\nInterrupted by user')\n\n\n# %%\n" }, { "path": "projects/dx7_nsp/evaluate.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/craggy-goldenrod-catfish_0_2020-04-28_02-22-57m8eftq1b/checkpoint_410/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\n\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nfor n in range(10):\n X = next(iter_X)['x']\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_d = torch.distributions.Categorical(logits=mask_parameters(torch.zeros(32, 155, 128)))\n\n X_a = torch.rand_like(X.float()) < 0.3\n X_a = torch.ones_like(X).bool()\n X_a[:,:-10] = 0\n\n X = X[[0]*32]\n X[~X_a] = X_d.sample()[~X_a]\n\n max_to_sample = max((~X_a).sum(-1))\n\n # for i in tqdm(range(max_to_sample)):\n\n logits = model.generate(X, X_a)\n samples = logits.sample()\n\n has_unsampled = ~X_a.all(-1)\n\n batch_idxs, sample_idx = (~X_a).nonzero().t()\n\n X[batch_idxs, sample_idx] = samples[batch_idxs, sample_idx]\n X_a[batch_idxs, sample_idx] = 1\n \n\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/np_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_nsp/experiment.py", "content": "#%%\nfrom os import environ\nenviron['MLFLOW_TRACKING_URI'] = 'http://tracking.olympus.nintorac.dev:9001/'\n\nfrom neuralDX7.constants import N_PARAMS, MAX_VALUE\nfrom agoge.utils import trial_name_creator\nfrom neuralDX7 import DEFAULTS\nfrom agoge import TrainWorker as Worker\nfrom ray import tune\nfrom neuralDX7.models import DX7NeuralSylvesterProcess as Model\nfrom neuralDX7.solvers import DX7NeuralSylvesterProcess as Solver\nfrom neuralDX7.datasets import DX7SysexDataset as Dataset\n\ndef config(experiment_name, trial_name, \n n_heads=8, n_features=32, \n batch_size=16, data_size=1.,\n latent_dim=8, num_flows=1,\n **kwargs):\n \n\n\n data_handler = {\n 'Dataset': Dataset,\n 'dataset_opts': {\n 'data_size': data_size\n },\n 'loader_opts': {\n 'batch_size': batch_size,\n },\n }\n\n ### MODEL FEATURES\n layer_features = n_heads * n_features\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': layer_features * 3\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 1\n }\n \n\n model = {\n 'Model': Model,\n 'features': layer_features,\n 'latent_dim': latent_dim,\n 'encoder': encoder,\n 'decoder': {\n 'c_features': latent_dim,\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 1\n },\n 'num_flows': num_flows,\n 'deterministic_path_drop_rate': 0.8\n }\n\n solver = {\n 'Solver': Solver,\n 'beta_temp': 1e-3,\n 'max_beta': 1\n }\n\n tracker = {\n 'metrics': [\n 'reconstruction_loss', \n 'accuracy', \n 'kl', \n 'beta', \n 'q_log_det', \n 'q_z',\n 'p_z',\n ],\n 'experiment_name': experiment_name,\n 'trial_name': trial_name\n }\n\n return {\n 'data_handler': data_handler,\n 'model': model,\n 'solver': solver,\n 'tracker': tracker,\n }\n\nif __name__=='__main__':\n # from ray import ray\n import sys\n postfix = sys.argv[1] if len(sys.argv)==2 else ''\n # ray.init()\n # from ray.tune.utils import validate_save_restore\n # validate_save_restore(Worker)\n # client = MlflowClient(tracking_uri='localhost:5000')\n experiment_name = f'dx7-nsp-00'#+experiment_name_creator()\n # experiment_id = client.create_experiment(experiment_name)\n\n\n experiment_metrics = dict(metric=\"loss/accuracy\", mode=\"max\")\n\n tune.run(Worker, \n config={\n 'config_generator': config,\n 'experiment_name': experiment_name,\n 'points_per_epoch': 10\n },\n trial_name_creator=trial_name_creator,\n resources_per_trial={\n 'gpu': 1\n },\n checkpoint_freq=2,\n checkpoint_at_end=True,\n keep_checkpoints_num=1,\n # search_alg=bohb_search, \n # scheduler=bohb_hyperband,\n num_samples=1,\n verbose=0,\n local_dir=DEFAULTS['ARTIFACTS_ROOT']\n # webui_host='127.0.0.1' ## supresses an error\n # stop={'loss/loss': 0}\n )\n# points_per_epoch" }, { "path": "projects/dx7_nsp/features.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nworker = InferenceWorker('~/agoge/artifacts/dx7-nsp/leaky-burgundy-coati.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\nfeatures_all = []\nfeatures_half = []\nfor x in map(lambda x: x['X'], tqdm(loader)):\n q = model.features(x, torch.ones_like(x.float()).bool()).q_z\n features_all += [(q.mean.numpy(), q.stddev.numpy())]\n # features_half += [model.features(x, torch.rand_like(x.float())>torch.linspace(0, 1, 32).unsqueeze(-1)).mean.numpy()]\n\nmus, vars = map(np.concatenate, zip(*features_all))\n\n# for item in loader:\n# %%\n" }, { "path": "projects/dx7_nsp/interpoalte.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/craggy-goldenrod-catfish_0_2020-04-28_02-22-57m8eftq1b/checkpoint_410/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nX_og = next(iter_X)['x']\nfor n in range(n_latents):\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_l = X_og[[0]].clone()\n\n X_a = torch.ones(1,155).bool()\n X_a[:,0] = 0\n q_l = model.features(X_l, X_a)\n\n z = q_l.mean[:,[0]*155][[0]*32]\n z[:,:,n] = torch.linspace(-4, 4, 32).unsqueeze(-1)\n\n X = model.generate_z(z).sample()\n X[...,-1] = 48 + torch.arange(32)\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/np_interp_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_nsp/live.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport mido\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nworker = InferenceWorker('~/agoge/artifacts/dx7-nsp/leaky-burgundy-coati.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\n\n\nfrom uuid import uuid4 as uuid\nuuid = lambda: hex(uuid) \n# self._event.set()\n\n\nclient = jack.Client('DX7Parameteriser')\nport = client.midi_outports.register('output')\ninport = client.midi_inports.register('input')\nevent = threading.Event()\nfs = None # sampling rate\noffset = 0\nfrom neuralDX7.constants import DX7Single, consume_syx\nimport torch\n\nname = torch.tensor([i for i in \"horns \".encode('ascii')])\n\nX = torch.zeros(1, 155).long()\nX[:,-(len(name)):] = name\nX_a = torch.zeros(1, 155).bool()\nX_a[:,-(len(name)):] = 1\nq = model.features(X, X_a)\n\nX_a[:,-29:] = 1\n# X_a = X_a & 0\niter_X = iter(loader)\nX_a\nsyx = list(consume_syx('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/SynprezFM/SynprezFM_01.syx'))\nsyx = torch.from_numpy(np.array([list(i.values()) for i in syx]))\n# m1, m2 = q.mean[:,0]\n# syx_iter = cycle(syx)\ndef slerp(val, low, high):\n omega = np.arccos(np.clip(np.dot(low/np.linalg.norm(low), high/np.linalg.norm(high)), -1, 1))\n so = np.sin(omega)\n if so == 0:\n return (1.0-val) * low + val * high # L'Hopital's rule/LERP\n return np.sin((1.0-val)*omega) / so * low + np.sin(val*omega) / so * high\nx_iter = cycle([*torch.linspace(0, 1, 7)[1:], *torch.linspace(1, 0, 7)[1:]])\n#%%\ni=0\n\nmu, std = \\\n(np.array([ 4.1554513, 4.1125965, -1.9699959, 2.8919716, -6.056072 ,\n -2.407577 , -5.1152377, 2.811712 ], dtype=np.float32),\n np.array([0.3197436 , 0.23426053, 0.25944906, 0.17878139, 0.3019972 ,\n 0.34080952, 0.32115632, 0.34698236], dtype=np.float32))\n\nvals = torch.from_numpy(mu + np.linspace(-3, 3, 128)[:,None] * std).float()\n\ncontroller_map = {}\n\nlatent = torch.full((1, 8), 64).long()\npatch_no = 0\nflow=None\nfrom neuralDX7.utils import mask_parameters\n@client.set_process_callback\ndef process(frames):\n global offset, i\n global msg\n global syx_iter\n global controller_map, patch_no, vals, latent, flow\n port.clear_buffer()\n needs_update = False\n X = syx[[patch_no]]\n a = X_a\n\n for offset, data in inport.incoming_midi_events():\n msg = mido.parse(bytes(data))\n\n if msg.type=='note_on':\n # print(msg.__dir__())\n patch_no = msg.note%32\n print(f\"patch set to {patch_no}\")\n needs_update = True\n\n a = X_a[[0]]\n flow = model.features(X, a|1)\n q = flow.q_z\n vals = q.mean + torch.linspace(-4, 4, 128)[:,None] * q.stddev \n \n if msg.type!='control_change':\n continue\n\n\n if msg.control not in controller_map:\n if len(controller_map) == 8:\n continue\n print(f\"latent {len(controller_map)} set to encoder {msg.control}\")\n controller_map[msg.control] = len(controller_map)\n l_i = list(controller_map).index(msg.control)\n print(f'Latent: {latent}')\n latent[:, controller_map[msg.control]] = msg.value\n needs_update = True\n \n # print(\"{0}: 0x{1}\".format(client.last_frame_time + offset,\n # binascii.hexlify(data).decode()))\n # print(time.time()-offset)\n inport.clear_buffer()\n if (needs_update):\n offset = time.time()\n\n # X = next(iter_X)['x'][[0]]\n # X_d = torch.distributions.Categorical(logits=mask_parameters(torch.zeros(1, 155, 128)))\n\n # X_a = torch.rand_like(X.float()) < 0.3\n # X_a = torch.ones_like(X).bool()\n # X_a[:,:-10] = 0\n \n\n\n # X[~X_a] = X_d.sample()[~X_a]\n\n # max_to_sample = max((~X_a).sum(-1))\n # # X = X[[0]*1]\n\n # # for i in tqdm(range(max_to_sample)):\n\n # logits = model.generate(X, X_a)\n # samples = logits.sample()\n \n # batch_idxs, sample_idx = (~X_a).nonzero().t()\n\n # X[batch_idxs, sample_idx] = samples[batch_idxs, sample_idx]\n # X_a[batch_idxs, sample_idx] = 1\n\n # z = slerp(next(x_iter), m1, m2).unsqueeze(-2)[...,[0]*155,:].unsqueeze(0)\n # z = q.mean\n # z = torch.from_numpy(latent).float()\n # print(q.stddev)\n # z = torch.randn_like(z)\n # z = z.mean().unsqueeze\n # z = q.mean# + torch.randn(q.mean.shape) * 0.1 + q.stddev\n # val = torch.from_numpy(vals).float()\n # print(latent)\n # print(time.time())\n z, _ = flow.flow(vals.gather(0, latent))\n # print(time.time())\n # print(a)\n msg = model.generate_z(X, a, z, t=0.001).sample()\n # print(time.time())\n msg[a] = X[a]\n # msg = X if i%2 else msg\n # msg[:,-(len(name)):] = name\n\n # msg = DX7Single.to_syx([list(next(syx_iter).values())])\n msg = DX7Single.to_syx(msg.numpy().tolist())\n # import mido\n # print([int(i, 2)mido.Message('control_change', control=123).bytes()])\n port.write_midi_event(0, msg.bytes())\n port.write_midi_event(1, mido.Message('control_change', control=123).bytes())\n port.write_midi_event(2, mido.Message('control_change', control=123).bytes())\n port.write_midi_event(3, mido.Message('control_change', control=123).bytes())\n port.write_midi_event(4, mido.Message('control_change', control=123).bytes())\n\n \n\n\n@client.set_samplerate_callback\ndef samplerate(samplerate):\n global fs\n fs = samplerate\n\n\n@client.set_shutdown_callback\ndef shutdown(status, reason):\n print('JACK shutdown:', reason, status)\n event.set()\n\ncapture_port = 'a2j:Arturia BeatStep [24] (capture): Arturia BeatStep MIDI 1'\nplayback_port = 'Carla:Dexed:events-in' \n\nwith client:\n # print(client.get_ports())\n offset = time.time()\n # if connect_to:\n port.connect(playback_port)\n inport.connect(capture_port)\n\n # print('Playing', repr(filename), '... press Ctrl+C to stop')\n try:\n event.wait()\n except KeyboardInterrupt:\n print('\\nInterrupted by user')\n\n\n# %%\n" }, { "path": "projects/dx7_patch_neural_process/evaluate.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/Worker/messy-firebrick-barracuda_0_2020-04-14_08-59-40ryp93n44/checkpoint_4/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.train\n\nn_samples = 32\n\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nfor n in range(10):\n X = next(iter_X)['x']\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_d = torch.distributions.Categorical(logits=mask_parameters(torch.zeros(32, 155, 128)))\n\n X_a = torch.rand_like(X.float()) < 0.3\n X_a = torch.ones_like(X).bool()\n X_a[:,:-10] = 0\n\n X = X[[0]*32]\n X[~X_a] = X_d.sample()[~X_a]\n\n max_to_sample = max((~X_a).sum(-1))\n\n for i in tqdm(range(max_to_sample)):\n\n logits = model.generate(X, X_a)\n samples = logits.sample()\n\n has_unsampled = ~X_a.all(-1)\n\n sample_idx = (torch.rand_like(X.float()) * (~X_a).float()).argmax(-1)[has_unsampled]\n batch_idxs = torch.arange(X.shape[0])[has_unsampled]\n\n\n X[batch_idxs, sample_idx] = samples[batch_idxs, sample_idx]\n X_a[batch_idxs, sample_idx] = 1\n # X_a = X_a | new_mask\n\n if X_a.all():\n break\n\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/gen_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_patch_neural_process/features_analysis.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/Worker/messy-firebrick-barracuda_0_2020-04-14_08-59-40ryp93n44/checkpoint_4/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\n\nXs = []\nfeatures = []\nfor X in tqdm(loader):\n Xs += [X['x']]\n features += [model.features(X['x'])]\nfeatures = torch.cat(features).flatten(-2, -1)\nX = torch.cat(Xs)\n \n#%%\n### --------TSNE-----------\n\nimport numpy as np\nfrom sklearn.manifold import TSNE\nX_embedded = TSNE(n_components=2).fit_transform(features)\nX_embedded.shape\n#%%\nplt.figure(figsize=(20,30))\nplt.scatter(*zip(*X_embedded), linewidths=0.1)\n\n# %%\n### ---------K-means---------------\n\nfrom sklearn.cluster import KMeans\nfrom neuralDX7.utils import dx7_bulk_pack\nimport mido\nn_clusters = 8\nkmeans = KMeans(n_clusters=n_clusters, random_state=0).fit(features)\nlabels = kmeans.labels_\ntasting = []\n\nout_template = '/home/nintorac/.local/share/DigitalSuburban/Dexed/cartridges/neuralDX7/group_{}.syx'\n\nfor i in range(n_clusters):\n in_cluster, = (labels == i).nonzero()\n if len(in_cluster) < 32:\n continue\n\n choices = np.random.choice(in_cluster, 32)\n voices = X[choices]\n\n patch_message = dx7_bulk_pack(voices)\n\n mido.write_syx_file(out_template.format(i), [patch_message])\n\n\n\n# %%\n" }, { "path": "projects/dx7_patch_neural_process/ray_train.py", "content": "#%%\nfrom os import environ\nenviron['MLFLOW_TRACKING_URI'] = 'http://tracking.olympus.nintorac.dev:9001/'\n# environ['MLFLOW_TRACKING_URI'] = 'http://localhost:9001/'\n#environ['ARTIFACTS_ROOT'] = '/content/gdrive/My Drive/audio/artifacts'\n# ARTIFACTS_ROOT='/content/gdrive/My Drive/audio/artifacts'\nfrom neuralDX7.constants import N_PARAMS, MAX_VALUE\nfrom agoge.utils import trial_name_creator\nfrom neuralDX7 import DEFAULTS\nfrom agoge import TrainWorker as Worker\nfrom ray import tune\nfrom neuralDX7.models import DX7PatchProcess as Model\nfrom neuralDX7.solvers import DX7PatchProcess as Solver\nfrom neuralDX7.datasets import DX7SysexDataset as Dataset\n\ndef config(experiment_name, trial_name, \n n_heads=8, n_features=32, \n batch_size=16, data_size=0.05,\n **kwargs):\n \n\n\n data_handler = {\n 'Dataset': Dataset,\n 'dataset_opts': {\n 'data_size': data_size\n },\n 'loader_opts': {\n 'batch_size': batch_size,\n },\n }\n\n ### MODEL FEATURES\n layer_features = n_heads * n_features\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': layer_features * 2\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 12\n }\n \n\n model = {\n 'Model': Model,\n 'features': layer_features,\n 'encoder': encoder\n }\n\n solver = {\n 'Solver': Solver,\n 'lr': 1e-3,\n }\n\n tracker = {\n 'metrics': ['reconstruction_loss', 'accuracy'],\n 'experiment_name': experiment_name,\n 'trial_name': trial_name\n }\n\n return {\n 'data_handler': data_handler,\n 'model': model,\n 'solver': solver,\n 'tracker': tracker,\n }\n\nif __name__=='__main__':\n # from ray import ray\n import sys\n import mlflow\n from mlflow.tracking import MlflowClient\n postfix = sys.argv[1] if len(sys.argv)==2 else ''\n\n # ray.init()\n # from ray.tune.utils import validate_save_restore\n # validate_save_restore(Worker)\n client = MlflowClient()\n experiment_name = f'dx7-vae-{postfix}'#+experiment_name_creator()\n resume=False\n try:\n experiment_id = client.create_experiment(experiment_name)\n except mlflow.exceptions.RestException:\n resume = True\n\n experiment_metrics = dict(metric=\"loss/accuracy\", mode=\"max\")\n import torch\n gpus = 0.5 if torch.cuda.is_available() else 0\n gpus = 1\n \n # import ray\n\n # ray.init()\n # ray.tune.utils.validate_save_restore(Worker)\n\n tune.run(Worker, \n config={\n 'config_generator': config,\n 'experiment_name': experiment_name,\n 'points_per_epoch': 10\n },\n trial_name_creator=trial_name_creator,\n resources_per_trial={\n # 'gpu': gpus,\n 'cpu': 6\n },\n checkpoint_freq=2,\n checkpoint_at_end=True,\n keep_checkpoints_num=1,\n # search_alg=bohb_search, \n # scheduler=bohb_hyperband,\n num_samples=1,\n verbose=1,\n local_dir=DEFAULTS['ARTIFACTS_ROOT'],\n resume=resume\n # webui_host='127.0.0.1' ## supresses an error\n # stop={'loss/loss': 0}\n )\n# points_per_epoch\n" }, { "path": "projects/dx7_vae/duplicate_test.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport mido\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nfrom numpy import array\nworker = InferenceWorker('hasty-copper-dogfish', 'dx7-vae', with_data=True)\nfloat32='float32'\nmodel = worker.model\n# data = worker.dataset\n# loader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\n# loader.batch_sampler.batch_size = n_samples\n\n\n# randoms = torch.cat([model.generate(torch.randn(2**11, 8)).logits.argmax(-1) for _ in tqdm(range(2**5))])\n\n# %%\nfrom matplotlib import pyplot as plt\nimport torch\n\nrand = torch.rand(100)\nrandn = torch.randn(100)\nplt.scatter(rand, torch.sigmoid(rand)+0.5)\nplt.scatter(randn, torch.sigmoid(randn)-0.5)\n\n# %%\n" }, { "path": "projects/dx7_vae/evaluate.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/craggy-goldenrod-catfish_0_2020-04-28_02-22-57m8eftq1b/checkpoint_410/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\n\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nfor n in range(10):\n X = next(iter_X)['x']\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_d = torch.distributions.Categorical(logits=mask_parameters(torch.zeros(32, 155, 128)))\n\n X_a = torch.rand_like(X.float()) < 0.3\n X_a = torch.ones_like(X).bool()\n X_a[:,:-10] = 0\n\n X = X[[0]*32]\n X[~X_a] = X_d.sample()[~X_a]\n\n max_to_sample = max((~X_a).sum(-1))\n\n # for i in tqdm(range(max_to_sample)):\n\n logits = model.generate(X, X_a)\n samples = logits.sample()\n\n has_unsampled = ~X_a.all(-1)\n\n batch_idxs, sample_idx = (~X_a).nonzero().t()\n\n X[batch_idxs, sample_idx] = samples[batch_idxs, sample_idx]\n X_a[batch_idxs, sample_idx] = 1\n \n\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/np_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_vae/experiment.py", "content": "#%%\nfrom os import environ\nenviron['MLFLOW_TRACKING_URI'] = 'http://tracking.olympus.nintorac.dev:9001/'\n\nfrom neuralDX7.constants import N_PARAMS, MAX_VALUE\nfrom agoge.utils import trial_name_creator\nfrom neuralDX7 import DEFAULTS\nfrom agoge import TrainWorker as Worker\nfrom ray import tune\nfrom neuralDX7.models import DX7VAE as Model\nfrom neuralDX7.solvers import DX7VAE as Solver\nfrom neuralDX7.datasets import DX7SysexDataset as Dataset\n\ndef config(experiment_name, trial_name, \n n_heads=8, n_features=64, \n batch_size=16, data_size=1.,\n latent_dim=8, num_flows=16,\n **kwargs):\n \n\n\n data_handler = {\n 'Dataset': Dataset,\n 'dataset_opts': {\n 'data_size': data_size\n },\n 'loader_opts': {\n 'batch_size': batch_size,\n },\n }\n\n ### MODEL FEATURES\n layer_features = n_heads * n_features\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': layer_features * 3\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 12\n }\n \n\n model = {\n 'Model': Model,\n 'features': layer_features,\n 'latent_dim': latent_dim,\n 'encoder': encoder,\n 'decoder': {\n 'c_features': latent_dim,\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': N_PARAMS,\n 'n_layers': 12\n },\n 'num_flows': num_flows,\n 'deterministic_path_drop_rate': 0.8\n }\n\n solver = {\n 'Solver': Solver,\n 'beta_temp': 6e-5,\n 'max_beta': 0.5\n }\n\n tracker = {\n 'metrics': [\n 'reconstruction_loss', \n 'accuracy', \n 'kl', \n 'beta', \n 'log_det', \n 'q_z_0',\n 'p_z_k',\n ],\n 'experiment_name': experiment_name,\n 'trial_name': trial_name\n }\n\n return {\n 'data_handler': data_handler,\n 'model': model,\n 'solver': solver,\n 'tracker': tracker,\n }\n\nif __name__=='__main__':\n # from ray import ray\n import sys\n postfix = sys.argv[1] if len(sys.argv)==2 else ''\n # ray.init()\n # from ray.tune.utils import validate_save_restore\n # validate_save_restore(Worker)\n # client = MlflowClient(tracking_uri='localhost:5000')\n experiment_name = f'dx7-vae-dev'#+experiment_name_creator()\n # experiment_id = client.create_experiment(experiment_name)\n\n\n experiment_metrics = dict(metric=\"loss/accuracy\", mode=\"max\")\n\n tune.run(Worker, \n config={\n 'config_generator': config,\n 'experiment_name': experiment_name,\n 'points_per_epoch': 10\n },\n trial_name_creator=trial_name_creator,\n resources_per_trial={\n # 'gpu': 1\n # 'cpu': 5\n },\n checkpoint_freq=2,\n checkpoint_at_end=True,\n keep_checkpoints_num=1,\n # search_alg=bohb_search, \n # scheduler=bohb_hyperband,\n num_samples=1,\n verbose=0,\n local_dir=DEFAULTS['ARTIFACTS_ROOT']\n # webui_host='127.0.0.1' ## supresses an error\n # stop={'loss/loss': 0}\n )\n# points_per_epoch\n\n# %%\n" }, { "path": "projects/dx7_vae/features.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nworker = InferenceWorker('~/agoge/artifacts/dx7-vae/hasty-copper-dogfish_0_2020-05-06_10-46-27o654hmde/checkpoint_204/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\nfeatures_all = []\nfeatures_half = []\nfor x in map(lambda x: x['X'], tqdm(loader)):\n q = model.features(x)\n features_all += [(q.mean.numpy(), q.stddev.numpy())]\n # features_half += [model.features(x, torch.rand_like(x.float())>torch.linspace(0, 1, 32).unsqueeze(-1)).mean.numpy()]\n\nmus, vars = map(np.concatenate, zip(*features_all))\nprint(mus.mean(0), vars.mean(0))\n# for item in loader:\n# %%\nfrom timeit import timeit\n\nwith timeit():\n model.generate(q.sample()[[0]])\n\n# %%\nfrom time import time\ns_time = time()\n\nfor i in range(100):\n model.generate(q.sample()[[0]])\n\nprint((time()-s_time)/100)\n# %%\n" }, { "path": "projects/dx7_vae/interpoalte.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport torch\nfrom tqdm import tqdm\nfrom matplotlib import pyplot as plt\nworker = InferenceWorker('/home/nintorac/agoge/artifacts/craggy-goldenrod-catfish_0_2020-04-28_02-22-57m8eftq1b/checkpoint_410/model.box', with_data=True)\n\nmodel = worker.model\ndata = worker.dataset\nloader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\nloader.batch_sampler.batch_size = n_samples\n# %%\n# batch = next(iter(loader))['x']\n\n# X_a = torch.rand_like(batch.float()) > torch.linspace(0, 1, n_samples).unsqueeze(-1)\n\n# logits = model.generate(batch, X_a)\n\n# # %%\n# from matplotlib import pyplot as plt\n# plt.imshow(X_a)\n\n# # %%\n# plt.scatter(torch.arange(n_samples), logits.log_prob(batch).mean(-1))\n\n# # %%\n# plt.imshow(logits.log_prob(batch))\n\n# %%\n\nfrom itertools import count \nfrom neuralDX7.utils import dx7_bulk_pack, mask_parameters\nimport mido\niter_X = iter(loader)\nX_og = next(iter_X)['x']\nfor n in range(n_latents):\n # syx = dx7_bulk_pack(X.numpy().tolist())\n # mido.write_syx_file('/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/OG.syx', [syx])\n\n X_l = X_og[[0]].clone()\n\n X_a = torch.ones(1,155).bool()\n X_a[:,0] = 0\n q_l = model.features(X_l, X_a)\n\n z = q_l.mean[:,[0]*155][[0]*32]\n z[:,:,n] = torch.linspace(-4, 4, 32).unsqueeze(-1)\n\n X = model.generate_z(z).sample()\n X[...,-1] = 48 + torch.arange(32)\n syx = dx7_bulk_pack(X.numpy().tolist())\n mido.write_syx_file(f'/home/nintorac/.local/share/DigitalSuburban/Dexed/Cartridges/neuralDX7/np_interp_{n}.syx', [syx])\n\n# # %%\n# from neuralDX7.constants import voice_struct, VOICE_KEYS, checksum\n# def dx7_bulk_pack(voices):\n\n# HEADER = int('0x43', 0), int('0x00', 0), int('0x09', 0), int('0x20', 0), int('0x00', 0)\n# assert len(voices)==32\n# voices_bytes = bytes()\n# for voice in voices:\n# voice_bytes = voice_struct.pack(dict(zip(VOICE_KEYS, voice)))\n# voices_bytes += voice_bytes\n \n \n# patch_checksum = [checksum(voices_bytes)]\n\n# # data = bytes(HEADER) + voices_bytes + bytes(patch_checksum)\n\n# # return mido.Message('sysex', data=data)\n\n# # %%\n\n# from neuralDX7.constants import VOICE_KEYS, MAX_VALUE, VOICE_PARAMETER_RANGES\n# def mask_parameters(x, voice_keys=VOICE_KEYS, inf=1e9):\n# device = x.device\n# mask_item_f = lambda x: torch.arange(MAX_VALUE).to(device) > max(x) \n# mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, voice_keys))\n\n# mask = torch.stack(list(mapper))\n \n# return torch.masked_fill(x, mask, -inf)\n\n# plt.imshow(mask_parameters(torch.randn(10, 155, 128))[0])\n# # %%\n\n\n# %%\n" }, { "path": "projects/dx7_vae/live.py", "content": "# %%\nfrom agoge import InferenceWorker\nimport threading\nimport torch\nimport mido\nimport time\nimport numpy as np\nfrom tqdm import tqdm\nimport jack\nfrom matplotlib import pyplot as plt\nfrom itertools import cycle\nfrom numpy import array\nworker = InferenceWorker('~/agoge/artifacts/dx7-vae/hasty-copper-dogfish_0_2020-05-06_10-46-27o654hmde/checkpoint_204/model.box', with_data=False)\nfloat32='float32'\nmodel = worker.model\n# data = worker.dataset\n# loader = data.loaders.test\n\nn_samples = 32\nn_latents = 8\n# loader.batch_sampler.batch_size = n_samples\n\n\nfrom uuid import uuid4 as uuid\nuuid = lambda: hex(uuid) \n# self._event.set()\n\n\nclient = jack.Client('DX7Parameteriser')\nport = client.midi_outports.register('output')\ninport = client.midi_inports.register('input')\nevent = threading.Event()\nfs = None # sampling rate\noffset = 0\nfrom neuralDX7.constants import DX7Single, consume_syx\nimport torch\n\nname = torch.tensor([i for i in \"horns \".encode('ascii')])\n\n\ndef slerp(val, low, high):\n omega = np.arccos(np.clip(np.dot(low/np.linalg.norm(low), high/np.linalg.norm(high)), -1, 1))\n so = np.sin(omega)\n if so == 0:\n return (1.0-val) * low + val * high # L'Hopital's rule/LERP\n return np.sin((1.0-val)*omega) / so * low + np.sin(val*omega) / so * high\nx_iter = cycle([*torch.linspace(0, 1, 7)[1:], *torch.linspace(1, 0, 7)[1:]])\n#%%\ni=0\n\nmu, std = \\\n(array([ 5.5626068e-02, 7.9248362e-04, -8.0890575e-04, 1.6684370e-01,\n 1.6537485e-01, -6.2455550e-02, 9.4467170e-05, -7.5367272e-02],\n dtype=float32),\n array([0.35453376, 0.3556142 , 0.35896832, 0.341505 , 0.3299536 ,\n 0.33990443, 0.3350083 , 0.339214 ], dtype=float32))\nvals = torch.from_numpy(mu + np.linspace(-3, 3, 128)[:,None] * std).float()\n\ncontroller_map = {}\n\nlatent = torch.full((1, 8), 64).long()\npatch_no = 0\n\nfrom neuralDX7.utils import mask_parameters\n@client.set_process_callback\ndef process(frames):\n global offset, i\n global msg\n global syx_iter\n global controller_map, patch_no, vals, latent\n port.clear_buffer()\n needs_update = False\n\n\n for offset, data in inport.incoming_midi_events():\n msg = mido.parse(bytes(data))\n\n if msg.type=='note_on':\n port.write_midi_event(0, mido.Message('control_change', control=123).bytes()) \n if msg.type!='control_change':\n continue\n\n\n if msg.control not in controller_map:\n if len(controller_map) == 8:\n continue\n print(f\"latent {len(controller_map)} set to encoder {msg.control}\")\n controller_map[msg.control] = len(controller_map)\n l_i = list(controller_map).index(msg.control)\n print(f'Latent: {latent}')\n latent[:, controller_map[msg.control]] = msg.value\n needs_update = True\n \n # print(\"{0}: 0x{1}\".format(client.last_frame_time + offset,\n # binascii.hexlify(data).decode()))\n # print(time.time()-offset)\n inport.clear_buffer()\n if (needs_update):\n offset = time.time()\n\n z = vals.gather(0, latent)\n msg = model.generate(z, t=0.001).sample()\n\n msg = DX7Single.to_syx(msg.numpy().tolist())\n\n port.write_midi_event(1, msg.bytes())\n mido.write_syx_file('example_single_voice.mid', [msg])\n # port.write_midi_event(1, mido.Message('control_change', control=123).bytes())\n # port.write_midi_event(2, mido.Message('control_change', control=123).bytes())\n # port.write_midi_event(3, mido.Message('control_change', control=123).bytes())\n # port.write_midi_event(4, mido.Message('control_change', control=123).bytes())\n\n \n\n\n@client.set_samplerate_callback\ndef samplerate(samplerate):\n global fs\n fs = samplerate\n\n\n@client.set_shutdown_callback\ndef shutdown(status, reason):\n print('JACK shutdown:', reason, status)\n event.set()\n\ncapture_port = 'a2j:Arturia BeatStep [24] (capture): Arturia BeatStep MIDI 1'\nplayback_port = 'Carla:Dexed:events-in' \n\nwith client:\n # print(client.get_ports())\n offset = time.time()\n # if connect_to:\n port.connect(playback_port)\n inport.connect(capture_port)\n\n # print('Playing', repr(filename), '... press Ctrl+C to stop')\n try:\n event.wait()\n except KeyboardInterrupt:\n print('\\nInterrupted by user')\n\n\n# %%\n" }, { "path": "projects/mnist_neural_process/experiment.py", "content": "#%%\nfrom os import environ\nenviron['MLFLOW_TRACKING_URI'] = 'http://tracking.olympus.nintorac.dev:9001/'\n# environ['MLFLOW_TRACKING_URI'] = 'http://localhost:9001/'\n#environ['ARTIFACTS_ROOT'] = '/content/gdrive/My Drive/audio/artifacts'\n# ARTIFACTS_ROOT='/content/gdrive/My Drive/audio/artifacts'\nfrom neuralDX7.constants import N_PARAMS, MAX_VALUE\nfrom agoge.utils import trial_name_creator\nfrom neuralDX7 import DEFAULTS\nfrom agoge import Worker\nfrom ray import tune\nfrom neuralDX7.models import DX7PatchProcess as Model\nfrom neuralDX7.solvers import DX7PatchProcess as Solver\nfrom neuralDX7.datasets import DX7SysexDataset as Dataset\n\nfrom pathlib import Path\nfrom torchvision import datasets, transforms\nfrom torch.utils.data import ConcatDataset\nfrom agoge import DEFAULTS\n\nimport numpy as np \nimport torch\n\nK=2\n\nclass MNISTDataset():\n\n def __init__(self, data_path=DEFAULTS['ARTIFACTS_ROOT'], transform=None):\n \n transform = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize((0.1307,), (0.3081,))\n ])\n\n if not isinstance(data_path, Path):\n data_path = Path(data_path).expanduser()\n\n train_dataset = datasets.MNIST(data_path.as_posix(), train=True, download=True,transform=transform)\n test_dataset = datasets.MNIST(data_path.as_posix(), train=False, download=True, transform=transform)\n\n self.dataset = ConcatDataset((train_dataset, test_dataset))\n\n def __getitem__(self, i):\n \n item = self.dataset[i][0]\n sub_item = item[:,14-K :14+K ,14-K :14+K ].flatten()\n digitzed = np.digitize(sub_item, np.linspace(-4, 4, 16))\n \n return {'x': torch.from_numpy(digitzed)}\n\n def __len__(self):\n\n return len(self.dataset)\n\nMNISTDataset()[0]\n\n#%%\nDataset = MNISTDataset\n\n\ndef config(experiment_name, trial_name, \n n_heads=8, n_features=32, \n batch_size=16, data_size=0.9999,\n **kwargs):\n \n\n\n data_handler = {\n 'Dataset': Dataset,\n 'dataset_opts': {\n # 'data_size': data_size\n },\n 'loader_opts': {\n 'batch_size': batch_size,\n },\n }\n\n ### MODEL FEATURES\n layer_features = n_heads * n_features\n\n head_features = layer_features // n_heads\n\n attention = {\n 'n_features': layer_features,\n 'n_hidden': head_features,\n 'n_heads': n_heads\n }\n \n attention_layer = {\n 'attention': attention,\n 'features': layer_features,\n 'hidden_dim': layer_features * 2\n }\n\n encoder = {\n 'features': layer_features,\n 'attention_layer': attention_layer,\n 'max_len': (K*2)**2,\n 'n_layers': 24\n }\n \n\n model = {\n 'Model': Model,\n 'features': layer_features,\n 'encoder': encoder\n }\n\n solver = {\n 'Solver': Solver,\n 'lr': 1e-3,\n }\n\n tracker = {\n 'metrics': ['reconstruction_loss', 'accuracy'],\n 'experiment_name': experiment_name,\n 'trial_name': trial_name\n }\n\n return {\n 'data_handler': data_handler,\n 'model': model,\n 'solver': solver,\n 'tracker': tracker,\n }\n\nif __name__=='__main__':\n # from ray import ray\n import sys\n import mlflow\n from mlflow.tracking import MlflowClient\n postfix = sys.argv[1] if len(sys.argv)==2 else ''\n\n # ray.init()\n # from ray.tune.utils import validate_save_restore\n # validate_save_restore(Worker)\n client = MlflowClient()\n experiment_name = f'dx7-vae-{postfix}'#+experiment_name_creator()\n resume=False\n try:\n experiment_id = client.create_experiment(experiment_name)\n except mlflow.exceptions.RestException:\n resume = True\n\n experiment_metrics = dict(metric=\"loss/accuracy\", mode=\"max\")\n import torch\n gpus = 0.5 if torch.cuda.is_available() else 0\n gpus = 1\n\n\n tune.run(Worker, \n config={\n 'config_generator': config,\n 'experiment_name': experiment_name,\n 'points_per_epoch': 10\n },\n trial_name_creator=trial_name_creator,\n resources_per_trial={\n 'gpu': gpus,\n 'cpu': 1\n },\n checkpoint_freq=2,\n checkpoint_at_end=True,\n keep_checkpoints_num=1,\n # search_alg=bohb_search, \n # scheduler=bohb_hyperband,\n num_samples=1,\n verbose=1,\n local_dir=DEFAULTS['ARTIFACTS_ROOT'],\n resume=resume\n # webui_host='127.0.0.1' ## supresses an error\n # stop={'loss/loss': 0}\n )\n# points_per_epoch\n" }, { "path": "requirements.txt", "content": "bitstruct==8.9.0\nagoge==0.0.6\nmido==1.2.9" }, { "path": "scratch/dx7-sysexformat.md", "content": "Sysex Documentation \n===================\n\n(Message GUS:472)\nReceived: from mailhub.iastate.edu by po-3.iastate.edu \n\tid AA06806; Sat, 25 Sep 93 16:13:53 -0500\nReceived: from Waisman.Wisc.EDU (don.waisman.wisc.edu) by mailhub.iastate.edu\n\tid AA23002; Sat, 25 Sep 1993 16:14:09 -0500\nReceived: from Waisman.Wisc.EDU by Waisman.Wisc.EDU (PMDF V4.2-10 #2484) id\n <01H3DDLUXLDSBMA3H1@Waisman.Wisc.EDU>; Sat, 25 Sep 1993 16:13:40 CDT\nDate: Sat, 25 Sep 1993 16:13:40 -0500 (CDT)\nFrom: \"Ewan A. Macpherson\" \nSubject: DX7 Data Format\nTo: xeno@iastate.edu\nMessage-Id: <01H3DDLUY4O2BMA3H1@Waisman.Wisc.EDU>\nOrganization: Waisman Center, University of Wisconsin-Madison\nX-Vms-To: IN::\"xeno@iastate.edu\"\nMime-Version: 1.0\nContent-Type: TEXT/PLAIN; CHARSET=US-ASCII\nContent-Transfer-Encoding: 7BIT\n\nGary:\n\nI don't know anything about the differences between the DX7 and DX7s, but this\nDX7 info may be useful. I posted this to r.m.s. before xmas.\n\nI've seen many requests for public domain / shareware DX editors, but I've\nnever seen a definitive reply. They're usually along the lines of \"I was\nroaching around on CompuServe last month, and I think I remember seeing one...\"\n\nAnyway, hope this helps ... \n\n=========================================================================\n\nFor those interested in unpacking the uscd.edu DX7 patch data, here is\nDX7 data format information.\n\n compiled from - the DX7 MIDI Data Format Sheet\n - article by Steve DeFuria (Keyboard Jan 87)\n - looking at what my DX7 spits out\n\nI have kept the kinda weird notation used in the DX7 Data Sheet to reduce\ntyping errors. Where it doesn't quite make sense to me I've added comments.\n(And I will not be liable for errors etc ....)\n\nContents: A: SYSEX Message: Bulk Data for 1 Voice\n B: SYSEX Message: Bulk Data for 32 Voices\n C: SYSEX Message: Parameter Change\n D: Data Structure: Single Voice Dump & Voice Parameter #'s\n E: Function Parameter #'s\n F: Data Structure: Bulk Dump Packed Format\n\n////////////////////////////////////////////////////////////\nA:\nSYSEX Message: Bulk Data for 1 Voice\n------------------------------------\n bits hex description\n\n 11110000 F0 Status byte - start sysex\n 0iiiiiii 43 ID # (i=67; Yamaha)\n 0sssnnnn 00 Sub-status (s=0) & channel number (n=0; ch 1)\n 0fffffff 00 format number (f=0; 1 voice)\n 0bbbbbbb 01 byte count MS byte\n 0bbbbbbb 1B byte count LS byte (b=155; 1 voice)\n 0ddddddd ** data byte 1\n\n | | |\n\n 0ddddddd ** data byte 155\n 0eeeeeee ** checksum (masked 2's complement of sum of 155 bytes)\n 11110111 F7 Status - end sysex\n\n\n\n///////////////////////////////////////////////////////////\nB:\nSYSEX Message: Bulk Data for 32 Voices\n--------------------------------------\n bits hex description\n\n 11110000 F0 Status byte - start sysex\n 0iiiiiii 43 ID # (i=67; Yamaha)\n 0sssnnnn 00 Sub-status (s=0) & channel number (n=0; ch 1)\n 0fffffff 09 format number (f=9; 32 voices)\n 0bbbbbbb 20 byte count MS byte\n 0bbbbbbb 00 byte count LS byte (b=4096; 32 voices)\n 0ddddddd ** data byte 1\n\n | | |\n\n 0ddddddd ** data byte 4096 (there are 128 bytes / voice)\n 0eeeeeee ** checksum (masked 2's comp. of sum of 4096 bytes)\n 11110111 F7 Status - end sysex\n\n\n/////////////////////////////////////////////////////////////\nC:\nSYSEX MESSAGE: Parameter Change\n-------------------------------\n bits hex description\n\n 11110000 F0 Status byte - start sysex\n 0iiiiiii 43 ID # (i=67; Yamaha)\n 0sssnnnn 10 Sub-status (s=1) & channel number (n=0; ch 1)\n 0gggggpp ** parameter group # (g=0; voice, g=2; function)\n 0ppppppp ** parameter # (these are listed in next section)\n Note that voice parameter #'s can go over 128 so\n the pp bits in the group byte are either 00 for\n par# 0-127 or 01 for par# 128-155. In the latter case\n you add 128 to the 0ppppppp byte to compute par#. \n 0ddddddd ** data byte\n 11110111 F7 Status - end sysex\n\n\n//////////////////////////////////////////////////////////////\n\nD:\nData Structure: Single Voice Dump & Parameter #'s (single voice format, g=0)\n-------------------------------------------------------------------------\n\nParameter\n Number Parameter Value Range\n--------- --------- -----------\n 0 OP6 EG rate 1 0-99\n 1 \" \" rate 2 \"\n 2 \" \" rate 3 \"\n 3 \" \" rate 4 \"\n 4 \" \" level 1 \"\n 5 \" \" level 2 \"\n 6 \" \" level 3 \"\n 7 \" \" level 4 \"\n 8 OP6 KBD LEV SCL BRK PT \" C3= $27\n 9 \" \" \" \" LFT DEPTH \"\n 10 \" \" \" \" RHT DEPTH \"\n 11 \" \" \" \" LFT CURVE 0-3 0=-LIN, -EXP, +EXP, +LIN\n 12 \" \" \" \" RHT CURVE \" \" \" \" \" \n 13 OP6 KBD RATE SCALING 0-7\n 14 OP6 AMP MOD SENSITIVITY 0-3\n 15 OP6 KEY VEL SENSITIVITY 0-7\n 16 OP6 OPERATOR OUTPUT LEVEL 0-99\n 17 OP6 OSC MODE (fixed/ratio) 0-1 0=ratio\n 18 OP6 OSC FREQ COARSE 0-31\n 19 OP6 OSC FREQ FINE 0-99\n 20 OP6 OSC DETUNE 0-14 0: det=-7\n 21 \\\n | > repeat above for OSC 5, OSC 4, ... OSC 1\n125 /\n126 PITCH EG RATE 1 0-99\n127 \" \" RATE 2 \"\n128 \" \" RATE 3 \"\n129 \" \" RATE 4 \"\n130 \" \" LEVEL 1 \"\n131 \" \" LEVEL 2 \"\n132 \" \" LEVEL 3 \"\n133 \" \" LEVEL 4 \"\n134 ALGORITHM # 0-31\n135 FEEDBACK 0-7\n136 OSCILLATOR SYNC 0-1\n137 LFO SPEED 0-99\n138 \" DELAY \"\n139 \" PITCH MOD DEPTH \"\n140 \" AMP MOD DEPTH \"\n141 LFO SYNC 0-1\n142 \" WAVEFORM 0-5, (data sheet claims 9-4 ?!?)\n 0:TR, 1:SD, 2:SU, 3:SQ, 4:SI, 5:SH\n143 PITCH MOD SENSITIVITY 0-7\n144 TRANSPOSE 0-48 12 = C2\n145 VOICE NAME CHAR 1 ASCII\n146 VOICE NAME CHAR 2 ASCII\n147 VOICE NAME CHAR 3 ASCII\n148 VOICE NAME CHAR 4 ASCII\n149 VOICE NAME CHAR 5 ASCII\n150 VOICE NAME CHAR 6 ASCII\n151 VOICE NAME CHAR 7 ASCII\n152 VOICE NAME CHAR 8 ASCII\n153 VOICE NAME CHAR 9 ASCII\n154 VOICE NAME CHAR 10 ASCII\n155 OPERATOR ON/OFF\n bit6 = 0 / bit 5: OP1 / ... / bit 0: OP6\n\nNote that there are actually 156 parameters listed here, one more than in \na single voice dump. The OPERATOR ON/OFF parameter is not stored with the\nvoice, and is only transmitted or received while editing a voice. So it\nonly shows up in parameter change SYS-EX's.\n\n\n////////////////////////////////////////////////////////\n\nE:\nFunction Parameters: (g=2)\n-------------------------\n\nParameter\nNumber Parameter Range\n--------- ---------- ------\n64 MONO/POLY MODE CHANGE 0-1 O=POLY\n65 PITCH BEND RANGE 0-12\n66 \" \" STEP 0-12\n67 PORTAMENTO MODE 0-1 0=RETAIN 1=FOLLOW\n68 \" GLISS 0-1\n69 \" TIME 0-99\n70 MOD WHEEL RANGE 0-99\n71 \" \" ASSIGN 0-7 b0: pitch, b1:amp, b2: EG bias\n72 FOOT CONTROL RANGE 0-99\n73 \" \" ASSIGN 0-7 \"\n74 BREATH CONT RANGE 0-99\n75 \" \" ASSIGN 0-7 \"\n76 AFTERTOUCH RANGE 0-99\n77 \" ASSIGN 0-7 \"\n\n///////////////////////////////////////////////////////////////\n\nF:\nData Structure: Bulk Dump Packed Format\n---------------------------------------\n\nOK, now the tricky bit. For a bulk dump the 155 voice parameters for each\n voice are packed into 32 consecutive 128 byte chunks as follows ...\n\nbyte bit #\n # 6 5 4 3 2 1 0 param A range param B range\n---- --- --- --- --- --- --- --- ------------ ----- ------------ -----\n 0 R1 OP6 EG R1 0-99\n 1 R2 OP6 EG R2 0-99\n 2 R3 OP6 EG R3 0-99\n 3 R4 OP6 EG R4 0-99\n 4 L1 OP6 EG L1 0-99\n 5 L2 OP6 EG L2 0-99\n 6 L3 OP6 EG L3 0-99\n 7 L4 OP6 EG L4 0-99\n 8 BP LEV SCL BRK PT 0-99\n 9 LD SCL LEFT DEPTH 0-99\n 10 RD SCL RGHT DEPTH 0-99\n 11 0 0 0 | RC | LC | SCL LEFT CURVE 0-3 SCL RGHT CURVE 0-3\n 12 | DET | RS | OSC DETUNE 0-14 OSC RATE SCALE 0-7\n 13 0 0 | KVS | AMS | KEY VEL SENS 0-7 AMP MOD SENS 0-3\n 14 OL OP6 OUTPUT LEV 0-99\n 15 0 | FC | M | FREQ COARSE 0-31 OSC MODE 0-1\n 16 FF FREQ FINE 0-99\n 17 \\\n | > these 17 bytes for OSC 5\n 33 /\n 34 \\\n | > these 17 bytes for OSC 4\n 50 /\n 51 \\\n | > these 17 bytes for OSC 3\n 67 /\n 68 \\\n | > these 17 bytes for OSC 2\n 84 /\n 85 \\\n | > these 17 bytes for OSC 1\n101 /\n\nbyte bit #\n # 6 5 4 3 2 1 0 param A range param B range\n---- --- --- --- --- --- --- --- ------------ ----- ------------ -----\n102 PR1 PITCH EG R1 0-99\n103 PR2 PITCH EG R2 0-99\n104 PR3 PITCH EG R3 0-99\n105 PR4 PITCH EG R4 0-99\n106 PL1 PITCH EG L1 0-99\n107 PL2 PITCH EG L2 0-99\n108 PL3 PITCH EG L3 0-99\n109 PL4 PITCH EG L4 0-99\n110 0 0 | ALG | ALGORITHM 0-31\n111 0 0 0 |OKS| FB | OSC KEY SYNC 0-1 FEEDBACK 0-7\n112 LFS LFO SPEED 0-99\n113 LFD LFO DELAY 0-99\n114 LPMD LF PT MOD DEP 0-99\n115 LAMD LF AM MOD DEP 0-99\n116 | LPMS | LFW |LKS| LF PT MOD SNS 0-7 WAVE 0-5, SYNC 0-1\n117 TRNSP TRANSPOSE 0-48\n118 NAME CHAR 1 VOICE NAME 1 ASCII\n119 NAME CHAR 2 VOICE NAME 2 ASCII\n120 NAME CHAR 3 VOICE NAME 3 ASCII\n121 NAME CHAR 4 VOICE NAME 4 ASCII\n122 NAME CHAR 5 VOICE NAME 5 ASCII\n123 NAME CHAR 6 VOICE NAME 6 ASCII\n124 NAME CHAR 7 VOICE NAME 7 ASCII\n125 NAME CHAR 8 VOICE NAME 8 ASCII\n126 NAME CHAR 9 VOICE NAME 9 ASCII\n127 NAME CHAR 10 VOICE NAME 10 ASCII\n\n/////////////////////////////////////////////////////////////////////\n\nAnd that's it.\n\nHope this is useful.\n\newan.\n\n\n\n 0 R1 OP6 EG R1 0-99\n 1 R2 OP6 EG R2 0-99\n 2 R3 OP6 EG R3 0-99\n 3 R4 OP6 EG R4 0-99\n 4 L1 OP6 EG L1 0-99\n 5 L2 OP6 EG L2 0-99\n 6 L3 OP6 EG L3 0-99\n 7 L4 OP6 EG L4 0-99\n 8 BP LEV SCL BRK PT 0-99\n 9 LD SCL LEFT DEPTH 0-99\n 10 RD SCL RGHT DEPTH 0-99\n 11 0 0 0 | RC | LC | SCL LEFT CURVE 0-3 SCL RGHT CURVE 0-3\n 12 | DET | RS | OSC DETUNE 0-14 OSC RATE SCALE 0-7\n 13 0 0 | KVS | AMS | KEY VEL SENS 0-7 AMP MOD SENS 0-3\n 14 OL OP6 OUTPUT LEV 0-99\n 15 0 | FC | M | FREQ COARSE 0-31 OSC MODE 0-1\n 16 FF FREQ FINE 0-99\n \n102 PR1 PITCH EG R1 0-99\n103 PR2 PITCH EG R2 0-99\n104 PR3 PITCH EG R3 0-99\n105 PR4 PITCH EG R4 0-99\n106 PL1 PITCH EG L1 0-99\n107 PL2 PITCH EG L2 0-99\n108 PL3 PITCH EG L3 0-99\n109 PL4 PITCH EG L4 0-99\n110 0 0 | ALG | ALGORITHM 0-31\n111 0 0 0 |OKS| FB | OSC KEY SYNC 0-1 FEEDBACK 0-7\n112 LFS LFO SPEED 0-99\n113 LFD LFO DELAY 0-99\n114 LPMD LF PT MOD DEP 0-99\n115 LAMD LF AM MOD DEP 0-99\n116 | LPMS | LFW |LKS| LF PT MOD SNS 0-7 WAVE 0-5, SYNC 0-1\n117 TRNSP TRANSPOSE 0-48\n118 NAME CHAR 1 VOICE NAME 1 ASCII\n119 NAME CHAR 2 VOICE NAME 2 ASCII\n120 NAME CHAR 3 VOICE NAME 3 ASCII\n121 NAME CHAR 4 VOICE NAME 4 ASCII\n122 NAME CHAR 5 VOICE NAME 5 ASCII\n123 NAME CHAR 6 VOICE NAME 6 ASCII\n124 NAME CHAR 7 VOICE NAME 7 ASCII\n125 NAME CHAR 8 VOICE NAME 8 ASCII\n126 NAME CHAR 9 VOICE NAME 9 ASCII\n127 NAME CHAR 10 VOICE NAME 10 ASCII" }, { "path": "scratch/dx7_constants.py", "content": "from pathlib import Path\nimport bitstruct\n\nARTIFACTS_ROOT = Path('/content/gdrive/My Drive/audio/artifacts').expanduser()\n\ndef take(take_from, n):\n for _ in range(n):\n yield next(take_from)\n\nN_OSC = 6\nN_VOICES = 32\n\ndef checksum(data):\n return (128-sum(data)&127)%128\n\nGLOBAL_VALID_RANGES = {\n 'PR1': range(0, 99+1),\n 'PR2': range(0, 99+1),\n 'PR3': range(0, 99+1),\n 'PR4': range(0, 99+1),\n 'PL1': range(0, 99+1),\n 'PL2': range(0, 99+1),\n 'PL3': range(0, 99+1),\n 'PL4': range(0, 99+1),\n 'ALG': range(0, 31+1),\n 'OKS': range(0, 1+1),\n 'FB': range(0, 7+1),\n 'LFS': range(0, 99+1),\n 'LFD': range(0, 99+1),\n 'LPMD': range(0, 99+1),\n 'LAMD': range(0, 99+1),\n 'LPMS': range(0, 7+1),\n 'LFW': range(0, 5+1),\n 'LKS': range(0, 1+1),\n 'TRNSP': range(0, 48+1),\n 'NAME CHAR 1': range(128),\n 'NAME CHAR 2': range(128),\n 'NAME CHAR 3': range(128),\n 'NAME CHAR 4': range(128),\n 'NAME CHAR 5': range(128),\n 'NAME CHAR 6': range(128),\n 'NAME CHAR 7': range(128),\n 'NAME CHAR 8': range(128),\n 'NAME CHAR 9': range(128),\n 'NAME CHAR 10': range(128),\n }\n\nOSCILLATOR_VALID_RANGES = {\n 'R1': range(0, 99+1),\n 'R2': range(0, 99+1),\n 'R3': range(0, 99+1),\n 'R4': range(0, 99+1),\n 'L1': range(0, 99+1),\n 'L2': range(0, 99+1),\n 'L3': range(0, 99+1),\n 'L4': range(0, 99+1),\n 'BP': range(0, 99+1),\n 'LD': range(0, 99+1),\n 'RD': range(0, 99+1),\n 'RC': range(0, 3+1),\n 'LC': range(0, 3+1),\n 'DET': range(0, 14+1),\n 'RS': range(0, 7+1),\n 'KVS': range(0, 7+1),\n 'AMS': range(0, 3+1),\n 'OL': range(0, 99+1),\n 'FC': range(0, 31+1),\n 'M': range(0, 1+1),\n 'FF': range(0, 99+1),\n}\n\nVOICE_PARAMETER_RANGES = {f'{i}_{key}': value for key, value in OSCILLATOR_VALID_RANGES.items() for i in range(N_OSC)}\nVOICE_PARAMETER_RANGES.update(GLOBAL_VALID_RANGES)\n\ndef verify(actual, ranges):\n assert set(actual.keys())==set(ranges.keys()), 'Params dont match'\n for key in actual:\n if not actual[key] in ranges[key]:\n return False\n return True\n\n\nHEADER_KEYS = [\n 'ID',\n 'Sub-status',\n 'format number',\n 'byte count',\n 'byte count',\n]\n\nGENERAL_KEYS = [\n 'PR1',\n 'PR2',\n 'PR3',\n 'PR4',\n 'PL1',\n 'PL2',\n 'PL3',\n 'PL4',\n 'ALG',\n 'OKS',\n 'FB',\n 'LFS',\n 'LFD',\n 'LPMD',\n 'LAMD',\n 'LPMS',\n 'LFW',\n 'LKS',\n 'TRNSP',\n 'NAME CHAR 1',\n 'NAME CHAR 2',\n 'NAME CHAR 3',\n 'NAME CHAR 4',\n 'NAME CHAR 5',\n 'NAME CHAR 6',\n 'NAME CHAR 7',\n 'NAME CHAR 8',\n 'NAME CHAR 9',\n 'NAME CHAR 10',\n]\n\nOSC_KEYS = [\n 'R1',\n 'R2',\n 'R3',\n 'R4',\n 'L1',\n 'L2',\n 'L3',\n 'L4',\n 'BP',\n 'LD',\n 'RD',\n 'RC',\n 'LC',\n 'DET',\n 'RS',\n 'KVS',\n 'AMS',\n 'OL',\n 'FC',\n 'M',\n 'FF',\n]\n\nFOOTER_KEYS = ['checksum']\n\n\nVOICE_KEYS = [f'{i}_{key}' for i in range(6) for key in OSC_KEYS] + \\\n GENERAL_KEYS \n\nKEYS = HEADER_KEYS + \\\n list(VOICE_KEYS * N_VOICES) + \\\n FOOTER_KEYS\n\n\n\nheader_bytes = [\n 'p1u7', # ID # (i=67; Yamaha)\n 'p1u7', # Sub-status (s=0) & channel number (n=0; ch 1)\n 'p1u7', # format number (f=9; 32 voices)\n 'p1u7', # byte count MS byte\n 'p1u7', # byte count LS byte (b=4096; 32 voices)\n]\n\n\n\n\ngeneral_parameter_bytes = [ \n 'p1u7', # PR1\n 'p1u7', # PR2\n 'p1u7', # PR3\n 'p1u7', # PR4\n 'p1u7', # PL1\n 'p1u7', # PL2\n 'p1u7', # PL3\n 'p1u7', # PL4\n 'p3u5', # ALG\n 'p4u1u3', # OKS| FB\n 'p1u7', # LFS\n 'p1u7', # LFD\n 'p1u7', # LPMD\n 'p1u7', # LAMD\n 'p1u3u3u1', # LPMS | LFW |LKS\n 'p1u7', # TRNSP\n 'p1u7', # NAME CHAR 1\n 'p1u7', # NAME CHAR 2\n 'p1u7', # NAME CHAR 3\n 'p1u7', # NAME CHAR 4\n 'p1u7', # NAME CHAR 5\n 'p1u7', # NAME CHAR 6\n 'p1u7', # NAME CHAR 7\n 'p1u7', # NAME CHAR 8\n 'p1u7', # NAME CHAR 9\n 'p1u7', # NAME CHAR 10\n]\n\nosc_parameter_bytes = [\n 'p1u7', # R1\n 'p1u7', # R2\n 'p1u7', # R3\n 'p1u7', # R4\n 'p1u7', # L1\n 'p1u7', # L2\n 'p1u7', # L3\n 'p1u7', # L4\n 'p1u7', # BP\n 'p1u7', # LD\n 'p1u7', # RD\n 'p4u2u2', # RC | LC \n 'p1u4u3', # DET | RS\n 'p3u3u2', # KVS | AMS\n 'p1u7', # OL\n 'p2u5u1', # FC | M\n 'p1u7' # FF\n]\n\nvoice_bytes = (osc_parameter_bytes * N_OSC) + general_parameter_bytes\n\ntail_bytes = [\n 'p1u7', # checksum\n]\n\n\nfull_string = ''.join(header_bytes + osc_parameter_bytes * 6 + general_parameter_bytes)\ndx7_struct = bitstruct.compile(full_string)\n\nvoice_struct = bitstruct.compile(''.join(voice_bytes), names=VOICE_KEYS)\nheader_struct = bitstruct.compile(''.join(header_bytes))\n\nif __name__==\"__main__\":\n print(VOICE_KEYS)" }, { "path": "scratch/dx7_syx.py", "content": "#%%\nimport bitstruct\nimport mido\nfrom pathlib import Path\nfrom itertools import chain\n\nfrom dx7_constants import voice_struct, verify, VOICE_PARAMETER_RANGES, header_struct,\\\n header_bytes, voice_bytes, take, VOICE_KEYS, ARTIFACTS_ROOT, N_VOICES, N_OSC\n\n# %%\n\n\ndef consume_syx(path):\n\n path = Path(path).expanduser()\n try:\n preset = mido.read_syx_file(path.as_posix())[0]\n except IndexError as e:\n return None\n except ValueError as e:\n return None\n if len(preset.data) == 0:\n return None\n\n def get_voice(data):\n \n unpacked = voice_struct.unpack(data)\n\n if not verify(unpacked, VOICE_PARAMETER_RANGES):\n return None\n \n return unpacked\n\n get_header = header_struct.unpack\n sysex_iter = iter(preset.data)\n \n try:\n header = get_header(bytes(take(sysex_iter, len(header_bytes))))\n yield from (get_voice(bytes(take(sysex_iter, len(voice_bytes)))) for _ in range(N_VOICES))\n except RuntimeError:\n return None\n#%%\nfrom tqdm import tqdm as tqdm\nfrom functools import reduce\nimport numpy as np\nif __name__=='__main__':\n DEV = False\n dataset_file = 'dx7.npy'\n# PRESETS_ROOT = '~/audio/artifacts/dx7-patches'\n# PRESETS_ROOT = '~/audio/artifacts/dx7-patches-dev'\n preset_root = ARTIFACTS_ROOT.joinpath('dx7-patches').expanduser()\n\n preset_paths = iter(tqdm(sorted(preset_root.glob('**/*.syx'))))\n if DEV:\n dataset_file = f'dev-{dataset_file}'\n preset_paths = take(preset_paths, 10)\n preset_paths = iter(filter(lambda preset_path: preset_path.is_file(), preset_paths))\n\n consume_chain = chain.from_iterable(map(consume_syx, preset_paths))\n consume_chain = filter(lambda x: x is not None, consume_chain)\n\n arr_dtype = list(zip(VOICE_KEYS, ['u8']*len(VOICE_KEYS)))\n to_arr = lambda voice_dict: np.array([tuple(voice_dict.values())], dtype=arr_dtype)\n arr = np.concatenate(list(map(to_arr, consume_chain)))\n arr = np.unique(arr)\n np.save(ARTIFACTS_ROOT.joinpath(dataset_file).as_posix(), arr)\n # print(to_arr(next(iter(consume_chain))))\n # print(type(consume_chain))\n # 1/0\n # # for i in list(map(list, outputs)):\n # # print(len(i))\n # paths, ns, data = zip(*outputs)\n # data = np.array(list(set(data)))\n # paths = np.array([path.as_posix() for path in paths])\n # ns = np.array(ns)\n # np.savez(ARTIFACTS_ROOT.joinpath('dx7.npy'), outputs, paths, ns)\n # # patch_map = dict(tqdm((iter(consume_chain))))\n # # random_key, *_ = patch_map.keys()\n # # # print(global_config['NAME'])\n\n # # name = items.pop('')\n # # print(global_config)\n # # print(json.dumps(oscillator_configs, indent=4))\n # # print(list(sysex_iter))\n\n\n\n\n\n# %%\n\n\n# %%\n" }, { "path": "scratch/fm-param-analysis.py", "content": "#%%\nimport torch\nfrom torch.utils.data import Subset as DataSubset\nfrom sklearn.model_selection import train_test_split\nfrom fm_param_vae import Net, DX7Dataset, ARTIFACTS_ROOT, VOICE_KEYS\nfrom dx7_constants import VOICE_PARAMETER_RANGES\n\nimport numpy as np\n\ndataset = DX7Dataset()\ntrain_idxs, test_idxs = train_test_split(range(len(dataset)), random_state=42)\ntrain_dataset = DataSubset(dataset, train_idxs)\ntest_dataset = DataSubset(dataset, test_idxs)\n\nmodel = Net()\nmodel.load_state_dict(torch.load(ARTIFACTS_ROOT.joinpath('fm-param-vae-8.pt')))\n\n\n# %%\nwith torch.no_grad():\n data = torch.stack([test_dataset[i] for i in range(len(test_dataset))])\n results, *_ = model(data)\n# %%\n\ncorrect = (results.argmax(-1) == data)\n# %%\n\np_z = torch.distributions.Normal(0, 1)\nz = p_z.sample((32, 8))\n\nx_hat = model.generate(z, 0.3)\n\n\n# %%\n" }, { "path": "scratch/fm_param_ae.py", "content": "#%%\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import StepLR\nfrom pathlib import Path\nfrom torch.utils.data import Subset as DataSubset\nfrom sklearn.model_selection import train_test_split\n\nfrom dx7_constants import VOICE_PARAMETER_RANGES, ARTIFACTS_ROOT, VOICE_KEYS\nimport numpy as np\nN_PARAMS = len(VOICE_PARAMETER_RANGES)\nMAX_VALUE = max([max(i) for i in VOICE_PARAMETER_RANGES.values()]) + 1\n#%%\n\n# class DataHandler()\n# def __init__(self, data_file, root=ARTIFACTS_ROOT):\n\n# if not isinstance(root, Path):\n# root = Path(root).expanduser()\n\n# data = np.load(ARTIFACTS_ROOT.joinpath(patch_file))\n\n\n\n\n\n\nclass DX7Dataset():\n \n\n def __init__(self, data_file='dx7.npy', root=ARTIFACTS_ROOT):\n\n if not isinstance(root, Path):\n root = Path(root).expanduser()\n\n self.data = np.load(root.joinpath(data_file)) \n \n\n def __getitem__(self, index):\n\n item = torch.tensor(self.data[index].item()).long()\n\n return item\n def __len__(self):\n return len(self.data)\n\n\n#%%\nclass Net(nn.Module):\n def __init__(self, latent_dim=16, n_params=N_PARAMS, max_value=MAX_VALUE):\n super(Net, self).__init__()\n\n self.n_params = n_params\n self.max_value = max_value\n\n self.embedder = nn.Embedding(max_value, 8)\n\n self.enc = nn.Sequential(\n nn.Linear(8*n_params, 512),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.Linear(512, latent_dim),\n )\n\n self.dec = nn.Sequential(\n nn.Linear(latent_dim, 512),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.Linear(512, max_value*n_params),\n )\n\n self.register_buffer('mask', self.generate_mask())\n\n @staticmethod\n def generate_mask():\n \n mask_item_f = lambda x: torch.arange(MAX_VALUE) <= max(x) \n mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, VOICE_KEYS))\n\n return torch.stack(list(mapper))\n\n def forward(self, x):\n \n x = self.embedder(x)\n x = x.flatten(-2, -1)\n z = self.enc(x)\n\n x_hat = self.dec(z)\n x_hat = x_hat.reshape(-1, self.n_params, self.max_value)\n\n x_hat = torch.masked_fill(x_hat, ~self.mask, -1e9)\n return x_hat\n\n#%%\ndef train(model, device, train_loader, optimizer, epoch):\n model.train()\n for batch_idx, (data) in enumerate(train_loader):\n data = data.to(device)\n optimizer.zero_grad()\n output = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n loss.backward()\n optimizer.step()\n if batch_idx % 100 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n epoch, batch_idx * len(data), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item()))\n\n\ndef test(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n with torch.no_grad():\n for data in test_loader:\n data = data.to(device)\n output = model(data)\n test_loss += F.cross_entropy(output.transpose(-1,-2), data, reduction='sum').item() # sum up batch loss\n pred = output.argmax(dim=-1, keepdim=True) # get the index of the max log-probability\n correct += pred.eq(data.view_as(pred)).sum().item() / 155\n\n test_loss /= len(test_loader.dataset)\n\n print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n test_loss, correct, len(test_loader.dataset),\n 100. * correct / len(test_loader.dataset)))\n\nif __name__==\"__main__\":\n # Training settings\n use_cuda = False\n batch_size = 32\n lr = 1\n gamma = 0.7\n epochs = 100\n\n device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n\n kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}\n\n \n dataset = DX7Dataset()\n train_idxs, test_idxs = train_test_split(range(len(dataset)), random_state=42)\n train_dataset = DataSubset(dataset, train_idxs)\n test_dataset = DataSubset(dataset, test_idxs)\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n test_loader = torch.utils.data.DataLoader(\n test_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n\n model = Net().to(device)\n optimizer = optim.Adadelta(model.parameters(), lr=lr)\n\n scheduler = StepLR(optimizer, step_size=1, gamma=gamma)\n for epoch in range(1, epochs + 1):\n train(model, device, train_loader, optimizer, epoch)\n test(model, device, test_loader)\n scheduler.step()\n\n # if args.save_model:\n # torch.save(model.state_dict(), \"mnist_cnn.pt\")\n\n\n# if __name__ == '__main__':\n# main()\n\n# %%\n\n\n# %%\n\n\n# %%\n\n\n# %%\n" }, { "path": "scratch/fm_param_agoge_vae_rnn.py", "content": "#%%\nimport os\nos.environ['MLFLOW_TRACKING_URI'] = 'http://localhost:9001'\nimport torch\ntorch.randn(10,10).cuda()\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.optim import AdamW\nfrom torch.optim.lr_scheduler import StepLR\nfrom pathlib import Path\nfrom torch.utils.data import Subset as DataSubset\nfrom ray import tune\n# from sklearn.model_selection import train_test_split\nimport numpy as np\nfrom dx7_constants import VOICE_PARAMETER_RANGES, ARTIFACTS_ROOT, VOICE_KEYS\n\nfrom ray.tune.schedulers import HyperBandForBOHB\nfrom ray.tune.suggest.bohb import TuneBOHB\nimport ConfigSpace as CS\n\nfrom agoge import AbstractModel, AbstractSolver, Worker\nfrom agoge.utils import uuid, trial_name_creator, experiment_name_creator, get_logger\nfrom itertools import starmap\n\nimport numpy as np\nN_PARAMS = len(VOICE_PARAMETER_RANGES)\nMAX_VALUE = max([max(i) for i in VOICE_PARAMETER_RANGES.values()]) + 1\n#%%\n\n# class DataHandler()\n# def __init__(self, data_file, root=ARTIFACTS_ROOT):\n\n# if not isinstance(root, Path):\n# root = Path(root).expanduser()\n\n# data = np.load(ARTIFACTS_ROOT.joinpath(patch_file))\n\n\n\nclass DX7Dataset():\n \n\n def __init__(self, data_file='dx7.npy', root=ARTIFACTS_ROOT, data_size=1.):\n\n assert data_size < 1\n\n self.data_size = data_size\n\n if not isinstance(root, Path):\n root = Path(root).expanduser()\n\n self.data = np.load(root.joinpath(data_file)) \n\n def __getitem__(self, index):\n\n item = torch.tensor(self.data[index].item()).long()\n\n return {'x': item}\n \n def __len__(self):\n return int(len(self.data) * self.data_size)\n\n\n#%%\n\n#%%\nclass DX7RecurrentVAE(AbstractModel):\n def __init__(self, latent_dim=8, n_params=N_PARAMS, max_value=MAX_VALUE, hidden_dim=128, params_ordering=None):\n super().__init__()\n\n self.n_params = n_params\n self.max_value = max_value\n\n self.embedder = nn.Embedding(max_value, hidden_dim)\n\n self.enc = nn.ModuleList(\n [nn.LSTM(hidden_dim, hidden_dim, batch_first=True, bidirectional=True),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.LSTM(hidden_dim, hidden_dim, batch_first=True, bidirectional=True),]\n )\n\n self.q_z = nn.Linear(hidden_dim, 2*latent_dim)\n self.z2x = nn.Linear(hidden_dim+latent_dim, hidden_dim)\n self.logits = nn.Linear(hidden_dim, max_value)\n\n self.dec = nn.ModuleList(\n [nn.LSTM(hidden_dim, hidden_dim, batch_first=True),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.LSTM(hidden_dim, hidden_dim, batch_first=True),\n ]\n )\n\n self.ordering = params_ordering\n if params_ordering is not None:\n assert len(params_ordering) == len(VOICE_KEYS), 'more or less params than expected'\n self.ordering = np.argsort(params_ordering)\n self.reverse_ordering = np.argsort(self.ordering)\n\n\n self.register_buffer('mask', self.generate_mask(self.ordering))\n\n\n def network(self, x, network):\n\n lstm, gelu, drop, lstm2 = network\n\n x_1, (h_1, _) = lstm(x)\n if lstm.bidirectional == True:\n x_1 = h_1.mean(0)\n x_1 = drop(gelu(x_1))\n\n x_2, (h_2, _) = lstm2(x)\n\n if lstm2.bidirectional == True:\n x_2 = h_2.mean(0)\n x = torch.ones_like(x_2)\n\n x_2 = drop(x_2)\n\n x = x_1 * x + x_2\n\n return x\n\n @staticmethod\n def generate_mask(ordering=None):\n \"\"\"\n ordering the index ordering of the parameters based on the index in the dx7_constants.VOICE_KEYS\n \"\"\"\n \n mask_item_f = lambda x: torch.arange(MAX_VALUE) <= max(x) \n mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, VOICE_KEYS))\n\n mask = torch.stack(list(mapper))\n\n if ordering is not None:\n return mask[ordering]\n return mask\n\n def forward(self, x):\n if self.ordering is not None:\n x = x[:, self.ordering]\n\n x = self.embedder(x)\n theta_z = self.network(x, self.enc)\n\n q_z_mu, q_z_std = self.q_z(theta_z).chunk(2, -1)\n\n q_z = torch.distributions.Normal(q_z_mu, (0.5*q_z_std.clamp(-5, 3)).exp())\n\n z = q_z.sample()\n z_in = z.unsqueeze(-2) + torch.zeros_like(x[...,0]).unsqueeze(-1)\n\n # x_endcut = x\n x_prepad = torch.cat([torch.zeros_like(x[:,[0]]), x], dim=-2)\n x_endcut = x_prepad[:,:-1]\n \n x_dec_in = torch.cat([x_endcut, z_in], dim=-1)\n x_dec_in = self.z2x(x_dec_in)\n x_hat = self.network(x_dec_in, self.dec)\n\n x_hat = self.logits(x_hat)\n\n x_hat = torch.masked_fill(x_hat, ~self.mask, -1e9)\n\n if self.ordering is not None:\n x_hat = x_hat[:, self.reverse_ordering]\n\n return x_hat, q_z, z\n\n def generate(self, z, t=1.):\n\n x_hat = self.dec(z)\n x_hat = x_hat.reshape(-1, self.n_params, self.max_value)\n x_hat = torch.masked_fill(x_hat, ~self.mask, -float('inf'))\n\n x_hat = torch.distributions.Categorical(logits=x_hat / t)\n\n return x_hat\n#%%\n\nclass DX7RecurrentVAESolver(AbstractSolver):\n\n def __init__(self, model,\n Optim=AdamW, optim_opts=dict(lr= 1e-4),\n max_beta=0.5,\n **kwargs):\n\n if isinstance(Optim, str):\n Optim = import_module(Optim)\n\n\n self.optim = Optim(params=model.parameters(), **optim_opts)\n self.schedule = self.scheduler()\n self.max_beta = max_beta\n self.model = model\n self._beta = self.schedule()\n\n @property\n def beta(self):\n\n return self._beta * self.max_beta\n\n @staticmethod\n def scheduler():\n n_steps = 0\n beta_steps = 37400\n\n def schedule():\n nonlocal n_steps\n n_steps += 1\n\n step = (n_steps)/beta_steps\n step = min(1, step)\n\n return 0.5 * (1 + np.sin((step*np.pi)-(np.pi/2)))\n return schedule\n \n def loss(self, x, x_hat, q_z, z):\n\n reconstruction_loss = F.cross_entropy(x_hat.transpose(-1,-2), x)\n\n p_z = torch.distributions.Normal(0, 1)\n\n log_q_z = q_z.log_prob(z)\n log_p_z = p_z.log_prob(z)\n\n kl = (log_q_z - log_p_z).mean()\n\n kl_tempered = kl * self.beta\n \n loss = reconstruction_loss + kl_tempered\n\n\n accuracy = (x_hat.argmax(-1)==x).float().mean()\n\n return loss, {\n 'log_q_z': log_q_z.mean(),\n 'log_p_z': log_p_z.mean(),\n 'kl': kl,\n 'reconstruction_loss': reconstruction_loss,\n 'beta': self.beta,\n 'accuracy': accuracy\n }\n \n\n def solve(self, x, **kwargs):\n \n x_hat, q_z, z = self.model(x)\n loss, L = self.loss(x, x_hat, q_z, z)\n\n if loss != loss:\n raise ValueError('Nan Values detected')\n\n if self.model.training:\n\n self.optim.zero_grad()\n loss.backward()\n self.optim.step()\n\n self._beta = self.schedule()\n \n return L\n\n \n def step(self):\n\n pass\n\n\n def state_dict(self):\n \n state_dict = {\n 'optim': self.optim.state_dict()\n }\n\n return state_dict\n\n def load_state_dict(self, state_dict):\n \n load_component = lambda component, state: getattr(self, component).load_state_dict(state)\n list(starmap(load_component, state_dict.items()))\n\n\n\ndef config(experiment_name, trial_name, batch_size=16, **kwargs):\n \n voice_params = {key.split('..')[-1]: value for key, value in kwargs.items() if 'VOICE..' in key}\n params_ordering = list(map(voice_params.get, VOICE_KEYS))\n\n\n data_handler = {\n 'Dataset': DX7Dataset,\n 'dataset_opts': {\n 'data_size': 0.2\n },\n 'loader_opts': {\n 'batch_size': batch_size,\n },\n }\n\n model = {\n 'Model': DX7RecurrentVAE,\n 'params_ordering': params_ordering\n # 'conv1': (1, 32, 3, 1)\n }\n\n solver = {\n 'Solver': DX7RecurrentVAESolver\n }\n\n tracker = {\n 'metrics': ['reconstruction_loss', 'log_q_z', 'log_p_z', 'kl', 'beta', 'accuracy'],\n 'experiment_name': experiment_name,\n 'trial_name': trial_name\n }\n\n return {\n 'data_handler': data_handler,\n 'model': model,\n 'solver': solver,\n 'tracker': tracker,\n }\n\nfrom mlflow.tracking import MlflowClient\nif __name__=='__main__':\n # from ray import ray\n import sys\n postfix = sys.argv[1]\n # ray.init()\n # from ray.tune.utils import validate_save_restore\n # validate_save_restore(Worker)\n # client = MlflowClient(tracking_uri='localhost:5000')\n experiment_name = f'dx7-vae-{postfix}'#+experiment_name_creator()\n # experiment_id = client.create_experiment(experiment_name)\n\n\n experiment_metrics = dict(metric=\"loss/accuracy\", mode=\"max\")\n\n config_space = CS.ConfigurationSpace()\n [config_space.add_hyperparameter(\n CS.UniformFloatHyperparameter(f'VOICE..{key}', lower=0., upper=1)\n ) for key in VOICE_KEYS]\n bohb_hyperband = HyperBandForBOHB(\n time_attr=\"training_iteration\", max_t=16, **experiment_metrics)\n bohb_search = TuneBOHB(\n config_space, max_concurrent=1, **experiment_metrics)\n\n\n tune.run(Worker, \n config={\n 'config_generator': config,\n 'experiment_name': experiment_name,\n 'points_per_epoch': 2\n },\n trial_name_creator=trial_name_creator,\n resources_per_trial={\n 'gpu': 1\n },\n checkpoint_freq=2,\n checkpoint_at_end=True,\n keep_checkpoints_num=1,\n search_alg=bohb_search, \n scheduler=bohb_hyperband,\n num_samples=4096,\n verbose=0,\n local_dir='~/ray_results'\n # webui_host='127.0.0.1' ## supresses an error\n # stop={'loss/loss': 0}\n )\n# points_per_epoch\n# %%\n\n# #################################################################################\n \n\n# if __name__==\"__main__\":\n# # Training settings\n# use_cuda = True\n# batch_size = 32\n# lr = 1e-4\n# gamma = 1.\n# epochs = 100\n# beta = 0.5\n# beta_steps = 37400\n \n# schedule = scheduler()\n# device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n\n# kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}\n\n \n# dataset = DX7Dataset()\n# train_idxs, test_idxs = train_test_split(range(len(dataset)), random_state=42)\n# train_dataset = DataSubset(dataset, train_idxs)\n# test_dataset = DataSubset(dataset, test_idxs)\n\n# train_loader = torch.utils.data.DataLoader(\n# train_dataset,\n# batch_size=batch_size, shuffle=True, **kwargs)\n# test_loader = torch.utils.data.DataLoader(\n# test_dataset,\n# batch_size=batch_size, shuffle=True, **kwargs)\n\n# model = Net().to(device)\n# optimizer = optim.AdamW(model.parameters(), lr=lr)\n\n# scheduler = StepLR(optimizer, step_size=1, gamma=gamma)\n# for epoch in range(1, epochs + 1):\n# train(model, device, train_loader, optimizer, epoch)\n# test(model, device, test_loader)\n# # scheduler.step()\n\n# # if args.save_model:\n# # torch.save(model.state_dict(), \"mnist_cnn.pt\")\n\n# torch.save(model.state_dict(), ARTIFACTS_ROOT.joinpath('fm-param-vae-8.pt'))\n# # if __name__ == '__main__':\n# # main()\n\n# # %%\n\n\n# # %%\n\n\n# # %%\n\n\n# # %%\n" }, { "path": "scratch/fm_param_rnn_decoder.py", "content": "#%%\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import StepLR\nfrom pathlib import Path\nfrom torch.utils.data import Subset as DataSubset\nfrom sklearn.model_selection import train_test_split\nimport numpy as np\nfrom dx7_constants import VOICE_PARAMETER_RANGES, ARTIFACTS_ROOT, VOICE_KEYS\nimport numpy as np\nN_PARAMS = len(VOICE_PARAMETER_RANGES)\nMAX_VALUE = max([max(i) for i in VOICE_PARAMETER_RANGES.values()]) + 1\n#%%\n\n# class DataHandler()\n# def __init__(self, data_file, root=ARTIFACTS_ROOT):\n\n# if not isinstance(root, Path):\n# root = Path(root).expanduser()\n\n# data = np.load(ARTIFACTS_ROOT.joinpath(patch_file))\n\n\n\nclass DX7Dataset():\n \n\n def __init__(self, data_file='dx7.npy', root=ARTIFACTS_ROOT):\n\n if not isinstance(root, Path):\n root = Path(root).expanduser()\n\n self.data = np.load(root.joinpath(data_file)) \n \n\n def __getitem__(self, index):\n\n item = torch.tensor(self.data[index].item()).long()\n\n return item\n def __len__(self):\n return len(self.data)\n\n\n#%%\n\n#%%\nclass Net(nn.Module):\n def __init__(self, latent_dim=8, n_params=N_PARAMS, max_value=MAX_VALUE):\n super(Net, self).__init__()\n\n self.n_params = n_params\n self.max_value = max_value\n\n self.embedder = nn.Embedding(max_value, 128)\n\n self.dec = nn.ModuleList(\n [nn.LSTM(128, 128, batch_first=True),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.LSTM(128, 128, batch_first=True),]\n )\n\n self.register_buffer('mask', self.generate_mask())\n\n def network(self, x, network):\n\n lstm, gelu, drop, lstm2 = network\n\n x_1, _ = lstm(x)\n if lstm.bidirectional == True:\n x_1 = torch.stack(x_1.chunk(2, -1)).sum(0)\n\n x_in = gelu(drop(x_1)) + x\n x_2, _ = lstm2(x)\n\n if lstm.bidirectional == True:\n x_2 = torch.stack(x_2.chunk(2, -1)).sum(0)\n\n x = gelu(drop(x_2)) * x_in\n\n return x\n\n @staticmethod\n def generate_mask():\n \n mask_item_f = lambda x: torch.arange(MAX_VALUE) <= max(x) \n mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, VOICE_KEYS))\n\n return torch.stack(list(mapper))\n\n def forward(self, x):\n \n x = self.embedder(x)\n x_z_sub_1 = torch.cat([x[:,[0]]*0, x[:,:-1]], dim=-2)\n\n\n x_hat = self.network(x_z_sub_1, self.dec)\n\n x_hat = torch.masked_fill(x_hat, ~self.mask, -1e9)\n return x_hat\n\n def generate(self, z, t=1.):\n\n x_hat = self.dec(z)\n x_hat = x_hat.reshape(-1, self.n_params, self.max_value)\n x_hat = torch.masked_fill(x_hat, ~self.mask, -float('inf'))\n\n x_hat = torch.distributions.Categorical(logits=x_hat / t)\n\n return x_hat\n#%%\ndef train(model, device, train_loader, optimizer, epoch):\n model.train()\n for batch_idx, (data) in enumerate(train_loader):\n data = data.to(device)\n optimizer.zero_grad()\n output = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n\n loss.backward()\n optimizer.step()\n if batch_idx % 100 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}\\tKL: {:.3f}'.format(\n epoch, batch_idx * len(data), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item(), 0))\n\n\ndef test(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n with torch.no_grad():\n for data in test_loader:\n data = data.to(device)\n output = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n \n test_loss += loss\n pred = output.argmax(dim=-1, keepdim=True) # get the index of the max log-probability\n correct += pred.eq(data.view_as(pred)).sum().item() / 155\n \n test_loss /= len(test_loader)\n\n print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n test_loss, correct, len(test_loader.dataset),\n 100. * correct / len(test_loader.dataset)))\n\nif __name__==\"__main__\":\n # Training settings\n use_cuda = True\n batch_size = 32\n lr = 1e-4\n gamma = 1.\n epochs = 100\n beta = 0.5\n beta_steps = 37400\n def scheduler():\n n_steps = 0\n def schedule():\n nonlocal n_steps\n n_steps += 1\n\n if n_steps < 1000:\n\n return 0\n\n step = (n_steps-1000)/beta_steps\n step = min(1, step)\n\n return 0.5 * (1 + np.sin((step*np.pi)-(np.pi/2)))\n return schedule\n schedule = scheduler()\n device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n\n kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}\n\n \n dataset = DX7Dataset()\n train_idxs, test_idxs = train_test_split(range(len(dataset)), random_state=42)\n train_dataset = DataSubset(dataset, train_idxs)\n test_dataset = DataSubset(dataset, test_idxs)\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n test_loader = torch.utils.data.DataLoader(\n test_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n\n model = Net().to(device)\n optimizer = optim.Adadelta(model.parameters(), lr=lr)\n\n scheduler = StepLR(optimizer, step_size=1, gamma=gamma)\n for epoch in range(1, epochs + 1):\n train(model, device, train_loader, optimizer, epoch)\n test(model, device, test_loader)\n # scheduler.step()\n\n # if args.save_model:\n # torch.save(model.state_dict(), \"mnist_cnn.pt\")\n\n torch.save(model.state_dict(), ARTIFACTS_ROOT.joinpath('fm-param-vae-8.pt'))\n# if __name__ == '__main__':\n# main()\n\n# %%\n\n\n# %%\n\n\n# %%\n\n\n# %%\n" }, { "path": "scratch/fm_param_vae.py", "content": "#%%\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import StepLR\nfrom pathlib import Path\nfrom torch.utils.data import Subset as DataSubset\nfrom sklearn.model_selection import train_test_split\n\nfrom dx7_constants import VOICE_PARAMETER_RANGES, ARTIFACTS_ROOT, VOICE_KEYS\nimport numpy as np\nN_PARAMS = len(VOICE_PARAMETER_RANGES)\nMAX_VALUE = max([max(i) for i in VOICE_PARAMETER_RANGES.values()]) + 1\n#%%\n\n# class DataHandler()\n# def __init__(self, data_file, root=ARTIFACTS_ROOT):\n\n# if not isinstance(root, Path):\n# root = Path(root).expanduser()\n\n# data = np.load(ARTIFACTS_ROOT.joinpath(patch_file))\n\n\n\nclass DX7Dataset():\n \n\n def __init__(self, data_file='dx7.npy', root=ARTIFACTS_ROOT):\n\n if not isinstance(root, Path):\n root = Path(root).expanduser()\n\n self.data = np.load(root.joinpath(data_file)) \n \n\n def __getitem__(self, index):\n\n item = torch.tensor(self.data[index].item()).long()\n\n return item\n def __len__(self):\n return len(self.data)\n\n\n#%%\n\n#%%\nclass Net(nn.Module):\n def __init__(self, latent_dim=8, n_params=N_PARAMS, max_value=MAX_VALUE):\n super(Net, self).__init__()\n\n self.n_params = n_params\n self.max_value = max_value\n\n self.embedder = nn.Embedding(max_value, 8)\n\n self.enc = nn.Sequential(\n nn.Linear(8*n_params, 512),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.Linear(512, latent_dim*2),\n )\n\n self.dec = nn.Sequential(\n nn.Linear(latent_dim, 512),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.Linear(512, max_value*n_params),\n )\n\n self.register_buffer('mask', self.generate_mask())\n\n @staticmethod\n def generate_mask():\n \n mask_item_f = lambda x: torch.arange(MAX_VALUE) <= max(x) \n mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, VOICE_KEYS))\n\n return torch.stack(list(mapper))\n\n def forward(self, x):\n \n x = self.embedder(x)\n x = x.flatten(-2, -1)\n q_z_mu, q_z_std = self.enc(x).chunk(2, -1)\n\n q_z = torch.distributions.Normal(q_z_mu, q_z_std.clamp(-3, 2).exp())\n\n z = q_z.sample()\n\n x_hat = self.dec(z)\n x_hat = x_hat.reshape(-1, self.n_params, self.max_value)\n\n x_hat = torch.masked_fill(x_hat, ~self.mask, -1e9)\n return x_hat, q_z, z\n\n def generate(self, z, t=1.):\n\n x_hat = self.dec(z)\n x_hat = x_hat.reshape(-1, self.n_params, self.max_value)\n x_hat = torch.masked_fill(x_hat, ~self.mask, -float('inf'))\n\n x_hat = torch.distributions.Categorical(logits=x_hat / t)\n\n return x_hat\n#%%\ndef train(model, device, train_loader, optimizer, epoch):\n model.train()\n for batch_idx, (data) in enumerate(train_loader):\n data = data.to(device)\n optimizer.zero_grad()\n output, q_z, z = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n\n p_z = torch.distributions.Normal(0, 1)\n\n loss = loss + (q_z.log_prob(z) - p_z.log_prob(z)).mean() * beta\n\n loss.backward()\n optimizer.step()\n if batch_idx % 100 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n epoch, batch_idx * len(data), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item()))\n\n\ndef test(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n with torch.no_grad():\n for data in test_loader:\n data = data.to(device)\n output, q_z, z = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n p_z = torch.distributions.Normal(0, 1)\n loss = loss + (q_z.log_prob(z) - p_z.log_prob(z)).mean() * beta\n \n test_loss += loss\n pred = output.argmax(dim=-1, keepdim=True) # get the index of the max log-probability\n correct += pred.eq(data.view_as(pred)).sum().item() / 155\n print(test_loss)\n test_loss /= len(test_loader)\n\n print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n test_loss, correct, len(test_loader.dataset),\n 100. * correct / len(test_loader.dataset)))\n\nif __name__==\"__main__\":\n # Training settings\n use_cuda = True\n batch_size = 32\n lr = 0.01\n gamma = 0.7\n epochs = 100\n beta = 0.5\n\n device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n\n kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}\n\n \n dataset = DX7Dataset()\n train_idxs, test_idxs = train_test_split(range(len(dataset)), random_state=42)\n train_dataset = DataSubset(dataset, train_idxs)\n test_dataset = DataSubset(dataset, test_idxs)\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n test_loader = torch.utils.data.DataLoader(\n test_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n\n model = Net().to(device)\n optimizer = optim.Adadelta(model.parameters(), lr=lr)\n\n scheduler = StepLR(optimizer, step_size=1, gamma=gamma)\n for epoch in range(1, epochs + 1):\n # train(model, device, train_loader, optimizer, epoch)\n test(model, device, test_loader)\n # scheduler.step()\n\n # if args.save_model:\n # torch.save(model.state_dict(), \"mnist_cnn.pt\")\n\n torch.save(model.state_dict(), ARTIFACTS_ROOT.joinpath('fm-param-vae-8.pt'))\n# if __name__ == '__main__':\n# main()\n\n# %%\n\n\n# %%\n\n\n# %%\n\n\n# %%\n" }, { "path": "scratch/fm_param_vae_rnn.py", "content": "#%%\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import StepLR\nfrom pathlib import Path\nfrom torch.utils.data import Subset as DataSubset\nfrom sklearn.model_selection import train_test_split\nimport numpy as np\nfrom dx7_constants import VOICE_PARAMETER_RANGES, ARTIFACTS_ROOT, VOICE_KEYS\nimport numpy as np\nN_PARAMS = len(VOICE_PARAMETER_RANGES)\nMAX_VALUE = max([max(i) for i in VOICE_PARAMETER_RANGES.values()]) + 1\n#%%\n\n# class DataHandler()\n# def __init__(self, data_file, root=ARTIFACTS_ROOT):\n\n# if not isinstance(root, Path):\n# root = Path(root).expanduser()\n\n# data = np.load(ARTIFACTS_ROOT.joinpath(patch_file))\n\n\n\nclass DX7Dataset():\n \n\n def __init__(self, data_file='dx7.npy', root=ARTIFACTS_ROOT):\n\n if not isinstance(root, Path):\n root = Path(root).expanduser()\n\n self.data = np.load(root.joinpath(data_file)) \n \n\n def __getitem__(self, index):\n\n item = torch.tensor(self.data[index].item()).long()\n\n return item\n def __len__(self):\n return len(self.data)\n\n\n#%%\n\n#%%\nclass Net(nn.Module):\n def __init__(self, latent_dim=8, n_params=N_PARAMS, max_value=MAX_VALUE, hidden_dim=128):\n super(Net, self).__init__()\n\n self.n_params = n_params\n self.max_value = max_value\n\n self.embedder = nn.Embedding(max_value, hidden_dim)\n\n self.enc = nn.ModuleList(\n [nn.LSTM(hidden_dim, hidden_dim, batch_first=True, bidirectional=True),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.LSTM(hidden_dim, hidden_dim, batch_first=True, bidirectional=True),]\n )\n\n self.q_z = nn.Linear(hidden_dim, 2*latent_dim)\n self.z2x = nn.Linear(hidden_dim+latent_dim, hidden_dim)\n self.logits = nn.Linear(hidden_dim, max_value)\n\n self.dec = nn.ModuleList(\n [nn.LSTM(hidden_dim, hidden_dim, batch_first=True),\n nn.GELU(),\n nn.Dropout(0.4),\n nn.LSTM(hidden_dim, hidden_dim, batch_first=True),\n ]\n )\n\n self.register_buffer('mask', self.generate_mask())\n\n def network(self, x, network):\n\n lstm, gelu, drop, lstm2 = network\n\n x_1, (h_1, _) = lstm(x)\n if lstm.bidirectional == True:\n x_1 = h_1.mean(0)\n x_1 = drop(gelu(x_1))\n\n x_2, (h_2, _) = lstm2(x)\n\n if lstm2.bidirectional == True:\n x_2 = h_2.mean(0)\n x = torch.ones_like(x_2)\n\n x_2 = drop(x_2)\n\n x = x_1 * x + x_2\n\n return x\n\n @staticmethod\n def generate_mask():\n \n mask_item_f = lambda x: torch.arange(MAX_VALUE) <= max(x) \n mapper = map(mask_item_f, map(VOICE_PARAMETER_RANGES.get, VOICE_KEYS))\n\n return torch.stack(list(mapper))\n\n def forward(self, x):\n \n x = self.embedder(x)\n theta_z = self.network(x, self.enc)\n\n q_z_mu, q_z_std = self.q_z(theta_z).chunk(2, -1)\n\n q_z = torch.distributions.Normal(q_z_mu, (0.5*q_z_std.clamp(-5, 3)).exp())\n\n z = q_z.sample()\n z_in = z.unsqueeze(-2) + torch.zeros_like(x[...,0]).unsqueeze(-1)\n\n # x_endcut = x\n x_prepad = torch.cat([torch.zeros_like(x[:,[0]]), x], dim=-2)\n x_endcut = x_prepad[:,:-1]\n \n x_dec_in = torch.cat([x_endcut, z_in], dim=-1)\n x_dec_in = self.z2x(x_dec_in)\n x_hat = self.network(x_dec_in, self.dec)\n\n x_hat = self.logits(x_hat)\n\n x_hat = torch.masked_fill(x_hat, ~self.mask, -1e9)\n return x_hat, q_z, z\n\n def generate(self, z, t=1.):\n\n x_hat = self.dec(z)\n x_hat = x_hat.reshape(-1, self.n_params, self.max_value)\n x_hat = torch.masked_fill(x_hat, ~self.mask, -float('inf'))\n\n x_hat = torch.distributions.Categorical(logits=x_hat / t)\n\n return x_hat\n#%%\ndef train(model, device, train_loader, optimizer, epoch):\n model.train()\n for batch_idx, (data) in enumerate(train_loader):\n data = data.to(device)\n optimizer.zero_grad()\n output, q_z, z = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n\n p_z = torch.distributions.Normal(0, 1)\n\n kl = (q_z.log_prob(z) - p_z.log_prob(z)).mean() \n kl_tempered = kl * beta * schedule()\n loss = loss + kl_tempered\n\n loss.backward()\n optimizer.step()\n if batch_idx % 100 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}\\tKL: {:.3f}'.format(\n epoch, batch_idx * len(data), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item(), kl.item()))\n\n\ndef test(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n with torch.no_grad():\n for data in test_loader:\n data = data.to(device)\n output, q_z, z = model(data)\n loss = F.cross_entropy(output.transpose(-1,-2), data)\n p_z = torch.distributions.Normal(0, 1)\n kl = (q_z.log_prob(z) - p_z.log_prob(z)).mean()\n loss = loss + kl * beta\n \n test_loss += loss\n pred = output.argmax(dim=-1, keepdim=True) # get the index of the max log-probability\n correct += pred.eq(data.view_as(pred)).sum().item() / 155\n \n test_loss /= len(test_loader)\n\n print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n test_loss, correct, len(test_loader.dataset),\n 100. * correct / len(test_loader.dataset)))\n\nif __name__==\"__main__\":\n # Training settings\n use_cuda = True\n batch_size = 32\n lr = 1e-4\n gamma = 1.\n epochs = 100\n beta = 0.5\n beta_steps = 37400\n def scheduler():\n n_steps = 0\n def schedule():\n nonlocal n_steps\n n_steps += 1\n\n if n_steps < 1000:\n\n return 0\n\n step = (n_steps-1000)/beta_steps\n step = min(1, step)\n\n return 0.5 * (1 + np.sin((step*np.pi)-(np.pi/2)))\n return schedule\n schedule = scheduler()\n device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n\n kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}\n\n \n dataset = DX7Dataset()\n train_idxs, test_idxs = train_test_split(range(len(dataset)), random_state=42)\n train_dataset = DataSubset(dataset, train_idxs)\n test_dataset = DataSubset(dataset, test_idxs)\n\n train_loader = torch.utils.data.DataLoader(\n train_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n test_loader = torch.utils.data.DataLoader(\n test_dataset,\n batch_size=batch_size, shuffle=True, **kwargs)\n\n model = Net().to(device)\n optimizer = optim.AdamW(model.parameters(), lr=lr)\n\n scheduler = StepLR(optimizer, step_size=1, gamma=gamma)\n for epoch in range(1, epochs + 1):\n train(model, device, train_loader, optimizer, epoch)\n test(model, device, test_loader)\n # scheduler.step()\n\n # if args.save_model:\n # torch.save(model.state_dict(), \"mnist_cnn.pt\")\n\n torch.save(model.state_dict(), ARTIFACTS_ROOT.joinpath('fm-param-vae-8.pt'))\n# if __name__ == '__main__':\n# main()\n\n# %%\n\n\n# %%\n\n\n# %%\n\n\n# %%\n" }, { "path": "scratch/syx_parser.py", "content": "#%%\nimport mido\nfrom pathlib import Path\nimport json\nfrom uuid import uuid4\nfrom itertools import chain\nfrom tqdm import tqdm as tqdm\nimport numpy as np\n\nPARAMETER_ORDER = ['PR1', 'PR2', 'PR3', 'PR4', 'PL1', 'PL2', 'PL3', 'PL4', 'ALG', 'OKS', 'FB', 'LFS', 'LFD', 'LPMD', 'LAMD', 'LPMS', 'LFW', 'LKS', 'TRNSP', 'NAME_0', 'NAME_1', 'NAME_2', 'NAME_3', 'NAME_4', 'NAME_5', 'NAME_6', 'NAME_7', 'NAME_8', 'NAME_9', '0_R1', '0_R2', '0_R3', '0_R4', '0_L1', '0_L2', '0_L3', '0_L4', '0_BP', '0_LD', '0_RD', '0_RC', '0_LC', '0_DET', '0_RS', '0_KVS', '0_AMS', '0_OL', '0_FC', '0_M', '0_FF', '1_R1', '1_R2', '1_R3', '1_R4', '1_L1', '1_L2', '1_L3', '1_L4', '1_BP', '1_LD', '1_RD', '1_RC', '1_LC', '1_DET', '1_RS', '1_KVS', '1_AMS', '1_OL', '1_FC', '1_M', '1_FF', '2_R1', '2_R2', '2_R3', '2_R4', '2_L1', '2_L2', '2_L3', '2_L4', '2_BP', '2_LD', '2_RD', '2_RC', '2_LC', '2_DET', '2_RS', '2_KVS', '2_AMS', '2_OL', '2_FC', '2_M', '2_FF', '3_R1', '3_R2', '3_R3', '3_R4', '3_L1', '3_L2', '3_L3', '3_L4', '3_BP', '3_LD', '3_RD', '3_RC', '3_LC', '3_DET', '3_RS', '3_KVS', '3_AMS', '3_OL', '3_FC', '3_M', '3_FF', '4_R1', '4_R2', '4_R3', '4_R4', '4_L1', '4_L2', '4_L3', '4_L4', '4_BP', '4_LD', '4_RD', '4_RC', '4_LC', '4_DET', '4_RS', '4_KVS', '4_AMS', '4_OL', '4_FC', '4_M', '4_FF', '5_R1', '5_R2', '5_R3', '5_R4', '5_L1', '5_L2', '5_L3', '5_L4', '5_BP', '5_LD', '5_RD', '5_RC', '5_LC', '5_DET', '5_RS', '5_KVS', '5_AMS', '5_OL', '5_FC', '5_M', '5_FF']\n\ndef uuid():\n\n return uuid4().hex\n\nARTIFACTS_ROOT = Path('~/audio/artifacts').expanduser()\n\n\n\nGLOBAL_VALID_RANGES = {\n 'PR1': range(0, 99+1),\n 'PR2': range(0, 99+1),\n 'PR3': range(0, 99+1),\n 'PR4': range(0, 99+1),\n 'PL1': range(0, 99+1),\n 'PL2': range(0, 99+1),\n 'PL3': range(0, 99+1),\n 'PL4': range(0, 99+1),\n 'ALG': range(0, 31+1),\n 'OKS': range(0, 1+1),\n 'FB': range(0, 7+1),\n 'LFS': range(0, 99+1),\n 'LFD': range(0, 99+1),\n 'LPMD': range(0, 99+1),\n 'LAMD': range(0, 99+1),\n 'LPMS': range(0, 7+1),\n 'LFW': range(0, 5+1),\n 'LKS': range(0, 1+1),\n 'TRNSP': range(0, 48+1),\n 'NAME_0': range(128),\n 'NAME_1': range(128),\n 'NAME_2': range(128),\n 'NAME_3': range(128),\n 'NAME_4': range(128),\n 'NAME_5': range(128),\n 'NAME_6': range(128),\n 'NAME_7': range(128),\n 'NAME_8': range(128),\n 'NAME_9': range(128),\n }\n\nOSCILLATOR_VALID_RANGES = {\n 'R1': range(0, 99+1),\n 'R2': range(0, 99+1),\n 'R3': range(0, 99+1),\n 'R4': range(0, 99+1),\n 'L1': range(0, 99+1),\n 'L2': range(0, 99+1),\n 'L3': range(0, 99+1),\n 'L4': range(0, 99+1),\n 'BP': range(0, 99+1),\n 'LD': range(0, 99+1),\n 'RD': range(0, 99+1),\n 'RC': range(0, 3+1),\n 'LC': range(0, 3+1),\n 'DET': range(0, 14+1),\n 'RS': range(0, 7+1),\n 'KVS': range(0, 7+1),\n 'AMS': range(0, 3+1),\n 'OL': range(0, 99+1),\n 'FC': range(0, 31+1),\n 'M': range(0, 1+1),\n 'FF': range(0, 99+1),\n}\n\ndef verify(actual, ranges, prefix=None):\n \n assert set(actual.keys())==set(ranges.keys()), 'Params dont match'\n\n for key in actual:\n if not actual[key] in ranges[key]:\n # print(f'{key} value {actual[key]} should be in {ranges[key]}')\n return False\n return True\n\n# # %%\n# presets = [mido.read_syx_file(patch.as_posix()) for patch in iter(dexed_presets)]\n\nN_OSC = 6\nN_VOICE = 32\n# # %%\n# preset = map(lambda x: x[0], presets[3][0]\n# %%\ndef consume_head(sysex_iter):\n \"\"\"\n ///////////////////////////////////////////////////////////\n B:\n SYSEX Message: Bulk Data for 32 Voices\n --------------------------------------\n bits hex description\n\n 11110000 F0 Status byte - start sysex\n 0iiiiiii 43 ID # (i=67; Yamaha)\n 0sssnnnn 00 Sub-status (s=0) & channel number (n=0; ch 1)\n 0fffffff 09 format number (f=9; 32 voices)\n 0bbbbbbb 20 byte count MS byte\n 0bbbbbbb 00 byte count LS byte (b=4096; 32 voices)\n 0ddddddd ** data byte 1\n\n | | |\n\n 0ddddddd ** data byte 4096 (there are 128 bytes / voice)\n 0eeeeeee ** checksum (masked 2's comp. of sum of 4096 bytes)\n 11110111 F7 Status - end sysex\n\n\n /////////////////////////////////////////////////////////////\n\n \"\"\"\n\n expected = ['0x43',\n '0x00',\n '0x09',\n '0x20',\n '0x00',]\n\n for i in expected:\n assert int(i, 0) == next(sysex_iter), 'unexpected header'\n\n# consume_head(sysex_iter)\n#%%\ndef consume_osc(sysex_iter):\n\n \"\"\"\n byte bit #\n # 6 5 4 3 2 1 0 param A range param B range\n ---- --- --- --- --- --- --- --- ------------ ----- ------------ -----\n 0 R1 OP6 EG R1 0-99\n 1 R2 OP6 EG R2 0-99\n 2 R3 OP6 EG R3 0-99\n 3 R4 OP6 EG R4 0-99\n 4 L1 OP6 EG L1 0-99\n 5 L2 OP6 EG L2 0-99\n 6 L3 OP6 EG L3 0-99\n 7 L4 OP6 EG L4 0-99\n 8 BP LEV SCL BRK PT 0-99\n 9 LD SCL LEFT DEPTH 0-99\n 10 RD SCL RGHT DEPTH 0-99\n 11 0 0 0 | RC | LC | SCL LEFT CURVE 0-3 SCL RGHT CURVE 0-3\n 12 | DET | RS | OSC DETUNE 0-14 OSC RATE SCALE 0-7\n 13 0 0 | KVS | AMS | KEY VEL SENS 0-7 AMP MOD SENS 0-3\n 14 OL OP6 OUTPUT LEV 0-99\n 15 0 | FC | M | FREQ COARSE 0-31 OSC MODE 0-1\n 16 FF FREQ FINE 0-99\n \"\"\"\n\n def process_byte(this_byte):\n\n this_byte = this_byte & int('0b1111111', 0)\n\n return int(this_byte)\n\n int_sysex_iter = iter(map(process_byte, sysex_iter))\n\n R1 = next(int_sysex_iter)\n R2 = next(int_sysex_iter)\n R3 = next(int_sysex_iter)\n R4 = next(int_sysex_iter)\n L1 = next(int_sysex_iter)\n L2 = next(int_sysex_iter)\n L3 = next(int_sysex_iter)\n L4 = next(int_sysex_iter)\n BP = next(int_sysex_iter)\n LD = next(int_sysex_iter)\n RD = next(int_sysex_iter)\n\n _RC_LC = next(int_sysex_iter) & int('0b1111', 0)\n RC = _RC_LC >> 2\n LC = _RC_LC & int('0b11', 0)\n\n\n\n _DET_RS = next(int_sysex_iter) & int('0b11111', 0)\n DET = _DET_RS >> 4\n RS = _DET_RS & int('0b111', 0)\n\n _KVS_AMS = next(int_sysex_iter) & int('0b11111', 0)\n KVS = _KVS_AMS >> 2\n AMS = _KVS_AMS & int('0b11', 0)\n\n OL = next(int_sysex_iter)\n\n _FC_M = next(int_sysex_iter) & int('0b111111', 0)\n FC = _FC_M >> 1\n M = _FC_M & int('0b1', 0)\n\n FF = next(int_sysex_iter)\n\n\n oscilattor_config = {\n 'R1': R1,\n 'R2': R2,\n 'R3': R3,\n 'R4': R4,\n 'L1': L1,\n 'L2': L2,\n 'L3': L3,\n 'L4': L4,\n 'BP': BP,\n 'LD': LD,\n 'RD': RD,\n 'RC': RC,\n 'LC': LC,\n 'DET': DET,\n 'RS': RS,\n 'KVS': KVS,\n 'AMS': AMS,\n 'OL': OL,\n 'FC': FC,\n 'M': M,\n 'FF': FF,\n }\n\n return oscilattor_config\n#%%\n\n\n\ndef consume_global(sysex_iter):\n \"\"\"\n byte bit #\n # 6 5 4 3 2 1 0 param A range param B range\n ---- --- --- --- --- --- --- --- ------------ ----- ------------ -----\n 102 PR1 PITCH EG R1 0-99\n 103 PR2 PITCH EG R2 0-99\n 104 PR3 PITCH EG R3 0-99\n 105 PR4 PITCH EG R4 0-99\n 106 PL1 PITCH EG L1 0-99\n 107 PL2 PITCH EG L2 0-99\n 108 PL3 PITCH EG L3 0-99\n 109 PL4 PITCH EG L4 0-99\n 110 0 0 | ALG | ALGORITHM 0-31\n 111 0 0 0 |OKS| FB | OSC KEY SYNC 0-1 FEEDBACK 0-7\n 112 LFS LFO SPEED 0-99\n 113 LFD LFO DELAY 0-99\n 114 LPMD LF PT MOD DEP 0-99\n 115 LAMD LF AM MOD DEP 0-99\n 116 | LPMS | LFW |LKS| LF PT MOD SNS 0-7 WAVE 0-5, SYNC 0-1\n 117 TRNSP TRANSPOSE 0-48\n 118 NAME CHAR 1 VOICE NAME 1 ASCII\n 119 NAME CHAR 2 VOICE NAME 2 ASCII\n 120 NAME CHAR 3 VOICE NAME 3 ASCII\n 121 NAME CHAR 4 VOICE NAME 4 ASCII\n 122 NAME CHAR 5 VOICE NAME 5 ASCII\n 123 NAME CHAR 6 VOICE NAME 6 ASCII\n 124 NAME CHAR 7 VOICE NAME 7 ASCII\n 125 NAME CHAR 8 VOICE NAME 8 ASCII\n 126 NAME CHAR 9 VOICE NAME 9 ASCII\n 127 NAME CHAR 10 VOICE NAME 10 ASCII\n \"\"\"\n\n def process_byte(this_byte):\n\n this_byte = this_byte & int('0b111111', 0)\n\n return this_byte\n\n sysex_iter = iter(map(process_byte, sysex_iter))\n\n\n PR1 = int(next(sysex_iter))\n PR2 = int(next(sysex_iter))\n PR3 = int(next(sysex_iter))\n PR4 = int(next(sysex_iter))\n PL1 = int(next(sysex_iter))\n PL2 = int(next(sysex_iter))\n PL3 = int(next(sysex_iter))\n PL4 = int(next(sysex_iter))\n\n ALG = int(next(sysex_iter)) & int('0b11111', 0)\n\n OKS_FB = int(next(sysex_iter))\n OKS = (OKS_FB & int('0b1000', 0)) >> 3\n FB = (OKS_FB & int('0b111', 0))\n\n LFS = int(next(sysex_iter))\n LFD = int(next(sysex_iter))\n LPMD = int(next(sysex_iter))\n LAMD = int(next(sysex_iter))\n\n LPMS_LFW_LKS = int(next(sysex_iter))\n LPMS = LPMS_LFW_LKS >> 4\n LFW = (LPMS_LFW_LKS >> 1) & int('0b111', 0)\n LKS = LPMS_LFW_LKS & int('0b1', 0)\n\n TRNSP = int(next(sysex_iter))\n\n to_ascii = lambda byte: bytes.fromhex(byte.decode(\"ascii\")).decode('ascii')\n to_ascii = lambda byte: ascii(byte.decode('ascii'))\n NAME = [(f'NAME_{i}', int(next(sysex_iter))) for i in range(10)]\n\n global_config = {\n 'PR1': PR1,\n 'PR2': PR2,\n 'PR3': PR3,\n 'PR4': PR4,\n 'PL1': PL1,\n 'PL2': PL2,\n 'PL3': PL3,\n 'PL4': PL4,\n 'ALG': ALG,\n 'OKS': OKS,\n 'FB': FB,\n 'LFS': LFS,\n 'LFD': LFD,\n 'LPMD': LPMD,\n 'LAMD': LAMD,\n 'LPMS': LPMS,\n 'LFW': LFW,\n 'LKS': LKS,\n 'TRNSP': TRNSP,\n }\n global_config.update(NAME)\n\n return uuid(), global_config\n\ndef consume_syx(path):\n\n path = Path(path).expanduser()\n\n try:\n preset = mido.read_syx_file(path.as_posix())[0]\n except IndexError as e:\n return None\n except ValueError as e:\n return None\n\n sysex_iter = iter(preset.data)\n # print(len(list(preset.bytes())))\n try:\n consume_head(sysex_iter)\n except AssertionError as e:\n return None\n\n for i in range(N_VOICE):\n def consume_oscillator():\n\n oscilattor_config = consume_osc(sysex_iter)\n\n if not verify(oscilattor_config, OSCILLATOR_VALID_RANGES):\n raise ValueError('Oscillator has values outside range')\n\n return oscilattor_config.items()\n\n prefix_oscillator = lambda n: [(f'{n}_{key}', value) for key,value in consume_oscillator()]\n oscilattor_mapper = chain.from_iterable(map(prefix_oscillator, range(N_OSC)))\n \n # oscillator_config = {i: consume_osc(sysex_iter) for i in range(N_OSC)}\n patch_config = {}\n has_error = False\n # oscilattor\n try:\n patch_config.update(oscilattor_mapper)\n except ValueError:\n # invalid range in oscillators\n has_error = True\n\n name, global_config = consume_global(sysex_iter)\n\n if not verify(global_config, GLOBAL_VALID_RANGES) or has_error:\n # print('eror')\n yield\n continue\n\n patch_config.update(global_config)\n\n\n\n\n yield ((path, i), patch_config)\n#%%\n\nif __name__=='__main__':\n DEV = False\n\n # PRESETS_ROOT = '~/audio/artifacts/dx7-patches'\n # PRESETS_ROOT = '~/audio/artifacts/dx7-patches-dev'\n preset_root = ARTIFACTS_ROOT.joinpath('dx7-patches').expanduser()\n\n preset_paths = sorted(preset_root.glob('**/*.syx'))\n if DEV:\n preset_paths = preset_paths[:10]\n preset_paths = iter(filter(lambda preset_path: preset_path.is_file(), preset_paths))\n\n consume_chain = chain.from_iterable(map(consume_syx, preset_paths))\n consume_chain = filter(lambda x: x is not None, consume_chain)\n\n outputs = []\n for (path, i), params in tqdm(consume_chain):\n # print(params)\n data = map(params.get, PARAMETER_ORDER)\n # _, data = zip(*params.items())\n # print(data)\n # break\n outputs += [(path, i, tuple(data))]\n\n # for i in list(map(list, outputs)):\n # print(len(i))\n paths, ns, data = zip(*outputs)\n data = np.array(list(set(data)))\n paths = np.array([path.as_posix() for path in paths])\n ns = np.array(ns)\n np.savez(ARTIFACTS_ROOT.joinpath('dx7.npy'), outputs, paths, ns)\n # patch_map = dict(tqdm((iter(consume_chain))))\n # random_key, *_ = patch_map.keys()\n # # print(global_config['NAME'])\n\n # name = items.pop('')\n # print(global_config)\n # print(json.dumps(oscillator_configs, indent=4))\n # print(list(sysex_iter))\n\n\n\n\n\n# %%\n\n\n# %%\n" }, { "path": "scratch/syx_write.py", "content": "#%%\nimport mido\nfrom pathlib import Path\nimport json\nfrom uuid import uuid4\nfrom itertools import chain\nfrom tqdm import tqdm as tqdm\nimport numpy as np\n\nfrom syx_parser import PARAMETER_ORDER, ARTIFACTS_ROOT\n\n# # %%\n# presets = [mido.read_syx_file(patch.as_posix()) for patch in iter(dexed_presets)]\n\nN_OSC = 6\nN_VOICE = 32\n# # %%\n\ndef checksum(data):\n return (128-sum(data)&127)%128\n\n# preset = map(lambda x: x[0], presets[3][0]\n# %%\ndef encode_head():\n \"\"\"\n ///////////////////////////////////////////////////////////\n B:\n SYSEX Message: Bulk Data for 32 Voices\n --------------------------------------\n bits hex description\n\n 11110000 F0 Status byte - start sysex\n 0iiiiiii 43 ID # (i=67; Yamaha)\n 0sssnnnn 00 Sub-status (s=0) & channel number (n=0; ch 1)\n 0fffffff 09 format number (f=9; 32 voices)\n 0bbbbbbb 20 byte count MS byte\n 0bbbbbbb 00 byte count LS byte (b=4096; 32 voices)\n 0ddddddd ** data byte 1\n\n | | |\n\n 0ddddddd ** data byte 4096 (there are 128 bytes / voice)\n 0eeeeeee ** checksum (masked 2's comp. of sum of 4096 bytes)\n 11110111 F7 Status - end sysex\n\n\n /////////////////////////////////////////////////////////////\n\n \"\"\"\n\n header = [ '0x43',\n '0x00',\n '0x09',\n '0x20',\n '0x00',]\n\n return [int(i, 0) for i in header]\n\n# consume_head(sysex_iter)\n#%%\ndef encode_osc(params, n):\n\n \"\"\"\n byte bit #\n # 6 5 4 3 2 1 0 param A range param B range\n ---- --- --- --- --- --- --- --- ------------ ----- ------------ -----\n 0 R1 OP6 EG R1 0-99\n 1 R2 OP6 EG R2 0-99\n 2 R3 OP6 EG R3 0-99\n 3 R4 OP6 EG R4 0-99\n 4 L1 OP6 EG L1 0-99\n 5 L2 OP6 EG L2 0-99\n 6 L3 OP6 EG L3 0-99\n 7 L4 OP6 EG L4 0-99\n 8 BP LEV SCL BRK PT 0-99\n 9 LD SCL LEFT DEPTH 0-99\n 10 RD SCL RGHT DEPTH 0-99\n 11 0 0 0 | RC | LC | SCL LEFT CURVE 0-3 SCL RGHT CURVE 0-3\n 12 | DET | RS | OSC DETUNE 0-14 OSC RATE SCALE 0-7\n 13 0 0 | KVS | AMS | KEY VEL SENS 0-7 AMP MOD SENS 0-3\n 14 OL OP6 OUTPUT LEV 0-99\n 15 0 | FC | M | FREQ COARSE 0-31 OSC MODE 0-1\n 16 FF FREQ FINE 0-99\n \"\"\"\n\n oscillator_params = []\n\n oscillator_params += [params[f'{n}_R1']]\n oscillator_params += [params[f'{n}_R2']]\n oscillator_params += [params[f'{n}_R3']]\n oscillator_params += [params[f'{n}_R4']]\n oscillator_params += [params[f'{n}_L1']]\n oscillator_params += [params[f'{n}_L2']]\n oscillator_params += [params[f'{n}_L3']]\n oscillator_params += [params[f'{n}_L4']]\n oscillator_params += [params[f'{n}_BP']]\n oscillator_params += [params[f'{n}_LD']]\n oscillator_params += [params[f'{n}_RD']]\n\n RC = params[f'{n}_RC'] << 2\n LC = params[f'{n}_LC']\n oscillator_params += [RC | LC]\n\n\n DET = params[f'{n}_DET'] << 3\n RS = params[f'{n}_RS']\n oscillator_params += [DET | RS]\n\n\n KVS = params[f'{n}_KVS'] << 2\n AMS = params[f'{n}_AMS'] \n oscillator_params += [KVS|AMS]\n\n oscillator_params += [params[f'{n}_OL']]\n\n\n FC = params[f'{n}_FC'] << 1\n M = params[f'{n}_M']\n oscillator_params += [FC|M]\n\n oscillator_params += [params[f'{n}_FF']]\n\n\n return oscillator_params\n\n#%%\ndef encode_global(params):\n \"\"\"\n byte bit #\n # 6 5 4 3 2 1 0 param A range param B range\n ---- --- --- --- --- --- --- --- ------------ ----- ------------ -----\n 102 PR1 PITCH EG R1 0-99\n 103 PR2 PITCH EG R2 0-99\n 104 PR3 PITCH EG R3 0-99\n 105 PR4 PITCH EG R4 0-99\n 106 PL1 PITCH EG L1 0-99\n 107 PL2 PITCH EG L2 0-99\n 108 PL3 PITCH EG L3 0-99\n 109 PL4 PITCH EG L4 0-99\n 110 0 0 | ALG | ALGORITHM 0-31\n 111 0 0 0 |OKS| FB | OSC KEY SYNC 0-1 FEEDBACK 0-7\n 112 LFS LFO SPEED 0-99\n 113 LFD LFO DELAY 0-99\n 114 LPMD LF PT MOD DEP 0-99\n 115 LAMD LF AM MOD DEP 0-99\n 116 | LPMS | LFW |LKS| LF PT MOD SNS 0-7 WAVE 0-5, SYNC 0-1\n 117 TRNSP TRANSPOSE 0-48\n 118 NAME CHAR 1 VOICE NAME 1 ASCII\n 119 NAME CHAR 2 VOICE NAME 2 ASCII\n 120 NAME CHAR 3 VOICE NAME 3 ASCII\n 121 NAME CHAR 4 VOICE NAME 4 ASCII\n 122 NAME CHAR 5 VOICE NAME 5 ASCII\n 123 NAME CHAR 6 VOICE NAME 6 ASCII\n 124 NAME CHAR 7 VOICE NAME 7 ASCII\n 125 NAME CHAR 8 VOICE NAME 8 ASCII\n 126 NAME CHAR 9 VOICE NAME 9 ASCII\n 127 NAME CHAR 10 VOICE NAME 10 ASCII\n \"\"\"\n\n global_params = []\n\n\n global_params += [params['PR1']]\n global_params += [params['PR2']]\n global_params += [params['PR3']]\n global_params += [params['PR4']]\n global_params += [params['PL1']]\n global_params += [params['PL2']]\n global_params += [params['PL3']]\n global_params += [params['PL4']]\n\n global_params += [params['ALG']]\n\n OKS = params['OKS'] << 3\n print(OKS, '------', params['OKS'])\n FB = params['FB']\n\n global_params += [OKS|FB]\n print(OKS|FB)\n\n\n global_params += [params['LFS']]\n global_params += [params['LFD']]\n global_params += [params['LPMD']]\n global_params += [params['LAMD']]\n\n LPMS = params['LPMS'] << 4\n LFW = params['LFW'] << 1\n LKS = params['LKS']\n # if (LPMS & LFW) or (LPMS & LKS):\n # print('que', LPMS | LFW | LKS)\n global_params += [LPMS | LFW | LKS]\n\n global_params += [params['TRNSP']]\n\n global_params += [params[f'NAME_{i}'] for i in range(10)]\n\n\n return global_params\n\ndef encode_syx(params_list):\n\n head = encode_head()\n\n data = []\n assert len(params_list) == N_VOICE\n\n # voices\n for params in params_list:\n \n for osc in range(N_OSC):\n data += encode_osc(params, osc)\n\n data += encode_global(params)\n\n\n this_checksum = checksum(data)\n\n\n return [*head, *data, this_checksum]\n \n#%%\n\n\n\ndata = np.load(ARTIFACTS_ROOT.joinpath('dev-dx7.npy'))\ndata = np.load(ARTIFACTS_ROOT.joinpath('dev-dx7.npy'))[[1]*32]\n\nparams_list = list(map(lambda params: dict(zip(PARAMETER_ORDER, list(params))), data))\n\nsyx = encode_syx(params_list)\n\nmessage = mido.Message('sysex', data=syx)\nmido.write_syx_file(ARTIFACTS_ROOT.joinpath('patch.syx'), [message])\n\n# %%\n\n\n# %%\n" }, { "path": "setup.cfg", "content": "[metadata]\ndescription-file = README.md" }, { "path": "setup.py", "content": "#!/usr/bin/env python\nfrom setuptools import setup, find_packages\nfrom pathlib import Path\n\nroot_path = Path(__file__).parent\n\n\nreadme = Path('README.md').read_text()\nrequirements = Path('requirements.txt').read_text().split('\\n')\n__version__ = Path('version').read_text().strip()\n\nsetup(\n name='neuralDX7',\n packages = find_packages(),\n license='MIT',\n version=__version__,\n author='Nintorac',\n author_email='neuralDX7@nintorac.dev',\n url='https://github.com/nintorac/neuralDX7',\n description='Models related to the DX7 fm synthesizer',\n long_description=readme,\n zip_safe=True,\n install_requires=requirements,\n classifiers=[\n 'Development Status :: 3 - Alpha', # Chose either \"3 - Alpha\", \"4 - Beta\" or \"5 - Production/Stable\" as the current state of your package\n 'Intended Audience :: Developers', # Define that your audience are developers\n 'Topic :: Software Development :: Build Tools',\n 'License :: OSI Approved :: MIT License', # Again, pick a license\n 'Programming Language :: Python :: 3', #Specify which pyhton versions that you want to support\n 'Programming Language :: Python :: 3.4',\n 'Programming Language :: Python :: 3.5',\n 'Programming Language :: Python :: 3.6',\n ],\n long_description_content_type='text/markdown'\n)" }, { "path": "version", "content": "0.0.8" } ]