Repository: RyanWangZf/transtab Branch: main Commit: fdb34cf38abd Files: 49 Total size: 510.6 KB Directory structure: gitextract_y454drjn/ ├── .gitignore ├── .readthedocs.yaml ├── LICENSE ├── README.md ├── blog/ │ └── README.md ├── docs/ │ ├── Makefile │ ├── make.bat │ ├── requirements.txt │ ├── source/ │ │ ├── about.rst │ │ ├── conf.py │ │ ├── data_preparation.rst │ │ ├── example_encode.rst │ │ ├── example_pretrain.rst │ │ ├── example_transfer.rst │ │ ├── fast_train.rst │ │ ├── index.rst │ │ ├── install.rst │ │ ├── main_func.rst │ │ ├── models.rst │ │ ├── transtab.basemodel.rst │ │ ├── transtab.build_classifier.rst │ │ ├── transtab.build_contrastive_learner.rst │ │ ├── transtab.build_encoder.rst │ │ ├── transtab.build_extractor.rst │ │ ├── transtab.classifier.rst │ │ ├── transtab.contrastive.rst │ │ ├── transtab.load_data.rst │ │ ├── transtab.predict.rst │ │ └── transtab.train.rst │ └── sphinx-commands.txt ├── examples/ │ ├── contrastive_learning.ipynb │ ├── fast_train.ipynb │ ├── table_embedding.ipynb │ ├── transfer_learning.ipynb │ └── transfer_learning_regressor.ipynb ├── pypi_build_commands.txt ├── requirements.txt ├── setup.py └── transtab/ ├── __init__.py ├── constants.py ├── dataset.py ├── evaluator.py ├── modeling_transtab.py ├── tokenizer/ │ ├── special_tokens_map.json │ ├── tokenizer_config.json │ └── vocab.txt ├── trainer.py ├── trainer_utils.py └── transtab.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ pip-wheel-metadata/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # PEP 582; used by e.g. github.com/David-OConnor/pyflow __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # checkpoint /ckpt /checkpoint # MAC .DS_Store ================================================ FILE: .readthedocs.yaml ================================================ # Read the Docs configuration file for Sphinx projects # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the OS, Python version and other tools you might need build: os: ubuntu-22.04 tools: python: "3.9" # You can also specify other tool versions: # nodejs: "20" # rust: "1.70" # golang: "1.20" # Build documentation in the "docs/" directory with Sphinx sphinx: configuration: docs/source/conf.py # You can configure Sphinx to use a different builder, for instance use the dirhtml builder for simpler URLs # builder: "dirhtml" # Fail on all warnings to avoid broken references # fail_on_warning: true # Optionally build your docs in additional formats such as PDF and ePub # formats: # - pdf # - epub # Optional but recommended, declare the Python requirements required # to build your documentation # See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html python: install: - requirements: docs/requirements.txt ================================================ FILE: LICENSE ================================================ BSD 2-Clause License Copyright (c) 2022, Zifeng All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ================================================ FILE: README.md ================================================ # TransTab: A flexible transferable tabular learning framework [[arxiv]](https://arxiv.org/pdf/2205.09328.pdf) [![PyPI version](https://badge.fury.io/py/transtab.svg)](https://badge.fury.io/py/transtab) [![Documentation Status](https://readthedocs.org/projects/transtab/badge/?version=latest)](https://transtab.readthedocs.io/en/latest/?badge=latest) [![License](https://img.shields.io/badge/License-BSD_2--Clause-orange.svg)](https://opensource.org/licenses/BSD-2-Clause) ![GitHub Repo stars](https://img.shields.io/github/stars/ryanwangzf/transtab) ![GitHub Repo forks](https://img.shields.io/github/forks/ryanwangzf/transtab) [![Downloads](https://pepy.tech/badge/transtab)](https://pepy.tech/project/transtab) [![Downloads](https://pepy.tech/badge/transtab/month)](https://pepy.tech/project/transtab) Document is available at https://transtab.readthedocs.io/en/latest/index.html. Paper is available at https://arxiv.org/pdf/2205.09328.pdf. 5 min blog to understand TransTab at [realsunlab.medium.com](https://realsunlab.medium.com/transtab-learning-transferable-tabular-transformers-across-tables-1e34eec161b8)! ### News! - [03/12/25] Version `0.0.7` with `TransTabRegressor` available for regression. Thanks @yuxinchenNU. - [05/04/23] Check the version `0.0.5` of `TransTab`! - [01/04/23] Check the version `0.0.3` of `TransTab`! - [12/03/22] Check out our [[blog]](https://realsunlab.medium.com/transtab-learning-transferable-tabular-transformers-across-tables-1e34eec161b8) for a quick understanding of TransTab! - [08/31/22] `0.0.2` Support encode tabular inputs into embeddings directly. An example is provided [here](examples/table_embedding.ipynb). Several bugs are fixed. ## TODO - [x] Table embedding. - [x] Add regression support. - [ ] Add support to direct process table with missing values. ### Features This repository provides the python package `transtab` for flexible tabular prediction model. The basic usage of `transtab` can be done in a couple of lines! ```python import transtab # load dataset by specifying dataset name allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-g') # build classifier model = transtab.build_classifier(cat_cols, num_cols, bin_cols) # build regressor # model = transtab.build_regressor(cat_cols, num_cols, bin_cols) # start training transtab.train(model, trainset, valset, **training_arguments) # make predictions, df_x is a pd.DataFrame with shape (n, d) # return the predictions ypred with shape (n, 1) if binary classification; # (n, n_class) if multiclass classification. ypred = transtab.predict(model, df_x) ``` It's easy, isn't it? ## How to install First, download the right ``pytorch`` version following the guide on https://pytorch.org/get-started/locally/. ~~Then try to install from pypi directly:~~ [Feb 2025: pypi version is not maintained, please try to install from github instead] ~~or~~ ```bash pip install git+https://github.com/RyanWangZf/transtab.git ``` Please refer to for [more guidance on installation](https://transtab.readthedocs.io/en/latest/install.html) and troubleshooting. ## Transfer learning across tables A novel feature of `transtab` is its ability to learn from multiple distinct tables. It is easy to trigger the training like ```python # load the pretrained transtab model model = transtab.build_classifier(checkpoint='./ckpt') # load a new tabular dataset allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-approval') # update categorical/numerical/binary column map of the loaded model model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols}) # then we just trigger the training on the new data transtab.train(model, trainset, valset, **training_arguments) ``` ## Contrastive pretraining on multiple tables We can also conduct contrastive pretraining on multiple distinct tables like ```python # load from multiple tabular datasets dataname_list = ['credit-g', 'credit-approval'] allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(dataname_list) # build contrastive learner, set supervised=True for supervised VPCL model, collate_fn = transtab.build_contrastive_learner( cat_cols, num_cols, bin_cols, supervised=True) # start contrastive pretraining training transtab.train(model, trainset, valset, collate_fn=collate_fn, **training_arguments) ``` ## Citation If you find this package useful, please consider citing the following paper: ```latex @inproceedings{wang2022transtab, title={TransTab: Learning Transferable Tabular Transformers Across Tables}, author={Wang, Zifeng and Sun, Jimeng}, booktitle={Advances in Neural Information Processing Systems}, year={2022} } ``` ================================================ FILE: blog/README.md ================================================ # NeurIPS'22 | How to perform transfer learning and zero-shot learning on tabular data? > This is our paper accepted by NeurIPS'22 with ratings 7/7/7, where we work on pretraining, transfer learning, and zero-shot learning on the tabular prediction task. The following are the links for this article and the codes. Paper: [TransTab: Learning Transferable Tabular Transformers Across Tables](https://arxiv.org/pdf/2205.09328.pdf) Code: [TransTab-github](https://github.com/RyanWangZf/transtab) Doc: [Transtab-doc](https://transtab.readthedocs.io/en/latest/) --- ## Tabular learning was not flexible In this article, we refer the term *tabular learning* to the predictive task on the tabular data. For instance, we might know Kaggle competitions, where a lot of competitions are based on tabular data, e.g., house price prediction, credit fault detection, CTR prediction, etc. Basically, this type of task is on predicting the target label through a couple of features, just like in the following table | index | feature A | feature B | feature C | label | | ----- | --------- | --------- | --------- | ----- | | 0 | $x_1$ | $x_2$ | $x_3$ | $y$ | one might take a linear regression to solve this problem as $$ y = ax_1 + b x_2 + c x_3 +d. $$ Compared with images and texts, tables are usually more frequently used in industrial applications. As we all know, recently there emerged the *pretrain+finetune* paradigm in the deep learning area, especially flourished in computer vision (CV) and natural language processing (NLP).
Figure 1: CV or NLP models usually share the same basic input unit, i.e., pixel for images and word/token for texts. However, tabular models only accept a fixed-structure table: the train and test tables should *always* have equal column sets, which prevents us from transfer learning or zero-shot learning on tabular data.
In CV & NLP, pretrained models like BERT, and ViT have become the strong baseline for almost all tasks. By contrast, in the tabular learning domain, we usually encounter the case "xgboost is all you need". GBDT models can achieve competent performances with less effort on data preprocessing and hyperparameter tuning than deep learning-based methods. In this circumstance, a lot of researchers have started to think about how we outperform GBDT using deep learning, especially leveraging the power of deep learning on big multi-sourced data. ## Recent efforts on transfer learning for tabular learning Of course, there have been some efforts on transfer learning for deep learning-based tabular learning. For example, VIME [[1]](#1), SCARF [[2]](#2), and SubTab [[3]](#3) all employ self-supervision for tabular learning. The common self-supervision can be categorized as *generative* and *discriminative* learning. For the first venue, we mask several cells in the table and ask the model to recover the missing values; for the second, we create positive samples by deleting or replacing cells. Nonetheless, they hardly apply to real application cases: all apply to fixed-structure tables. We do not have a large table without labels, instead, we often have multiple heterogenous labeled labels. The core challenge is how to leverage as much labeled data as possible and get rid of heavy data preprocessing and missing value imputation. The nature of only receiving fixed-structure tables causes all existing tabular methods to be incapable of dealing with pretraining on multi-sourced tables. Once there is a minor change in the table's structure, e.g., a column named *age* changed to *ages*, the pretrained model becomes useless. And we need to roll back to the process of *data processing* $\to$ *feature engineering* $\to$ *model training*, which is costly in terms of time and money. Therefore, we ask, if it is possible to propose a tabular model that encodes **arbitrary** input tables needless of any adaptions? ### Tabular learning is flexible In fact, if we look back on the tabular data, we shall identify the column names are rich in semantics, which were long neglected by previous methods. | index | gender | age | is_citizen | | ----- | ------ | ---- | ---------- | | 0 | male | 25 | 0 | In this example, we include three common types of features: *categorical*, *numerical*, and *binary*. We argue that interpreting features considering column names is necessary. We know *25* under the column *age* means 25 years old instead of 25 km or 25 kg. We know *0* under the column *is_citizen* means the person is not a citizen instead of is not anything else. Previous methods drop column names and enforce the model to learn semantics from the raw features for decision-making, which is easy to implement but not transferable. On the contrary, we ask why not just explicitly include the column names in the modeling. Surprisingly, we do not find any prior arts doing that in tabular learning. Formally, we process three types of features through - For categorical: we concatenate column names and features, i.e., *gender is male*. - For numerical, we tokenize and embed column names, then multiply the column embeddings with the feature value. - For bool: we tokenize and embed column names, they decide if pass this embedding to the encoder based on feature. If 0, then we drop this embedding.
Figure 2: The input feature processing module of *TransTab*.
With this processing module, we can linearize, tokenize, and embed any tabular data, which serves as the inputs for the encoder and the predictor. ## Pretraining for TransTab Thanks to its flexibility, *TransTab* is capable of learning across multiple heterogeneous tables. However, it is nontrivial to design an appropriate pretraining algorithm for it.
Figure 3: Learning across tables using naive supervised learning is harmful for representation learning.
The most straightforward way is illustrated as above: we train a shared backbone encoder plus task-specific classifiers across tabular datasets. Nevertheless, we soon find this paradigm is suboptimal. The flaw comes from the heterogeneity of the target labels: two datasets might have opposite definition of labels. Accounting for this issue, we propose a novel **supervised contrastive learning** approach, namely **vertical partition contrastive learning (VPCL)** in this paper.
Figure 4: The proposed vertical partition contrastive learning (VPCL) approach for pretraining TransTab in our paper.
Its principle is: - We split each raw (sample) into several partitions vertically, each partition is a sample for contrastive learning. - The partition comes from the same-label raw are positive, and vice versa. VPCL has the following merits: - It significantly expand the number of pairs for contrastive learning. - It is much more efficient and robust because it does not add additional task-specific classifiers. ## Which new tasks that TransTab can solve? Thanks to the flexibility of TransTab, it now handles many new tasks.
Figure 5: The new tasks that are amenable to TransTab.
- Learning across multiple labeled datasets (share the same label) based on supervised learning, and finetuned on each specific dataset. - Learning from an incremental set of features and data, which usually originates from the updated measurements over time. - Pretrained on multiple labeled/unlabeled datasets (can have distinct labels) based on supervised VPCL, and finetuned on each dataset. - Learning from multiple labeled datasets (share the same label) based on supervised learning, and making predictions for brand new data without any further parameter updates. ## Some experiment results For the complete experiment results, please refer to [our paper](https://arxiv.org/pdf/2205.09328.pdf). Here we tell two interesting findings. ### Pretraining
Figure 6: Experiment results of the pretraining+finetuning performances of TransTab.
The above figure illustrates the average performance (AUC) on multiple datasets. Left: on clinical trial patient outcome prediction datasets. Right: on many open datasets. The red dotted line indicates the naive supervised learning performance. X-axis is the number of partitions made for VPCL. We find: - Supervised VPCL generally improves predictive performances. - It is not an universally optimal number of partitions for VPCL. - Compared with open datasets, the pretraining on the left introduces much more improvements. That implies that it is still crucial to transfer knowledge from datasets coming from the similar domain. While the open datasets are very heterogeneous. ### Zero-shot prediction
Figure 7: Experiment results of the zero-shot learning performances of TransTab.
The above figures demonstrate the zero-shot prediction performances of TransTab. We split one dataset into two parts and vary the overlap ratio of their column sets: from 0% to 100%. We find: - TransTab can even make reasonable predictions when there is **no column overlapping** between the train and test data, which is really amazing. - When the overlap ratio increases, we witness better performances, which is reasonable. ## Use TransTab based on our package We opensourced our package on [github](https://github.com/RyanWangZf/transtab) with the [documentations](https://transtab.readthedocs.io/en/latest/). It can be downloaded through ```shell pip install git+https://github.com/RyanWangZf/transtab.git ``` And it is rather easy to use it in tabular prediction tasks on multiple distinct tables: ```python import transtab # load multiple datasets by passing a list of data names allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(['credit-g','credit-approval']) # build transtab classifier model model = transtab.build_classifier(cat_cols, num_cols, bin_cols) # specify training arguments, take validation loss for early stopping training_arguments = { 'num_epoch':5, 'eval_metric':'val_loss', 'eval_less_is_better':True, 'output_dir':'./checkpoint' } # start training transtab.train(model, trainset, valset[0], **training_arguments) ``` For pretraining based on VPCL, we have ```python import transtab # load multiple datasets by passing a list of data names allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(['credit-g','credit-approval']) # build contrastive learner, set supervised=True for supervised VPCL model, collate_fn = transtab.build_contrastive_learner( cat_cols, num_cols, bin_cols, supervised=True, # if take supervised CL num_partition=4, # num of column partitions for pos/neg sampling overlap_ratio=0.5, # specify the overlap ratio of column partitions during the CL ) # start contrastive pretraining training training_arguments = { 'num_epoch':50, 'batch_size':64, 'lr':1e-4, 'eval_metric':'val_loss', 'eval_less_is_better':True, 'output_dir':'./checkpoint' # save the pretrained model } # pass the collate function to the train function transtab.train(model, trainset, valset, collate_fn=collate_fn, **training_arguments) ``` And after the pretraining completes, we can build a new classifier based on the pretrained model: ```python # load the pretrained model and finetune on a target dataset allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-approval') # build transtab classifier model, and load from the pretrained dir model = transtab.build_classifier(checkpoint='./checkpoint') # update model's categorical/numerical/binary column dict model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols}) ``` It is easy 😎 ! ## Conclusion Duplicating the success of deep learning in CV & NLP to tabular learning domain still requires rethinking the basic elements. In CV, we have pixel; in NLP, we have token/word. In this paper, we propose a simple yet effective algorithm to model tabular data. Our method explores using NLP techniques for enhancing tabular learning, with flexibility to handle arbitrary input tables. We hope it appeals to more attention in deep learning for tabular learning. ## References [1] Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. VIME: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:11033–11043, 2020. [2] Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. SCARF: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022. [3] Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. SubTab: Subsetting features of tabular data for self-supervised representation learning. Advances in Neural Information Processing Systems, 34, 2021. ================================================ FILE: docs/Makefile ================================================ # Minimal makefile for Sphinx documentation # # You can set these variables from the command line, and also # from the environment for the first two. SPHINXOPTS ?= SPHINXBUILD ?= sphinx-build SOURCEDIR = source BUILDDIR = build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) ================================================ FILE: docs/make.bat ================================================ @ECHO OFF pushd %~dp0 REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set SOURCEDIR=source set BUILDDIR=build %SPHINXBUILD% >NUL 2>NUL if errorlevel 9009 ( echo. echo.The 'sphinx-build' command was not found. Make sure you have Sphinx echo.installed, then set the SPHINXBUILD environment variable to point echo.to the full path of the 'sphinx-build' executable. Alternatively you echo.may add the Sphinx directory to PATH. echo. echo.If you don't have Sphinx installed, grab it from echo.https://www.sphinx-doc.org/ exit /b 1 ) if "%1" == "" goto help %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% goto end :help %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% :end popd ================================================ FILE: docs/requirements.txt ================================================ sphinx-markdown-tables recommonmark sphinx==4.2.0 sphinx_rtd_theme==1.0.0 readthedocs-sphinx-search==0.1.1 loguru numpy scikit_learn setuptools transformers tqdm pandas>=1.3.* openml>=0.10.0 torch ================================================ FILE: docs/source/about.rst ================================================ About Us ======== This package was developed and maintained by Zifeng Wang (Ph.D. Student @ UIUC). Please refer to his `Homepage `_ for more details. ================================================ FILE: docs/source/conf.py ================================================ # Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # import os import sys import pdb sys.path.insert(0, os.path.abspath('../../')) # -- Project information ----------------------------------------------------- project = 'transtab' copyright = '2022, Zifeng Wang' author = 'Zifeng Wang' # The full version, including alpha/beta/rc tags release = 'alpha' # Override the RTD default master doc master_doc = 'index' # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'recommonmark', 'sphinx_markdown_tables', 'sphinx.ext.intersphinx', 'sphinx.ext.imgmath', 'sphinx.ext.viewcode', 'sphinx.ext.napoleon', 'sphinx.ext.autodoc', ] napoleon_google_docstring = False napoleon_numpy_docstring = True # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. exclude_patterns = [] # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # # html_theme = 'alabaster' html_theme = 'sphinx_rtd_theme' # html_theme = 'furo' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] ================================================ FILE: docs/source/data_preparation.rst ================================================ Custom Dataset ============== Here is the best practice to build your own datasets for `transtab`. :: project | ├── run_your_model.py | └─── data | ├── dataset1 | | data_processed.csv | | binary_feature.txt | └─── numerical_feature.txt | ├── dataset2 | ... where the ``run_your_model.py`` is the code where you will load the dataset and train your models. You should put the preprocessed table into ``data_processed.csv``, which is better to follow the protocols: * All the column names to be represented by meaningful natural languge. * All the categorical features to be represented by meaningful natural language. * All the binary features to be represented by 0 or 1. * All the numerical features to be represented by continuous values. * Store the processed table into ``data_processed.csv``. * Store the binary column names into ``binary_feature.txt``. No need to create this file if no binary feature. * Store the numerical column names into ``numerical_feature.txt``. No need to create this file if no numerical feature. * All the other columns will be treated as categorical or textual. After that, you can try to load the dataset by .. code-block:: python transtab.load_data('./data/dataset1') About ``dataset_config``, an example is provided as .. code-block:: python EXAMPLE_DATACONFIG = { "example": { # dataset name "bin": ["bin1", "bin2"], # binary column names "cat": ["cat1", "cat2"], # categorical column names "num": ["num1", "num2"], # numerical column names "cols": ["bin1", "bin2", "cat1", "cat2", "num1", "num2"], # all column names "binary_indicator": ["1", "yes", "true", "positive", "t", "y"], # binary indicators in the binary columns, which will be converted to 1 "data_split_idx": { "train":[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], # row indices for training set "val":[10, 11, 12, 13, 14, 15, 16, 17, 18, 19], # row indices for validation set "test":[20, 21, 22, 23, 24, 25, 26, 27, 28, 29], # row indices for test set } } } ================================================ FILE: docs/source/example_encode.rst ================================================ Encode Tables ============= *transtab* is able to take pd.DataFrame as inputs and outputs the encoded sample-level embeddings. The full code is available at `Notebook Example `_. .. code-block:: python import transtab # load a dataset and start vanilla supervised training allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-g') # build transtab classifier model model, collate_fn = transtab.build_contrastive_learner(cat_cols, num_cols, bin_cols) # start training training_arguments = { 'num_epoch':50, 'batch_size':64, 'lr':1e-4, 'eval_metric':'val_loss', 'eval_less_is_better':True, 'output_dir':'./checkpoint' } transtab.train(model, trainset, valset, collate_fn=collate_fn, **training_arguments) Now we have obtained the pretrained model saved in './checkpoint', we can load the model from this path and use it to encode tables. .. code-block:: python # load the pretrained model enc = transtab.build_encoder( binary_columns=bin_cols, checkpoint = './checkpoint' ) Then we can take the whole pretrained model and output the cls token embedding at the last layer's outputs .. code-block:: python # encode tables to sample-level embeddings df = trainset[0] output = enc(df) ================================================ FILE: docs/source/example_pretrain.rst ================================================ Tabular Pretraining =================== When encountering multiple distinct tables which may have different number of classes, performing contrastive pretraining (called Vertical-Partition Contrastive Learning, VPCL in the paper) is often a better choice. This can be done using the transtab contrastive learner model. The full code is available at `Notebook Example `_. .. code-block:: python import transtab # load multiple datasets by passing a list of data names allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(['credit-g','credit-approval']) # build contrastive learner, set supervised=True for supervised VPCL model, collate_fn = transtab.build_contrastive_learner( cat_cols, num_cols, bin_cols, supervised=True, # if take supervised CL num_partition=4, # num of column partitions for pos/neg sampling overlap_ratio=0.5, # specify the overlap ratio of column partitions during the CL ) The function transtab.build_contrastive_learner returns both the CL model and the collate function for the training dataloaders. We then train the model like .. code-block:: python # start contrastive pretraining training training_arguments = { 'num_epoch':50, 'batch_size':64, 'lr':1e-4, 'eval_metric':'val_loss', 'eval_less_is_better':True, 'output_dir':'./checkpoint' # save the pretrained model } # pass the collate function to the train function transtab.train(model, trainset, valset, collate_fn=collate_fn, **training_arguments) After this pretrain completes, we shall build a classifier from the checkpoint. .. code-block:: python # load the pretrained model and finetune on a target dataset allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-approval') # build transtab classifier model, and load from the pretrained dir model = transtab.build_classifier(checkpoint='./checkpoint') # update model's categorical/numerical/binary column dict model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols}) ================================================ FILE: docs/source/example_transfer.rst ================================================ Tabular Transfer Learning ========================= *transtab* is able to leverage the knowledge learned from broad data sources than finetunes on the target data. It is also easy to fulfill it by this package. The full code is available at `Notebook Example `_. .. code-block:: python import transtab # load a dataset and start vanilla supervised training allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-g') # build transtab classifier model model = transtab.build_classifier(cat_cols, num_cols, bin_cols) # start training training_arguments = { 'num_epoch':50, 'eval_metric':'val_loss', 'eval_less_is_better':True, 'output_dir':'./checkpoint' } transtab.train(model, trainset, valset, **training_arguments) Now we have obtained the pretrained model saved in './checkpoint', we can load the model from this path and update the model with new samples and columns. .. code-block:: python # now let's load another data and try to leverage the pretrained model for finetuning allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-approval') # load the pretrained model model.load('./checkpoint') # update model's categorical/numerical/binary column dict model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols}) It should be noted if the finetune data differs the pretrain data on the number of classes, this should be explicitly claimed in the update. .. code-block:: python model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols, 'num_class':2}) Then we can continue to train the model just as same as done for supervised learning. .. code-block:: python transtab.train(model, trainset, valset, **training_arguments) ================================================ FILE: docs/source/fast_train.rst ================================================ Fast Train with TransTab ========================= *transtab* is featured for accepting variable-column tables for training and predicting. This is easy to be done by this package. The full code is available at `Notebook Example `_. .. code-block:: python import transtab # load multiple datasets by passing a list of data names allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(['credit-g','credit-approval']) # build transtab classifier model model = transtab.build_classifier(cat_cols, num_cols, bin_cols) # specify training arguments, take validation loss for early stopping training_arguments = { 'num_epoch':5, 'eval_metric':'val_loss', 'eval_less_is_better':True, 'output_dir':'./checkpoint' } One can take the validation loss on the validation data of the first dataset *credit-g* only: .. code-block:: python transtab.train(model, trainset, valset[0], **training_arguments) or take the macro average loss on the validation set of both two datasets: .. code-block:: python transtab.train(model, trainset, valset, **training_arguments) After the training completes, we can load the best checkpoint judged by validation loss from the predefined *output_dir* and make predictions. .. code-block:: python model.load('./checkpoint') x_test, y_test = testset[0] ypred = transtab.predict(x_test) .. warning:: Under this pure supervised learning setting, all the passed datasets should have the same **number of label classes**. For instance, here *credit-g* and *credit-approval* are both binary classification task. It is because the classifier of `transtab` only keeps one classification head during the training and predicting. ================================================ FILE: docs/source/index.rst ================================================ Welcome to transtab documentation! ================================== `transtab` is an easy-to-use **Python package** for flexible tabular prediction framework. **Tabular data** dominates the applications of machine learning in research & development, including healthcare, finance, advertising, engineering, etc. `transtab` is featured for the following scenarios of tabular predictions: * **Supervised learning**: the vanilla train and predict on tables with the identical columns. * **Transfer learning**: given multiple labeled tables partially share columns, we enhance models for each of those tables by leveraging other tables. * **Incremental learning**: as a table incrementally grows with more columns, we update the existing model to handle the new table with more columns. * **Table Pretraining**: we pretrain models on many tables with distinct columns and identifiers for the target tabular prediction task. * **Zero-shot inference**: we build a model for an unseen table that only has partial overlaps with training tables. .. figure:: ../images/transtab_tasks.png The demonstration of ML modeling on different tabular data settings. Previous tabular methods only do vanilla supervised training or pretraining on the same table due to they only accept **fixed-column tables**. By contrast, \method covers more new tasks (1) to (4) as it accepts **variable-column** tables. The basic usage of `transtab` can be done in a couple of lines: .. code-block:: python import transtab # load dataset by specifying dataset name allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-g') # build classifier model = transtab.build_classifier(cat_cols, num_cols, bin_cols) # start training transtab.train(model, trainset, valset, **training_arguments) # make predictions, df_x is a pd.DataFrame with shape (n, d) # return the predictions ypred with shape (n, 1) if binary classification; # (n, n_class) if multiclass classification. ypred = transtab.predict(model, df_x) It's easy, isn't it? Let's start the journey from the `installation `_ and the `first demo on supervised tabular learning `_ ! We also provide the examples on `tabular transfer learning `_ and `tabular pretraining `_ for the quick start. ---- **Citing transtab**: If you use `transtab` in a scientific publication, we would appreciate citations to the following paper:: @article{wang2022transtab, author = {Wang, Zifeng and Sun, Jimeng}, title = {TransTab: Learning Transferable Tabular Transformers Across Tables}, journal={arXiv preprint arXiv:2205.09328}, year = {2022}, } .. toctree:: :maxdepth: 2 :hidden: :caption: Getting Started install fast_train example_transfer example_pretrain example_encode data_preparation .. toctree:: :maxdepth: 2 :hidden: :caption: Documentation main_func models .. toctree:: :maxdepth: 2 :hidden: :caption: Additional Information about ================================================ FILE: docs/source/install.rst ================================================ Installation ============ *transtab* was tested on Python 3.7+, PyTorch 1.8.0+. Please follow the Installation instructions below for the torch version and CUDA device you are using: `PyTorch Installation Instructions `_. After that, *transtab* can be downloaded directly using **pip**. [Feb 2025, the PyPI version is no longer maintained, please try to install it from github]: .. code-block:: bash pip install git+https://github.com/RyanWangZf/transtab.git Alternatively, you can clone the project and install it from local .. code-block:: bash git clone https://github.com/RyanWangZf/transtab.git cd transtab pip install . **Troubleshooting**: 1. If encountering ``ERROR: Failed building wheel for tokenizers`` on MAC/Linux, please call .. code-block:: bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh then restart the terminal and call ``pip`` again. ================================================ FILE: docs/source/main_func.rst ================================================ Main Functions ============== .. toctree:: load_data build_classifier build_contrastive_learner build_encoder build_extractor train predict ================================================ FILE: docs/source/models.rst ================================================ Models ====== .. toctree:: BaseModel TransTabClassifier TransTabForCL ================================================ FILE: docs/source/transtab.basemodel.rst ================================================ TransTabModel ============= .. automodule:: transtab.modeling_transtab :members: TransTabModel :no-undoc-members: :no-show-inheritance: ================================================ FILE: docs/source/transtab.build_classifier.rst ================================================ build_classifier ================ .. autofunction:: transtab.build_classifier .. warning:: If ``categorical_columns``, ``numerical_columns``, and ``binary_columns`` are **ALL** not specified, the model takes **ALL** as ``categorical columns``, which may undermine the performance significantly. ================================================ FILE: docs/source/transtab.build_contrastive_learner.rst ================================================ build_contrastive_learner ========================= .. autofunction:: transtab.build_contrastive_learner ================================================ FILE: docs/source/transtab.build_encoder.rst ================================================ build_extractor =============== .. autofunction:: transtab.build_encoder The returned feature extractor takes pd.DataFrame as inputs and outputs the encoded sample-level embeddings. .. code-block:: python # build the feature extractor enc = transtab.build_encoder(categorical_columns=['gender'], numerical_columns=['age']) # build a table for inputs df = pd.DataFrame({'age':[1,2], 'gender':['male','female']}) # extract the outputs outputs = enc(df) print(outputs.shape) ''' torch.Size([2, 128]) ''' ================================================ FILE: docs/source/transtab.build_extractor.rst ================================================ build_extractor =============== .. autofunction:: transtab.build_extractor The returned feature extractor takes pd.DataFrame as inputs and outputs the encoded outputs in dict. .. code-block:: python # build the feature extractor extractor = transtab.build_extractor(categorical_columns=['gender'], numerical_columns=['age']) # build a table for inputs df = pd.DataFrame({'age':[1,2], 'gender':['male','female']}) # extract the outputs outputs = extractor(df) print(outputs) ''' { 'x_num': tensor([[1.],[2.]], dtype=torch.float64), 'num_col_input_ids': tensor([[2287]]), 'x_cat_input_ids': tensor([[5907, 3287], [5907, 2931]]), 'x_bin_input_ids': None, 'num_att_mask': tensor([[1]]), 'cat_att_mask': tensor([[1, 1], [1, 1]]) } ''' ================================================ FILE: docs/source/transtab.classifier.rst ================================================ TransTabClassifier ================== .. autoclass:: transtab.modeling_transtab.TransTabClassifier :members: :no-undoc-members: :no-show-inheritance: ================================================ FILE: docs/source/transtab.contrastive.rst ================================================ TransTabForCL ============= .. autoclass:: transtab.modeling_transtab.TransTabForCL :members: :no-undoc-members: :no-show-inheritance: ================================================ FILE: docs/source/transtab.load_data.rst ================================================ load_data ========= .. autofunction:: transtab.load_data *transtab* provides flexible data loading function. It can be used to load arbitrary datasets from `openml `_ supported by `openml.datasets API `_. .. code-block:: python # specify the dataname allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-g') # or specify the dataset index (in openml) allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(31) It can also be used to load datasets from the local device. .. code-block:: python # specify the dataset dir allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('./data/credit-g') Another important feature is to use this function to load multiple datasets .. code-block:: python # specify the dataset dir allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data(['./data/credit-g','./data/credit-approval']) One can also pass ``dataset_config`` to the ``load_data`` function to manipulate the input table directly. .. code-block:: python # customize dataset configuration dataset_config = { 'credit-g':{ 'columns':['a','b','c'], # specify the new columns for the table, should keep the same dimension as the original table. 'cat':['a'], # specify all the categorical columns 'bin':['b'], # specify all the binary columns 'num':['c']} # specify all the numerical columns } allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \ = transtab.load_data('credit-g', dataset_config=dataset_config) While this operation is not recommended. To avoid making errors, you'd better deposit all these configurations to the local following the guidance of `custom dataset `_. ================================================ FILE: docs/source/transtab.predict.rst ================================================ predict ======= .. autofunction:: transtab.predict ================================================ FILE: docs/source/transtab.train.rst ================================================ train ===== .. autofunction:: transtab.train ================================================ FILE: docs/sphinx-commands.txt ================================================ # build html files sphinx-build -b html source build ================================================ FILE: examples/contrastive_learning.ipynb ================================================ { "cells": [ { "cell_type": "code", "execution_count": 1, "id": "0c0001bb", "metadata": {}, "outputs": [], "source": [ "import os\n", "os.chdir('../')\n", "\n", "import transtab\n", "\n", "# set random seed\n", "transtab.random_seed(42)" ] }, { "cell_type": "code", "execution_count": 2, "id": "865b42a8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "########################################\n", "openml data index: 31\n", "load data from credit-g\n", "# data: 1000, # feat: 20, # cate: 11, # bin: 2, # numerical: 7, pos rate: 0.70\n", "########################################\n", "openml data index: 29\n", "load data from credit-approval\n", "# data: 690, # feat: 15, # cate: 9, # bin: 0, # numerical: 6, pos rate: 0.56\n" ] } ], "source": [ "# load multiple datasets by passing a list of data names\n", "allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \\\n", " = transtab.load_data(['credit-g','credit-approval'])\n", "\n", "# build contrastive learner, set supervised=True for supervised VPCL\n", "model, collate_fn = transtab.build_contrastive_learner(\n", " cat_cols, num_cols, bin_cols, \n", " supervised=True, # if take supervised CL\n", " num_partition=4, # num of column partitions for pos/neg sampling\n", " overlap_ratio=0.5, # specify the overlap ratio of column partitions during the CL\n", ")" ] }, { "cell_type": "code", "execution_count": 3, "id": "78d0bc6c", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "1a6a12cd244e4672b360c68222c7b7f8", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Epoch: 0%| | 0/50 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
own_telephoneforeign_workerdurationcredit_amountinstallment_commitmentresidence_sinceageexisting_creditsnum_dependentschecking_statuscredit_historypurposesavings_statusemploymentpersonal_statusother_partiesproperty_magnitudeother_payment_planshousingjob
636010.2941180.0619571.0000000.0000000.1607140.0000000.0no checkingexisting paidradio/tv500<=X<10004<=X<7female div/dep/marnonecarnoneownskilled
182010.2500000.0768681.0000000.3333330.3750000.3333331.0<0all paidnew carno known savings1<=X<4male singlenonelife insurancenoneownunskilled resident
736010.2941180.6223180.0000001.0000000.0714290.3333330.00<=X<200existing paidused car<1001<=X<4female div/dep/marnonecarnonerenthigh qualif/self emp/mgmt
922010.0735290.0614060.6666671.0000000.0535710.0000000.0<0existing paidradio/tv<100<1female div/dep/marnonelife insurancenonerentskilled
511110.4705880.2440850.3333330.3333330.2321430.0000000.0no checkingexisting paidused car<1001<=X<4male singlenoneno known propertynonefor freehigh qualif/self emp/mgmt
...............................................................
845110.2500000.2050180.3333330.6666670.2857140.0000000.00<=X<200existing paidfurniture/equipmentno known savings4<=X<7male singlenonecarnoneownskilled
492010.0294120.0543080.0000000.0000000.1428570.3333330.0no checkingcritical/other existing creditradio/tv100<=X<5001<=X<4female div/dep/marnonelife insurancenoneownskilled
849010.1176470.0252561.0000001.0000000.6785710.0000000.0<0existing paidradio/tv<100>=7male singlenonereal estatestoresownunskilled resident
297000.0882350.0570601.0000000.3333330.4642860.0000000.0no checkingexisting paidnew carno known savings>=7male singleco applicantlife insurancenoneownunskilled resident
98010.4705880.1148341.0000001.0000000.3035710.0000000.00<=X<200critical/other existing creditradio/tv<100>=7male singlenonereal estatenoneownskilled
\n", "

700 rows × 20 columns

\n", "" ], "text/plain": [ " own_telephone foreign_worker duration credit_amount \\\n", "636 0 1 0.294118 0.061957 \n", "182 0 1 0.250000 0.076868 \n", "736 0 1 0.294118 0.622318 \n", "922 0 1 0.073529 0.061406 \n", "511 1 1 0.470588 0.244085 \n", ".. ... ... ... ... \n", "845 1 1 0.250000 0.205018 \n", "492 0 1 0.029412 0.054308 \n", "849 0 1 0.117647 0.025256 \n", "297 0 0 0.088235 0.057060 \n", "98 0 1 0.470588 0.114834 \n", "\n", " installment_commitment residence_since age existing_credits \\\n", "636 1.000000 0.000000 0.160714 0.000000 \n", "182 1.000000 0.333333 0.375000 0.333333 \n", "736 0.000000 1.000000 0.071429 0.333333 \n", "922 0.666667 1.000000 0.053571 0.000000 \n", "511 0.333333 0.333333 0.232143 0.000000 \n", ".. ... ... ... ... \n", "845 0.333333 0.666667 0.285714 0.000000 \n", "492 0.000000 0.000000 0.142857 0.333333 \n", "849 1.000000 1.000000 0.678571 0.000000 \n", "297 1.000000 0.333333 0.464286 0.000000 \n", "98 1.000000 1.000000 0.303571 0.000000 \n", "\n", " num_dependents checking_status credit_history \\\n", "636 0.0 no checking existing paid \n", "182 1.0 <0 all paid \n", "736 0.0 0<=X<200 existing paid \n", "922 0.0 <0 existing paid \n", "511 0.0 no checking existing paid \n", ".. ... ... ... \n", "845 0.0 0<=X<200 existing paid \n", "492 0.0 no checking critical/other existing credit \n", "849 0.0 <0 existing paid \n", "297 0.0 no checking existing paid \n", "98 0.0 0<=X<200 critical/other existing credit \n", "\n", " purpose savings_status employment personal_status \\\n", "636 radio/tv 500<=X<1000 4<=X<7 female div/dep/mar \n", "182 new car no known savings 1<=X<4 male single \n", "736 used car <100 1<=X<4 female div/dep/mar \n", "922 radio/tv <100 <1 female div/dep/mar \n", "511 used car <100 1<=X<4 male single \n", ".. ... ... ... ... \n", "845 furniture/equipment no known savings 4<=X<7 male single \n", "492 radio/tv 100<=X<500 1<=X<4 female div/dep/mar \n", "849 radio/tv <100 >=7 male single \n", "297 new car no known savings >=7 male single \n", "98 radio/tv <100 >=7 male single \n", "\n", " other_parties property_magnitude other_payment_plans housing \\\n", "636 none car none own \n", "182 none life insurance none own \n", "736 none car none rent \n", "922 none life insurance none rent \n", "511 none no known property none for free \n", ".. ... ... ... ... \n", "845 none car none own \n", "492 none life insurance none own \n", "849 none real estate stores own \n", "297 co applicant life insurance none own \n", "98 none real estate none own \n", "\n", " job \n", "636 skilled \n", "182 unskilled resident \n", "736 high qualif/self emp/mgmt \n", "922 skilled \n", "511 high qualif/self emp/mgmt \n", ".. ... \n", "845 skilled \n", "492 skilled \n", "849 unskilled resident \n", "297 unskilled resident \n", "98 skilled \n", "\n", "[700 rows x 20 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainset[0][0]" ] }, { "cell_type": "code", "execution_count": 4, "id": "058f667e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
own_telephoneforeign_workerdurationcredit_amountinstallment_commitmentresidence_sinceageexisting_creditsnum_dependentschecking_statuscredit_historypurposesavings_statusemploymentpersonal_statusother_partiesproperty_magnitudeother_payment_planshousingjob
32110.2058820.3090130.3333330.3333330.1964290.3333330.00<=X<200existing paidnew car100<=X<5001<=X<4male singlenonecarnoneownskilled
924110.2941180.3643670.3333330.0000000.6428570.0000000.0<0all paidfurniture/equipment<100<1male div/sepnonelife insurancebankownskilled
931110.0735290.0781341.0000000.3333330.0535710.0000000.00<=X<200existing paidradio/tv<100<1female div/dep/marnonecarnoneownskilled
796110.2058820.3995270.0000001.0000000.5714290.0000001.0<0existing paidused carno known savings>=7male singlenonelife insurancenonefor freeskilled
226110.6470590.5893580.0000000.3333330.1428570.3333330.00<=X<200existing paidradio/tv>=10004<=X<7male singleco applicantno known propertybankownskilled
...............................................................
380110.2352940.1079561.0000001.0000000.3571430.0000000.0<0existing paidfurniture/equipmentno known savings4<=X<7male singlenonecarnoneownskilled
768110.1176470.1852650.0000001.0000000.1607140.6666670.00<=X<200critical/other existing creditfurniture/equipment<100>=7male singlenonecarnonerentskilled
85110.1176470.0639371.0000000.3333330.1785710.3333330.0no checkingcritical/other existing creditbusiness<1001<=X<4female div/dep/marguarantorreal estatenoneownhigh qualif/self emp/mgmt
527010.0000000.0689450.3333330.0000000.4107140.3333331.0no checkingcritical/other existing creditradio/tv<1004<=X<7male singlenonereal estatenoneownunskilled resident
117000.0882350.1035550.3333330.6666670.1428570.3333330.0<0critical/other existing creditfurniture/equipmentno known savings<1female div/dep/marco applicantreal estatenonerentskilled
\n", "

100 rows × 20 columns

\n", "
" ], "text/plain": [ " own_telephone foreign_worker duration credit_amount \\\n", "32 1 1 0.205882 0.309013 \n", "924 1 1 0.294118 0.364367 \n", "931 1 1 0.073529 0.078134 \n", "796 1 1 0.205882 0.399527 \n", "226 1 1 0.647059 0.589358 \n", ".. ... ... ... ... \n", "380 1 1 0.235294 0.107956 \n", "768 1 1 0.117647 0.185265 \n", "85 1 1 0.117647 0.063937 \n", "527 0 1 0.000000 0.068945 \n", "117 0 0 0.088235 0.103555 \n", "\n", " installment_commitment residence_since age existing_credits \\\n", "32 0.333333 0.333333 0.196429 0.333333 \n", "924 0.333333 0.000000 0.642857 0.000000 \n", "931 1.000000 0.333333 0.053571 0.000000 \n", "796 0.000000 1.000000 0.571429 0.000000 \n", "226 0.000000 0.333333 0.142857 0.333333 \n", ".. ... ... ... ... \n", "380 1.000000 1.000000 0.357143 0.000000 \n", "768 0.000000 1.000000 0.160714 0.666667 \n", "85 1.000000 0.333333 0.178571 0.333333 \n", "527 0.333333 0.000000 0.410714 0.333333 \n", "117 0.333333 0.666667 0.142857 0.333333 \n", "\n", " num_dependents checking_status credit_history \\\n", "32 0.0 0<=X<200 existing paid \n", "924 0.0 <0 all paid \n", "931 0.0 0<=X<200 existing paid \n", "796 1.0 <0 existing paid \n", "226 0.0 0<=X<200 existing paid \n", ".. ... ... ... \n", "380 0.0 <0 existing paid \n", "768 0.0 0<=X<200 critical/other existing credit \n", "85 0.0 no checking critical/other existing credit \n", "527 1.0 no checking critical/other existing credit \n", "117 0.0 <0 critical/other existing credit \n", "\n", " purpose savings_status employment personal_status \\\n", "32 new car 100<=X<500 1<=X<4 male single \n", "924 furniture/equipment <100 <1 male div/sep \n", "931 radio/tv <100 <1 female div/dep/mar \n", "796 used car no known savings >=7 male single \n", "226 radio/tv >=1000 4<=X<7 male single \n", ".. ... ... ... ... \n", "380 furniture/equipment no known savings 4<=X<7 male single \n", "768 furniture/equipment <100 >=7 male single \n", "85 business <100 1<=X<4 female div/dep/mar \n", "527 radio/tv <100 4<=X<7 male single \n", "117 furniture/equipment no known savings <1 female div/dep/mar \n", "\n", " other_parties property_magnitude other_payment_plans housing \\\n", "32 none car none own \n", "924 none life insurance bank own \n", "931 none car none own \n", "796 none life insurance none for free \n", "226 co applicant no known property bank own \n", ".. ... ... ... ... \n", "380 none car none own \n", "768 none car none rent \n", "85 guarantor real estate none own \n", "527 none real estate none own \n", "117 co applicant real estate none rent \n", "\n", " job \n", "32 skilled \n", "924 skilled \n", "931 skilled \n", "796 skilled \n", "226 skilled \n", ".. ... \n", "380 skilled \n", "768 skilled \n", "85 high qualif/self emp/mgmt \n", "527 unskilled resident \n", "117 skilled \n", "\n", "[100 rows x 20 columns]" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "valset[0][0]" ] }, { "cell_type": "code", "execution_count": 5, "id": "af2eed94", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3018579e308d4eb995ed65b3581b7f06", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Epoch: 0%| | 0/50 [00:00)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Then take the encoder to get the input embedding\n", "df = trainset[0]\n", "output = enc(df)\n", "print(output.shape)\n", "output[:2]" ] }, { "cell_type": "code", "execution_count": 6, "id": "4aadae44", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
own_telephoneforeign_workerdurationcredit_amountinstallment_commitmentresidence_sinceageexisting_creditsnum_dependentschecking_statuscredit_historypurposesavings_statusemploymentpersonal_statusother_partiesproperty_magnitudeother_payment_planshousingjob
636010.2941180.0619571.0000000.0000000.1607140.0000000.0no checkingexisting paidradio/tv500<=X<10004<=X<7female div/dep/marnonecarnoneownskilled
182010.2500000.0768681.0000000.3333330.3750000.3333331.0<0all paidnew carno known savings1<=X<4male singlenonelife insurancenoneownunskilled resident
736010.2941180.6223180.0000001.0000000.0714290.3333330.00<=X<200existing paidused car<1001<=X<4female div/dep/marnonecarnonerenthigh qualif/self emp/mgmt
922010.0735290.0614060.6666671.0000000.0535710.0000000.0<0existing paidradio/tv<100<1female div/dep/marnonelife insurancenonerentskilled
511110.4705880.2440850.3333330.3333330.2321430.0000000.0no checkingexisting paidused car<1001<=X<4male singlenoneno known propertynonefor freehigh qualif/self emp/mgmt
\n", "
" ], "text/plain": [ " own_telephone foreign_worker duration credit_amount \\\n", "636 0 1 0.294118 0.061957 \n", "182 0 1 0.250000 0.076868 \n", "736 0 1 0.294118 0.622318 \n", "922 0 1 0.073529 0.061406 \n", "511 1 1 0.470588 0.244085 \n", "\n", " installment_commitment residence_since age existing_credits \\\n", "636 1.000000 0.000000 0.160714 0.000000 \n", "182 1.000000 0.333333 0.375000 0.333333 \n", "736 0.000000 1.000000 0.071429 0.333333 \n", "922 0.666667 1.000000 0.053571 0.000000 \n", "511 0.333333 0.333333 0.232143 0.000000 \n", "\n", " num_dependents checking_status credit_history purpose \\\n", "636 0.0 no checking existing paid radio/tv \n", "182 1.0 <0 all paid new car \n", "736 0.0 0<=X<200 existing paid used car \n", "922 0.0 <0 existing paid radio/tv \n", "511 0.0 no checking existing paid used car \n", "\n", " savings_status employment personal_status other_parties \\\n", "636 500<=X<1000 4<=X<7 female div/dep/mar none \n", "182 no known savings 1<=X<4 male single none \n", "736 <100 1<=X<4 female div/dep/mar none \n", "922 <100 <1 female div/dep/mar none \n", "511 <100 1<=X<4 male single none \n", "\n", " property_magnitude other_payment_plans housing \\\n", "636 car none own \n", "182 life insurance none own \n", "736 car none rent \n", "922 life insurance none rent \n", "511 no known property none for free \n", "\n", " job \n", "636 skilled \n", "182 unskilled resident \n", "736 high qualif/self emp/mgmt \n", "922 skilled \n", "511 high qualif/self emp/mgmt " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": 8, "id": "4f3e1e91", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2022-08-31 14:16:28.124 | INFO | transtab.modeling_transtab:load:222 - load feature extractor from ./checkpoint/extractor/extractor.json\n", "2022-08-31 14:16:28.134 | INFO | transtab.modeling_transtab:load:523 - missing keys: []\n", "2022-08-31 14:16:28.135 | INFO | transtab.modeling_transtab:load:524 - unexpected keys: []\n", "2022-08-31 14:16:28.136 | INFO | transtab.modeling_transtab:load:525 - load model from ./checkpoint\n" ] } ], "source": [ "# Second, if we only want to the embeded token level embeddings (embeddings before going to transformers)\n", "enc = transtab.build_encoder(\n", " binary_columns=bin_cols,\n", " checkpoint = './checkpoint',\n", " num_layer = 0,\n", ")" ] }, { "cell_type": "code", "execution_count": 12, "id": "39a0172b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "torch.Size([700, 85, 128])\n" ] }, { "data": { "text/plain": [ "tensor([[[ 0.1370, 0.0427, -0.0106, ..., -0.0806, 0.0518, -0.1315],\n", " [ 0.0657, 0.0341, -0.0128, ..., -0.0207, 0.0102, -0.0046],\n", " [ 0.1494, 0.4290, 0.2463, ..., 0.1992, -0.0848, -0.0840],\n", " ...,\n", " [ 1.1575, 0.0165, 0.9202, ..., -0.2052, 1.0815, -1.0268],\n", " [ 1.1575, 0.0165, 0.9202, ..., -0.2052, 1.0815, -1.0268],\n", " [ 1.1575, 0.0165, 0.9202, ..., -0.2052, 1.0815, -1.0268]],\n", "\n", " [[ 0.1204, 0.0388, -0.0098, ..., -0.0738, 0.0400, -0.1099],\n", " [ 0.0752, 0.0383, -0.0145, ..., -0.0174, 0.0190, -0.0085],\n", " [ 0.1494, 0.4290, 0.2463, ..., 0.1992, -0.0848, -0.0840],\n", " ...,\n", " [ 1.1575, 0.0165, 0.9202, ..., -0.2052, 1.0815, -1.0268],\n", " [ 1.1575, 0.0165, 0.9202, ..., -0.2052, 1.0815, -1.0268],\n", " [ 1.1575, 0.0165, 0.9202, ..., -0.2052, 1.0815, -1.0268]]],\n", " device='cuda:0', grad_fn=)" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "output = enc(df)\n", "print(output['embedding'].shape)\n", "output['embedding'][:2]" ] }, { "cell_type": "code", "execution_count": null, "id": "55936f1e", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" } }, "nbformat": 4, "nbformat_minor": 5 } ================================================ FILE: examples/transfer_learning.ipynb ================================================ { "cells": [ { "cell_type": "code", "execution_count": 6, "id": "134f979d", "metadata": {}, "outputs": [], "source": [ "import os\n", "os.chdir('../')\n", "\n", "import transtab\n", "\n", "# set random seed\n", "transtab.random_seed(42)" ] }, { "cell_type": "code", "execution_count": 7, "id": "42c60011", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "########################################\n", "openml data index: 31\n", "load data from credit-g\n", "# data: 1000, # feat: 20, # cate: 11, # bin: 2, # numerical: 7, pos rate: 0.70\n", "########################################\n", "openml data index: 29\n", "load data from credit-approval\n", "# data: 690, # feat: 15, # cate: 9, # bin: 0, # numerical: 6, pos rate: 0.56\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "dd62a8df24d14e22a69d77088bd1b220", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Epoch: 0%| | 0/50 [00:00=2.4.0 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (2.5.0)\n", "Requirement already satisfied: xmltodict in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (0.14.2)\n", "Requirement already satisfied: requests in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (2.32.3)\n", "Requirement already satisfied: scikit-learn>=0.18 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (1.6.1)\n", "Requirement already satisfied: python-dateutil in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (2.9.0.post0)\n", "Requirement already satisfied: pandas>=1.0.0 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (2.2.3)\n", "Requirement already satisfied: scipy>=0.13.3 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (1.15.2)\n", "Requirement already satisfied: numpy>=1.6.2 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (2.2.3)\n", "Requirement already satisfied: minio in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (7.2.15)\n", "Requirement already satisfied: pyarrow in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (19.0.1)\n", "Requirement already satisfied: tqdm in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (4.67.1)\n", "Requirement already satisfied: packaging in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from openml) (24.2)\n", "Requirement already satisfied: pytz>=2020.1 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from pandas>=1.0.0->openml) (2025.1)\n", "Requirement already satisfied: tzdata>=2022.7 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from pandas>=1.0.0->openml) (2025.1)\n", "Requirement already satisfied: six>=1.5 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from python-dateutil->openml) (1.17.0)\n", "Requirement already satisfied: joblib>=1.2.0 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from scikit-learn>=0.18->openml) (1.4.2)\n", "Requirement already satisfied: threadpoolctl>=3.1.0 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from scikit-learn>=0.18->openml) (3.5.0)\n", "Requirement already satisfied: certifi in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from minio->openml) (2025.1.31)\n", "Requirement already satisfied: urllib3 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from minio->openml) (2.3.0)\n", "Requirement already satisfied: argon2-cffi in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from minio->openml) (23.1.0)\n", "Requirement already satisfied: pycryptodome in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from minio->openml) (3.21.0)\n", "Requirement already satisfied: typing-extensions in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from minio->openml) (4.12.2)\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from requests->openml) (3.4.1)\n", "Requirement already satisfied: idna<4,>=2.5 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from requests->openml) (3.10)\n", "Requirement already satisfied: argon2-cffi-bindings in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from argon2-cffi->minio->openml) (21.2.0)\n", "Requirement already satisfied: cffi>=1.0.1 in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from argon2-cffi-bindings->argon2-cffi->minio->openml) (1.17.1)\n", "Requirement already satisfied: pycparser in /home/zifengw2/miniconda3/envs/digitaltwin/lib/python3.10/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->minio->openml) (2.22)\n", "########################################\n" ] }, { "ename": "ImportError", "evalue": "OpenML is required for this functionality. Please install it with: pip install openml", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[7], line 3\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# load a dataset and start vanilla supervised training\u001b[39;00m\n\u001b[1;32m 2\u001b[0m get_ipython()\u001b[38;5;241m.\u001b[39msystem(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mpip install openml\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \u001b[38;5;241m=\u001b[39m \u001b[43mtranstab\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mload_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mcredit-g\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mcredit-approval\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/github/transtab/transtab/dataset.py:95\u001b[0m, in \u001b[0;36mload_data\u001b[0;34m(dataname, dataset_config, encode_cat, data_cut, seed)\u001b[0m\n\u001b[1;32m 92\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m dataname_ \u001b[38;5;129;01min\u001b[39;00m dataname:\n\u001b[1;32m 93\u001b[0m data_config \u001b[38;5;241m=\u001b[39m dataset_config\u001b[38;5;241m.\u001b[39mget(dataname_, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m 94\u001b[0m allset, trainset, valset, testset, cat_cols, num_cols, bin_cols \u001b[38;5;241m=\u001b[39m \\\n\u001b[0;32m---> 95\u001b[0m \u001b[43mload_single_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdataname_\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdataset_config\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdata_config\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mencode_cat\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mencode_cat\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdata_cut\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdata_cut\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mseed\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mseed\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 96\u001b[0m num_col_list\u001b[38;5;241m.\u001b[39mextend(num_cols)\n\u001b[1;32m 97\u001b[0m cat_col_list\u001b[38;5;241m.\u001b[39mextend(cat_cols)\n", "File \u001b[0;32m~/github/transtab/transtab/dataset.py:159\u001b[0m, in \u001b[0;36mload_single_data\u001b[0;34m(dataname, dataset_config, encode_cat, data_cut, seed)\u001b[0m\n\u001b[1;32m 157\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 158\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m _has_openml:\n\u001b[0;32m--> 159\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m(\n\u001b[1;32m 160\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mOpenML is required for this functionality. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 161\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPlease install it with: pip install openml\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 162\u001b[0m )\n\u001b[1;32m 163\u001b[0m dataset \u001b[38;5;241m=\u001b[39m openml\u001b[38;5;241m.\u001b[39mdatasets\u001b[38;5;241m.\u001b[39mget_dataset(dataname)\n\u001b[1;32m 164\u001b[0m X,y,categorical_indicator, attribute_names \u001b[38;5;241m=\u001b[39m dataset\u001b[38;5;241m.\u001b[39mget_data(dataset_format\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mdataframe\u001b[39m\u001b[38;5;124m'\u001b[39m, target\u001b[38;5;241m=\u001b[39mdataset\u001b[38;5;241m.\u001b[39mdefault_target_attribute)\n", "\u001b[0;31mImportError\u001b[0m: OpenML is required for this functionality. Please install it with: pip install openml" ] } ], "source": [ "# load a dataset and start vanilla supervised training\n", "# !pip install openml\n", "allset, trainset, valset, testset, cat_cols, num_cols, bin_cols = transtab.load_data(['credit-g', 'credit-approval'])\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "521fb369", "metadata": {}, "outputs": [ { "ename": "NameError", "evalue": "name 'trainset' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[3], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m trainset_reg \u001b[38;5;241m=\u001b[39m [(\u001b[43mtrainset\u001b[49m[\u001b[38;5;241m0\u001b[39m][\u001b[38;5;241m0\u001b[39m], pd\u001b[38;5;241m.\u001b[39mSeries(np\u001b[38;5;241m.\u001b[39mrandom\u001b[38;5;241m.\u001b[39mrandn(trainset[\u001b[38;5;241m0\u001b[39m][\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m]))), (trainset[\u001b[38;5;241m1\u001b[39m][\u001b[38;5;241m0\u001b[39m], pd\u001b[38;5;241m.\u001b[39mSeries(np\u001b[38;5;241m.\u001b[39mrandom\u001b[38;5;241m.\u001b[39mrandn(trainset[\u001b[38;5;241m1\u001b[39m][\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m0\u001b[39m])))]\n", "\u001b[0;31mNameError\u001b[0m: name 'trainset' is not defined" ] } ], "source": [ "trainset_reg = [(trainset[0][0], pd.Series(np.random.randn(trainset[0][0].shape[0]))), (trainset[1][0], pd.Series(np.random.randn(trainset[1][0].shape[0])))]" ] }, { "cell_type": "code", "execution_count": 38, "id": "cadc940f", "metadata": {}, "outputs": [], "source": [ "valset_reg = [(valset[0][0], pd.Series(np.random.randn(valset[0][0].shape[0]))), (valset[1][0], pd.Series(np.random.randn(valset[1][0].shape[0])))]\n", "testset_reg = [(testset[0][0], pd.Series(np.random.randn(testset[0][0].shape[0]))), (testset[1][0], pd.Series(np.random.randn(testset[1][0].shape[0])))]" ] }, { "cell_type": "code", "execution_count": 39, "id": "42c60011", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 2%|▏ | 1/50 [00:01<01:33, 1.91s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 0, test val_loss: 1.372377\n", "epoch: 0, train loss: 6.7940, lr: 0.000100, spent: 1.9 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 4%|▍ | 2/50 [00:03<01:22, 1.72s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 1, test val_loss: 1.184756\n", "epoch: 1, train loss: 6.0480, lr: 0.000100, spent: 3.5 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 6%|▌ | 3/50 [00:05<01:18, 1.66s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 2, test val_loss: 1.194661\n", "EarlyStopping counter: 1 out of 5\n", "epoch: 2, train loss: 6.1002, lr: 0.000100, spent: 5.1 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 8%|▊ | 4/50 [00:06<01:14, 1.63s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 3, test val_loss: 1.218926\n", "EarlyStopping counter: 2 out of 5\n", "epoch: 3, train loss: 5.8850, lr: 0.000100, spent: 6.7 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 10%|█ | 5/50 [00:08<01:12, 1.61s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 4, test val_loss: 1.198663\n", "EarlyStopping counter: 3 out of 5\n", "epoch: 4, train loss: 5.9642, lr: 0.000100, spent: 8.3 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 12%|█▏ | 6/50 [00:09<01:10, 1.61s/it]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 5, test val_loss: 1.205427\n", "EarlyStopping counter: 4 out of 5\n", "epoch: 5, train loss: 5.8004, lr: 0.000100, spent: 9.9 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 12%|█▏ | 6/50 [00:11<01:24, 1.92s/it]\n", "\u001b[32m2024-03-08 16:32:55.367\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m136\u001b[0m - \u001b[1mload best at last from ./checkpoint\u001b[0m\n", "\u001b[32m2024-03-08 16:32:55.378\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.trainer\u001b[0m:\u001b[36msave_model\u001b[0m:\u001b[36m247\u001b[0m - \u001b[1msaving model checkpoint to ./checkpoint\u001b[0m\n", "\u001b[32m2024-03-08 16:32:55.471\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m141\u001b[0m - \u001b[1mtraining complete, cost 11.6 secs.\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 6, test val_loss: 1.198403\n", "EarlyStopping counter: 5 out of 5\n", "early stopped\n" ] } ], "source": [ "\n", "# build transtab classifier model\n", "model = transtab.build_regressor(cat_cols, num_cols, bin_cols, device='cpu')\n", "\n", "# start training\n", "training_arguments = {\n", " 'num_epoch':50,\n", " 'eval_metric':'val_loss',\n", " 'eval_less_is_better':True,\n", " 'output_dir':'./checkpoint',\n", " 'batch_size':128,\n", " 'lr':1e-4,\n", " 'weight_decay':1e-4,\n", " }\n", "transtab.train(model, trainset_reg[0], valset_reg[0], **training_arguments)\n", "\n", "# save model\n", "model.save('./ckpt/pretrained')" ] }, { "cell_type": "code", "execution_count": 40, "id": "d6bdc971", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m2024-03-08 16:33:11.448\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.modeling_transtab\u001b[0m:\u001b[36mload\u001b[0m:\u001b[36m787\u001b[0m - \u001b[1mmissing keys: []\u001b[0m\n", "\u001b[32m2024-03-08 16:33:11.448\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.modeling_transtab\u001b[0m:\u001b[36mload\u001b[0m:\u001b[36m788\u001b[0m - \u001b[1munexpected keys: []\u001b[0m\n", "\u001b[32m2024-03-08 16:33:11.449\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.modeling_transtab\u001b[0m:\u001b[36mload\u001b[0m:\u001b[36m789\u001b[0m - \u001b[1mload model from ./ckpt/pretrained\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\u001b[32m2024-03-08 16:33:11.468\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.modeling_transtab\u001b[0m:\u001b[36mload\u001b[0m:\u001b[36m222\u001b[0m - \u001b[1mload feature extractor from ./ckpt/pretrained/extractor/extractor.json\u001b[0m\n", "\u001b[32m2024-03-08 16:33:11.470\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.modeling_transtab\u001b[0m:\u001b[36m_adapt_to_new_num_class\u001b[0m:\u001b[36m886\u001b[0m - \u001b[1mBuild a new classifier with num 2 classes outputs, need further finetune to work.\u001b[0m\n" ] } ], "source": [ "# now let's use another data and try to leverage the pretrained model for finetuning\n", "# here we have loaded the required data `credit-approval` before, no need to load again.\n", "\n", "# load the pretrained model\n", "model.load('./ckpt/pretrained')\n", "\n", "# update model's categorical/numerical/binary column dict\n", "# need to specify the number of classes if the new dataset has different # of classes from the \n", "# pretrained one.\n", "model.update({'cat':cat_cols,'num':num_cols,'bin':bin_cols, 'num_class':2})" ] }, { "cell_type": "code", "execution_count": 44, "id": "f399d02e", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 2%|▏ | 1/50 [00:00<00:37, 1.32it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 0, test mse: 0.814842\n", "epoch: 0, train loss: 2.9249, lr: 0.000200, spent: 0.8 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 4%|▍ | 2/50 [00:01<00:31, 1.54it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 1, test mse: 0.803411\n", "EarlyStopping counter: 1 out of 5\n", "epoch: 1, train loss: 0.1003, lr: 0.000200, spent: 1.3 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 6%|▌ | 3/50 [00:01<00:29, 1.57it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 2, test mse: 0.802998\n", "EarlyStopping counter: 2 out of 5\n", "epoch: 2, train loss: -0.3084, lr: 0.000200, spent: 2.0 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 8%|▊ | 4/50 [00:02<00:28, 1.61it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 3, test mse: 0.802881\n", "EarlyStopping counter: 3 out of 5\n", "epoch: 3, train loss: -0.3803, lr: 0.000200, spent: 2.6 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 10%|█ | 5/50 [00:03<00:28, 1.60it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 4, test mse: 0.802826\n", "EarlyStopping counter: 4 out of 5\n", "epoch: 4, train loss: -0.2638, lr: 0.000200, spent: 3.2 secs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Epoch: 10%|█ | 5/50 [00:03<00:33, 1.34it/s]\n", "\u001b[32m2024-03-08 16:37:52.614\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m136\u001b[0m - \u001b[1mload best at last from ./checkpoint\u001b[0m\n", "\u001b[32m2024-03-08 16:37:52.621\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.trainer\u001b[0m:\u001b[36msave_model\u001b[0m:\u001b[36m247\u001b[0m - \u001b[1msaving model checkpoint to ./checkpoint\u001b[0m\n", "\u001b[32m2024-03-08 16:37:52.718\u001b[0m | \u001b[1mINFO \u001b[0m | \u001b[36mtranstab.trainer\u001b[0m:\u001b[36mtrain\u001b[0m:\u001b[36m141\u001b[0m - \u001b[1mtraining complete, cost 3.9 secs.\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "epoch: 5, test mse: 0.802803\n", "EarlyStopping counter: 5 out of 5\n", "early stopped\n" ] } ], "source": [ "# start training\n", "training_arguments = {\n", " 'num_epoch':50,\n", " 'eval_metric':'mse',\n", " 'eval_less_is_better':False,\n", " 'output_dir':'./checkpoint',\n", " 'batch_size':128,\n", " 'lr':2e-4,\n", " }\n", "\n", "transtab.train(model, trainset_reg[1], valset_reg[1], **training_arguments)\n" ] }, { "cell_type": "code", "execution_count": 46, "id": "3aa87021", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.9819256995837686\n" ] } ], "source": [ "# evaluation\n", "x_test, y_test = testset_reg[1]\n", "ypred = transtab.predict(model, x_test, y_test)\n", "transtab.evaluate(ypred, y_test, metric='mse')\n", "\n", "from sklearn.metrics import mean_squared_error\n", "print(mean_squared_error(y_test, ypred))" ] }, { "cell_type": "code", "execution_count": null, "id": "d4bf1d31", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "digitaltwin", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.16" } }, "nbformat": 4, "nbformat_minor": 5 } ================================================ FILE: pypi_build_commands.txt ================================================ # This is a command list for building pypi packages python setup.py sdist bdist_wheel twine check dist/* # upload to pypi-test python -m twine upload --repository-url https://test.pypi.org/legacy/ dist/* # install from test-pypi pip install --index-url https://test.pypi.org/simple/ transtab==0.0.2c # upload to pypi twine upload dist/* ================================================ FILE: requirements.txt ================================================ numpy scikit_learn setuptools transformers<=4.30.0 tqdm pandas>=1.3.0 openml>=0.10.0 ================================================ FILE: setup.py ================================================ import os import setuptools this_directory = os.path.abspath(os.path.dirname(__file__)) with open("README.md", "r") as f: long_description = f.read() # read the contents of requirements.txt with open(os.path.join(this_directory, 'requirements.txt'), encoding='utf-8') as f: requirements = f.read().splitlines() setuptools.setup( name = 'transtab', version = '0.0.7', author = 'Zifeng Wang', author_email = 'zifengw2@illinois.edu', description = 'A flexible tabular prediction model that handles variable-column input tables.', url = 'https://github.com/RyanWangZf/transtab', keywords=['tabular data', 'machine learning', 'data mining', 'data science'], long_description=long_description, long_description_content_type='text/markdown', packages=setuptools.find_packages(exclude=['test']), install_requires=requirements, classifiers=[ "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", ], ) ================================================ FILE: transtab/__init__.py ================================================ name = 'transtab' version = '0.0.6' from .transtab import * ================================================ FILE: transtab/constants.py ================================================ # Name of the files used for checkpointing TRAINING_ARGS_NAME = "training_args.json" TRAINER_STATE_NAME = "trainer_state.json" OPTIMIZER_NAME = "optimizer.pt" SCHEDULER_NAME = "scheduler.pt" WEIGHTS_NAME = "pytorch_model.bin" TOKENIZER_DIR = 'tokenizer' EXTRACTOR_STATE_DIR = 'extractor' EXTRACTOR_STATE_NAME = 'extractor.json' INPUT_ENCODER_NAME = 'input_encoder.bin' ================================================ FILE: transtab/dataset.py ================================================ import os import pdb import pandas as pd import numpy as np from sklearn.preprocessing import LabelEncoder, OrdinalEncoder, MinMaxScaler from sklearn.model_selection import train_test_split try: import openml _has_openml = True except ImportError: _has_openml = False import logging logger = logging.getLogger(__name__) OPENML_DATACONFIG = { 'credit-g': {'bin': ['own_telephone', 'foreign_worker']}, } EXAMPLE_DATACONFIG = { "example": { "bin": ["bin1", "bin2"], "cat": ["cat1", "cat2"], "num": ["num1", "num2"], "cols": ["bin1", "bin2", "cat1", "cat2", "num1", "num2"], "binary_indicator": ["1", "yes", "true", "positive", "t", "y"], "data_split_idx": { "train":[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], "val":[10, 11, 12, 13, 14, 15, 16, 17, 18, 19], "test":[20, 21, 22, 23, 24, 25, 26, 27, 28, 29], } } } def load_data(dataname, dataset_config=None, encode_cat=False, data_cut=None, seed=123): '''Load datasets from the local device or from openml.datasets. Parameters ---------- dataname: str or int the dataset name/index intended to be loaded from openml. or the directory to the local dataset. dataset_config: dict the dataset configuration to specify for loading. Please note that this variable will override the configuration loaded from the local files or from the openml.dataset. encode_cat: bool whether encoder the categorical/binary columns to be discrete indices, keep False for TransTab models. data_cut: int how many to split the raw tables into partitions equally; set None will not execute partition. seed: int the random seed set to ensure the fixed train/val/test split. Returns ------- all_list: list or tuple the complete dataset, be (x,y) or [(x1,y1),(x2,y2),...]. train_list: list or tuple the train dataset, be (x,y) or [(x1,y1),(x2,y2),...]. val_list: list or tuple the validation dataset, be (x,y) or [(x1,y1),(x2,y2),...]. test_list: list the test dataset, be (x,y) or [(x1,y1),(x2,y2),...]. cat_col_list: list the list of categorical column names. num_col_list: list the list of numerical column names. bin_col_list: list the list of binary column names. ''' if dataset_config is None: dataset_config = OPENML_DATACONFIG if isinstance(dataname, str): # load a single tabular data return load_single_data(dataname=dataname, dataset_config=dataset_config, encode_cat=encode_cat, data_cut=data_cut, seed=seed) if isinstance(dataname, list): # load a list of datasets, combine together and outputs num_col_list, cat_col_list, bin_col_list = [], [], [] all_list = [] train_list, val_list, test_list = [], [], [] for dataname_ in dataname: data_config = dataset_config.get(dataname_, None) allset, trainset, valset, testset, cat_cols, num_cols, bin_cols = \ load_single_data(dataname_, dataset_config=data_config, encode_cat=encode_cat, data_cut=data_cut, seed=seed) num_col_list.extend(num_cols) cat_col_list.extend(cat_cols) bin_col_list.extend(bin_cols) all_list.append(allset) train_list.append(trainset) val_list.append(valset) test_list.append(testset) return all_list, train_list, val_list, test_list, cat_col_list, num_col_list, bin_col_list def load_single_data(dataname, dataset_config=None, encode_cat=False, data_cut=None, seed=123): '''Load tabular dataset from local or from openml public database. args: dataname: Can either be the data directory on `./data/{dataname}` or the dataname which can be found from the openml database. dataset_config: A dict like {'dataname':{'bin': [col1,col2,...]}} to indicate the binary columns for the data obtained from openml. Also can be used to {'dataname':{'cols':[col1,col2,..]}} to assign a new set of column names to the data encode_cat: Set `False` if we are using transtab, otherwise we set it True to encode categorical values into indexes. data_cut: The number of cuts of the training set. Cut is performed on both rows and columns. outputs: allset: (X,y) that contains all samples of this dataset trainset, valset, testset: the train/val/test split num_cols, cat_cols, bin_cols: the list of numerical/categorical/binary column names ''' print('####'*10) if os.path.exists(dataname): print(f'load from local data dir {dataname}') filename = os.path.join(dataname, 'data_processed.csv') df = pd.read_csv(filename, index_col=0) y = df['target_label'] X = df.drop(['target_label'],axis=1) all_cols = [col.lower() for col in X.columns.tolist()] X.columns = all_cols attribute_names = all_cols ftfile = os.path.join(dataname, 'numerical_feature.txt') if os.path.exists(ftfile): with open(ftfile,'r') as f: num_cols = [x.strip().lower() for x in f.readlines()] else: num_cols = [] bnfile = os.path.join(dataname, 'binary_feature.txt') if os.path.exists(bnfile): with open(bnfile,'r') as f: bin_cols = [x.strip().lower() for x in f.readlines()] else: bin_cols = [] cat_cols = [col for col in all_cols if col not in num_cols and col not in bin_cols] # update cols by loading dataset_config if dataset_config is not None: if 'columns' in dataset_config: new_cols = dataset_config['columns'] X.columns = new_cols if 'bin' in dataset_config: bin_cols = dataset_config['bin'] if 'cat' in dataset_config: cat_cols = dataset_config['cat'] if 'num' in dataset_config: num_cols = dataset_config['num'] else: if not _has_openml: raise ImportError( "OpenML is required for this functionality. " "Please install it with: pip install openml" ) dataset = openml.datasets.get_dataset(dataname) X,y,categorical_indicator, attribute_names = dataset.get_data(dataset_format='dataframe', target=dataset.default_target_attribute) if isinstance(dataname, int): openml_list = openml.datasets.list_datasets(output_format="dataframe") # returns a dict dataname = openml_list.loc[openml_list.did == dataname].name.values[0] else: openml_list = openml.datasets.list_datasets(output_format="dataframe") # returns a dict print(f'openml data index: {openml_list.loc[openml_list.name == dataname].index[0]}') print(f'load data from {dataname}') # drop cols which only have one unique value drop_cols = [col for col in attribute_names if X[col].nunique()<=1] all_cols = np.array(attribute_names) categorical_indicator = np.array(categorical_indicator) cat_cols = [col for col in all_cols[categorical_indicator] if col not in drop_cols] num_cols = [col for col in all_cols[~categorical_indicator] if col not in drop_cols] all_cols = [col for col in all_cols if col not in drop_cols] if dataset_config is not None: if 'bin' in dataset_config: bin_cols = [c for c in cat_cols if c in dataset_config['bin']] else: bin_cols = [] cat_cols = [c for c in cat_cols if c not in bin_cols] # encode target label y = LabelEncoder().fit_transform(y.values) y = pd.Series(y,index=X.index) # start processing features # process num if len(num_cols) > 0: for col in num_cols: X[col].fillna(X[col].mode()[0], inplace=True) X[num_cols] = MinMaxScaler().fit_transform(X[num_cols]) if len(cat_cols) > 0: for col in cat_cols: X[col].fillna(X[col].mode()[0], inplace=True) # process cate if encode_cat: X[cat_cols] = OrdinalEncoder().fit_transform(X[cat_cols]) else: X[cat_cols] = X[cat_cols].astype(str) if len(bin_cols) > 0: for col in bin_cols: X[col].fillna(X[col].mode()[0], inplace=True) if 'binary_indicator' in dataset_config: X[bin_cols] = X[bin_cols].astype(str).applymap(lambda x: 1 if x.lower() in dataset_config['binary_indicator'] else 0).values else: X[bin_cols] = X[bin_cols].astype(str).applymap(lambda x: 1 if x.lower() in ['yes','true','1','t'] else 0).values # if no dataset_config given, keep its original format # raise warning if there is not only 0/1 in the binary columns if (~X[bin_cols].isin([0,1])).any().any(): raise ValueError(f'binary columns {bin_cols} contains values other than 0/1.') X = X[bin_cols + num_cols + cat_cols] # rename column names if is given if dataset_config is not None: data_config = dataset_config if 'columns' in data_config: new_cols = data_config['columns'] X.columns = new_cols attribute_names = new_cols if 'bin' in data_config: bin_cols = data_config['bin'] if 'cat' in data_config: cat_cols = data_config['cat'] if 'num' in data_config: num_cols = data_config['num'] # split train/val/test data_split_idx = None if dataset_config is not None: data_split_idx = dataset_config.get('data_split_idx', None) if data_split_idx is not None: train_idx = data_split_idx.get('train', None) val_idx = data_split_idx.get('val', None) test_idx = data_split_idx.get('test', None) if train_idx is None or test_idx is None: raise ValueError('train/test split indices must be provided together') else: train_dataset = X.iloc[train_idx] y_train = y[train_idx] test_dataset = X.iloc[test_idx] y_test = y[test_idx] if val_idx is not None: val_dataset = X.iloc[val_idx] y_val = y[val_idx] else: val_dataset = None y_val = None else: # split train/val/test train_dataset, test_dataset, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y, shuffle=True) val_size = int(len(y)*0.1) val_dataset = train_dataset.iloc[-val_size:] y_val = y_train[-val_size:] train_dataset = train_dataset.iloc[:-val_size] y_train = y_train[:-val_size] if data_cut is not None: np.random.shuffle(all_cols) sp_size=int(len(all_cols)/data_cut) col_splits = np.split(all_cols, range(0,len(all_cols),sp_size))[1:] new_col_splits = [] for split in col_splits: candidate_cols = np.random.choice(np.setdiff1d(all_cols, split), int(sp_size/2), replace=False) new_col_splits.append(split.tolist() + candidate_cols.tolist()) if len(col_splits) > data_cut: for i in range(len(col_splits[-1])): new_col_splits[i] += [col_splits[-1][i]] new_col_splits[i] = np.unique(new_col_splits[i]).tolist() new_col_splits = new_col_splits[:-1] # cut subset trainset_splits = np.array_split(train_dataset, data_cut) train_subset_list = [] for i in range(data_cut): train_subset_list.append( (trainset_splits[i][new_col_splits[i]], y_train.loc[trainset_splits[i].index]) ) print('# data: {}, # feat: {}, # cate: {}, # bin: {}, # numerical: {}, pos rate: {:.2f}'.format(len(X), len(attribute_names), len(cat_cols), len(bin_cols), len(num_cols), (y==1).sum()/len(y))) return (X, y), train_subset_list, (val_dataset,y_val), (test_dataset, y_test), cat_cols, num_cols, bin_cols else: print('# data: {}, # feat: {}, # cate: {}, # bin: {}, # numerical: {}, pos rate: {:.2f}'.format(len(X), len(attribute_names), len(cat_cols), len(bin_cols), len(num_cols), (y==1).sum()/len(y))) return (X,y), (train_dataset,y_train), (val_dataset,y_val), (test_dataset, y_test), cat_cols, num_cols, bin_cols ================================================ FILE: transtab/evaluator.py ================================================ from collections import defaultdict import os import pdb import torch import numpy as np from sklearn.metrics import roc_auc_score, accuracy_score, mean_squared_error from transtab import constants def predict(clf, x_test, y_test=None, return_loss=False, eval_batch_size=256, ): '''Make predictions by TransTabClassifier. Parameters ---------- clf: TransTabClassifier the classifier model to make predictions. x_test: pd.DataFrame input tabular data. y_test: pd.Series target labels for input x_test. will be ignored if ``return_loss=False``. return_loss: bool set True will return the loss if y_test is given. eval_batch_size: int the batch size for inference. Returns ------- pred_all: np.array if ``return_loss=False``, return the predictions made by TransTabClassifier. avg_loss: float if ``return_loss=True``, return the mean loss of the predictions made by TransTabClassifier. ''' clf.eval() pred_list, loss_list = [], [] for i in range(0, len(x_test), eval_batch_size): bs_x_test = x_test.iloc[i:i+eval_batch_size] bs_y_test = y_test.iloc[i:i+eval_batch_size] if y_test is not None else None with torch.no_grad(): logits, loss = clf(bs_x_test, bs_y_test) if loss is not None: loss_list.append(loss.item()) if logits.shape[-1] == 1: # binary classification pred_list.append(logits.sigmoid().detach().cpu().numpy()) else: # multi-class classification pred_list.append(torch.softmax(logits,-1).detach().cpu().numpy()) pred_all = np.concatenate(pred_list, 0) if logits.shape[-1] == 1: pred_all = pred_all.flatten() if return_loss: avg_loss = np.mean(loss_list) return avg_loss else: return pred_all def evaluate(ypred, y_test, metric='auc', seed=123, bootstrap=False): np.random.seed(seed) eval_fn = get_eval_metric_fn(metric) res_list = [] stats_dict = defaultdict(list) if bootstrap: for i in range(10): sub_idx = np.random.choice(np.arange(len(ypred)), len(ypred), replace=True) sub_ypred = ypred[sub_idx] sub_ytest = y_test.iloc[sub_idx] try: sub_res = eval_fn(sub_ytest, sub_ypred) except ValueError: print('evaluation went wrong!') stats_dict[metric].append(sub_res) for key in stats_dict.keys(): stats = stats_dict[key] alpha = 0.95 p = ((1-alpha)/2) * 100 lower = max(0, np.percentile(stats, p)) p = (alpha+((1.0-alpha)/2.0)) * 100 upper = min(1.0, np.percentile(stats, p)) print('{} {:.2f} mean/interval {:.4f}({:.2f})'.format(key, alpha, (upper+lower)/2, (upper-lower)/2)) if key == metric: res_list.append((upper+lower)/2) else: res = eval_fn(y_test, ypred) res_list.append(res) return res_list def get_eval_metric_fn(eval_metric): fn_dict = { 'acc': acc_fn, 'auc': auc_fn, 'mse': mse_fn, 'val_loss': None, } return fn_dict[eval_metric] def acc_fn(y, p): y_p = np.argmax(p, -1) return accuracy_score(y, y_p) def auc_fn(y, p): return roc_auc_score(y, p) def mse_fn(y, p): return mean_squared_error(y, p) class EarlyStopping: """Early stops the training if validation loss doesn't improve after a given patience.""" def __init__(self, patience=7, verbose=False, delta=0, output_dir='ckpt', trace_func=print, less_is_better=False): """ Args: patience (int): How long to wait after last time validation loss improved. Default: 7 verbose (bool): If True, prints a message for each validation loss improvement. Default: False delta (float): Minimum change in the monitored quantity to qualify as an improvement. Default: 0 path (str): Path for the checkpoint to be saved to. Default: 'checkpoint.pt' trace_func (function): trace print function. Default: print less_is_better (bool): If True (e.g., val loss), the metric is less the better. """ self.patience = patience self.verbose = verbose self.counter = 0 self.best_score = None self.early_stop = False self.val_loss_min = np.inf self.delta = delta self.path = output_dir self.trace_func = trace_func self.less_is_better = less_is_better def __call__(self, val_loss, model): if self.patience < 0: # no early stop self.early_stop = False return if self.less_is_better: score = val_loss else: score = -val_loss if self.best_score is None: self.best_score = score self.save_checkpoint(val_loss, model) elif score < self.best_score + self.delta: self.counter += 1 self.trace_func(f'EarlyStopping counter: {self.counter} out of {self.patience}') if self.counter >= self.patience: self.early_stop = True else: self.best_score = score self.save_checkpoint(val_loss, model) self.counter = 0 def save_checkpoint(self, val_loss, model): '''Saves model when validation loss decrease.''' if self.verbose: self.trace_func(f'Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}). Saving model ...') torch.save(model.state_dict(), os.path.join(self.path, constants.WEIGHTS_NAME)) self.val_loss_min = val_loss ================================================ FILE: transtab/modeling_transtab.py ================================================ import os, pdb import math import collections import json from typing import Dict, Optional, Any, Union, Callable, List from transformers import BertTokenizer, BertTokenizerFast import torch from torch import nn from torch import Tensor import torch.nn.init as nn_init import torch.nn.functional as F import numpy as np import pandas as pd from transtab import constants import logging logger = logging.getLogger(__name__) class TransTabWordEmbedding(nn.Module): r''' Encode tokens drawn from column names, categorical and binary features. ''' def __init__(self, vocab_size, hidden_dim, padding_idx=0, hidden_dropout_prob=0, layer_norm_eps=1e-5, ) -> None: super().__init__() self.word_embeddings = nn.Embedding(vocab_size, hidden_dim, padding_idx) nn_init.kaiming_normal_(self.word_embeddings.weight) self.norm = nn.LayerNorm(hidden_dim, eps=layer_norm_eps) self.dropout = nn.Dropout(hidden_dropout_prob) def forward(self, input_ids) -> Tensor: embeddings = self.word_embeddings(input_ids) embeddings = self.norm(embeddings) embeddings = self.dropout(embeddings) return embeddings class TransTabNumEmbedding(nn.Module): r''' Encode tokens drawn from column names and the corresponding numerical features. ''' def __init__(self, hidden_dim) -> None: super().__init__() self.norm = nn.LayerNorm(hidden_dim) self.num_bias = nn.Parameter(Tensor(1, 1, hidden_dim)) # add bias nn_init.uniform_(self.num_bias, a=-1/math.sqrt(hidden_dim), b=1/math.sqrt(hidden_dim)) def forward(self, num_col_emb, x_num_ts, num_mask=None) -> Tensor: '''args: num_col_emb: numerical column embedding, (# numerical columns, emb_dim) x_num_ts: numerical features, (bs, emb_dim) num_mask: the mask for NaN numerical features, (bs, # numerical columns) ''' num_col_emb = num_col_emb.unsqueeze(0).expand((x_num_ts.shape[0],-1,-1)) num_feat_emb = num_col_emb * x_num_ts.unsqueeze(-1).float() + self.num_bias return num_feat_emb class TransTabFeatureExtractor: r''' Process input dataframe to input indices towards transtab encoder, usually used to build dataloader for paralleling loading. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, disable_tokenizer_parallel=False, ignore_duplicate_cols=False, **kwargs, ) -> None: '''args: categorical_columns: a list of categories feature names numerical_columns: a list of numerical feature names binary_columns: a list of yes or no feature names, accept binary indicators like (yes,no); (true,false); (0,1). disable_tokenizer_parallel: true if use extractor for collator function in torch.DataLoader ignore_duplicate_cols: check if exists one col belongs to both cat/num or cat/bin or num/bin, if set `true`, the duplicate cols will be deleted, else throws errors. ''' if os.path.exists('./transtab/tokenizer'): self.tokenizer = BertTokenizerFast.from_pretrained('./transtab/tokenizer') else: self.tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased') self.tokenizer.save_pretrained('./transtab/tokenizer') self.tokenizer.__dict__['model_max_length'] = 512 if disable_tokenizer_parallel: # disable tokenizer parallel os.environ["TOKENIZERS_PARALLELISM"] = "false" self.vocab_size = self.tokenizer.vocab_size self.pad_token_id = self.tokenizer.pad_token_id self.categorical_columns = categorical_columns self.numerical_columns = numerical_columns self.binary_columns = binary_columns self.ignore_duplicate_cols = ignore_duplicate_cols if categorical_columns is not None: self.categorical_columns = list(set(categorical_columns)) if numerical_columns is not None: self.numerical_columns = list(set(numerical_columns)) if binary_columns is not None: self.binary_columns = list(set(binary_columns)) # check if column exists overlap col_no_overlap, duplicate_cols = self._check_column_overlap(self.categorical_columns, self.numerical_columns, self.binary_columns) if not self.ignore_duplicate_cols: for col in duplicate_cols: logger.error(f'Find duplicate cols named `{col}`, please process the raw data or set `ignore_duplicate_cols` to True!') assert col_no_overlap, 'The assigned categorical_columns, numerical_columns, binary_columns should not have overlap! Please check your input.' else: self._solve_duplicate_cols(duplicate_cols) def __call__(self, x, shuffle=False) -> Dict: ''' Parameters ---------- x: pd.DataFrame with column names and features. shuffle: bool if shuffle column order during the training. Returns ------- encoded_inputs: a dict with { 'x_num': tensor contains numerical features, 'num_col_input_ids': tensor contains numerical column tokenized ids, 'x_cat_input_ids': tensor contains categorical column + feature ids, 'x_bin_input_ids': tesnor contains binary column + feature ids, } ''' encoded_inputs = { 'x_num':None, 'num_col_input_ids':None, 'x_cat_input_ids':None, 'x_bin_input_ids':None, } col_names = x.columns.tolist() cat_cols = [c for c in col_names if c in self.categorical_columns] if self.categorical_columns is not None else [] num_cols = [c for c in col_names if c in self.numerical_columns] if self.numerical_columns is not None else [] bin_cols = [c for c in col_names if c in self.binary_columns] if self.binary_columns is not None else [] if len(cat_cols+num_cols+bin_cols) == 0: # take all columns as categorical columns! cat_cols = col_names if shuffle: np.random.shuffle(cat_cols) np.random.shuffle(num_cols) np.random.shuffle(bin_cols) # TODO: # mask out NaN values like done in binary columns if len(num_cols) > 0: x_num = x[num_cols] x_num = x_num.fillna(0) # fill Nan with zero x_num_ts = torch.tensor(x_num.values, dtype=float) num_col_ts = self.tokenizer(num_cols, padding=True, truncation=True, add_special_tokens=False, return_tensors='pt') encoded_inputs['x_num'] = x_num_ts encoded_inputs['num_col_input_ids'] = num_col_ts['input_ids'] encoded_inputs['num_att_mask'] = num_col_ts['attention_mask'] # mask out attention if len(cat_cols) > 0: x_cat = x[cat_cols].astype(str) x_mask = (~pd.isna(x_cat)).astype(int) x_cat = x_cat.fillna('') x_cat = x_cat.apply(lambda x: x.name + ' '+ x) * x_mask # mask out nan features x_cat_str = x_cat.agg(' '.join, axis=1).values.tolist() x_cat_ts = self.tokenizer(x_cat_str, padding=True, truncation=True, add_special_tokens=False, return_tensors='pt') encoded_inputs['x_cat_input_ids'] = x_cat_ts['input_ids'] encoded_inputs['cat_att_mask'] = x_cat_ts['attention_mask'] if len(bin_cols) > 0: x_bin = x[bin_cols] # x_bin should already be integral (binary values in 0 & 1) x_bin_str = x_bin.apply(lambda x: x.name + ' ') * x_bin x_bin_str = x_bin_str.agg(' '.join, axis=1).values.tolist() x_bin_ts = self.tokenizer(x_bin_str, padding=True, truncation=True, add_special_tokens=False, return_tensors='pt') if x_bin_ts['input_ids'].shape[1] > 0: # not all false encoded_inputs['x_bin_input_ids'] = x_bin_ts['input_ids'] encoded_inputs['bin_att_mask'] = x_bin_ts['attention_mask'] return encoded_inputs def save(self, path): '''save the feature extractor configuration to local dir. ''' save_path = os.path.join(path, constants.EXTRACTOR_STATE_DIR) if not os.path.exists(save_path): os.makedirs(save_path) # save tokenizer tokenizer_path = os.path.join(save_path, constants.TOKENIZER_DIR) self.tokenizer.save_pretrained(tokenizer_path) # save other configurations coltype_path = os.path.join(save_path, constants.EXTRACTOR_STATE_NAME) col_type_dict = { 'categorical': self.categorical_columns, 'binary': self.binary_columns, 'numerical': self.numerical_columns, } with open(coltype_path, 'w', encoding='utf-8') as f: f.write(json.dumps(col_type_dict)) def load(self, path): '''load the feature extractor configuration from local dir. ''' tokenizer_path = os.path.join(path, constants.TOKENIZER_DIR) coltype_path = os.path.join(path, constants.EXTRACTOR_STATE_NAME) self.tokenizer = BertTokenizerFast.from_pretrained(tokenizer_path) with open(coltype_path, 'r', encoding='utf-8') as f: col_type_dict = json.loads(f.read()) self.categorical_columns = col_type_dict['categorical'] self.numerical_columns = col_type_dict['numerical'] self.binary_columns = col_type_dict['binary'] logger.info(f'load feature extractor from {coltype_path}') def update(self, cat=None, num=None, bin=None): '''update cat/num/bin column maps. ''' if cat is not None: self.categorical_columns.extend(cat) self.categorical_columns = list(set(self.categorical_columns)) if num is not None: self.numerical_columns.extend(num) self.numerical_columns = list(set(self.numerical_columns)) if bin is not None: self.binary_columns.extend(bin) self.binary_columns = list(set(self.binary_columns)) col_no_overlap, duplicate_cols = self._check_column_overlap(self.categorical_columns, self.numerical_columns, self.binary_columns) if not self.ignore_duplicate_cols: for col in duplicate_cols: logger.error(f'Find duplicate cols named `{col}`, please process the raw data or set `ignore_duplicate_cols` to True!') assert col_no_overlap, 'The assigned categorical_columns, numerical_columns, binary_columns should not have overlap! Please check your input.' else: self._solve_duplicate_cols(duplicate_cols) def _check_column_overlap(self, cat_cols=None, num_cols=None, bin_cols=None): all_cols = [] if cat_cols is not None: all_cols.extend(cat_cols) if num_cols is not None: all_cols.extend(num_cols) if bin_cols is not None: all_cols.extend(bin_cols) org_length = len(all_cols) if org_length == 0: logger.warning('No cat/num/bin cols specified, will take ALL columns as categorical! Ignore this warning if you specify the `checkpoint` to load the model.') return True, [] unq_length = len(list(set(all_cols))) duplicate_cols = [item for item, count in collections.Counter(all_cols).items() if count > 1] return org_length == unq_length, duplicate_cols def _solve_duplicate_cols(self, duplicate_cols): for col in duplicate_cols: logger.warning('Find duplicate cols named `{col}`, will ignore it during training!') if col in self.categorical_columns: self.categorical_columns.remove(col) self.categorical_columns.append(f'[cat]{col}') if col in self.numerical_columns: self.numerical_columns.remove(col) self.numerical_columns.append(f'[num]{col}') if col in self.binary_columns: self.binary_columns.remove(col) self.binary_columns.append(f'[bin]{col}') class TransTabFeatureProcessor(nn.Module): r''' Process inputs from feature extractor to map them to embeddings. ''' def __init__(self, vocab_size=None, hidden_dim=128, hidden_dropout_prob=0, pad_token_id=0, device='cuda:0', ) -> None: '''args: categorical_columns: a list of categories feature names numerical_columns: a list of numerical feature names binary_columns: a list of yes or no feature names, accept binary indicators like (yes,no); (true,false); (0,1). ''' super().__init__() self.word_embedding = TransTabWordEmbedding( vocab_size=vocab_size, hidden_dim=hidden_dim, hidden_dropout_prob=hidden_dropout_prob, padding_idx=pad_token_id ) self.num_embedding = TransTabNumEmbedding(hidden_dim) self.align_layer = nn.Linear(hidden_dim, hidden_dim, bias=False) self.device = device def _avg_embedding_by_mask(self, embs, att_mask=None): if att_mask is None: return embs.mean(1) else: embs[att_mask==0] = 0 embs = embs.sum(1) / att_mask.sum(1,keepdim=True).to(embs.device) return embs def forward(self, x_num=None, num_col_input_ids=None, num_att_mask=None, x_cat_input_ids=None, cat_att_mask=None, x_bin_input_ids=None, bin_att_mask=None, **kwargs, ) -> Tensor: '''args: x: pd.DataFrame with column names and features. shuffle: if shuffle column order during the training. num_mask: indicate the NaN place of numerical features, 0: NaN 1: normal. ''' num_feat_embedding = None cat_feat_embedding = None bin_feat_embedding = None if x_num is not None and num_col_input_ids is not None: num_col_emb = self.word_embedding(num_col_input_ids.to(self.device)) # number of cat col, num of tokens, embdding size x_num = x_num.to(self.device) num_col_emb = self._avg_embedding_by_mask(num_col_emb, num_att_mask) num_feat_embedding = self.num_embedding(num_col_emb, x_num) num_feat_embedding = self.align_layer(num_feat_embedding) if x_cat_input_ids is not None: cat_feat_embedding = self.word_embedding(x_cat_input_ids.to(self.device)) cat_feat_embedding = self.align_layer(cat_feat_embedding) if x_bin_input_ids is not None: if x_bin_input_ids.shape[1] == 0: # all false, pad zero x_bin_input_ids = torch.zeros(x_bin_input_ids.shape[0],dtype=int)[:,None] bin_feat_embedding = self.word_embedding(x_bin_input_ids.to(self.device)) bin_feat_embedding = self.align_layer(bin_feat_embedding) # concat all embeddings emb_list = [] att_mask_list = [] if num_feat_embedding is not None: emb_list += [num_feat_embedding] att_mask_list += [torch.ones(num_feat_embedding.shape[0], num_feat_embedding.shape[1])] if cat_feat_embedding is not None: emb_list += [cat_feat_embedding] att_mask_list += [cat_att_mask] if bin_feat_embedding is not None: emb_list += [bin_feat_embedding] att_mask_list += [bin_att_mask] if len(emb_list) == 0: raise Exception('no feature found belonging into numerical, categorical, or binary, check your data!') all_feat_embedding = torch.cat(emb_list, 1).float() attention_mask = torch.cat(att_mask_list, 1).to(all_feat_embedding.device) return {'embedding': all_feat_embedding, 'attention_mask': attention_mask} def _get_activation_fn(activation): if activation == "relu": return F.relu elif activation == "gelu": return F.gelu elif activation == 'selu': return F.selu elif activation == 'leakyrelu': return F.leaky_relu raise RuntimeError("activation should be relu/gelu/selu/leakyrelu, not {}".format(activation)) class TransTabTransformerLayer(nn.Module): __constants__ = ['batch_first', 'norm_first'] def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1, activation=F.relu, layer_norm_eps=1e-5, batch_first=True, norm_first=False, device=None, dtype=None, use_layer_norm=True) -> None: factory_kwargs = {'device': device, 'dtype': dtype} super().__init__() self.self_attn = nn.MultiheadAttention(d_model, nhead, batch_first=batch_first, **factory_kwargs) # Implementation of Feedforward model self.linear1 = nn.Linear(d_model, dim_feedforward, **factory_kwargs) self.dropout = nn.Dropout(dropout) self.linear2 = nn.Linear(dim_feedforward, d_model, **factory_kwargs) # Implementation of gates self.gate_linear = nn.Linear(d_model, 1, bias=False) self.gate_act = nn.Sigmoid() self.norm_first = norm_first self.use_layer_norm = use_layer_norm if self.use_layer_norm: self.norm1 = nn.LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs) self.norm2 = nn.LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs) self.dropout1 = nn.Dropout(dropout) self.dropout2 = nn.Dropout(dropout) # Legacy string support for activation function. if isinstance(activation, str): self.activation = _get_activation_fn(activation) else: self.activation = activation # self-attention block def _sa_block(self, x: Tensor, attn_mask: Optional[Tensor], key_padding_mask: Optional[Tensor]) -> Tensor: src = x key_padding_mask = ~key_padding_mask.bool() x = self.self_attn(x, x, x, attn_mask=attn_mask, key_padding_mask=key_padding_mask, )[0] return self.dropout1(x) # feed forward block def _ff_block(self, x: Tensor) -> Tensor: g = self.gate_act(self.gate_linear(x)) h = self.linear1(x) h = h * g # add gate h = self.linear2(self.dropout(self.activation(h))) return self.dropout2(h) def __setstate__(self, state): if 'activation' not in state: state['activation'] = F.relu super().__setstate__(state) def forward(self, src, src_mask= None, src_key_padding_mask= None, is_causal=None, **kwargs) -> Tensor: r"""Pass the input through the encoder layer. Args: src: the sequence to the encoder layer (required). src_mask: the mask for the src sequence (optional). src_key_padding_mask: the mask for the src keys per batch (optional). Shape: see the docs in Transformer class. """ # see Fig. 1 of https://arxiv.org/pdf/2002.04745v1.pdf x = src if self.use_layer_norm: if self.norm_first: x = x + self._sa_block(self.norm1(x), src_mask, src_key_padding_mask) x = x + self._ff_block(self.norm2(x)) else: x = self.norm1(x + self._sa_block(x, src_mask, src_key_padding_mask)) x = self.norm2(x + self._ff_block(x)) else: # do not use layer norm x = x + self._sa_block(x, src_mask, src_key_padding_mask) x = x + self._ff_block(x) return x class TransTabInputEncoder(nn.Module): ''' Build a feature encoder that maps inputs tabular samples to embeddings. Parameters: ----------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). ignore_duplicate_cols: bool if there is one column assigned to more than one type, e.g., the feature age is both nominated as categorical and binary columns, the model will raise errors. set True to avoid this error as the model will ignore this duplicate feature. disable_tokenizer_parallel: bool if the returned feature extractor is leveraged by the collate function for a dataloader, try to set this False in case the dataloader raises errors because the dataloader builds multiple workers and the tokenizer builds multiple workers at the same time. hidden_dim: int the dimension of hidden embeddings. hidden_dropout_prob: float the dropout ratio in the transformer encoder. device: str the device, ``"cpu"`` or ``"cuda:0"``. ''' def __init__(self, feature_extractor, feature_processor, device='cuda:0', ): super().__init__() self.feature_extractor = feature_extractor self.feature_processor = feature_processor self.device = device self.to(device) def forward(self, x): ''' Encode input tabular samples into embeddings. Parameters ---------- x: pd.DataFrame with column names and features. ''' tokenized = self.feature_extractor(x) embeds = self.feature_processor(**tokenized) return embeds def load(self, ckpt_dir): # load feature extractor self.feature_extractor.load(os.path.join(ckpt_dir, constants.EXTRACTOR_STATE_DIR)) # load embedding layer model_name = os.path.join(ckpt_dir, constants.INPUT_ENCODER_NAME) state_dict = torch.load(model_name, map_location='cpu') missing_keys, unexpected_keys = self.load_state_dict(state_dict, strict=False) logger.info(f'missing keys: {missing_keys}') logger.info(f'unexpected keys: {unexpected_keys}') logger.info(f'load model from {ckpt_dir}') class TransTabEncoder(nn.Module): def __init__(self, hidden_dim=128, num_layer=2, num_attention_head=2, hidden_dropout_prob=0, ffn_dim=256, activation='relu', ): super().__init__() self.transformer_encoder = nn.ModuleList( [ TransTabTransformerLayer( d_model=hidden_dim, nhead=num_attention_head, dropout=hidden_dropout_prob, dim_feedforward=ffn_dim, batch_first=True, layer_norm_eps=1e-5, norm_first=False, use_layer_norm=True, activation=activation,) ] ) if num_layer > 1: encoder_layer = TransTabTransformerLayer(d_model=hidden_dim, nhead=num_attention_head, dropout=hidden_dropout_prob, dim_feedforward=ffn_dim, batch_first=True, layer_norm_eps=1e-5, norm_first=False, use_layer_norm=True, activation=activation, ) stacked_transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layer-1) self.transformer_encoder.append(stacked_transformer) def forward(self, embedding, attention_mask=None, **kwargs) -> Tensor: '''args: embedding: bs, num_token, hidden_dim ''' outputs = embedding for i, mod in enumerate(self.transformer_encoder): outputs = mod(outputs, src_key_padding_mask=attention_mask) return outputs class TransTabLinearClassifier(nn.Module): def __init__(self, num_class, hidden_dim=128) -> None: super().__init__() if num_class <= 2: self.fc = nn.Linear(hidden_dim, 1) else: self.fc = nn.Linear(hidden_dim, num_class) self.norm = nn.LayerNorm(hidden_dim) def forward(self, x) -> Tensor: x = x[:,0,:] # take the cls token embedding x = self.norm(x) logits = self.fc(x) return logits class TransTabLinearRegressor(nn.Module): def __init__(self, hidden_dim=128) -> None: super().__init__() self.fc = nn.Linear(hidden_dim, 1) self.norm = nn.LayerNorm(hidden_dim) def forward(self, x) -> Tensor: x = x[:,0,:] # take the cls token embedding x = self.norm(x) output = self.fc(x) return output class TransTabProjectionHead(nn.Module): def __init__(self, hidden_dim=128, projection_dim=128): super().__init__() self.dense = nn.Linear(hidden_dim, projection_dim, bias=False) def forward(self, x) -> Tensor: h = self.dense(x) return h class TransTabCLSToken(nn.Module): '''add a learnable cls token embedding at the end of each sequence. ''' def __init__(self, hidden_dim) -> None: super().__init__() self.weight = nn.Parameter(Tensor(hidden_dim)) nn_init.uniform_(self.weight, a=-1/math.sqrt(hidden_dim),b=1/math.sqrt(hidden_dim)) self.hidden_dim = hidden_dim def expand(self, *leading_dimensions): new_dims = (1,) * (len(leading_dimensions)-1) return self.weight.view(*new_dims, -1).expand(*leading_dimensions, -1) def forward(self, embedding, attention_mask=None, **kwargs) -> Tensor: embedding = torch.cat([self.expand(len(embedding), 1), embedding], dim=1) outputs = {'embedding': embedding} if attention_mask is not None: attention_mask = torch.cat([torch.ones(attention_mask.shape[0],1).to(attention_mask.device), attention_mask], 1) outputs['attention_mask'] = attention_mask return outputs class TransTabModel(nn.Module): '''The base transtab model for downstream tasks like contrastive learning, binary classification, etc. All models subclass this basemodel and usually rewrite the ``forward`` function. Refer to the source code of :class:`transtab.modeling_transtab.TransTabClassifier` or :class:`transtab.modeling_transtab.TransTabForCL` for the implementation details. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. Returns ------- A TransTabModel model. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, feature_extractor=None, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0.1, ffn_dim=256, activation='relu', device='cuda:0', **kwargs, ) -> None: super().__init__() self.categorical_columns=categorical_columns self.numerical_columns=numerical_columns self.binary_columns=binary_columns if categorical_columns is not None: self.categorical_columns = list(set(categorical_columns)) if numerical_columns is not None: self.numerical_columns = list(set(numerical_columns)) if binary_columns is not None: self.binary_columns = list(set(binary_columns)) if feature_extractor is None: feature_extractor = TransTabFeatureExtractor( categorical_columns=self.categorical_columns, numerical_columns=self.numerical_columns, binary_columns=self.binary_columns, **kwargs, ) feature_processor = TransTabFeatureProcessor( vocab_size=feature_extractor.vocab_size, pad_token_id=feature_extractor.pad_token_id, hidden_dim=hidden_dim, hidden_dropout_prob=hidden_dropout_prob, device=device, ) self.input_encoder = TransTabInputEncoder( feature_extractor=feature_extractor, feature_processor=feature_processor, device=device, ) self.encoder = TransTabEncoder( hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, ) self.cls_token = TransTabCLSToken(hidden_dim=hidden_dim) self.device = device self.to(device) def forward(self, x, y=None): '''Extract the embeddings based on input tables. Parameters ---------- x: pd.DataFrame a batch of samples stored in pd.DataFrame. y: pd.Series the corresponding labels for each sample in ``x``. ignored for the basemodel. Returns ------- final_cls_embedding: torch.Tensor the [CLS] embedding at the end of transformer encoder. ''' embeded = self.input_encoder(x) embeded = self.cls_token(**embeded) # go through transformers, get final cls embedding encoder_output = self.encoder(**embeded) # get cls token final_cls_embedding = encoder_output[:,0,:] return final_cls_embedding def load(self, ckpt_dir): '''Load the model state_dict and feature_extractor configuration from the ``ckpt_dir``. Parameters ---------- ckpt_dir: str the directory path to load. Returns ------- None ''' # load model weight state dict model_name = os.path.join(ckpt_dir, constants.WEIGHTS_NAME) state_dict = torch.load(model_name, map_location='cpu') missing_keys, unexpected_keys = self.load_state_dict(state_dict, strict=False) logger.info(f'missing keys: {missing_keys}') logger.info(f'unexpected keys: {unexpected_keys}') logger.info(f'load model from {ckpt_dir}') # load feature extractor self.input_encoder.feature_extractor.load(os.path.join(ckpt_dir, constants.EXTRACTOR_STATE_DIR)) self.binary_columns = self.input_encoder.feature_extractor.binary_columns self.categorical_columns = self.input_encoder.feature_extractor.categorical_columns self.numerical_columns = self.input_encoder.feature_extractor.numerical_columns def save(self, ckpt_dir): '''Save the model state_dict and feature_extractor configuration to the ``ckpt_dir``. Parameters ---------- ckpt_dir: str the directory path to save. Returns ------- None ''' # save model weight state dict if not os.path.exists(ckpt_dir): os.makedirs(ckpt_dir, exist_ok=True) state_dict = self.state_dict() torch.save(state_dict, os.path.join(ckpt_dir, constants.WEIGHTS_NAME)) if self.input_encoder.feature_extractor is not None: self.input_encoder.feature_extractor.save(ckpt_dir) # save the input encoder separately state_dict_input_encoder = self.input_encoder.state_dict() torch.save(state_dict_input_encoder, os.path.join(ckpt_dir, constants.INPUT_ENCODER_NAME)) return None def update(self, config): '''Update the configuration of feature extractor's column map for cat, num, and bin cols. Or update the number of classes for the output classifier layer. Parameters ---------- config: dict a dict of configurations: keys cat:list, num:list, bin:list are to specify the new column names; key num_class:int is to specify the number of classes for finetuning on a new dataset. Returns ------- None ''' col_map = {} for k,v in config.items(): if k in ['cat','num','bin']: col_map[k] = v self.input_encoder.feature_extractor.update(**col_map) self.binary_columns = self.input_encoder.feature_extractor.binary_columns self.categorical_columns = self.input_encoder.feature_extractor.categorical_columns self.numerical_columns = self.input_encoder.feature_extractor.numerical_columns if 'num_class' in config: num_class = config['num_class'] self._adapt_to_new_num_class(num_class) return None def _check_column_overlap(self, cat_cols=None, num_cols=None, bin_cols=None): all_cols = [] if cat_cols is not None: all_cols.extend(cat_cols) if num_cols is not None: all_cols.extend(num_cols) if bin_cols is not None: all_cols.extend(bin_cols) org_length = len(all_cols) unq_length = len(list(set(all_cols))) duplicate_cols = [item for item, count in collections.Counter(all_cols).items() if count > 1] return org_length == unq_length, duplicate_cols def _solve_duplicate_cols(self, duplicate_cols): for col in duplicate_cols: logger.warning('Find duplicate cols named `{col}`, will ignore it during training!') if col in self.categorical_columns: self.categorical_columns.remove(col) self.categorical_columns.append(f'[cat]{col}') if col in self.numerical_columns: self.numerical_columns.remove(col) self.numerical_columns.append(f'[num]{col}') if col in self.binary_columns: self.binary_columns.remove(col) self.binary_columns.append(f'[bin]{col}') def _adapt_to_new_num_class(self, num_class): if num_class != self.num_class: self.num_class = num_class self.clf = TransTabLinearClassifier(num_class, hidden_dim=self.cls_token.hidden_dim) self.clf.to(self.device) if self.num_class > 2: self.loss_fn = nn.CrossEntropyLoss(reduction='none') else: self.loss_fn = nn.BCEWithLogitsLoss(reduction='none') logger.info(f'Build a new classifier with num {num_class} classes outputs, need further finetune to work.') class TransTabClassifier(TransTabModel): '''The classifier model subclass from :class:`transtab.modeling_transtab.TransTabModel`. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. num_class: int number of output classes to be predicted. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. Returns ------- A TransTabClassifier model. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, feature_extractor=None, num_class=2, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, activation='relu', device='cuda:0', **kwargs, ) -> None: super().__init__( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, feature_extractor=feature_extractor, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, device=device, **kwargs, ) self.num_class = num_class self.clf = TransTabLinearClassifier(num_class=num_class, hidden_dim=hidden_dim) if self.num_class > 2: self.loss_fn = nn.CrossEntropyLoss(reduction='none') else: self.loss_fn = nn.BCEWithLogitsLoss(reduction='none') self.to(device) def forward(self, x, y=None): '''Make forward pass given the input feature ``x`` and label ``y`` (optional). Parameters ---------- x: pd.DataFrame or dict pd.DataFrame: a batch of raw tabular samples; dict: the output of TransTabFeatureExtractor. y: pd.Series the corresponding labels for each sample in ``x``. if label is given, the model will return the classification loss by ``self.loss_fn``. Returns ------- logits: torch.Tensor the [CLS] embedding at the end of transformer encoder. loss: torch.Tensor or None the classification loss. ''' if isinstance(x, dict): # input is the pre-tokenized encoded inputs inputs = x elif isinstance(x, pd.DataFrame): # input is dataframe inputs = self.input_encoder.feature_extractor(x) else: raise ValueError(f'TransTabClassifier takes inputs with dict or pd.DataFrame, find {type(x)}.') outputs = self.input_encoder.feature_processor(**inputs) outputs = self.cls_token(**outputs) # go through transformers, get the first cls embedding encoder_output = self.encoder(**outputs) # bs, seqlen+1, hidden_dim # classifier logits = self.clf(encoder_output) if y is not None: # compute classification loss if self.num_class == 2: y_ts = torch.tensor(y.values).to(self.device).float() loss = self.loss_fn(logits.flatten(), y_ts) else: y_ts = torch.tensor(y.values).to(self.device).long() loss = self.loss_fn(logits, y_ts) loss = loss.mean() else: loss = None return logits, loss class TransTabRegressor(TransTabModel): '''The regression model subclass from :class:`transtab.modeling_transtab.TransTabModel`. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. num_class: int number of output classes to be predicted. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. Returns ------- A TransTabRegressor model. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, feature_extractor=None, num_class=1, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, activation='relu', device='cuda:0', **kwargs, ) -> None: super().__init__( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, feature_extractor=feature_extractor, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, device=device, **kwargs, ) self.num_class = num_class self.regressor = TransTabLinearRegressor(hidden_dim=hidden_dim) self.loss_fn = nn.MSELoss() self.to(device) def forward(self, x, y=None): '''Make forward pass given the input feature ``x`` and label ``y`` (optional). Parameters ---------- x: pd.DataFrame or dict pd.DataFrame: a batch of raw tabular samples; dict: the output of TransTabFeatureExtractor. y: pd.Series the corresponding labels for each sample in ``x``. if label is given, the model will return the classification loss by ``self.loss_fn``. Returns ------- logits: torch.Tensor the [CLS] embedding at the end of transformer encoder. loss: torch.Tensor or None the classification loss. ''' if isinstance(x, dict): # input is the pre-tokenized encoded inputs inputs = x elif isinstance(x, pd.DataFrame): # input is dataframe inputs = self.input_encoder.feature_extractor(x) else: raise ValueError(f'TransTabRegressor takes inputs with dict or pd.DataFrame, find {type(x)}.') outputs = self.input_encoder.feature_processor(**inputs) outputs = self.cls_token(**outputs) # go through transformers, get the first cls embedding encoder_output = self.encoder(**outputs) # bs, seqlen+1, hidden_dim # regression output = self.regressor(encoder_output) if y is not None: # compute regression loss y_ts = torch.tensor(y.values).to(self.device).float() loss = self.loss_fn(output.flatten(), y_ts) loss = loss.mean() else: loss = None return output, loss class TransTabForCL(TransTabModel): '''The contrasstive learning model subclass from :class:`transtab.modeling_transtab.TransTabModel`. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. projection_dim: int the dimension of projection head on the top of encoder. overlap_ratio: float the overlap ratio of columns of different partitions when doing subsetting. num_partition: int the number of partitions made for vertical-partition contrastive learning. supervised: bool whether or not to take supervised VPCL, otherwise take self-supervised VPCL. temperature: float temperature used to compute logits for contrastive learning. base_temperature: float base temperature used to normalize the temperature. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. Returns ------- A TransTabForCL model. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, feature_extractor=None, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, projection_dim=128, overlap_ratio=0.1, num_partition=2, supervised=True, temperature=10, base_temperature=10, activation='relu', device='cuda:0', **kwargs, ) -> None: super().__init__( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, feature_extractor=feature_extractor, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, device=device, **kwargs, ) assert num_partition > 0, f'number of contrastive subsets must be greater than 0, got {num_partition}' assert isinstance(num_partition,int), f'number of constrative subsets must be int, got {type(num_partition)}' assert overlap_ratio >= 0 and overlap_ratio < 1, f'overlap_ratio must be in [0, 1), got {overlap_ratio}' self.projection_head = TransTabProjectionHead(hidden_dim, projection_dim) self.cross_entropy_loss = nn.CrossEntropyLoss() self.temperature = temperature self.base_temperature = base_temperature self.num_partition = num_partition self.overlap_ratio = overlap_ratio self.supervised = supervised self.device = device self.to(device) def forward(self, x, y=None): '''Make forward pass given the input feature ``x`` and label ``y`` (optional). Parameters ---------- x: pd.DataFrame or dict pd.DataFrame: a batch of raw tabular samples; dict: the output of TransTabFeatureExtractor. y: pd.Series the corresponding labels for each sample in ``x``. if label is given, the model will return the classification loss by ``self.loss_fn``. Returns ------- logits: None this CL model does NOT return logits. loss: torch.Tensor the supervised or self-supervised VPCL loss. ''' # do positive sampling feat_x_list = [] if isinstance(x, pd.DataFrame): sub_x_list = self._build_positive_pairs(x, self.num_partition) for sub_x in sub_x_list: # encode two subset feature samples feat_x = self.input_encoder(sub_x) feat_x = self.cls_token(**feat_x) feat_x = self.encoder(**feat_x) feat_x_proj = feat_x[:,0,:] # take cls embedding feat_x_proj = self.projection_head(feat_x_proj) # bs, projection_dim feat_x_list.append(feat_x_proj) elif isinstance(x, dict): # pretokenized inputs for input_x in x['input_sub_x']: feat_x = self.input_encoder.feature_processor(**input_x) feat_x = self.cls_token(**feat_x) feat_x = self.encoder(**feat_x) feat_x_proj = feat_x[:, 0, :] feat_x_proj = self.projection_head(feat_x_proj) feat_x_list.append(feat_x_proj) else: raise ValueError(f'expect input x to be pd.DataFrame or dict(pretokenized), get {type(x)} instead') feat_x_multiview = torch.stack(feat_x_list, axis=1) # bs, n_view, emb_dim if y is not None and self.supervised: # take supervised loss y = torch.tensor(y.values, device=feat_x_multiview.device) loss = self.supervised_contrastive_loss(feat_x_multiview, y) else: # compute cl loss (multi-view InfoNCE loss) loss = self.self_supervised_contrastive_loss(feat_x_multiview) return None, loss def _build_positive_pairs(self, x, n): x_cols = x.columns.tolist() sub_col_list = np.array_split(np.array(x_cols), n) len_cols = len(sub_col_list[0]) overlap = int(np.ceil(len_cols * (self.overlap_ratio))) sub_x_list = [] for i, sub_col in enumerate(sub_col_list): if overlap > 0 and i < n-1: sub_col = np.concatenate([sub_col, sub_col_list[i+1][:overlap]]) elif overlap >0 and i == n-1: sub_col = np.concatenate([sub_col, sub_col_list[i-1][-overlap:]]) sub_x = x.copy()[sub_col] sub_x_list.append(sub_x) return sub_x_list def cos_sim(self, a, b): if not isinstance(a, torch.Tensor): a = torch.tensor(a) if not isinstance(b, torch.Tensor): b = torch.tensor(b) if len(a.shape) == 1: a = a.unsqueeze(0) if len(b.shape) == 1: b = b.unsqueeze(0) a_norm = torch.nn.functional.normalize(a, p=2, dim=1) b_norm = torch.nn.functional.normalize(b, p=2, dim=1) return torch.mm(a_norm, b_norm.transpose(0, 1)) def self_supervised_contrastive_loss(self, features): '''Compute the self-supervised VPCL loss. Parameters ---------- features: torch.Tensor the encoded features of multiple partitions of input tables, with shape ``(bs, n_partition, proj_dim)``. Returns ------- loss: torch.Tensor the computed self-supervised VPCL loss. ''' batch_size = features.shape[0] labels = torch.arange(batch_size, dtype=torch.long, device=self.device).view(-1,1) mask = torch.eq(labels, labels.T).float().to(labels.device) contrast_count = features.shape[1] # [[0,1],[2,3]] -> [0,2,1,3] contrast_feature = torch.cat(torch.unbind(features,dim=1),dim=0) anchor_feature = contrast_feature anchor_count = contrast_count anchor_dot_contrast = torch.div(torch.matmul(anchor_feature, contrast_feature.T), self.temperature) logits_max, _ = torch.max(anchor_dot_contrast, dim=1, keepdim=True) logits = anchor_dot_contrast - logits_max.detach() mask = mask.repeat(anchor_count, contrast_count) logits_mask = torch.scatter(torch.ones_like(mask), 1, torch.arange(batch_size * anchor_count).view(-1, 1).to(features.device), 0) mask = mask * logits_mask # compute log_prob exp_logits = torch.exp(logits) * logits_mask log_prob = logits - torch.log(exp_logits.sum(1, keepdim=True)) # compute mean of log-likelihood over positive mean_log_prob_pos = (mask * log_prob).sum(1) / mask.sum(1) loss = - (self.temperature / self.base_temperature) * mean_log_prob_pos loss = loss.view(anchor_count, batch_size).mean() return loss def supervised_contrastive_loss(self, features, labels): '''Compute the supervised VPCL loss. Parameters ---------- features: torch.Tensor the encoded features of multiple partitions of input tables, with shape ``(bs, n_partition, proj_dim)``. labels: torch.Tensor the class labels to be used for building positive/negative pairs in VPCL. Returns ------- loss: torch.Tensor the computed VPCL loss. ''' labels = labels.contiguous().view(-1,1) batch_size = features.shape[0] mask = torch.eq(labels, labels.T).float().to(labels.device) contrast_count = features.shape[1] contrast_feature = torch.cat(torch.unbind(features,dim=1),dim=0) # contrast_mode == 'all' anchor_feature = contrast_feature anchor_count = contrast_count # compute logits anchor_dot_contrast = torch.div( torch.matmul(anchor_feature, contrast_feature.T), self.temperature) # for numerical stability logits_max, _ = torch.max(anchor_dot_contrast, dim=1, keepdim=True) logits = anchor_dot_contrast - logits_max.detach() # tile mask mask = mask.repeat(anchor_count, contrast_count) # mask-out self-contrast cases logits_mask = torch.scatter( torch.ones_like(mask), 1, torch.arange(batch_size * anchor_count).view(-1, 1).to(features.device), 0, ) mask = mask * logits_mask # compute log_prob exp_logits = torch.exp(logits) * logits_mask log_prob = logits - torch.log(exp_logits.sum(1, keepdim=True)) # compute mean of log-likelihood over positive mean_log_prob_pos = (mask * log_prob).sum(1) / mask.sum(1) loss = - (self.temperature / self.base_temperature) * mean_log_prob_pos loss = loss.view(anchor_count, batch_size).mean() return loss ================================================ FILE: transtab/tokenizer/special_tokens_map.json ================================================ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"} ================================================ FILE: transtab/tokenizer/tokenizer_config.json ================================================ {"do_lower_case": true, "model_max_length": 512} ================================================ FILE: transtab/tokenizer/vocab.txt ================================================ [PAD] [unused0] [unused1] [unused2] [unused3] [unused4] [unused5] [unused6] [unused7] [unused8] [unused9] [unused10] [unused11] [unused12] [unused13] [unused14] [unused15] [unused16] [unused17] [unused18] [unused19] [unused20] [unused21] [unused22] [unused23] [unused24] [unused25] [unused26] [unused27] [unused28] [unused29] [unused30] [unused31] [unused32] [unused33] [unused34] [unused35] [unused36] [unused37] [unused38] [unused39] [unused40] [unused41] [unused42] [unused43] [unused44] [unused45] [unused46] [unused47] [unused48] [unused49] [unused50] [unused51] [unused52] [unused53] [unused54] [unused55] [unused56] [unused57] [unused58] [unused59] [unused60] [unused61] [unused62] [unused63] [unused64] [unused65] [unused66] [unused67] [unused68] [unused69] [unused70] [unused71] [unused72] [unused73] [unused74] [unused75] [unused76] [unused77] [unused78] [unused79] [unused80] [unused81] [unused82] [unused83] [unused84] [unused85] [unused86] [unused87] [unused88] [unused89] [unused90] [unused91] [unused92] [unused93] [unused94] [unused95] [unused96] [unused97] [unused98] [UNK] [CLS] [SEP] [MASK] [unused99] [unused100] [unused101] [unused102] [unused103] [unused104] [unused105] [unused106] [unused107] [unused108] [unused109] [unused110] [unused111] [unused112] [unused113] [unused114] [unused115] [unused116] [unused117] [unused118] [unused119] [unused120] [unused121] [unused122] [unused123] [unused124] [unused125] [unused126] [unused127] [unused128] [unused129] [unused130] [unused131] [unused132] [unused133] [unused134] [unused135] [unused136] [unused137] [unused138] [unused139] [unused140] [unused141] [unused142] [unused143] [unused144] [unused145] [unused146] [unused147] [unused148] [unused149] [unused150] [unused151] [unused152] [unused153] [unused154] [unused155] [unused156] [unused157] [unused158] [unused159] [unused160] [unused161] [unused162] [unused163] [unused164] [unused165] [unused166] [unused167] [unused168] [unused169] [unused170] [unused171] [unused172] [unused173] [unused174] [unused175] [unused176] [unused177] [unused178] [unused179] [unused180] [unused181] [unused182] [unused183] [unused184] [unused185] [unused186] [unused187] [unused188] [unused189] [unused190] [unused191] [unused192] [unused193] [unused194] [unused195] [unused196] [unused197] [unused198] [unused199] [unused200] [unused201] [unused202] [unused203] [unused204] [unused205] [unused206] [unused207] [unused208] [unused209] [unused210] [unused211] [unused212] [unused213] [unused214] [unused215] [unused216] [unused217] [unused218] [unused219] [unused220] [unused221] [unused222] [unused223] [unused224] [unused225] [unused226] [unused227] [unused228] [unused229] [unused230] [unused231] [unused232] [unused233] [unused234] [unused235] [unused236] [unused237] [unused238] [unused239] [unused240] [unused241] [unused242] [unused243] [unused244] [unused245] [unused246] [unused247] [unused248] [unused249] [unused250] [unused251] [unused252] [unused253] [unused254] [unused255] [unused256] [unused257] [unused258] [unused259] [unused260] [unused261] [unused262] [unused263] [unused264] [unused265] [unused266] [unused267] [unused268] [unused269] [unused270] [unused271] [unused272] [unused273] [unused274] [unused275] [unused276] [unused277] [unused278] [unused279] [unused280] [unused281] [unused282] [unused283] [unused284] [unused285] [unused286] [unused287] [unused288] [unused289] [unused290] [unused291] [unused292] [unused293] [unused294] [unused295] [unused296] [unused297] [unused298] [unused299] [unused300] [unused301] [unused302] [unused303] [unused304] [unused305] [unused306] [unused307] [unused308] [unused309] [unused310] [unused311] [unused312] [unused313] [unused314] [unused315] [unused316] [unused317] [unused318] [unused319] [unused320] [unused321] [unused322] [unused323] [unused324] [unused325] [unused326] [unused327] [unused328] [unused329] [unused330] [unused331] [unused332] [unused333] [unused334] [unused335] [unused336] [unused337] [unused338] [unused339] [unused340] [unused341] [unused342] [unused343] [unused344] [unused345] [unused346] [unused347] [unused348] [unused349] [unused350] [unused351] [unused352] [unused353] [unused354] [unused355] [unused356] [unused357] [unused358] [unused359] [unused360] [unused361] [unused362] [unused363] [unused364] [unused365] [unused366] [unused367] [unused368] [unused369] [unused370] [unused371] [unused372] [unused373] [unused374] [unused375] [unused376] [unused377] [unused378] [unused379] [unused380] [unused381] [unused382] [unused383] [unused384] [unused385] [unused386] [unused387] [unused388] [unused389] [unused390] [unused391] [unused392] [unused393] [unused394] [unused395] [unused396] [unused397] [unused398] [unused399] [unused400] [unused401] [unused402] [unused403] [unused404] [unused405] [unused406] [unused407] [unused408] [unused409] [unused410] [unused411] [unused412] [unused413] [unused414] [unused415] [unused416] [unused417] [unused418] [unused419] [unused420] [unused421] [unused422] [unused423] [unused424] [unused425] [unused426] [unused427] [unused428] [unused429] [unused430] [unused431] [unused432] [unused433] [unused434] [unused435] [unused436] [unused437] [unused438] [unused439] [unused440] [unused441] [unused442] [unused443] [unused444] [unused445] [unused446] [unused447] [unused448] [unused449] [unused450] [unused451] [unused452] [unused453] [unused454] [unused455] [unused456] [unused457] [unused458] [unused459] [unused460] [unused461] [unused462] [unused463] [unused464] [unused465] [unused466] [unused467] [unused468] [unused469] [unused470] [unused471] [unused472] [unused473] [unused474] [unused475] [unused476] [unused477] [unused478] [unused479] [unused480] [unused481] [unused482] [unused483] [unused484] [unused485] [unused486] [unused487] [unused488] [unused489] [unused490] [unused491] [unused492] [unused493] [unused494] [unused495] [unused496] [unused497] [unused498] [unused499] [unused500] [unused501] [unused502] [unused503] [unused504] [unused505] [unused506] [unused507] [unused508] [unused509] [unused510] [unused511] [unused512] [unused513] [unused514] [unused515] [unused516] [unused517] [unused518] [unused519] [unused520] [unused521] [unused522] [unused523] [unused524] [unused525] [unused526] [unused527] [unused528] [unused529] [unused530] [unused531] [unused532] [unused533] [unused534] [unused535] [unused536] [unused537] [unused538] [unused539] [unused540] [unused541] [unused542] [unused543] [unused544] [unused545] [unused546] [unused547] [unused548] [unused549] [unused550] [unused551] [unused552] [unused553] [unused554] [unused555] [unused556] [unused557] [unused558] [unused559] [unused560] [unused561] [unused562] [unused563] [unused564] [unused565] [unused566] [unused567] [unused568] [unused569] [unused570] [unused571] [unused572] [unused573] [unused574] [unused575] [unused576] [unused577] [unused578] [unused579] [unused580] [unused581] [unused582] [unused583] [unused584] [unused585] [unused586] [unused587] [unused588] [unused589] [unused590] [unused591] [unused592] [unused593] [unused594] [unused595] [unused596] [unused597] [unused598] [unused599] [unused600] [unused601] [unused602] [unused603] [unused604] [unused605] [unused606] [unused607] [unused608] [unused609] [unused610] [unused611] [unused612] [unused613] [unused614] [unused615] [unused616] [unused617] [unused618] [unused619] [unused620] [unused621] [unused622] [unused623] [unused624] [unused625] [unused626] [unused627] [unused628] [unused629] [unused630] [unused631] [unused632] [unused633] [unused634] [unused635] [unused636] [unused637] [unused638] [unused639] [unused640] [unused641] [unused642] [unused643] [unused644] [unused645] [unused646] [unused647] [unused648] [unused649] [unused650] [unused651] [unused652] [unused653] [unused654] [unused655] [unused656] [unused657] [unused658] [unused659] [unused660] [unused661] [unused662] [unused663] [unused664] [unused665] [unused666] [unused667] [unused668] [unused669] [unused670] [unused671] [unused672] [unused673] [unused674] [unused675] [unused676] [unused677] [unused678] [unused679] [unused680] [unused681] [unused682] [unused683] [unused684] [unused685] [unused686] [unused687] [unused688] [unused689] [unused690] [unused691] [unused692] [unused693] [unused694] [unused695] [unused696] [unused697] [unused698] [unused699] [unused700] [unused701] [unused702] [unused703] [unused704] [unused705] [unused706] [unused707] [unused708] [unused709] [unused710] [unused711] [unused712] [unused713] [unused714] [unused715] [unused716] [unused717] [unused718] [unused719] [unused720] [unused721] [unused722] [unused723] [unused724] [unused725] [unused726] [unused727] [unused728] [unused729] [unused730] [unused731] [unused732] [unused733] [unused734] [unused735] [unused736] [unused737] [unused738] [unused739] [unused740] [unused741] [unused742] [unused743] [unused744] [unused745] [unused746] [unused747] [unused748] [unused749] [unused750] [unused751] [unused752] [unused753] [unused754] [unused755] [unused756] [unused757] [unused758] [unused759] [unused760] [unused761] [unused762] [unused763] [unused764] [unused765] [unused766] [unused767] [unused768] [unused769] [unused770] [unused771] [unused772] [unused773] [unused774] [unused775] [unused776] [unused777] [unused778] [unused779] [unused780] [unused781] [unused782] [unused783] [unused784] [unused785] [unused786] [unused787] [unused788] [unused789] [unused790] [unused791] [unused792] [unused793] [unused794] [unused795] [unused796] [unused797] [unused798] [unused799] [unused800] [unused801] [unused802] [unused803] [unused804] [unused805] [unused806] [unused807] [unused808] [unused809] [unused810] [unused811] [unused812] [unused813] [unused814] [unused815] [unused816] [unused817] [unused818] [unused819] [unused820] [unused821] [unused822] [unused823] [unused824] [unused825] [unused826] [unused827] [unused828] [unused829] [unused830] [unused831] [unused832] [unused833] [unused834] [unused835] [unused836] [unused837] [unused838] [unused839] [unused840] [unused841] [unused842] [unused843] [unused844] [unused845] [unused846] [unused847] [unused848] [unused849] [unused850] [unused851] [unused852] [unused853] [unused854] [unused855] [unused856] [unused857] [unused858] [unused859] [unused860] [unused861] [unused862] [unused863] [unused864] [unused865] [unused866] [unused867] [unused868] [unused869] [unused870] [unused871] [unused872] [unused873] [unused874] [unused875] [unused876] [unused877] [unused878] [unused879] [unused880] [unused881] [unused882] [unused883] [unused884] [unused885] [unused886] [unused887] [unused888] [unused889] [unused890] [unused891] [unused892] [unused893] [unused894] [unused895] [unused896] [unused897] [unused898] [unused899] [unused900] [unused901] [unused902] [unused903] [unused904] [unused905] [unused906] [unused907] [unused908] [unused909] [unused910] [unused911] [unused912] [unused913] [unused914] [unused915] [unused916] [unused917] [unused918] [unused919] [unused920] [unused921] [unused922] [unused923] [unused924] [unused925] [unused926] [unused927] [unused928] [unused929] [unused930] [unused931] [unused932] [unused933] [unused934] [unused935] [unused936] [unused937] [unused938] [unused939] [unused940] [unused941] [unused942] [unused943] [unused944] [unused945] [unused946] [unused947] [unused948] [unused949] [unused950] [unused951] [unused952] [unused953] [unused954] [unused955] [unused956] [unused957] [unused958] [unused959] [unused960] [unused961] [unused962] [unused963] [unused964] [unused965] [unused966] [unused967] [unused968] [unused969] [unused970] [unused971] [unused972] [unused973] [unused974] [unused975] [unused976] [unused977] [unused978] [unused979] [unused980] [unused981] [unused982] [unused983] [unused984] [unused985] [unused986] [unused987] [unused988] [unused989] [unused990] [unused991] [unused992] [unused993] ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ® ° ± ² ³ ´ µ ¶ · ¹ º » ¼ ½ ¾ ¿ × ß æ ð ÷ ø þ đ ħ ı ł ŋ œ ƒ ɐ ɑ ɒ ɔ ɕ ə ɛ ɡ ɣ ɨ ɪ ɫ ɬ ɯ ɲ ɴ ɹ ɾ ʀ ʁ ʂ ʃ ʉ ʊ ʋ ʌ ʎ ʐ ʑ ʒ ʔ ʰ ʲ ʳ ʷ ʸ ʻ ʼ ʾ ʿ ˈ ː ˡ ˢ ˣ ˤ α β γ δ ε ζ η θ ι κ λ μ ν ξ ο π ρ ς σ τ υ φ χ ψ ω а б в г д е ж з и к л м н о п р с т у ф х ц ч ш щ ъ ы ь э ю я ђ є і ј љ њ ћ ӏ ա բ գ դ ե թ ի լ կ հ մ յ ն ո պ ս վ տ ր ւ ք ־ א ב ג ד ה ו ז ח ט י ך כ ל ם מ ן נ ס ע ף פ ץ צ ק ר ש ת ، ء ا ب ة ت ث ج ح خ د ذ ر ز س ش ص ض ط ظ ع غ ـ ف ق ك ل م ن ه و ى ي ٹ پ چ ک گ ں ھ ہ ی ے अ आ उ ए क ख ग च ज ट ड ण त थ द ध न प ब भ म य र ल व श ष स ह ा ि ी ो । ॥ ং অ আ ই উ এ ও ক খ গ চ ছ জ ট ড ণ ত থ দ ধ ন প ব ভ ম য র ল শ ষ স হ া ি ী ে க ச ட த ந ன ப ம ய ர ல ள வ ா ி ு ே ை ನ ರ ಾ ක ය ර ල ව ා ก ง ต ท น พ ม ย ร ล ว ส อ า เ ་ ། ག ང ད ན པ བ མ འ ར ལ ས မ ა ბ გ დ ე ვ თ ი კ ლ მ ნ ო რ ს ტ უ ᄀ ᄂ ᄃ ᄅ ᄆ ᄇ ᄉ ᄊ ᄋ ᄌ ᄎ ᄏ ᄐ ᄑ ᄒ ᅡ ᅢ ᅥ ᅦ ᅧ ᅩ ᅪ ᅭ ᅮ ᅯ ᅲ ᅳ ᅴ ᅵ ᆨ ᆫ ᆯ ᆷ ᆸ ᆼ ᴬ ᴮ ᴰ ᴵ ᴺ ᵀ ᵃ ᵇ ᵈ ᵉ ᵍ ᵏ ᵐ ᵒ ᵖ ᵗ ᵘ ᵢ ᵣ ᵤ ᵥ ᶜ ᶠ ‐ ‑ ‒ – — ― ‖ ‘ ’ ‚ “ ” „ † ‡ • … ‰ ′ ″ › ‿ ⁄ ⁰ ⁱ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ⁺ ⁻ ⁿ ₀ ₁ ₂ ₃ ₄ ₅ ₆ ₇ ₈ ₉ ₊ ₍ ₎ ₐ ₑ ₒ ₓ ₕ ₖ ₗ ₘ ₙ ₚ ₛ ₜ ₤ ₩ € ₱ ₹ ℓ № ℝ ™ ⅓ ⅔ ← ↑ → ↓ ↔ ↦ ⇄ ⇌ ⇒ ∂ ∅ ∆ ∇ ∈ − ∗ ∘ √ ∞ ∧ ∨ ∩ ∪ ≈ ≡ ≤ ≥ ⊂ ⊆ ⊕ ⊗ ⋅ ─ │ ■ ▪ ● ★ ☆ ☉ ♠ ♣ ♥ ♦ ♭ ♯ ⟨ ⟩ ⱼ ⺩ ⺼ ⽥ 、 。 〈 〉 《 》 「 」 『 』 〜 あ い う え お か き く け こ さ し す せ そ た ち っ つ て と な に ぬ ね の は ひ ふ へ ほ ま み む め も や ゆ よ ら り る れ ろ を ん ァ ア ィ イ ウ ェ エ オ カ キ ク ケ コ サ シ ス セ タ チ ッ ツ テ ト ナ ニ ノ ハ ヒ フ ヘ ホ マ ミ ム メ モ ャ ュ ョ ラ リ ル レ ロ ワ ン ・ ー 一 三 上 下 不 世 中 主 久 之 也 事 二 五 井 京 人 亻 仁 介 代 仮 伊 会 佐 侍 保 信 健 元 光 八 公 内 出 分 前 劉 力 加 勝 北 区 十 千 南 博 原 口 古 史 司 合 吉 同 名 和 囗 四 国 國 土 地 坂 城 堂 場 士 夏 外 大 天 太 夫 奈 女 子 学 宀 宇 安 宗 定 宣 宮 家 宿 寺 將 小 尚 山 岡 島 崎 川 州 巿 帝 平 年 幸 广 弘 張 彳 後 御 德 心 忄 志 忠 愛 成 我 戦 戸 手 扌 政 文 新 方 日 明 星 春 昭 智 曲 書 月 有 朝 木 本 李 村 東 松 林 森 楊 樹 橋 歌 止 正 武 比 氏 民 水 氵 氷 永 江 沢 河 治 法 海 清 漢 瀬 火 版 犬 王 生 田 男 疒 発 白 的 皇 目 相 省 真 石 示 社 神 福 禾 秀 秋 空 立 章 竹 糹 美 義 耳 良 艹 花 英 華 葉 藤 行 街 西 見 訁 語 谷 貝 貴 車 軍 辶 道 郎 郡 部 都 里 野 金 鈴 镇 長 門 間 阝 阿 陳 陽 雄 青 面 風 食 香 馬 高 龍 龸 fi fl ！ （ ） ， － ． ／ ： ？ ～ the of and in to was he is as for on with that it his by at from her ##s she you had an were but be this are not my they one which or have him me first all also their has up who out been when after there into new two its ##a time would no what about said we over then other so more ##e can if like back them only some could ##i where just ##ing during before ##n do ##o made school through than now years most world may between down well three ##d year while will ##ed ##r ##y later ##t city under around did such being used state people part know against your many second university both national ##er these don known off way until re how even get head ... didn ##ly team american because de ##l born united film since still long work south us became any high again day family see right man eyes house season war states including took life north same each called name much place however go four group another found won area here going 10 away series left home music best make hand number company several never last john 000 very album take end good too following released game played little began district ##m old want those side held own early county ll league use west ##u face think ##es 2010 government ##h march came small general town june ##on line based something ##k september thought looked along international 2011 air july club went january october our august april york 12 few 2012 2008 east show member college 2009 father public ##us come men five set station church ##c next former november room party located december 2013 age got 2007 ##g system let love 2006 though every 2014 look song water century without body black night within great women single ve building large population river named band white started ##an once 15 20 should 18 2015 service top built british open death king moved local times children february book why 11 door need president order final road wasn although due major died village third knew 2016 asked turned st wanted say ##p together received main son served different ##en behind himself felt members power football law voice play ##in near park history 30 having 2005 16 ##man saw mother ##al army point front help english street art late hands games award ##ia young 14 put published country division across told 13 often ever french london center six red 2017 led days include light 25 find tell among species really according central half 2004 form original gave office making enough lost full opened must included live given german player run business woman community cup might million land 2000 court development 17 short round ii km seen class story always become sure research almost director council la ##2 career things using island ##z couldn car ##is 24 close force ##1 better free support control field students 2003 education married ##b nothing worked others record big inside level anything continued give james ##3 military established non returned feel does title written thing feet william far co association hard already 2002 ##ra championship human western 100 ##na department hall role various production 21 19 heart 2001 living fire version ##ers ##f television royal ##4 produced working act case society region present radio period looking least total keep england wife program per brother mind special 22 ##le am works soon ##6 political george services taken created ##7 further able reached david union joined upon done important social information either ##ic ##x appeared position ground lead rock dark election 23 board france hair course arms site police girl instead real sound ##v words moment ##te someone ##8 summer project announced san less wrote past followed ##5 blue founded al finally india taking records america ##ne 1999 design considered northern god stop battle toward european outside described track today playing language 28 call 26 heard professional low australia miles california win yet green ##ie trying blood ##ton southern science maybe everything match square 27 mouth video race recorded leave above ##9 daughter points space 1998 museum change middle common ##0 move tv post ##ta lake seven tried elected closed ten paul minister ##th months start chief return canada person sea release similar modern brought rest hit formed mr ##la 1997 floor event doing thomas 1996 robert care killed training star week needed turn finished railway rather news health sent example ran term michael coming currently yes forces despite gold areas 50 stage fact 29 dead says popular 2018 originally germany probably developed result pulled friend stood money running mi signed word songs child eventually met tour average teams minutes festival current deep kind 1995 decided usually eastern seemed ##ness episode bed added table indian private charles route available idea throughout centre addition appointed style 1994 books eight construction press mean wall friends remained schools study ##ch ##um institute oh chinese sometimes events possible 1992 australian type brown forward talk process food debut seat performance committee features character arts herself else lot strong russian range hours peter arm ##da morning dr sold ##ry quickly directed 1993 guitar china ##w 31 list ##ma performed media uk players smile ##rs myself 40 placed coach province towards wouldn leading whole boy official designed grand census ##el europe attack japanese henry 1991 ##re ##os cross getting alone action lower network wide washington japan 1990 hospital believe changed sister ##ar hold gone sir hadn ship ##ka studies academy shot rights below base bad involved kept largest ##ist bank future especially beginning mark movement section female magazine plan professor lord longer ##ian sat walked hill actually civil energy model families size thus aircraft completed includes data captain ##or fight vocals featured richard bridge fourth 1989 officer stone hear ##ism means medical groups management self lips competition entire lived technology leaving federal tournament bit passed hot independent awards kingdom mary spent fine doesn reported ##ling jack fall raised itself stay true studio 1988 sports replaced paris systems saint leader theatre whose market capital parents spanish canadian earth ##ity cut degree writing bay christian awarded natural higher bill ##as coast provided previous senior ft valley organization stopped onto countries parts conference queen security interest saying allowed master earlier phone matter smith winning try happened moving campaign los ##ley breath nearly mid 1987 certain girls date italian african standing fell artist ##ted shows deal mine industry 1986 ##ng everyone republic provide collection library student ##ville primary owned older via heavy 1st makes ##able attention anyone africa ##ri stated length ended fingers command staff skin foreign opening governor okay medal kill sun cover job 1985 introduced chest hell feeling ##ies success meet reason standard meeting novel 1984 trade source buildings ##land rose guy goal ##ur chapter native husband previously unit limited entered weeks producer operations mountain takes covered forced related roman complete successful key texas cold ##ya channel 1980 traditional films dance clear approximately 500 nine van prince question active tracks ireland regional silver author personal sense operation ##ine economic 1983 holding twenty isbn additional speed hour edition regular historic places whom shook movie km² secretary prior report chicago read foundation view engine scored 1982 units ask airport property ready immediately lady month listed contract ##de manager themselves lines ##ki navy writer meant ##ts runs ##ro practice championships singer glass commission required forest starting culture generally giving access attended test couple stand catholic martin caught executive ##less eye ##ey thinking chair quite shoulder 1979 hope decision plays defeated municipality whether structure offered slowly pain ice direction ##ion paper mission 1981 mostly 200 noted individual managed nature lives plant ##ha helped except studied computer figure relationship issue significant loss die smiled gun ago highest 1972 ##am male bring goals mexico problem distance commercial completely location annual famous drive 1976 neck 1978 surface caused italy understand greek highway wrong hotel comes appearance joseph double issues musical companies castle income review assembly bass initially parliament artists experience 1974 particular walk foot engineering talking window dropped ##ter miss baby boys break 1975 stars edge remember policy carried train stadium bar sex angeles evidence ##ge becoming assistant soviet 1977 upper step wing 1970 youth financial reach ##ll actor numerous ##se ##st nodded arrived ##ation minute ##nt believed sorry complex beautiful victory associated temple 1968 1973 chance perhaps metal ##son 1945 bishop ##et lee launched particularly tree le retired subject prize contains yeah theory empire ##ce suddenly waiting trust recording ##to happy terms camp champion 1971 religious pass zealand names 2nd port ancient tom corner represented watch legal anti justice cause watched brothers 45 material changes simply response louis fast ##ting answer 60 historical 1969 stories straight create feature increased rate administration virginia el activities cultural overall winner programs basketball legs guard beyond cast doctor mm flight results remains cost effect winter ##ble larger islands problems chairman grew commander isn 1967 pay failed selected hurt fort box regiment majority journal 35 edward plans ##ke ##ni shown pretty irish characters directly scene likely operated allow spring ##j junior matches looks mike houses fellow ##tion beach marriage ##ham ##ive rules oil 65 florida expected nearby congress sam peace recent iii wait subsequently cell ##do variety serving agreed please poor joe pacific attempt wood democratic piece prime ##ca rural mile touch appears township 1964 1966 soldiers ##men ##ized 1965 pennsylvania closer fighting claimed score jones physical editor ##ous filled genus specific sitting super mom ##va therefore supported status fear cases store meaning wales minor spain tower focus vice frank follow parish separate golden horse fifth remaining branch 32 presented stared ##id uses secret forms ##co baseball exactly ##ck choice note discovered travel composed truth russia ball color kiss dad wind continue ring referred numbers digital greater ##ns metres slightly direct increase 1960 responsible crew rule trees troops ##no broke goes individuals hundred weight creek sleep memory defense provides ordered code value jewish windows 1944 safe judge whatever corps realized growing pre ##ga cities alexander gaze lies spread scott letter showed situation mayor transport watching workers extended ##li expression normal ##ment chart multiple border ##ba host ##ner daily mrs walls piano ##ko heat cannot ##ate earned products drama era authority seasons join grade ##io sign difficult machine 1963 territory mainly ##wood stations squadron 1962 stepped iron 19th ##led serve appear sky speak broken charge knowledge kilometres removed ships article campus simple ##ty pushed britain ##ve leaves recently cd soft boston latter easy acquired poland ##sa quality officers presence planned nations mass broadcast jean share image influence wild offer emperor electric reading headed ability promoted yellow ministry 1942 throat smaller politician ##by latin spoke cars williams males lack pop 80 ##ier acting seeing consists ##ti estate 1961 pressure johnson newspaper jr chris olympics online conditions beat elements walking vote ##field needs carolina text featuring global block shirt levels francisco purpose females et dutch duke ahead gas twice safety serious turning highly lieutenant firm maria amount mixed daniel proposed perfect agreement affairs 3rd seconds contemporary paid 1943 prison save kitchen label administrative intended constructed academic nice teacher races 1956 formerly corporation ben nation issued shut 1958 drums housing victoria seems opera 1959 graduated function von mentioned picked build recognized shortly protection picture notable exchange elections 1980s loved percent racing fish elizabeth garden volume hockey 1941 beside settled ##ford 1940 competed replied drew 1948 actress marine scotland steel glanced farm steve 1957 risk tonight positive magic singles effects gray screen dog ##ja residents bus sides none secondary literature polish destroyed flying founder households 1939 lay reserve usa gallery ##ler 1946 industrial younger approach appearances urban ones 1950 finish avenue powerful fully growth page honor jersey projects advanced revealed basic 90 infantry pair equipment visit 33 evening search grant effort solo treatment buried republican primarily bottom owner 1970s israel gives jim dream bob remain spot 70 notes produce champions contact ed soul accepted ways del ##ally losing split price capacity basis trial questions ##ina 1955 20th guess officially memorial naval initial ##ization whispered median engineer ##ful sydney ##go columbia strength 300 1952 tears senate 00 card asian agent 1947 software 44 draw warm supposed com pro ##il transferred leaned ##at candidate escape mountains asia potential activity entertainment seem traffic jackson murder 36 slow product orchestra haven agency bbc taught website comedy unable storm planning albums rugby environment scientific grabbed protect ##hi boat typically 1954 1953 damage principal divided dedicated mount ohio ##berg pick fought driver ##der empty shoulders sort thank berlin prominent account freedom necessary efforts alex headquarters follows alongside des simon andrew suggested operating learning steps 1949 sweet technical begin easily 34 teeth speaking settlement scale ##sh renamed ray max enemy semi joint compared ##rd scottish leadership analysis offers georgia pieces captured animal deputy guest organized ##lin tony combined method challenge 1960s huge wants battalion sons rise crime types facilities telling path 1951 platform sit 1990s ##lo tells assigned rich pull ##ot commonly alive ##za letters concept conducted wearing happen bought becomes holy gets ocean defeat languages purchased coffee occurred titled ##q declared applied sciences concert sounds jazz brain ##me painting fleet tax nick ##ius michigan count animals leaders episodes ##line content ##den birth ##it clubs 64 palace critical refused fair leg laughed returning surrounding participated formation lifted pointed connected rome medicine laid taylor santa powers adam tall shared focused knowing yards entrance falls ##wa calling ##ad sources chosen beneath resources yard ##ite nominated silence zone defined ##que gained thirty 38 bodies moon ##ard adopted christmas widely register apart iran premier serves du unknown parties ##les generation ##ff continues quick fields brigade quiet teaching clothes impact weapons partner flat theater supreme 1938 37 relations ##tor plants suffered 1936 wilson kids begins ##age 1918 seats armed internet models worth laws 400 communities classes background knows thanks quarter reaching humans carry killing format kong hong setting 75 architecture disease railroad inc possibly wish arthur thoughts harry doors density ##di crowd illinois stomach tone unique reports anyway ##ir liberal der vehicle thick dry drug faced largely facility theme holds creation strange colonel ##mi revolution bell politics turns silent rail relief independence combat shape write determined sales learned 4th finger oxford providing 1937 heritage fiction situated designated allowing distribution hosted ##est sight interview estimated reduced ##ria toronto footballer keeping guys damn claim motion sport sixth stayed ##ze en rear receive handed twelve dress audience granted brazil ##well spirit ##ated noticed etc olympic representative eric tight trouble reviews drink vampire missing roles ranked newly household finals wave critics ##ee phase massachusetts pilot unlike philadelphia bright guns crown organizations roof 42 respectively clearly tongue marked circle fox korea bronze brian expanded sexual supply yourself inspired labour fc ##ah reference vision draft connection brand reasons 1935 classic driving trip jesus cells entry 1920 neither trail claims atlantic orders labor nose afraid identified intelligence calls cancer attacked passing stephen positions imperial grey jason 39 sunday 48 swedish avoid extra uncle message covers allows surprise materials fame hunter ##ji 1930 citizens figures davis environmental confirmed shit titles di performing difference acts attacks ##ov existing votes opportunity nor shop entirely trains opposite pakistan ##pa develop resulted representatives actions reality pressed ##ish barely wine conversation faculty northwest ends documentary nuclear stock grace sets eat alternative ##ps bag resulting creating surprised cemetery 1919 drop finding sarah cricket streets tradition ride 1933 exhibition target ear explained rain composer injury apartment municipal educational occupied netherlands clean billion constitution learn 1914 maximum classical francis lose opposition jose ontario bear core hills rolled ending drawn permanent fun ##tes ##lla lewis sites chamber ryan ##way scoring height 1934 ##house lyrics staring 55 officials 1917 snow oldest ##tic orange ##ger qualified interior apparently succeeded thousand dinner lights existence fans heavily 41 greatest conservative send bowl plus enter catch ##un economy duty 1929 speech authorities princess performances versions shall graduate pictures effective remembered poetry desk crossed starring starts passenger sharp ##ant acres ass weather falling rank fund supporting check adult publishing heads cm southeast lane ##burg application bc ##ura les condition transfer prevent display ex regions earl federation cool relatively answered besides 1928 obtained portion ##town mix ##ding reaction liked dean express peak 1932 ##tte counter religion chain rare miller convention aid lie vehicles mobile perform squad wonder lying crazy sword ##ping attempted centuries weren philosophy category ##ize anna interested 47 sweden wolf frequently abandoned kg literary alliance task entitled ##ay threw promotion factory tiny soccer visited matt fm achieved 52 defence internal persian 43 methods ##ging arrested otherwise cambridge programming villages elementary districts rooms criminal conflict worry trained 1931 attempts waited signal bird truck subsequent programme ##ol ad 49 communist details faith sector patrick carrying laugh ##ss controlled korean showing origin fuel evil 1927 ##ent brief identity darkness address pool missed publication web planet ian anne wings invited ##tt briefly standards kissed ##be ideas climate causing walter worse albert articles winners desire aged northeast dangerous gate doubt 1922 wooden multi ##ky poet rising funding 46 communications communication violence copies prepared ford investigation skills 1924 pulling electronic ##ak ##ial ##han containing ultimately offices singing understanding restaurant tomorrow fashion christ ward da pope stands 5th flow studios aired commissioned contained exist fresh americans ##per wrestling approved kid employed respect suit 1925 angel asking increasing frame angry selling 1950s thin finds ##nd temperature statement ali explain inhabitants towns extensive narrow 51 jane flowers images promise somewhere object fly closely ##ls 1912 bureau cape 1926 weekly presidential legislative 1921 ##ai ##au launch founding ##ny 978 ##ring artillery strike un institutions roll writers landing chose kevin anymore pp ##ut attorney fit dan billboard receiving agricultural breaking sought dave admitted lands mexican ##bury charlie specifically hole iv howard credit moscow roads accident 1923 proved wear struck hey guards stuff slid expansion 1915 cat anthony ##kin melbourne opposed sub southwest architect failure plane 1916 ##ron map camera tank listen regarding wet introduction metropolitan link ep fighter inch grown gene anger fixed buy dvd khan domestic worldwide chapel mill functions examples ##head developing 1910 turkey hits pocket antonio papers grow unless circuit 18th concerned attached journalist selection journey converted provincial painted hearing aren bands negative aside wondered knight lap survey ma ##ow noise billy ##ium shooting guide bedroom priest resistance motor homes sounded giant ##mer 150 scenes equal comic patients hidden solid actual bringing afternoon touched funds wedding consisted marie canal sr kim treaty turkish recognition residence cathedral broad knees incident shaped fired norwegian handle cheek contest represent ##pe representing beauty ##sen birds advantage emergency wrapped drawing notice pink broadcasting ##ong somehow bachelor seventh collected registered establishment alan assumed chemical personnel roger retirement jeff portuguese wore tied device threat progress advance ##ised banks hired manchester nfl teachers structures forever ##bo tennis helping saturday sale applications junction hip incorporated neighborhood dressed ceremony ##ds influenced hers visual stairs decades inner kansas hung hoped gain scheduled downtown engaged austria clock norway certainly pale protected 1913 victor employees plate putting surrounded ##ists finishing blues tropical ##ries minnesota consider philippines accept 54 retrieved 1900 concern anderson properties institution gordon successfully vietnam ##dy backing outstanding muslim crossing folk producing usual demand occurs observed lawyer educated ##ana kelly string pleasure budget items quietly colorado philip typical ##worth derived 600 survived asks mental ##ide 56 jake jews distinguished ltd 1911 sri extremely 53 athletic loud thousands worried shadow transportation horses weapon arena importance users tim objects contributed dragon douglas aware senator johnny jordan sisters engines flag investment samuel shock capable clark row wheel refers session familiar biggest wins hate maintained drove hamilton request expressed injured underground churches walker wars tunnel passes stupid agriculture softly cabinet regarded joining indiana ##ea ##ms push dates spend behavior woods protein gently chase morgan mention burning wake combination occur mirror leads jimmy indeed impossible singapore paintings covering ##nes soldier locations attendance sell historian wisconsin invasion argued painter diego changing egypt ##don experienced inches ##ku missouri vol grounds spoken switzerland ##gan reform rolling ha forget massive resigned burned allen tennessee locked values improved ##mo wounded universe sick dating facing pack purchase user ##pur moments ##ul merged anniversary 1908 coal brick understood causes dynasty queensland establish stores crisis promote hoping views cards referee extension ##si raise arizona improve colonial formal charged ##rt palm lucky hide rescue faces 95 feelings candidates juan ##ell goods 6th courses weekend 59 luke cash fallen ##om delivered affected installed carefully tries swiss hollywood costs lincoln responsibility ##he shore file proper normally maryland assistance jump constant offering friendly waters persons realize contain trophy 800 partnership factor 58 musicians cry bound oregon indicated hero houston medium ##ure consisting somewhat ##ara 57 cycle ##che beer moore frederick gotten eleven worst weak approached arranged chin loan universal bond fifteen pattern disappeared ##ney translated ##zed lip arab capture interests insurance ##chi shifted cave prix warning sections courts coat plot smell feed golf favorite maintain knife vs voted degrees finance quebec opinion translation manner ruled operate productions choose musician discovery confused tired separated stream techniques committed attend ranking kings throw passengers measure horror fan mining sand danger salt calm decade dam require runner ##ik rush associate greece ##ker rivers consecutive matthew ##ski sighed sq documents steam edited closing tie accused 1905 ##ini islamic distributed directors organisation bruce 7th breathing mad lit arrival concrete taste 08 composition shaking faster amateur adjacent stating 1906 twin flew ##ran tokyo publications ##tone obviously ridge storage 1907 carl pages concluded desert driven universities ages terminal sequence borough 250 constituency creative cousin economics dreams margaret notably reduce montreal mode 17th ears saved jan vocal ##ica 1909 andy ##jo riding roughly threatened ##ise meters meanwhile landed compete repeated grass czech regularly charges tea sudden appeal ##ung solution describes pierre classification glad parking ##ning belt physics 99 rachel add hungarian participate expedition damaged gift childhood 85 fifty ##red mathematics jumped letting defensive mph ##ux ##gh testing ##hip hundreds shoot owners matters smoke israeli kentucky dancing mounted grandfather emma designs profit argentina ##gs truly li lawrence cole begun detroit willing branches smiling decide miami enjoyed recordings ##dale poverty ethnic gay ##bi gary arabic 09 accompanied ##one ##ons fishing determine residential acid ##ary alice returns starred mail ##ang jonathan strategy ##ue net forty cook businesses equivalent commonwealth distinct ill ##cy seriously ##ors ##ped shift harris replace rio imagine formula ensure ##ber additionally scheme conservation occasionally purposes feels favor ##and ##ore 1930s contrast hanging hunt movies 1904 instruments victims danish christopher busy demon sugar earliest colony studying balance duties ##ks belgium slipped carter 05 visible stages iraq fifa ##im commune forming zero 07 continuing talked counties legend bathroom option tail clay daughters afterwards severe jaw visitors ##ded devices aviation russell kate ##vi entering subjects ##ino temporary swimming forth smooth ghost audio bush operates rocks movements signs eddie ##tz ann voices honorary 06 memories dallas pure measures racial promised 66 harvard ceo 16th parliamentary indicate benefit flesh dublin louisiana 1902 1901 patient sleeping 1903 membership coastal medieval wanting element scholars rice 62 limit survive makeup rating definitely collaboration obvious ##tan boss ms baron birthday linked soil diocese ##lan ncaa ##mann offensive shell shouldn waist ##tus plain ross organ resolution manufacturing adding relative kennedy 98 whilst moth marketing gardens crash 72 heading partners credited carlos moves cable ##zi marshall ##out depending bottle represents rejected responded existed 04 jobs denmark lock ##ating treated graham routes talent commissioner drugs secure tests reign restored photography ##gi contributions oklahoma designer disc grin seattle robin paused atlanta unusual ##gate praised las laughing satellite hungary visiting ##sky interesting factors deck poems norman ##water stuck speaker rifle domain premiered ##her dc comics actors 01 reputation eliminated 8th ceiling prisoners script ##nce leather austin mississippi rapidly admiral parallel charlotte guilty tools gender divisions fruit ##bs laboratory nelson fantasy marry rapid aunt tribe requirements aspects suicide amongst adams bone ukraine abc kick sees edinburgh clothing column rough gods hunting broadway gathered concerns ##ek spending ty 12th snapped requires solar bones cavalry ##tta iowa drinking waste index franklin charity thompson stewart tip flash landscape friday enjoy singh poem listening ##back eighth fred differences adapted bomb ukrainian surgery corporate masters anywhere ##more waves odd sean portugal orleans dick debate kent eating puerto cleared 96 expect cinema 97 guitarist blocks electrical agree involving depth dying panel struggle ##ged peninsula adults novels emerged vienna metro debuted shoes tamil songwriter meets prove beating instance heaven scared sending marks artistic passage superior 03 significantly shopping ##tive retained ##izing malaysia technique cheeks ##ola warren maintenance destroy extreme allied 120 appearing ##yn fill advice alabama qualifying policies cleveland hat battery smart authors 10th soundtrack acted dated lb glance equipped coalition funny outer ambassador roy possibility couples campbell dna loose ethan supplies 1898 gonna 88 monster ##res shake agents frequency springs dogs practices 61 gang plastic easier suggests gulf blade exposed colors industries markets pan nervous electoral charts legislation ownership ##idae mac appointment shield copy assault socialist abbey monument license throne employment jay 93 replacement charter cloud powered suffering accounts oak connecticut strongly wright colour crystal 13th context welsh networks voiced gabriel jerry ##cing forehead mp ##ens manage schedule totally remix ##ii forests occupation print nicholas brazilian strategic vampires engineers 76 roots seek correct instrumental und alfred backed hop ##des stanley robinson traveled wayne welcome austrian achieve 67 exit rates 1899 strip whereas ##cs sing deeply adventure bobby rick jamie careful components cap useful personality knee ##shi pushing hosts 02 protest ca ottoman symphony ##sis 63 boundary 1890 processes considering considerable tons ##work ##ft ##nia cooper trading dear conduct 91 illegal apple revolutionary holiday definition harder ##van jacob circumstances destruction ##lle popularity grip classified liverpool donald baltimore flows seeking honour approval 92 mechanical till happening statue critic increasingly immediate describe commerce stare ##ster indonesia meat rounds boats baker orthodox depression formally worn naked claire muttered sentence 11th emily document 77 criticism wished vessel spiritual bent virgin parker minimum murray lunch danny printed compilation keyboards false blow belonged 68 raising 78 cutting ##board pittsburgh ##up 9th shadows 81 hated indigenous jon 15th barry scholar ah ##zer oliver ##gy stick susan meetings attracted spell romantic ##ver ye 1895 photo demanded customers ##ac 1896 logan revival keys modified commanded jeans ##ious upset raw phil detective hiding resident vincent ##bly experiences diamond defeating coverage lucas external parks franchise helen bible successor percussion celebrated il lift profile clan romania ##ied mills ##su nobody achievement shrugged fault 1897 rhythm initiative breakfast carbon 700 69 lasted violent 74 wound ken killer gradually filmed °c dollars processing 94 remove criticized guests sang chemistry ##vin legislature disney ##bridge uniform escaped integrated proposal purple denied liquid karl influential morris nights stones intense experimental twisted 71 84 ##ld pace nazi mitchell ny blind reporter newspapers 14th centers burn basin forgotten surviving filed collections monastery losses manual couch description appropriate merely tag missions sebastian restoration replacing triple 73 elder julia warriors benjamin julian convinced stronger amazing declined versus merchant happens output finland bare barbara absence ignored dawn injuries ##port producers ##ram 82 luis ##ities kw admit expensive electricity nba exception symbol ##ving ladies shower sheriff characteristics ##je aimed button ratio effectively summit angle jury bears foster vessels pants executed evans dozen advertising kicked patrol 1889 competitions lifetime principles athletics ##logy birmingham sponsored 89 rob nomination 1893 acoustic ##sm creature longest ##tra credits harbor dust josh ##so territories milk infrastructure completion thailand indians leon archbishop ##sy assist pitch blake arrangement girlfriend serbian operational hence sad scent fur dj sessions hp refer rarely ##ora exists 1892 ##ten scientists dirty penalty burst portrait seed 79 pole limits rival 1894 stable alpha grave constitutional alcohol arrest flower mystery devil architectural relationships greatly habitat ##istic larry progressive remote cotton ##ics ##ok preserved reaches ##ming cited 86 vast scholarship decisions cbs joy teach 1885 editions knocked eve searching partly participation gap animated fate excellent ##ett na 87 alternate saints youngest ##ily climbed ##ita ##tors suggest ##ct discussion staying choir lakes jacket revenue nevertheless peaked instrument wondering annually managing neil 1891 signing terry ##ice apply clinical brooklyn aim catherine fuck farmers figured ninth pride hugh evolution ordinary involvement comfortable shouted tech encouraged taiwan representation sharing ##lia ##em panic exact cargo competing fat cried 83 1920s occasions pa cabin borders utah marcus ##isation badly muscles ##ance victorian transition warner bet permission ##rin slave terrible similarly shares seth uefa possession medals benefits colleges lowered perfectly mall transit ##ye ##kar publisher ##ened harrison deaths elevation ##ae asleep machines sigh ash hardly argument occasion parent leo decline 1888 contribution ##ua concentration 1000 opportunities hispanic guardian extent emotions hips mason volumes bloody controversy diameter steady mistake phoenix identify violin ##sk departure richmond spin funeral enemies 1864 gear literally connor random sergeant grab confusion 1865 transmission informed op leaning sacred suspended thinks gates portland luck agencies yours hull expert muscle layer practical sculpture jerusalem latest lloyd statistics deeper recommended warrior arkansas mess supports greg eagle 1880 recovered rated concerts rushed ##ano stops eggs files premiere keith ##vo delhi turner pit affair belief paint ##zing mate ##ach ##ev victim ##ology withdrew bonus styles fled ##ud glasgow technologies funded nbc adaptation ##ata portrayed cooperation supporters judges bernard justin hallway ralph ##ick graduating controversial distant continental spider bite ##ho recognize intention mixing ##ese egyptian bow tourism suppose claiming tiger dominated participants vi ##ru nurse partially tape ##rum psychology ##rn essential touring duo voting civilian emotional channels ##king apparent hebrew 1887 tommy carrier intersection beast hudson ##gar ##zo lab nova bench discuss costa ##ered detailed behalf drivers unfortunately obtain ##lis rocky ##dae siege friendship honey ##rian 1861 amy hang posted governments collins respond wildlife preferred operator ##po laura pregnant videos dennis suspected boots instantly weird automatic businessman alleged placing throwing ph mood 1862 perry venue jet remainder ##lli ##ci passion biological boyfriend 1863 dirt buffalo ron segment fa abuse ##era genre thrown stroke colored stress exercise displayed ##gen struggled ##tti abroad dramatic wonderful thereafter madrid component widespread ##sed tale citizen todd monday 1886 vancouver overseas forcing crying descent ##ris discussed substantial ranks regime 1870 provinces switch drum zane ted tribes proof lp cream researchers volunteer manor silk milan donated allies venture principle delivery enterprise ##ves ##ans bars traditionally witch reminded copper ##uk pete inter links colin grinned elsewhere competitive frequent ##oy scream ##hu tension texts submarine finnish defending defend pat detail 1884 affiliated stuart themes villa periods tool belgian ruling crimes answers folded licensed resort demolished hans lucy 1881 lion traded photographs writes craig ##fa trials generated beth noble debt percentage yorkshire erected ss viewed grades confidence ceased islam telephone retail ##ible chile m² roberts sixteen ##ich commented hampshire innocent dual pounds checked regulations afghanistan sung rico liberty assets bigger options angels relegated tribute wells attending leaf ##yan butler romanian forum monthly lisa patterns gmina ##tory madison hurricane rev ##ians bristol ##ula elite valuable disaster democracy awareness germans freyja ##ins loop absolutely paying populations maine sole prayer spencer releases doorway bull ##ani lover midnight conclusion ##sson thirteen lily mediterranean ##lt nhl proud sample ##hill drummer guinea ##ova murphy climb ##ston instant attributed horn ain railways steven ##ao autumn ferry opponent root traveling secured corridor stretched tales sheet trinity cattle helps indicates manhattan murdered fitted 1882 gentle grandmother mines shocked vegas produces ##light caribbean ##ou belong continuous desperate drunk historically trio waved raf dealing nathan bat murmured interrupted residing scientist pioneer harold aaron ##net delta attempting minority mini believes chorus tend lots eyed indoor load shots updated jail ##llo concerning connecting wealth ##ved slaves arrive rangers sufficient rebuilt ##wick cardinal flood muhammad whenever relation runners moral repair viewers arriving revenge punk assisted bath fairly breathe lists innings illustrated whisper nearest voters clinton ties ultimate screamed beijing lions andre fictional gathering comfort radar suitable dismissed hms ban pine wrist atmosphere voivodeship bid timber ##ned ##nan giants ##ane cameron recovery uss identical categories switched serbia laughter noah ensemble therapy peoples touching ##off locally pearl platforms everywhere ballet tables lanka herbert outdoor toured derek 1883 spaces contested swept 1878 exclusive slight connections ##dra winds prisoner collective bangladesh tube publicly wealthy thai ##ys isolated select ##ric insisted pen fortune ticket spotted reportedly animation enforcement tanks 110 decides wider lowest owen ##time nod hitting ##hn gregory furthermore magazines fighters solutions ##ery pointing requested peru reed chancellor knights mask worker eldest flames reduction 1860 volunteers ##tis reporting ##hl wire advisory endemic origins settlers pursue knock consumer 1876 eu compound creatures mansion sentenced ivan deployed guitars frowned involves mechanism kilometers perspective shops maps terminus duncan alien fist bridges ##pers heroes fed derby swallowed ##ros patent sara illness characterized adventures slide hawaii jurisdiction ##op organised ##side adelaide walks biology se ##ties rogers swing tightly boundaries ##rie prepare implementation stolen ##sha certified colombia edwards garage ##mm recalled ##ball rage harm nigeria breast ##ren furniture pupils settle ##lus cuba balls client alaska 21st linear thrust celebration latino genetic terror ##cia ##ening lightning fee witness lodge establishing skull ##ique earning hood ##ei rebellion wang sporting warned missile devoted activist porch worship fourteen package 1871 decorated ##shire housed ##ock chess sailed doctors oscar joan treat garcia harbour jeremy ##ire traditions dominant jacques ##gon ##wan relocated 1879 amendment sized companion simultaneously volleyball spun acre increases stopping loves belongs affect drafted tossed scout battles 1875 filming shoved munich tenure vertical romance pc ##cher argue ##ical craft ranging www opens honest tyler yesterday virtual ##let muslims reveal snake immigrants radical screaming speakers firing saving belonging ease lighting prefecture blame farmer hungry grows rubbed beam sur subsidiary ##cha armenian sao dropping conventional ##fer microsoft reply qualify spots 1867 sweat festivals ##ken immigration physician discover exposure sandy explanation isaac implemented ##fish hart initiated connect stakes presents heights householder pleased tourist regardless slip closest ##ction surely sultan brings riley preparation aboard slammed baptist experiment ongoing interstate organic playoffs ##ika 1877 130 ##tar hindu error tours tier plenty arrangements talks trapped excited sank ho athens 1872 denver welfare suburb athletes trick diverse belly exclusively yelled 1868 ##med conversion ##ette 1874 internationally computers conductor abilities sensitive hello dispute measured globe rocket prices amsterdam flights tigers inn municipalities emotion references 3d ##mus explains airlines manufactured pm archaeological 1873 interpretation devon comment ##ites settlements kissing absolute improvement suite impressed barcelona sullivan jefferson towers jesse julie ##tin ##lu grandson hi gauge regard rings interviews trace raymond thumb departments burns serial bulgarian scores demonstrated ##ix 1866 kyle alberta underneath romanized ##ward relieved acquisition phrase cliff reveals han cuts merger custom ##dar nee gilbert graduation ##nts assessment cafe difficulty demands swung democrat jennifer commons 1940s grove ##yo completing focuses sum substitute bearing stretch reception ##py reflected essentially destination pairs ##ched survival resource ##bach promoting doubles messages tear ##down ##fully parade florence harvey incumbent partial framework 900 pedro frozen procedure olivia controls ##mic shelter personally temperatures ##od brisbane tested sits marble comprehensive oxygen leonard ##kov inaugural iranian referring quarters attitude ##ivity mainstream lined mars dakota norfolk unsuccessful ##° explosion helicopter congressional ##sing inspector bitch seal departed divine ##ters coaching examination punishment manufacturer sink columns unincorporated signals nevada squeezed dylan dining photos martial manuel eighteen elevator brushed plates ministers ivy congregation ##len slept specialized taxes curve restricted negotiations likes statistical arnold inspiration execution bold intermediate significance margin ruler wheels gothic intellectual dependent listened eligible buses widow syria earn cincinnati collapsed recipient secrets accessible philippine maritime goddess clerk surrender breaks playoff database ##ified ##lon ideal beetle aspect soap regulation strings expand anglo shorter crosses retreat tough coins wallace directions pressing ##oon shipping locomotives comparison topics nephew ##mes distinction honors travelled sierra ibn ##over fortress sa recognised carved 1869 clients ##dan intent ##mar coaches describing bread ##ington beaten northwestern ##ona merit youtube collapse challenges em historians objective submitted virus attacking drake assume ##ere diseases marc stem leeds ##cus ##ab farming glasses ##lock visits nowhere fellowship relevant carries restaurants experiments 101 constantly bases targets shah tenth opponents verse territorial ##ira writings corruption ##hs instruction inherited reverse emphasis ##vic employee arch keeps rabbi watson payment uh ##ala nancy ##tre venice fastest sexy banned adrian properly ruth touchdown dollar boards metre circles edges favour comments ok travels liberation scattered firmly ##ular holland permitted diesel kenya den originated ##ral demons resumed dragged rider ##rus servant blinked extend torn ##ias ##sey input meal everybody cylinder kinds camps ##fe bullet logic ##wn croatian evolved healthy fool chocolate wise preserve pradesh ##ess respective 1850 ##ew chicken artificial gross corresponding convicted cage caroline dialogue ##dor narrative stranger mario br christianity failing trent commanding buddhist 1848 maurice focusing yale bike altitude ##ering mouse revised ##sley veteran ##ig pulls theology crashed campaigns legion ##ability drag excellence customer cancelled intensity excuse ##lar liga participating contributing printing ##burn variable ##rk curious bin legacy renaissance ##my symptoms binding vocalist dancer ##nie grammar gospel democrats ya enters sc diplomatic hitler ##ser clouds mathematical quit defended oriented ##heim fundamental hardware impressive equally convince confederate guilt chuck sliding ##ware magnetic narrowed petersburg bulgaria otto phd skill ##ama reader hopes pitcher reservoir hearts automatically expecting mysterious bennett extensively imagined seeds monitor fix ##ative journalism struggling signature ranch encounter photographer observation protests ##pin influences ##hr calendar ##all cruz croatia locomotive hughes naturally shakespeare basement hook uncredited faded theories approaches dare phillips filling fury obama ##ain efficient arc deliver min raid breeding inducted leagues efficiency axis montana eagles ##ked supplied instructions karen picking indicating trap anchor practically christians tomb vary occasional electronics lords readers newcastle faint innovation collect situations engagement 160 claude mixture ##feld peer tissue logo lean ##ration °f floors ##ven architects reducing ##our ##ments rope 1859 ottawa ##har samples banking declaration proteins resignation francois saudi advocate exhibited armor twins divorce ##ras abraham reviewed jo temporarily matrix physically pulse curled ##ena difficulties bengal usage ##ban annie riders certificate ##pi holes warsaw distinctive jessica ##mon mutual 1857 customs circular eugene removal loaded mere vulnerable depicted generations dame heir enormous lightly climbing pitched lessons pilots nepal ram google preparing brad louise renowned ##₂ liam ##ably plaza shaw sophie brilliant bills ##bar ##nik fucking mainland server pleasant seized veterans jerked fail beta brush radiation stored warmth southeastern nate sin raced berkeley joke athlete designation trunk ##low roland qualification archives heels artwork receives judicial reserves ##bed woke installation abu floating fake lesser excitement interface concentrated addressed characteristic amanda saxophone monk auto ##bus releasing egg dies interaction defender ce outbreak glory loving ##bert sequel consciousness http awake ski enrolled ##ress handling rookie brow somebody biography warfare amounts contracts presentation fabric dissolved challenged meter psychological lt elevated rally accurate ##tha hospitals undergraduate specialist venezuela exhibit shed nursing protestant fluid structural footage jared consistent prey ##ska succession reflect exile lebanon wiped suspect shanghai resting integration preservation marvel variant pirates sheep rounded capita sailing colonies manuscript deemed variations clarke functional emerging boxing relaxed curse azerbaijan heavyweight nickname editorial rang grid tightened earthquake flashed miguel rushing ##ches improvements boxes brooks 180 consumption molecular felix societies repeatedly variation aids civic graphics professionals realm autonomous receiver delayed workshop militia chairs trump canyon ##point harsh extending lovely happiness ##jan stake eyebrows embassy wellington hannah ##ella sony corners bishops swear cloth contents xi namely commenced 1854 stanford nashville courage graphic commitment garrison ##bin hamlet clearing rebels attraction literacy cooking ruins temples jenny humanity celebrate hasn freight sixty rebel bastard ##art newton ##ada deer ##ges ##ching smiles delaware singers ##ets approaching assists flame ##ph boulevard barrel planted ##ome pursuit ##sia consequences posts shallow invitation rode depot ernest kane rod concepts preston topic chambers striking blast arrives descendants montgomery ranges worlds ##lay ##ari span chaos praise ##ag fewer 1855 sanctuary mud fbi ##ions programmes maintaining unity harper bore handsome closure tournaments thunder nebraska linda facade puts satisfied argentine dale cork dome panama ##yl 1858 tasks experts ##ates feeding equation ##las ##ida ##tu engage bryan ##ax um quartet melody disbanded sheffield blocked gasped delay kisses maggie connects ##non sts poured creator publishers ##we guided ellis extinct hug gaining ##ord complicated ##bility poll clenched investigate ##use thereby quantum spine cdp humor kills administered semifinals ##du encountered ignore ##bu commentary ##maker bother roosevelt 140 plains halfway flowing cultures crack imprisoned neighboring airline ##ses ##view ##mate ##ec gather wolves marathon transformed ##ill cruise organisations carol punch exhibitions numbered alarm ratings daddy silently ##stein queens colours impression guidance liu tactical ##rat marshal della arrow ##ings rested feared tender owns bitter advisor escort ##ides spare farms grants ##ene dragons encourage colleagues cameras ##und sucked pile spirits prague statements suspension landmark fence torture recreation bags permanently survivors pond spy predecessor bombing coup ##og protecting transformation glow ##lands ##book dug priests andrea feat barn jumping ##chen ##ologist ##con casualties stern auckland pipe serie revealing ba ##bel trevor mercy spectrum yang consist governing collaborated possessed epic comprises blew shane ##ack lopez honored magical sacrifice judgment perceived hammer mtv baronet tune das missionary sheets 350 neutral oral threatening attractive shade aims seminary ##master estates 1856 michel wounds refugees manufacturers ##nic mercury syndrome porter ##iya ##din hamburg identification upstairs purse widened pause cared breathed affiliate santiago prevented celtic fisher 125 recruited byzantine reconstruction farther ##mp diet sake au spite sensation ##ert blank separation 105 ##hon vladimir armies anime ##lie accommodate orbit cult sofia archive ##ify ##box founders sustained disorder honours northeastern mia crops violet threats blanket fires canton followers southwestern prototype voyage assignment altered moderate protocol pistol ##eo questioned brass lifting 1852 math authored ##ual doug dimensional dynamic ##san 1851 pronounced grateful quest uncomfortable boom presidency stevens relating politicians chen barrier quinn diana mosque tribal cheese palmer portions sometime chester treasure wu bend download millions reforms registration ##osa consequently monitoring ate preliminary brandon invented ps eaten exterior intervention ports documented log displays lecture sally favourite ##itz vermont lo invisible isle breed ##ator journalists relay speaks backward explore midfielder actively stefan procedures cannon blond kenneth centered servants chains libraries malcolm essex henri slavery ##hal facts fairy coached cassie cats washed cop ##fi announcement item 2000s vinyl activated marco frontier growled curriculum ##das loyal accomplished leslie ritual kenny ##00 vii napoleon hollow hybrid jungle stationed friedrich counted ##ulated platinum theatrical seated col rubber glen 1840 diversity healing extends id provisions administrator columbus ##oe tributary te assured org ##uous prestigious examined lectures grammy ronald associations bailey allan essays flute believing consultant proceedings travelling 1853 kit kerala yugoslavia buddy methodist ##ith burial centres batman ##nda discontinued bo dock stockholm lungs severely ##nk citing manga ##ugh steal mumbai iraqi robot celebrity bride broadcasts abolished pot joel overhead franz packed reconnaissance johann acknowledged introduce handled doctorate developments drinks alley palestine ##nis ##aki proceeded recover bradley grain patch afford infection nationalist legendary ##ath interchange virtually gen gravity exploration amber vital wishes powell doctrine elbow screenplay ##bird contribute indonesian pet creates ##com enzyme kylie discipline drops manila hunger ##ien layers suffer fever bits monica keyboard manages ##hood searched appeals ##bad testament grande reid ##war beliefs congo ##ification ##dia si requiring ##via casey 1849 regret streak rape depends syrian sprint pound tourists upcoming pub ##xi tense ##els practiced echo nationwide guild motorcycle liz ##zar chiefs desired elena bye precious absorbed relatives booth pianist ##mal citizenship exhausted wilhelm ##ceae ##hed noting quarterback urge hectares ##gue ace holly ##tal blonde davies parked sustainable stepping twentieth airfield galaxy nest chip ##nell tan shaft paulo requirement ##zy paradise tobacco trans renewed vietnamese ##cker ##ju suggesting catching holmes enjoying md trips colt holder butterfly nerve reformed cherry bowling trailer carriage goodbye appreciate toy joshua interactive enabled involve ##kan collar determination bunch facebook recall shorts superintendent episcopal frustration giovanni nineteenth laser privately array circulation ##ovic armstrong deals painful permit discrimination ##wi aires retiring cottage ni ##sta horizon ellen jamaica ripped fernando chapters playstation patron lecturer navigation behaviour genes georgian export solomon rivals swift seventeen rodriguez princeton independently sox 1847 arguing entity casting hank criteria oakland geographic milwaukee reflection expanding conquest dubbed ##tv halt brave brunswick doi arched curtis divorced predominantly somerset streams ugly zoo horrible curved buenos fierce dictionary vector theological unions handful stability chan punjab segments ##lly altar ignoring gesture monsters pastor ##stone thighs unexpected operators abruptly coin compiled associates improving migration pin ##ose compact collegiate reserved ##urs quarterfinals roster restore assembled hurry oval ##cies 1846 flags martha ##del victories sharply ##rated argues deadly neo drawings symbols performer ##iel griffin restrictions editing andrews java journals arabia compositions dee pierce removing hindi casino runway civilians minds nasa hotels ##zation refuge rent retain potentially conferences suburban conducting ##tto ##tions ##tle descended massacre ##cal ammunition terrain fork souls counts chelsea durham drives cab ##bank perth realizing palestinian finn simpson ##dal betty ##ule moreover particles cardinals tent evaluation extraordinary ##oid inscription ##works wednesday chloe maintains panels ashley trucks ##nation cluster sunlight strikes zhang ##wing dialect canon ##ap tucked ##ws collecting ##mas ##can ##sville maker quoted evan franco aria buying cleaning eva closet provision apollo clinic rat ##ez necessarily ac ##gle ##ising venues flipped cent spreading trustees checking authorized ##sco disappointed ##ado notion duration trumpet hesitated topped brussels rolls theoretical hint define aggressive repeat wash peaceful optical width allegedly mcdonald strict copyright ##illa investors mar jam witnesses sounding miranda michelle privacy hugo harmony ##pp valid lynn glared nina 102 headquartered diving boarding gibson ##ncy albanian marsh routine dealt enhanced er intelligent substance targeted enlisted discovers spinning observations pissed smoking rebecca capitol visa varied costume seemingly indies compensation surgeon thursday arsenal westminster suburbs rid anglican ##ridge knots foods alumni lighter fraser whoever portal scandal ##ray gavin advised instructor flooding terrorist ##ale teenage interim senses duck teen thesis abby eager overcome ##ile newport glenn rises shame ##cc prompted priority forgot bomber nicolas protective 360 cartoon katherine breeze lonely trusted henderson richardson relax banner candy palms remarkable ##rio legends cricketer essay ordained edmund rifles trigger ##uri ##away sail alert 1830 audiences penn sussex siblings pursued indianapolis resist rosa consequence succeed avoided 1845 ##ulation inland ##tie ##nna counsel profession chronicle hurried ##una eyebrow eventual bleeding innovative cure ##dom committees accounting con scope hardy heather tenor gut herald codes tore scales wagon ##oo luxury tin prefer fountain triangle bonds darling convoy dried traced beings troy accidentally slam findings smelled joey lawyers outcome steep bosnia configuration shifting toll brook performers lobby philosophical construct shrine aggregate boot cox phenomenon savage insane solely reynolds lifestyle ##ima nationally holdings consideration enable edgar mo mama ##tein fights relegation chances atomic hub conjunction awkward reactions currency finale kumar underwent steering elaborate gifts comprising melissa veins reasonable sunshine chi solve trails inhabited elimination ethics huh ana molly consent apartments layout marines ##ces hunters bulk ##oma hometown ##wall ##mont cracked reads neighbouring withdrawn admission wingspan damned anthology lancashire brands batting forgive cuban awful ##lyn 104 dimensions imagination ##ade dante ##ship tracking desperately goalkeeper ##yne groaned workshops confident burton gerald milton circus uncertain slope copenhagen sophia fog philosopher portraits accent cycling varying gripped larvae garrett specified scotia mature luther kurt rap ##kes aerial 750 ferdinand heated es transported ##shan safely nonetheless ##orn ##gal motors demanding ##sburg startled ##brook ally generate caps ghana stained demo mentions beds ap afterward diary ##bling utility ##iro richards 1837 conspiracy conscious shining footsteps observer cyprus urged loyalty developer probability olive upgraded gym miracle insects graves 1844 ourselves hydrogen amazon katie tickets poets ##pm planes ##pan prevention witnessed dense jin randy tang warehouse monroe bang archived elderly investigations alec granite mineral conflicts controlling aboriginal carlo ##zu mechanics stan stark rhode skirt est ##berry bombs respected ##horn imposed limestone deny nominee memphis grabbing disabled ##als amusement aa frankfurt corn referendum varies slowed disk firms unconscious incredible clue sue ##zhou twist ##cio joins idaho chad developers computing destroyer 103 mortal tucker kingston choices yu carson 1800 os whitney geneva pretend dimension staged plateau maya ##une freestyle ##bc rovers hiv ##ids tristan classroom prospect ##hus honestly diploma lied thermal auxiliary feast unlikely iata ##tel morocco pounding treasury lithuania considerably 1841 dish 1812 geological matching stumbled destroying marched brien advances cake nicole belle settling measuring directing ##mie tuesday bassist capabilities stunned fraud torpedo ##list ##phone anton wisdom surveillance ruined ##ulate lawsuit healthcare theorem halls trend aka horizontal dozens acquire lasting swim hawk gorgeous fees vicinity decrease adoption tactics ##ography pakistani ##ole draws ##hall willie burke heath algorithm integral powder elliott brigadier jackie tate varieties darker ##cho lately cigarette specimens adds ##ree ##ensis ##inger exploded finalist cia murders wilderness arguments nicknamed acceptance onwards manufacture robertson jets tampa enterprises blog loudly composers nominations 1838 ai malta inquiry automobile hosting viii rays tilted grief museums strategies furious euro equality cohen poison surrey wireless governed ridiculous moses ##esh ##room vanished ##ito barnes attract morrison istanbul ##iness absent rotation petition janet ##logical satisfaction custody deliberately observatory comedian surfaces pinyin novelist strictly canterbury oslo monks embrace ibm jealous photograph continent dorothy marina doc excess holden allegations explaining stack avoiding lance storyline majesty poorly spike dos bradford raven travis classics proven voltage pillow fists butt 1842 interpreted ##car 1839 gage telegraph lens promising expelled casual collector zones ##min silly nintendo ##kh ##bra downstairs chef suspicious afl flies vacant uganda pregnancy condemned lutheran estimates cheap decree saxon proximity stripped idiot deposits contrary presenter magnus glacier im offense edwin ##ori upright ##long bolt ##ois toss geographical ##izes environments delicate marking abstract xavier nails windsor plantation occurring equity saskatchewan fears drifted sequences vegetation revolt ##stic 1843 sooner fusion opposing nato skating 1836 secretly ruin lease ##oc edit ##nne flora anxiety ruby ##ological ##mia tel bout taxi emmy frost rainbow compounds foundations rainfall assassination nightmare dominican ##win achievements deserve orlando intact armenia ##nte calgary valentine 106 marion proclaimed theodore bells courtyard thigh gonzalez console troop minimal monte everyday ##ence ##if supporter terrorism buck openly presbyterian activists carpet ##iers rubbing uprising ##yi cute conceived legally ##cht millennium cello velocity ji rescued cardiff 1835 rex concentrate senators beard rendered glowing battalions scouts competitors sculptor catalogue arctic ion raja bicycle wow glancing lawn ##woman gentleman lighthouse publish predicted calculated ##val variants ##gne strain ##ui winston deceased ##nus touchdowns brady caleb sinking echoed crush hon blessed protagonist hayes endangered magnitude editors ##tine estimate responsibilities ##mel backup laying consumed sealed zurich lovers frustrated ##eau ahmed kicking mit treasurer 1832 biblical refuse terrified pump agrees genuine imprisonment refuses plymouth ##hen lou ##nen tara trembling antarctic ton learns ##tas crap crucial faction atop ##borough wrap lancaster odds hopkins erik lyon ##eon bros ##ode snap locality tips empress crowned cal acclaimed chuckled ##ory clara sends mild towel ##fl ##day ##а wishing assuming interviewed ##bal ##die interactions eden cups helena ##lf indie beck ##fire batteries filipino wizard parted ##lam traces ##born rows idol albany delegates ##ees ##sar discussions ##ex notre instructed belgrade highways suggestion lauren possess orientation alexandria abdul beats salary reunion ludwig alright wagner intimate pockets slovenia hugged brighton merchants cruel stole trek slopes repairs enrollment politically underlying promotional counting boeing ##bb isabella naming ##и keen bacteria listing separately belfast ussr 450 lithuanian anybody ribs sphere martinez cock embarrassed proposals fragments nationals ##fs ##wski premises fin 1500 alpine matched freely bounded jace sleeve ##af gaming pier populated evident ##like frances flooded ##dle frightened pour trainer framed visitor challenging pig wickets ##fold infected email ##pes arose ##aw reward ecuador oblast vale ch shuttle ##usa bach rankings forbidden cornwall accordance salem consumers bruno fantastic toes machinery resolved julius remembering propaganda iceland bombardment tide contacts wives ##rah concerto macdonald albania implement daisy tapped sudan helmet angela mistress ##lic crop sunk finest ##craft hostile ##ute ##tsu boxer fr paths adjusted habit ballot supervision soprano ##zen bullets wicked sunset regiments disappear lamp performs app ##gia ##oa rabbit digging incidents entries ##cion dishes ##oi introducing ##ati ##fied freshman slot jill tackles baroque backs ##iest lone sponsor destiny altogether convert ##aro consensus shapes demonstration basically feminist auction artifacts ##bing strongest twitter halifax 2019 allmusic mighty smallest precise alexandra viola ##los ##ille manuscripts ##illo dancers ari managers monuments blades barracks springfield maiden consolidated electron ##end berry airing wheat nobel inclusion blair payments geography bee cc eleanor react ##hurst afc manitoba ##yu su lineup fitness recreational investments airborne disappointment ##dis edmonton viewing ##row renovation ##cast infant bankruptcy roses aftermath pavilion ##yer carpenter withdrawal ladder ##hy discussing popped reliable agreements rochester ##abad curves bombers 220 rao reverend decreased choosing 107 stiff consulting naples crawford tracy ka ribbon cops ##lee crushed deciding unified teenager accepting flagship explorer poles sanchez inspection revived skilled induced exchanged flee locals tragedy swallow loading hanna demonstrate ##ela salvador flown contestants civilization ##ines wanna rhodes fletcher hector knocking considers ##ough nash mechanisms sensed mentally walt unclear ##eus renovated madame ##cks crews governmental ##hin undertaken monkey ##ben ##ato fatal armored copa caves governance grasp perception certification froze damp tugged wyoming ##rg ##ero newman ##lor nerves curiosity graph 115 ##ami withdraw tunnels dull meredith moss exhibits neighbors communicate accuracy explored raiders republicans secular kat superman penny criticised ##tch freed update conviction wade ham likewise delegation gotta doll promises technological myth nationality resolve convent ##mark sharon dig sip coordinator entrepreneur fold ##dine capability councillor synonym blown swan cursed 1815 jonas haired sofa canvas keeper rivalry ##hart rapper speedway swords postal maxwell estonia potter recurring ##nn ##ave errors ##oni cognitive 1834 ##² claws nadu roberto bce wrestler ellie ##ations infinite ink ##tia presumably finite staircase 108 noel patricia nacional ##cation chill eternal tu preventing prussia fossil limbs ##logist ernst frog perez rene ##ace pizza prussian ##ios ##vy molecules regulatory answering opinions sworn lengths supposedly hypothesis upward habitats seating ancestors drank yield hd synthesis researcher modest ##var mothers peered voluntary homeland ##the acclaim ##igan static valve luxembourg alto carroll fe receptor norton ambulance ##tian johnston catholics depicting jointly elephant gloria mentor badge ahmad distinguish remarked councils precisely allison advancing detection crowded ##10 cooperative ankle mercedes dagger surrendered pollution commit subway jeffrey lesson sculptures provider ##fication membrane timothy rectangular fiscal heating teammate basket particle anonymous deployment ##ple missiles courthouse proportion shoe sec ##ller complaints forbes blacks abandon remind sizes overwhelming autobiography natalie ##awa risks contestant countryside babies scorer invaded enclosed proceed hurling disorders ##cu reflecting continuously cruiser graduates freeway investigated ore deserved maid blocking phillip jorge shakes dove mann variables lacked burden accompanying que consistently organizing provisional complained endless ##rm tubes juice georges krishna mick labels thriller ##uch laps arcade sage snail ##table shannon fi laurence seoul vacation presenting hire churchill surprisingly prohibited savannah technically ##oli 170 ##lessly testimony suited speeds toys romans mlb flowering measurement talented kay settings charleston expectations shattered achieving triumph ceremonies portsmouth lanes mandatory loser stretching cologne realizes seventy cornell careers webb ##ulating americas budapest ava suspicion ##ison yo conrad ##hai sterling jessie rector ##az 1831 transform organize loans christine volcanic warrant slender summers subfamily newer danced dynamics rhine proceeds heinrich gastropod commands sings facilitate easter ra positioned responses expense fruits yanked imported 25th velvet vic primitive tribune baldwin neighbourhood donna rip hay pr ##uro 1814 espn welcomed ##aria qualifier glare highland timing ##cted shells eased geometry louder exciting slovakia ##sion ##iz ##lot savings prairie ##ques marching rafael tonnes ##lled curtain preceding shy heal greene worthy ##pot detachment bury sherman ##eck reinforced seeks bottles contracted duchess outfit walsh ##sc mickey ##ase geoffrey archer squeeze dawson eliminate invention ##enberg neal ##eth stance dealer coral maple retire polo simplified ##ht 1833 hid watts backwards jules ##oke genesis mt frames rebounds burma woodland moist santos whispers drained subspecies ##aa streaming ulster burnt correspondence maternal gerard denis stealing ##load genius duchy ##oria inaugurated momentum suits placement sovereign clause thames ##hara confederation reservation sketch yankees lets rotten charm hal verses ultra commercially dot salon citation adopt winnipeg mist allocated cairo ##boy jenkins interference objectives ##wind 1820 portfolio armoured sectors ##eh initiatives ##world integrity exercises robe tap ab gazed ##tones distracted rulers 111 favorable jerome tended cart factories ##eri diplomat valued gravel charitable ##try calvin exploring chang shepherd terrace pdf pupil ##ural reflects ups ##rch governors shelf depths ##nberg trailed crest tackle ##nian ##ats hatred ##kai clare makers ethiopia longtime detected embedded lacking slapped rely thomson anticipation iso morton successive agnes screenwriter straightened philippe playwright haunted licence iris intentions sutton 112 logical correctly ##weight branded licked tipped silva ricky narrator requests ##ents greeted supernatural cow ##wald lung refusing employer strait gaelic liner ##piece zoe sabha ##mba driveway harvest prints bates reluctantly threshold algebra ira wherever coupled 240 assumption picks ##air designers raids gentlemen ##ean roller blowing leipzig locks screw dressing strand ##lings scar dwarf depicts ##nu nods ##mine differ boris ##eur yuan flip ##gie mob invested questioning applying ##ture shout ##sel gameplay blamed illustrations bothered weakness rehabilitation ##of ##zes envelope rumors miners leicester subtle kerry ##ico ferguson ##fu premiership ne ##cat bengali prof catches remnants dana ##rily shouting presidents baltic ought ghosts dances sailors shirley fancy dominic ##bie madonna ##rick bark buttons gymnasium ashes liver toby oath providence doyle evangelical nixon cement carnegie embarked hatch surroundings guarantee needing pirate essence ##bee filter crane hammond projected immune percy twelfth ##ult regent doctoral damon mikhail ##ichi lu critically elect realised abortion acute screening mythology steadily ##fc frown nottingham kirk wa minneapolis ##rra module algeria mc nautical encounters surprising statues availability shirts pie alma brows munster mack soup crater tornado sanskrit cedar explosive bordered dixon planets stamp exam happily ##bble carriers kidnapped ##vis accommodation emigrated ##met knockout correspondent violation profits peaks lang specimen agenda ancestry pottery spelling equations obtaining ki linking 1825 debris asylum ##20 buddhism teddy ##ants gazette ##nger ##sse dental eligibility utc fathers averaged zimbabwe francesco coloured hissed translator lynch mandate humanities mackenzie uniforms lin ##iana ##gio asset mhz fitting samantha genera wei rim beloved shark riot entities expressions indo carmen slipping owing abbot neighbor sidney ##av rats recommendations encouraging squadrons anticipated commanders conquered ##oto donations diagnosed ##mond divide ##iva guessed decoration vernon auditorium revelation conversations ##kers ##power herzegovina dash alike protested lateral herman accredited mg ##gent freeman mel fiji crow crimson ##rine livestock ##pped humanitarian bored oz whip ##lene ##ali legitimate alter grinning spelled anxious oriental wesley ##nin ##hole carnival controller detect ##ssa bowed educator kosovo macedonia ##sin occupy mastering stephanie janeiro para unaware nurses noon 135 cam hopefully ranger combine sociology polar rica ##eer neill ##sman holocaust ##ip doubled lust 1828 109 decent cooling unveiled ##card 1829 nsw homer chapman meyer ##gin dive mae reagan expertise ##gled darwin brooke sided prosecution investigating comprised petroleum genres reluctant differently trilogy johns vegetables corpse highlighted lounge pension unsuccessfully elegant aided ivory beatles amelia cain dubai sunny immigrant babe click ##nder underwater pepper combining mumbled atlas horns accessed ballad physicians homeless gestured rpm freak louisville corporations patriots prizes rational warn modes decorative overnight din troubled phantom ##ort monarch sheer ##dorf generals guidelines organs addresses ##zon enhance curling parishes cord ##kie linux caesar deutsche bavaria ##bia coleman cyclone ##eria bacon petty ##yama ##old hampton diagnosis 1824 throws complexity rita disputed ##₃ pablo ##sch marketed trafficking ##ulus examine plague formats ##oh vault faithful ##bourne webster ##ox highlights ##ient ##ann phones vacuum sandwich modeling ##gated bolivia clergy qualities isabel ##nas ##ars wears screams reunited annoyed bra ##ancy ##rate differential transmitter tattoo container poker ##och excessive resides cowboys ##tum augustus trash providers statute retreated balcony reversed void storey preceded masses leap laughs neighborhoods wards schemes falcon santo battlefield pad ronnie thread lesbian venus ##dian beg sandstone daylight punched gwen analog stroked wwe acceptable measurements dec toxic ##kel adequate surgical economist parameters varsity ##sberg quantity ella ##chy ##rton countess generating precision diamonds expressway ga ##ı 1821 uruguay talents galleries expenses scanned colleague outlets ryder lucien ##ila paramount ##bon syracuse dim fangs gown sweep ##sie toyota missionaries websites ##nsis sentences adviser val trademark spells ##plane patience starter slim ##borg toe incredibly shoots elliot nobility ##wyn cowboy endorsed gardner tendency persuaded organisms emissions kazakhstan amused boring chips themed ##hand llc constantinople chasing systematic guatemala borrowed erin carey ##hard highlands struggles 1810 ##ifying ##ced wong exceptions develops enlarged kindergarten castro ##ern ##rina leigh zombie juvenile ##most consul ##nar sailor hyde clarence intensive pinned nasty useless jung clayton stuffed exceptional ix apostolic 230 transactions ##dge exempt swinging cove religions ##ash shields dairy bypass 190 pursuing bug joyce bombay chassis southampton chat interact redesignated ##pen nascar pray salmon rigid regained malaysian grim publicity constituted capturing toilet delegate purely tray drift loosely striker weakened trinidad mitch itv defines transmitted ming scarlet nodding fitzgerald fu narrowly sp tooth standings virtue ##₁ ##wara ##cting chateau gloves lid ##nel hurting conservatory ##pel sinclair reopened sympathy nigerian strode advocated optional chronic discharge ##rc suck compatible laurel stella shi fails wage dodge 128 informal sorts levi buddha villagers ##aka chronicles heavier summoned gateway 3000 eleventh jewelry translations accordingly seas ##ency fiber pyramid cubic dragging ##ista caring ##ops android contacted lunar ##dt kai lisbon patted 1826 sacramento theft madagascar subtropical disputes ta holidays piper willow mare cane itunes newfoundland benny companions dong raj observe roar charming plaque tibetan fossils enacted manning bubble tina tanzania ##eda ##hir funk swamp deputies cloak ufc scenario par scratch metals anthem guru engaging specially ##boat dialects nineteen cecil duet disability messenger unofficial ##lies defunct eds moonlight drainage surname puzzle honda switching conservatives mammals knox broadcaster sidewalk cope ##ried benson princes peterson ##sal bedford sharks eli wreck alberto gasp archaeology lgbt teaches securities madness compromise waving coordination davidson visions leased possibilities eighty jun fernandez enthusiasm assassin sponsorship reviewer kingdoms estonian laboratories ##fy ##nal applies verb celebrations ##zzo rowing lightweight sadness submit mvp balanced dude ##vas explicitly metric magnificent mound brett mohammad mistakes irregular ##hing ##ass sanders betrayed shipped surge ##enburg reporters termed georg pity verbal bulls abbreviated enabling appealed ##are ##atic sicily sting heel sweetheart bart spacecraft brutal monarchy ##tter aberdeen cameo diane ##ub survivor clyde ##aries complaint ##makers clarinet delicious chilean karnataka coordinates 1818 panties ##rst pretending ar dramatically kiev bella tends distances 113 catalog launching instances telecommunications portable lindsay vatican ##eim angles aliens marker stint screens bolton ##rne judy wool benedict plasma europa spark imaging filmmaker swiftly ##een contributor ##nor opted stamps apologize financing butter gideon sophisticated alignment avery chemicals yearly speculation prominence professionally ##ils immortal institutional inception wrists identifying tribunal derives gains ##wo papal preference linguistic vince operative brewery ##ont unemployment boyd ##ured ##outs albeit prophet 1813 bi ##rr ##face ##rad quarterly asteroid cleaned radius temper ##llen telugu jerk viscount menu ##ote glimpse ##aya yacht hawaiian baden ##rl laptop readily ##gu monetary offshore scots watches ##yang ##arian upgrade needle xbox lea encyclopedia flank fingertips ##pus delight teachings confirm roth beaches midway winters ##iah teasing daytime beverly gambling bonnie ##backs regulated clement hermann tricks knot ##shing ##uring ##vre detached ecological owed specialty byron inventor bats stays screened unesco midland trim affection ##ander ##rry jess thoroughly feedback ##uma chennai strained heartbeat wrapping overtime pleaded ##sworth mon leisure oclc ##tate ##ele feathers angelo thirds nuts surveys clever gill commentator ##dos darren rides gibraltar ##nc ##mu dissolution dedication shin meals saddle elvis reds chaired taller appreciation functioning niece favored advocacy robbie criminals suffolk yugoslav passport constable congressman hastings vera ##rov consecrated sparks ecclesiastical confined ##ovich muller floyd nora 1822 paved 1827 cumberland ned saga spiral ##flow appreciated yi collaborative treating similarities feminine finishes ##ib jade import ##nse ##hot champagne mice securing celebrities helsinki attributes ##gos cousins phases ache lucia gandhi submission vicar spear shine tasmania biting detention constitute tighter seasonal ##gus terrestrial matthews ##oka effectiveness parody philharmonic ##onic 1816 strangers encoded consortium guaranteed regards shifts tortured collision supervisor inform broader insight theaters armour emeritus blink incorporates mapping ##50 ##ein handball flexible ##nta substantially generous thief ##own carr loses 1793 prose ucla romeo generic metallic realization damages mk commissioners zach default ##ther helicopters lengthy stems spa partnered spectators rogue indication penalties teresa 1801 sen ##tric dalton ##wich irving photographic ##vey dell deaf peters excluded unsure ##vable patterson crawled ##zio resided whipped latvia slower ecole pipes employers maharashtra comparable va textile pageant ##gel alphabet binary irrigation chartered choked antoine offs waking supplement ##wen quantities demolition regain locate urdu folks alt 114 ##mc scary andreas whites ##ava classrooms mw aesthetic publishes valleys guides cubs johannes bryant conventions affecting ##itt drain awesome isolation prosecutor ambitious apology captive downs atmospheric lorenzo aisle beef foul ##onia kidding composite disturbed illusion natives ##ffer emi rockets riverside wartime painters adolf melted ##ail uncertainty simulation hawks progressed meantime builder spray breach unhappy regina russians ##urg determining ##tation tram 1806 ##quin aging ##12 1823 garion rented mister diaz terminated clip 1817 depend nervously disco owe defenders shiva notorious disbelief shiny worcester ##gation ##yr trailing undertook islander belarus limitations watershed fuller overlooking utilized raphael 1819 synthetic breakdown klein ##nate moaned memoir lamb practicing ##erly cellular arrows exotic ##graphy witches 117 charted rey hut hierarchy subdivision freshwater giuseppe aloud reyes qatar marty sideways utterly sexually jude prayers mccarthy softball blend damien ##gging ##metric wholly erupted lebanese negro revenues tasted comparative teamed transaction labeled maori sovereignty parkway trauma gran malay 121 advancement descendant 2020 buzz salvation inventory symbolic ##making antarctica mps ##gas ##bro mohammed myanmar holt submarines tones ##lman locker patriarch bangkok emerson remarks predators kin afghan confession norwich rental emerge advantages ##zel rca ##hold shortened storms aidan ##matic autonomy compliance ##quet dudley atp ##osis 1803 motto documentation summary professors spectacular christina archdiocese flashing innocence remake ##dell psychic reef scare employ rs sticks meg gus leans ##ude accompany bergen tomas ##iko doom wages pools ##nch ##bes breasts scholarly alison outline brittany breakthrough willis realistic ##cut ##boro competitor ##stan pike picnic icon designing commercials washing villain skiing micro costumes auburn halted executives ##hat logistics cycles vowel applicable barrett exclaimed eurovision eternity ramon ##umi ##lls modifications sweeping disgust ##uck torch aviv ensuring rude dusty sonic donovan outskirts cu pathway ##band ##gun ##lines disciplines acids cadet paired ##40 sketches ##sive marriages ##⁺ folding peers slovak implies admired ##beck 1880s leopold instinct attained weston megan horace ##ination dorsal ingredients evolutionary ##its complications deity lethal brushing levy deserted institutes posthumously delivering telescope coronation motivated rapids luc flicked pays volcano tanner weighed ##nica crowds frankie gifted addressing granddaughter winding ##rna constantine gomez ##front landscapes rudolf anthropology slate werewolf ##lio astronomy circa rouge dreaming sack knelt drowned naomi prolific tracked freezing herb ##dium agony randall twisting wendy deposit touches vein wheeler ##bbled ##bor batted retaining tire presently compare specification daemon nigel ##grave merry recommendation czechoslovakia sandra ng roma ##sts lambert inheritance sheikh winchester cries examining ##yle comeback cuisine nave ##iv ko retrieve tomatoes barker polished defining irene lantern personalities begging tract swore 1809 175 ##gic omaha brotherhood ##rley haiti ##ots exeter ##ete ##zia steele dumb pearson 210 surveyed elisabeth trends ##ef fritz ##rf premium bugs fraction calmly viking ##birds tug inserted unusually ##ield confronted distress crashing brent turks resign ##olo cambodia gabe sauce ##kal evelyn 116 extant clusters quarry teenagers luna ##lers ##ister affiliation drill ##ashi panthers scenic libya anita strengthen inscriptions ##cated lace sued judith riots ##uted mint ##eta preparations midst dub challenger ##vich mock cf displaced wicket breaths enables schmidt analyst ##lum ag highlight automotive axe josef newark sufficiently resembles 50th ##pal flushed mum traits ##ante commodore incomplete warming titular ceremonial ethical 118 celebrating eighteenth cao lima medalist mobility strips snakes ##city miniature zagreb barton escapes umbrella automated doubted differs cooled georgetown dresden cooked fade wyatt rna jacobs carlton abundant stereo boost madras inning ##hia spur ip malayalam begged osaka groan escaping charging dose vista ##aj bud papa communists advocates edged tri ##cent resemble peaking necklace fried montenegro saxony goose glances stuttgart curator recruit grocery sympathetic ##tting ##fort 127 lotus randolph ancestor ##rand succeeding jupiter 1798 macedonian ##heads hiking 1808 handing fischer ##itive garbage node ##pies prone singular papua inclined attractions italia pouring motioned grandma garnered jacksonville corp ego ringing aluminum ##hausen ordering ##foot drawer traders synagogue ##play ##kawa resistant wandering fragile fiona teased var hardcore soaked jubilee decisive exposition mercer poster valencia hale kuwait 1811 ##ises ##wr ##eed tavern gamma 122 johan ##uer airways amino gil ##ury vocational domains torres ##sp generator folklore outcomes ##keeper canberra shooter fl beams confrontation ##lling ##gram feb aligned forestry pipeline jax motorway conception decay ##tos coffin ##cott stalin 1805 escorted minded ##nam sitcom purchasing twilight veronica additions passive tensions straw 123 frequencies 1804 refugee cultivation ##iate christie clary bulletin crept disposal ##rich ##zong processor crescent ##rol bmw emphasized whale nazis aurora ##eng dwelling hauled sponsors toledo mega ideology theatres tessa cerambycidae saves turtle cone suspects kara rusty yelling greeks mozart shades cocked participant ##tro shire spit freeze necessity ##cos inmates nielsen councillors loaned uncommon omar peasants botanical offspring daniels formations jokes 1794 pioneers sigma licensing ##sus wheelchair polite 1807 liquor pratt trustee ##uta forewings balloon ##zz kilometre camping explicit casually shawn foolish teammates nm hassan carrie judged satisfy vanessa knives selective cnn flowed ##lice eclipse stressed eliza mathematician cease cultivated ##roy commissions browns ##ania destroyers sheridan meadow ##rius minerals ##cial downstream clash gram memoirs ventures baha seymour archie midlands edith fare flynn invite canceled tiles stabbed boulder incorporate amended camden facial mollusk unreleased descriptions yoga grabs 550 raises ramp shiver ##rose coined pioneering tunes qing warwick tops 119 melanie giles ##rous wandered ##inal annexed nov 30th unnamed ##ished organizational airplane normandy stoke whistle blessing violations chased holders shotgun ##ctic outlet reactor ##vik tires tearing shores fortified mascot constituencies nc columnist productive tibet ##rta lineage hooked oct tapes judging cody ##gger hansen kashmir triggered ##eva solved cliffs ##tree resisted anatomy protesters transparent implied ##iga injection mattress excluding ##mbo defenses helpless devotion ##elli growl liberals weber phenomena atoms plug ##iff mortality apprentice howe convincing aaa swimmer barber leone promptly sodium def nowadays arise ##oning gloucester corrected dignity norm erie ##ders elders evacuated sylvia compression ##yar hartford pose backpack reasoning accepts 24th wipe millimetres marcel ##oda dodgers albion 1790 overwhelmed aerospace oaks 1795 showcase acknowledge recovering nolan ashe hurts geology fashioned disappearance farewell swollen shrug marquis wimbledon 124 rue 1792 commemorate reduces experiencing inevitable calcutta intel ##court murderer sticking fisheries imagery bloom 280 brake ##inus gustav hesitation memorable po viral beans accidents tunisia antenna spilled consort treatments aye perimeter ##gard donation hostage migrated banker addiction apex lil trout ##ously conscience ##nova rams sands genome passionate troubles ##lets ##set amid ##ibility ##ret higgins exceed vikings ##vie payne ##zan muscular ##ste defendant sucking ##wal ibrahim fuselage claudia vfl europeans snails interval ##garh preparatory statewide tasked lacrosse viktor ##lation angola ##hra flint implications employs teens patrons stall weekends barriers scrambled nucleus tehran jenna parsons lifelong robots displacement 5000 ##bles precipitation ##gt knuckles clutched 1802 marrying ecology marx accusations declare scars kolkata mat meadows bermuda skeleton finalists vintage crawl coordinate affects subjected orchestral mistaken ##tc mirrors dipped relied 260 arches candle ##nick incorporating wildly fond basilica owl fringe rituals whispering stirred feud tertiary slick goat honorable whereby skip ricardo stripes parachute adjoining submerged synthesizer ##gren intend positively ninety phi beaver partition fellows alexis prohibition carlisle bizarre fraternity ##bre doubts icy cbc aquatic sneak sonny combines airports crude supervised spatial merge alfonso ##bic corrupt scan undergo ##ams disabilities colombian comparing dolphins perkins ##lish reprinted unanimous bounced hairs underworld midwest semester bucket paperback miniseries coventry demise ##leigh demonstrations sensor rotating yan ##hler arrange soils ##idge hyderabad labs ##dr brakes grandchildren ##nde negotiated rover ferrari continuation directorate augusta stevenson counterpart gore ##rda nursery rican ave collectively broadly pastoral repertoire asserted discovering nordic styled fiba cunningham harley middlesex survives tumor tempo zack aiming lok urgent ##rade ##nto devils ##ement contractor turin ##wl ##ool bliss repaired simmons moan astronomical cr negotiate lyric 1890s lara bred clad angus pbs ##ience engineered posed ##lk hernandez possessions elbows psychiatric strokes confluence electorate lifts campuses lava alps ##ep ##ution ##date physicist woody ##page ##ographic ##itis juliet reformation sparhawk 320 complement suppressed jewel ##½ floated ##kas continuity sadly ##ische inability melting scanning paula flour judaism safer vague ##lm solving curb ##stown financially gable bees expired miserable cassidy dominion 1789 cupped 145 robbery facto amos warden resume tallest marvin ing pounded usd declaring gasoline ##aux darkened 270 650 sophomore ##mere erection gossip televised risen dial ##eu pillars ##link passages profound ##tina arabian ashton silicon nail ##ead ##lated ##wer ##hardt fleming firearms ducked circuits blows waterloo titans ##lina atom fireplace cheshire financed activation algorithms ##zzi constituent catcher cherokee partnerships sexuality platoon tragic vivian guarded whiskey meditation poetic ##late ##nga ##ake porto listeners dominance kendra mona chandler factions 22nd salisbury attitudes derivative ##ido ##haus intake paced javier illustrator barrels bias cockpit burnett dreamed ensuing ##anda receptors someday hawkins mattered ##lal slavic 1799 jesuit cameroon wasted tai wax lowering victorious freaking outright hancock librarian sensing bald calcium myers tablet announcing barack shipyard pharmaceutical ##uan greenwich flush medley patches wolfgang pt speeches acquiring exams nikolai ##gg hayden kannada ##type reilly ##pt waitress abdomen devastated capped pseudonym pharmacy fulfill paraguay 1796 clicked ##trom archipelago syndicated ##hman lumber orgasm rejection clifford lorraine advent mafia rodney brock ##ght ##used ##elia cassette chamberlain despair mongolia sensors developmental upstream ##eg ##alis spanning 165 trombone basque seeded interred renewable rhys leapt revision molecule ##ages chord vicious nord shivered 23rd arlington debts corpus sunrise bays blackburn centimetres ##uded shuddered gm strangely gripping cartoons isabelle orbital ##ppa seals proving ##lton refusal strengthened bust assisting baghdad batsman portrayal mara pushes spears og ##cock reside nathaniel brennan 1776 confirmation caucus ##worthy markings yemen nobles ku lazy viewer catalan encompasses sawyer ##fall sparked substances patents braves arranger evacuation sergio persuade dover tolerance penguin cum jockey insufficient townships occupying declining plural processed projection puppet flanders introduces liability ##yon gymnastics antwerp taipei hobart candles jeep wes observers 126 chaplain bundle glorious ##hine hazel flung sol excavations dumped stares sh bangalore triangular icelandic intervals expressing turbine ##vers songwriting crafts ##igo jasmine ditch rite ##ways entertaining comply sorrow wrestlers basel emirates marian rivera helpful ##some caution downward networking ##atory ##tered darted genocide emergence replies specializing spokesman convenient unlocked fading augustine concentrations resemblance elijah investigator andhra ##uda promotes bean ##rrell fleeing wan simone announcer ##ame ##bby lydia weaver 132 residency modification ##fest stretches ##ast alternatively nat lowe lacks ##ented pam tile concealed inferior abdullah residences tissues vengeance ##ided moisture peculiar groove zip bologna jennings ninja oversaw zombies pumping batch livingston emerald installations 1797 peel nitrogen rama ##fying ##star schooling strands responding werner ##ost lime casa accurately targeting ##rod underway ##uru hemisphere lester ##yard occupies 2d griffith angrily reorganized ##owing courtney deposited ##dd ##30 estadio ##ifies dunn exiled ##ying checks ##combe ##о ##fly successes unexpectedly blu assessed ##flower ##ه observing sacked spiders kn ##tail mu nodes prosperity audrey divisional 155 broncos tangled adjust feeds erosion paolo surf directory snatched humid admiralty screwed gt reddish ##nese modules trench lamps bind leah bucks competes ##nz ##form transcription ##uc isles violently clutching pga cyclist inflation flats ragged unnecessary ##hian stubborn coordinated harriet baba disqualified 330 insect wolfe ##fies reinforcements rocked duel winked embraced bricks ##raj hiatus defeats pending brightly jealousy ##xton ##hm ##uki lena gdp colorful ##dley stein kidney ##shu underwear wanderers ##haw ##icus guardians m³ roared habits ##wise permits gp uranium punished disguise bundesliga elise dundee erotic partisan pi collectors float individually rendering behavioral bucharest ser hare valerie corporal nutrition proportional ##isa immense ##kis pavement ##zie ##eld sutherland crouched 1775 ##lp suzuki trades endurance operas crosby prayed priory rory socially ##urn gujarat ##pu walton cube pasha privilege lennon floods thorne waterfall nipple scouting approve ##lov minorities voter dwight extensions assure ballroom slap dripping privileges rejoined confessed demonstrating patriotic yell investor ##uth pagan slumped squares ##cle ##kins confront bert embarrassment ##aid aston urging sweater starr yuri brains williamson commuter mortar structured selfish exports ##jon cds ##him unfinished ##rre mortgage destinations ##nagar canoe solitary buchanan delays magistrate fk ##pling motivation ##lier ##vier recruiting assess ##mouth malik antique 1791 pius rahman reich tub zhou smashed airs galway xii conditioning honduras discharged dexter ##pf lionel 129 debates lemon tiffany volunteered dom dioxide procession devi sic tremendous advertisements colts transferring verdict hanover decommissioned utter relate pac racism ##top beacon limp similarity terra occurrence ant ##how becky capt updates armament richie pal ##graph halloween mayo ##ssen ##bone cara serena fcc dolls obligations ##dling violated lafayette jakarta exploitation ##ime infamous iconic ##lah ##park kitty moody reginald dread spill crystals olivier modeled bluff equilibrium separating notices ordnance extinction onset cosmic attachment sammy expose privy anchored ##bil abbott admits bending baritone emmanuel policeman vaughan winged climax dresses denny polytechnic mohamed burmese authentic nikki genetics grandparents homestead gaza postponed metacritic una ##sby ##bat unstable dissertation ##rial ##cian curls obscure uncovered bronx praying disappearing ##hoe prehistoric coke turret mutations nonprofit pits monaco ##ي ##usion prominently dispatched podium ##mir uci ##uation 133 fortifications birthplace kendall ##lby ##oll preacher rack goodman ##rman persistent ##ott countless jaime recorder lexington persecution jumps renewal wagons ##11 crushing ##holder decorations ##lake abundance wrath laundry £1 garde ##rp jeanne beetles peasant ##sl splitting caste sergei ##rer ##ema scripts ##ively rub satellites ##vor inscribed verlag scrapped gale packages chick potato slogan kathleen arabs ##culture counterparts reminiscent choral ##tead rand retains bushes dane accomplish courtesy closes ##oth slaughter hague krakow lawson tailed elias ginger ##ttes canopy betrayal rebuilding turf ##hof frowning allegiance brigades kicks rebuild polls alias nationalism td rowan audition bowie fortunately recognizes harp dillon horrified ##oro renault ##tics ropes ##α presumed rewarded infrared wiping accelerated illustration ##rid presses practitioners badminton ##iard detained ##tera recognizing relates misery ##sies ##tly reproduction piercing potatoes thornton esther manners hbo ##aan ours bullshit ernie perennial sensitivity illuminated rupert ##jin ##iss ##ear rfc nassau ##dock staggered socialism ##haven appointments nonsense prestige sharma haul ##tical solidarity gps ##ook ##rata igor pedestrian ##uit baxter tenants wires medication unlimited guiding impacts diabetes ##rama sasha pas clive extraction 131 continually constraints ##bilities sonata hunted sixteenth chu planting quote mayer pretended abs spat ##hua ceramic ##cci curtains pigs pitching ##dad latvian sore dayton ##sted ##qi patrols slice playground ##nted shone stool apparatus inadequate mates treason ##ija desires ##liga ##croft somalia laurent mir leonardo oracle grape obliged chevrolet thirteenth stunning enthusiastic ##ede accounted concludes currents basil ##kovic drought ##rica mai ##aire shove posting ##shed pilgrimage humorous packing fry pencil wines smells 144 marilyn aching newest clung bon neighbours sanctioned ##pie mug ##stock drowning ##mma hydraulic ##vil hiring reminder lilly investigators ##ncies sour ##eous compulsory packet ##rion ##graphic ##elle cannes ##inate depressed ##rit heroic importantly theresa ##tled conway saturn marginal rae ##xia corresponds royce pact jasper explosives packaging aluminium ##ttered denotes rhythmic spans assignments hereditary outlined originating sundays lad reissued greeting beatrice ##dic pillar marcos plots handbook alcoholic judiciary avant slides extract masculine blur ##eum ##force homage trembled owens hymn trey omega signaling socks accumulated reacted attic theo lining angie distraction primera talbot ##key 1200 ti creativity billed ##hey deacon eduardo identifies proposition dizzy gunner hogan ##yam ##pping ##hol ja ##chan jensen reconstructed ##berger clearance darius ##nier abe harlem plea dei circled emotionally notation fascist neville exceeded upwards viable ducks ##fo workforce racer limiting shri ##lson possesses 1600 kerr moths devastating laden disturbing locking ##cture gal fearing accreditation flavor aide 1870s mountainous ##baum melt ##ures motel texture servers soda ##mb herd ##nium erect puzzled hum peggy examinations gould testified geoff ren devised sacks ##law denial posters grunted cesar tutor ec gerry offerings byrne falcons combinations ct incoming pardon rocking 26th avengers flared mankind seller uttar loch nadia stroking exposing ##hd fertile ancestral instituted ##has noises prophecy taxation eminent vivid pol ##bol dart indirect multimedia notebook upside displaying adrenaline referenced geometric ##iving progression ##ddy blunt announce ##far implementing ##lav aggression liaison cooler cares headache plantations gorge dots impulse thickness ashamed averaging kathy obligation precursor 137 fowler symmetry thee 225 hears ##rai undergoing ads butcher bowler ##lip cigarettes subscription goodness ##ically browne ##hos ##tech kyoto donor ##erty damaging friction drifting expeditions hardened prostitution 152 fauna blankets claw tossing snarled butterflies recruits investigative coated healed 138 communal hai xiii academics boone psychologist restless lahore stephens mba brendan foreigners printer ##pc ached explode 27th deed scratched dared ##pole cardiac 1780 okinawa proto commando compelled oddly electrons ##base replica thanksgiving ##rist sheila deliberate stafford tidal representations hercules ou ##path ##iated kidnapping lenses ##tling deficit samoa mouths consuming computational maze granting smirk razor fixture ideals inviting aiden nominal ##vs issuing julio pitt ramsey docks ##oss exhaust ##owed bavarian draped anterior mating ethiopian explores noticing ##nton discarded convenience hoffman endowment beasts cartridge mormon paternal probe sleeves interfere lump deadline ##rail jenks bulldogs scrap alternating justified reproductive nam seize descending secretariat kirby coupe grouped smash panther sedan tapping ##18 lola cheer germanic unfortunate ##eter unrelated ##fan subordinate ##sdale suzanne advertisement ##ility horsepower ##lda cautiously discourse luigi ##mans ##fields noun prevalent mao schneider everett surround governorate kira ##avia westward ##take misty rails sustainability 134 unused ##rating packs toast unwilling regulate thy suffrage nile awe assam definitions travelers affordable ##rb conferred sells undefeated beneficial torso basal repeating remixes ##pass bahrain cables fang ##itated excavated numbering statutory ##rey deluxe ##lian forested ramirez derbyshire zeus slamming transfers astronomer banana lottery berg histories bamboo ##uchi resurrection posterior bowls vaguely ##thi thou preserving tensed offence ##inas meyrick callum ridden watt langdon tying lowland snorted daring truman ##hale ##girl aura overly filing weighing goa infections philanthropist saunders eponymous ##owski latitude perspectives reviewing mets commandant radial ##kha flashlight reliability koch vowels amazed ada elaine supper ##rth ##encies predator debated soviets cola ##boards ##nah compartment crooked arbitrary fourteenth ##ctive havana majors steelers clips profitable ambush exited packers ##tile nude cracks fungi ##е limb trousers josie shelby tens frederic ##ος definite smoothly constellation insult baton discs lingering ##nco conclusions lent staging becker grandpa shaky ##tron einstein obstacles sk adverse elle economically ##moto mccartney thor dismissal motions readings nostrils treatise ##pace squeezing evidently prolonged 1783 venezuelan je marguerite beirut takeover shareholders ##vent denise digit airplay norse ##bbling imaginary pills hubert blaze vacated eliminating ##ello vine mansfield ##tty retrospective barrow borne clutch bail forensic weaving ##nett ##witz desktop citadel promotions worrying dorset ieee subdivided ##iating manned expeditionary pickup synod chuckle 185 barney ##rz ##ffin functionality karachi litigation meanings uc lick turbo anders ##ffed execute curl oppose ankles typhoon ##د ##ache ##asia linguistics compassion pressures grazing perfection ##iting immunity monopoly muddy backgrounds 136 namibia francesca monitors attracting stunt tuition ##ии vegetable ##mates ##quent mgm jen complexes forts ##ond cellar bites seventeenth royals flemish failures mast charities ##cular peruvian capitals macmillan ipswich outward frigate postgraduate folds employing ##ouse concurrently fiery ##tai contingent nightmares monumental nicaragua ##kowski lizard mal fielding gig reject ##pad harding ##ipe coastline ##cin ##nos beethoven humphrey innovations ##tam ##nge norris doris solicitor huang obey 141 ##lc niagara ##tton shelves aug bourbon curry nightclub specifications hilton ##ndo centennial dispersed worm neglected briggs sm font kuala uneasy plc ##nstein ##bound ##aking ##burgh awaiting pronunciation ##bbed ##quest eh optimal zhu raped greens presided brenda worries ##life venetian marxist turnout ##lius refined braced sins grasped sunderland nickel speculated lowell cyrillic communism fundraising resembling colonists mutant freddie usc ##mos gratitude ##run mural ##lous chemist wi reminds 28th steals tess pietro ##ingen promoter ri microphone honoured rai sant ##qui feather ##nson burlington kurdish terrorists deborah sickness ##wed ##eet hazard irritated desperation veil clarity ##rik jewels xv ##gged ##ows ##cup berkshire unfair mysteries orchid winced exhaustion renovations stranded obe infinity ##nies adapt redevelopment thanked registry olga domingo noir tudor ole ##atus commenting behaviors ##ais crisp pauline probable stirling wigan ##bian paralympics panting surpassed ##rew luca barred pony famed ##sters cassandra waiter carolyn exported ##orted andres destructive deeds jonah castles vacancy suv ##glass 1788 orchard yep famine belarusian sprang ##forth skinny ##mis administrators rotterdam zambia zhao boiler discoveries ##ride ##physics lucius disappointing outreach spoon ##frame qualifications unanimously enjoys regency ##iidae stade realism veterinary rodgers dump alain chestnut castile censorship rumble gibbs ##itor communion reggae inactivated logs loads ##houses homosexual ##iano ale informs ##cas phrases plaster linebacker ambrose kaiser fascinated 850 limerick recruitment forge mastered ##nding leinster rooted threaten ##strom borneo ##hes suggestions scholarships propeller documentaries patronage coats constructing invest neurons comet entirety shouts identities annoying unchanged wary ##antly ##ogy neat oversight ##kos phillies replay constance ##kka incarnation humble skies minus ##acy smithsonian ##chel guerrilla jar cadets ##plate surplus audit ##aru cracking joanna louisa pacing ##lights intentionally ##iri diner nwa imprint australians tong unprecedented bunker naive specialists ark nichols railing leaked pedal ##uka shrub longing roofs v8 captains neural tuned ##ntal ##jet emission medina frantic codex definitive sid abolition intensified stocks enrique sustain genoa oxide ##written clues cha ##gers tributaries fragment venom ##rity ##ente ##sca muffled vain sire laos ##ingly ##hana hastily snapping surfaced sentiment motive ##oft contests approximate mesa luckily dinosaur exchanges propelled accord bourne relieve tow masks offended ##ues cynthia ##mmer rains bartender zinc reviewers lois ##sai legged arrogant rafe rosie comprise handicap blockade inlet lagoon copied drilling shelley petals ##inian mandarin obsolete ##inated onward arguably productivity cindy praising seldom busch discusses raleigh shortage ranged stanton encouragement firstly conceded overs temporal ##uke cbe ##bos woo certainty pumps ##pton stalked ##uli lizzie periodic thieves weaker ##night gases shoving chooses wc ##chemical prompting weights ##kill robust flanked sticky hu tuberculosis ##eb ##eal christchurch resembled wallet reese inappropriate pictured distract fixing fiddle giggled burger heirs hairy mechanic torque apache obsessed chiefly cheng logging ##tag extracted meaningful numb ##vsky gloucestershire reminding ##bay unite ##lit breeds diminished clown glove 1860s ##ن ##ug archibald focal freelance sliced depiction ##yk organism switches sights stray crawling ##ril lever leningrad interpretations loops anytime reel alicia delighted ##ech inhaled xiv suitcase bernie vega licenses northampton exclusion induction monasteries racecourse homosexuality ##right ##sfield ##rky dimitri michele alternatives ions commentators genuinely objected pork hospitality fencing stephan warships peripheral wit drunken wrinkled quentin spends departing chung numerical spokesperson ##zone johannesburg caliber killers ##udge assumes neatly demographic abigail bloc ##vel mounting ##lain bentley slightest xu recipients ##jk merlin ##writer seniors prisons blinking hindwings flickered kappa ##hel 80s strengthening appealing brewing gypsy mali lashes hulk unpleasant harassment bio treaties predict instrumentation pulp troupe boiling mantle ##ffe ins ##vn dividing handles verbs ##onal coconut senegal 340 thorough gum momentarily ##sto cocaine panicked destined ##turing teatro denying weary captained mans ##hawks ##code wakefield bollywood thankfully ##16 cyril ##wu amendments ##bahn consultation stud reflections kindness 1787 internally ##ovo tex mosaic distribute paddy seeming 143 ##hic piers ##15 ##mura ##verse popularly winger kang sentinel mccoy ##anza covenant ##bag verge fireworks suppress thrilled dominate ##jar swansea ##60 142 reconciliation ##ndi stiffened cue dorian ##uf damascus amor ida foremost ##aga porsche unseen dir ##had ##azi stony lexi melodies ##nko angular integer podcast ants inherent jaws justify persona ##olved josephine ##nr ##ressed customary flashes gala cyrus glaring backyard ariel physiology greenland html stir avon atletico finch methodology ked ##lent mas catholicism townsend branding quincy fits containers 1777 ashore aragon ##19 forearm poisoning ##sd adopting conquer grinding amnesty keller finances evaluate forged lankan instincts ##uto guam bosnian photographed workplace desirable protector ##dog allocation intently encourages willy ##sten bodyguard electro brighter ##ν bihar ##chev lasts opener amphibious sal verde arte ##cope captivity vocabulary yields ##tted agreeing desmond pioneered ##chus strap campaigned railroads ##ович emblem ##dre stormed 501 ##ulous marijuana northumberland ##gn ##nath bowen landmarks beaumont ##qua danube ##bler attorneys th ge flyers critique villains cass mutation acc ##0s colombo mckay motif sampling concluding syndicate ##rell neon stables ds warnings clint mourning wilkinson ##tated merrill leopard evenings exhaled emil sonia ezra discrete stove farrell fifteenth prescribed superhero ##rier worms helm wren ##duction ##hc expo ##rator hq unfamiliar antony prevents acceleration fiercely mari painfully calculations cheaper ign clifton irvine davenport mozambique ##np pierced ##evich wonders ##wig ##cate ##iling crusade ware ##uel enzymes reasonably mls ##coe mater ambition bunny eliot kernel ##fin asphalt headmaster torah aden lush pins waived ##care ##yas joao substrate enforce ##grad ##ules alvarez selections epidemic tempted ##bit bremen translates ensured waterfront 29th forrest manny malone kramer reigning cookies simpler absorption 205 engraved ##ffy evaluated 1778 haze 146 comforting crossover ##abe thorn ##rift ##imo ##pop suppression fatigue cutter ##tr 201 wurttemberg ##orf enforced hovering proprietary gb samurai syllable ascent lacey tick lars tractor merchandise rep bouncing defendants ##yre huntington ##ground ##oko standardized ##hor ##hima assassinated nu predecessors rainy liar assurance lyrical ##uga secondly flattened ios parameter undercover ##mity bordeaux punish ridges markers exodus inactive hesitate debbie nyc pledge savoy nagar offset organist ##tium hesse marin converting ##iver diagram propulsion pu validity reverted supportive ##dc ministries clans responds proclamation ##inae ##ø ##rea ein pleading patriot sf birch islanders strauss hates ##dh brandenburg concession rd ##ob 1900s killings textbook antiquity cinematography wharf embarrassing setup creed farmland inequality centred signatures fallon 370 ##ingham ##uts ceylon gazing directive laurie ##tern globally ##uated ##dent allah excavation threads ##cross 148 frantically icc utilize determines respiratory thoughtful receptions ##dicate merging chandra seine 147 builders builds diagnostic dev visibility goddamn analyses dhaka cho proves chancel concurrent curiously canadians pumped restoring 1850s turtles jaguar sinister spinal traction declan vows 1784 glowed capitalism swirling install universidad ##lder ##oat soloist ##genic ##oor coincidence beginnings nissan dip resorts caucasus combustion infectious ##eno pigeon serpent ##itating conclude masked salad jew ##gr surreal toni ##wc harmonica 151 ##gins ##etic ##coat fishermen intending bravery ##wave klaus titan wembley taiwanese ransom 40th incorrect hussein eyelids jp cooke dramas utilities ##etta ##print eisenhower principally granada lana ##rak openings concord ##bl bethany connie morality sega ##mons ##nard earnings ##kara ##cine wii communes ##rel coma composing softened severed grapes ##17 nguyen analyzed warlord hubbard heavenly behave slovenian ##hit ##ony hailed filmmakers trance caldwell skye unrest coward likelihood ##aging bern sci taliban honolulu propose ##wang 1700 browser imagining cobra contributes dukes instinctively conan violinist ##ores accessories gradual ##amp quotes sioux ##dating undertake intercepted sparkling compressed 139 fungus tombs haley imposing rests degradation lincolnshire retailers wetlands tulsa distributor dungeon nun greenhouse convey atlantis aft exits oman dresser lyons ##sti joking eddy judgement omitted digits ##cts ##game juniors ##rae cents stricken une ##ngo wizards weir breton nan technician fibers liking royalty ##cca 154 persia terribly magician ##rable ##unt vance cafeteria booker camille warmer ##static consume cavern gaps compass contemporaries foyer soothing graveyard maj plunged blush ##wear cascade demonstrates ordinance ##nov boyle ##lana rockefeller shaken banjo izzy ##ense breathless vines ##32 ##eman alterations chromosome dwellings feudal mole 153 catalonia relics tenant mandated ##fm fridge hats honesty patented raul heap cruisers accusing enlightenment infants wherein chatham contractors zen affinity hc osborne piston 156 traps maturity ##rana lagos ##zal peering ##nay attendant dealers protocols subset prospects biographical ##cre artery ##zers insignia nuns endured ##eration recommend schwartz serbs berger cromwell crossroads ##ctor enduring clasped grounded ##bine marseille twitched abel choke https catalyst moldova italians ##tist disastrous wee ##oured ##nti wwf nope ##piration ##asa expresses thumbs 167 ##nza coca 1781 cheating ##ption skipped sensory heidelberg spies satan dangers semifinal 202 bohemia whitish confusing shipbuilding relies surgeons landings ravi baku moor suffix alejandro ##yana litre upheld ##unk rajasthan ##rek coaster insists posture scenarios etienne favoured appoint transgender elephants poked greenwood defences fulfilled militant somali 1758 chalk potent ##ucci migrants wink assistants nos restriction activism niger ##ario colon shaun ##sat daphne ##erated swam congregations reprise considerations magnet playable xvi ##р overthrow tobias knob chavez coding ##mers propped katrina orient newcomer ##suke temperate ##pool farmhouse interrogation ##vd committing ##vert forthcoming strawberry joaquin macau ponds shocking siberia ##cellular chant contributors ##nant ##ologists sped absorb hail 1782 spared ##hore barbados karate opus originates saul ##xie evergreen leaped ##rock correlation exaggerated weekday unification bump tracing brig afb pathways utilizing ##ners mod mb disturbance kneeling ##stad ##guchi 100th pune ##thy decreasing 168 manipulation miriam academia ecosystem occupational rbi ##lem rift ##14 rotary stacked incorporation awakening generators guerrero racist ##omy cyber derivatives culminated allie annals panzer sainte wikipedia pops zu austro ##vate algerian politely nicholson mornings educate tastes thrill dartmouth ##gating db ##jee regan differing concentrating choreography divinity ##media pledged alexandre routing gregor madeline ##idal apocalypse ##hora gunfire culminating elves fined liang lam programmed tar guessing transparency gabrielle ##gna cancellation flexibility ##lining accession shea stronghold nets specializes ##rgan abused hasan sgt ling exceeding ##₄ admiration supermarket ##ark photographers specialised tilt resonance hmm perfume 380 sami threatens garland botany guarding boiled greet puppy russo supplier wilmington vibrant vijay ##bius paralympic grumbled paige faa licking margins hurricanes ##gong fest grenade ripping ##uz counseling weigh ##sian needles wiltshire edison costly ##not fulton tramway redesigned staffordshire cache gasping watkins sleepy candidacy ##group monkeys timeline throbbing ##bid ##sos berth uzbekistan vanderbilt bothering overturned ballots gem ##iger sunglasses subscribers hooker compelling ang exceptionally saloon stab ##rdi carla terrifying rom ##vision coil ##oids satisfying vendors 31st mackay deities overlooked ambient bahamas felipe olympia whirled botanist advertised tugging ##dden disciples morales unionist rites foley morse motives creepy ##₀ soo ##sz bargain highness frightening turnpike tory reorganization ##cer depict biographer ##walk unopposed manifesto ##gles institut emile accidental kapoor ##dam kilkenny cortex lively ##13 romanesque jain shan cannons ##ood ##ske petrol echoing amalgamated disappears cautious proposes sanctions trenton ##ر flotilla aus contempt tor canary cote theirs ##hun conceptual deleted fascinating paso blazing elf honourable hutchinson ##eiro ##outh ##zin surveyor tee amidst wooded reissue intro ##ono cobb shelters newsletter hanson brace encoding confiscated dem caravan marino scroll melodic cows imam ##adi ##aneous northward searches biodiversity cora 310 roaring ##bers connell theologian halo compose pathetic unmarried dynamo ##oot az calculation toulouse deserves humour nr forgiveness tam undergone martyr pamela myths whore counselor hicks 290 heavens battleship electromagnetic ##bbs stellar establishments presley hopped ##chin temptation 90s wills nas ##yuan nhs ##nya seminars ##yev adaptations gong asher lex indicator sikh tobago cites goin ##yte satirical ##gies characterised correspond bubbles lure participates ##vid eruption skate therapeutic 1785 canals wholesale defaulted sac 460 petit ##zzled virgil leak ravens 256 portraying ##yx ghetto creators dams portray vicente ##rington fae namesake bounty ##arium joachim ##ota ##iser aforementioned axle snout depended dismantled reuben 480 ##ibly gallagher ##lau ##pd earnest ##ieu ##iary inflicted objections ##llar asa gritted ##athy jericho ##sea ##was flick underside ceramics undead substituted 195 eastward undoubtedly wheeled chimney ##iche guinness cb ##ager siding ##bell traitor baptiste disguised inauguration 149 tipperary choreographer perched warmed stationary eco ##ike ##ntes bacterial ##aurus flores phosphate ##core attacker invaders alvin intersects a1 indirectly immigrated businessmen cornelius valves narrated pill sober ul nationale monastic applicants scenery ##jack 161 motifs constitutes cpu ##osh jurisdictions sd tuning irritation woven ##uddin fertility gao ##erie antagonist impatient glacial hides boarded denominations interception ##jas cookie nicola ##tee algebraic marquess bahn parole buyers bait turbines paperwork bestowed natasha renee oceans purchases 157 vaccine 215 ##tock fixtures playhouse integrate jai oswald intellectuals ##cky booked nests mortimer ##isi obsession sept ##gler ##sum 440 scrutiny simultaneous squinted ##shin collects oven shankar penned remarkably ##я slips luggage spectral 1786 collaborations louie consolidation ##ailed ##ivating 420 hoover blackpool harness ignition vest tails belmont mongol skinner ##nae visually mage derry ##tism ##unce stevie transitional ##rdy redskins drying prep prospective ##21 annoyance oversee ##loaded fills ##books ##iki announces fda scowled respects prasad mystic tucson ##vale revue springer bankrupt 1772 aristotle salvatore habsburg ##geny dal natal nut pod chewing darts moroccan walkover rosario lenin punjabi ##ße grossed scattering wired invasive hui polynomial corridors wakes gina portrays ##cratic arid retreating erich irwin sniper ##dha linen lindsey maneuver butch shutting socio bounce commemorative postseason jeremiah pines 275 mystical beads bp abbas furnace bidding consulted assaulted empirical rubble enclosure sob weakly cancel polly yielded ##emann curly prediction battered 70s vhs jacqueline render sails barked detailing grayson riga sloane raging ##yah herbs bravo ##athlon alloy giggle imminent suffers assumptions waltz ##itate accomplishments ##ited bathing remixed deception prefix ##emia deepest ##tier ##eis balkan frogs ##rong slab ##pate philosophers peterborough grains imports dickinson rwanda ##atics 1774 dirk lan tablets ##rove clone ##rice caretaker hostilities mclean ##gre regimental treasures norms impose tsar tango diplomacy variously complain 192 recognise arrests 1779 celestial pulitzer ##dus bing libretto ##moor adele splash ##rite expectation lds confronts ##izer spontaneous harmful wedge entrepreneurs buyer ##ope bilingual translate rugged conner circulated uae eaton ##gra ##zzle lingered lockheed vishnu reelection alonso ##oom joints yankee headline cooperate heinz laureate invading ##sford echoes scandinavian ##dham hugging vitamin salute micah hind trader ##sper radioactive ##ndra militants poisoned ratified remark campeonato deprived wander prop ##dong outlook ##tani ##rix ##eye chiang darcy ##oping mandolin spice statesman babylon 182 walled forgetting afro ##cap 158 giorgio buffer ##polis planetary ##gis overlap terminals kinda centenary ##bir arising manipulate elm ke 1770 ak ##tad chrysler mapped moose pomeranian quad macarthur assemblies shoreline recalls stratford ##rted noticeable ##evic imp ##rita ##sque accustomed supplying tents disgusted vogue sipped filters khz reno selecting luftwaffe mcmahon tyne masterpiece carriages collided dunes exercised flare remembers muzzle ##mobile heck ##rson burgess lunged middleton boycott bilateral ##sity hazardous lumpur multiplayer spotlight jackets goldman liege porcelain rag waterford benz attracts hopeful battling ottomans kensington baked hymns cheyenne lattice levine borrow polymer clashes michaels monitored commitments denounced ##25 ##von cavity ##oney hobby akin ##holders futures intricate cornish patty ##oned illegally dolphin ##lag barlow yellowish maddie apologized luton plagued ##puram nana ##rds sway fanny łodz ##rino psi suspicions hanged ##eding initiate charlton ##por nak competent 235 analytical annex wardrobe reservations ##rma sect 162 fairfax hedge piled buckingham uneven bauer simplicity snyder interpret accountability donors moderately byrd continents ##cite ##max disciple hr jamaican ping nominees ##uss mongolian diver attackers eagerly ideological pillows miracles apartheid revolver sulfur clinics moran 163 ##enko ile katy rhetoric ##icated chronology recycling ##hrer elongated mughal pascal profiles vibration databases domination ##fare ##rant matthias digest rehearsal polling weiss initiation reeves clinging flourished impress ngo ##hoff ##ume buckley symposium rhythms weed emphasize transforming ##taking ##gence ##yman accountant analyze flicker foil priesthood voluntarily decreases ##80 ##hya slater sv charting mcgill ##lde moreno ##iu besieged zur robes ##phic admitting api deported turmoil peyton earthquakes ##ares nationalists beau clair brethren interrupt welch curated galerie requesting 164 ##ested impending steward viper ##vina complaining beautifully brandy foam nl 1660 ##cake alessandro punches laced explanations ##lim attribute clit reggie discomfort ##cards smoothed whales ##cene adler countered duffy disciplinary widening recipe reliance conducts goats gradient preaching ##shaw matilda quasi striped meridian cannabis cordoba certificates ##agh ##tering graffiti hangs pilgrims repeats ##ych revive urine etat ##hawk fueled belts fuzzy susceptible ##hang mauritius salle sincere beers hooks ##cki arbitration entrusted advise sniffed seminar junk donnell processors principality strapped celia mendoza everton fortunes prejudice starving reassigned steamer ##lund tuck evenly foreman ##ffen dans 375 envisioned slit ##xy baseman liberia rosemary ##weed electrified periodically potassium stride contexts sperm slade mariners influx bianca subcommittee ##rane spilling icao estuary ##nock delivers iphone ##ulata isa mira bohemian dessert ##sbury welcoming proudly slowing ##chs musee ascension russ ##vian waits ##psy africans exploit ##morphic gov eccentric crab peck ##ull entrances formidable marketplace groom bolted metabolism patton robbins courier payload endure ##ifier andes refrigerator ##pr ornate ##uca ruthless illegitimate masonry strasbourg bikes adobe ##³ apples quintet willingly niche bakery corpses energetic ##cliffe ##sser ##ards 177 centimeters centro fuscous cretaceous rancho ##yde andrei telecom tottenham oasis ordination vulnerability presiding corey cp penguins sims ##pis malawi piss ##48 correction ##cked ##ffle ##ryn countdown detectives psychiatrist psychedelic dinosaurs blouse ##get choi vowed ##oz randomly ##pol 49ers scrub blanche bruins dusseldorf ##using unwanted ##ums 212 dominique elevations headlights om laguna ##oga 1750 famously ignorance shrewsbury ##aine ajax breuning che confederacy greco overhaul ##screen paz skirts disagreement cruelty jagged phoebe shifter hovered viruses ##wes mandy ##lined ##gc landlord squirrel dashed ##ι ornamental gag wally grange literal spurs undisclosed proceeding yin ##text billie orphan spanned humidity indy weighted presentations explosions lucian ##tary vaughn hindus ##anga ##hell psycho 171 daytona protects efficiently rematch sly tandem ##oya rebranded impaired hee metropolis peach godfrey diaspora ethnicity prosperous gleaming dar grossing playback ##rden stripe pistols ##tain births labelled ##cating 172 rudy alba ##onne aquarium hostility ##gb ##tase shudder sumatra hardest lakers consonant creeping demos homicide capsule zeke liberties expulsion pueblo ##comb trait transporting ##ddin ##neck ##yna depart gregg mold ledge hangar oldham playboy termination analysts gmbh romero ##itic insist cradle filthy brightness slash shootout deposed bordering ##truct isis microwave tumbled sheltered cathy werewolves messy andersen convex clapped clinched satire wasting edo vc rufus ##jak mont ##etti poznan ##keeping restructuring transverse ##rland azerbaijani slovene gestures roommate choking shear ##quist vanguard oblivious ##hiro disagreed baptism ##lich coliseum ##aceae salvage societe cory locke relocation relying versailles ahl swelling ##elo cheerful ##word ##edes gin sarajevo obstacle diverted ##nac messed thoroughbred fluttered utrecht chewed acquaintance assassins dispatch mirza ##wart nike salzburg swell yen ##gee idle ligue samson ##nds ##igh playful spawned ##cise tease ##case burgundy ##bot stirring skeptical interceptions marathi ##dies bedrooms aroused pinch ##lik preferences tattoos buster digitally projecting rust ##ital kitten priorities addison pseudo ##guard dusk icons sermon ##psis ##iba bt ##lift ##xt ju truce rink ##dah ##wy defects psychiatry offences calculate glucose ##iful ##rized ##unda francaise ##hari richest warwickshire carly 1763 purity redemption lending ##cious muse bruises cerebral aero carving ##name preface terminology invade monty ##int anarchist blurred ##iled rossi treats guts shu foothills ballads undertaking premise cecilia affiliates blasted conditional wilder minors drone rudolph buffy swallowing horton attested ##hop rutherford howell primetime livery penal ##bis minimize hydro wrecked wrought palazzo ##gling cans vernacular friedman nobleman shale walnut danielle ##ection ##tley sears ##kumar chords lend flipping streamed por dracula gallons sacrifices gamble orphanage ##iman mckenzie ##gible boxers daly ##balls ##ان 208 ##ific ##rative ##iq exploited slated ##uity circling hillary pinched goldberg provost campaigning lim piles ironically jong mohan successors usaf ##tem ##ught autobiographical haute preserves ##ending acquitted comparisons 203 hydroelectric gangs cypriot torpedoes rushes chrome derive bumps instability fiat pets ##mbe silas dye reckless settler ##itation info heats ##writing 176 canonical maltese fins mushroom stacy aspen avid ##kur ##loading vickers gaston hillside statutes wilde gail kung sabine comfortably motorcycles ##rgo 169 pneumonia fetch ##sonic axel faintly parallels ##oop mclaren spouse compton interdisciplinary miner ##eni 181 clamped ##chal ##llah separates versa ##mler scarborough labrador ##lity ##osing rutgers hurdles como 166 burt divers ##100 wichita cade coincided ##erson bruised mla ##pper vineyard ##ili ##brush notch mentioning jase hearted kits doe ##acle pomerania ##ady ronan seizure pavel problematic ##zaki domenico ##ulin catering penelope dependence parental emilio ministerial atkinson ##bolic clarkson chargers colby grill peeked arises summon ##aged fools ##grapher faculties qaeda ##vial garner refurbished ##hwa geelong disasters nudged bs shareholder lori algae reinstated rot ##ades ##nous invites stainless 183 inclusive ##itude diocesan til ##icz denomination ##xa benton floral registers ##ider ##erman ##kell absurd brunei guangzhou hitter retaliation ##uled ##eve blanc nh consistency contamination ##eres ##rner dire palermo broadcasters diaries inspire vols brewer tightening ky mixtape hormone ##tok stokes ##color ##dly ##ssi pg ##ometer ##lington sanitation ##tility intercontinental apps ##adt ¹⁄₂ cylinders economies favourable unison croix gertrude odyssey vanity dangling ##logists upgrades dice middleweight practitioner ##ight 206 henrik parlor orion angered lac python blurted ##rri sensual intends swings angled ##phs husky attain peerage precinct textiles cheltenham shuffled dai confess tasting bhutan ##riation tyrone segregation abrupt ruiz ##rish smirked blackwell confidential browning amounted ##put vase scarce fabulous raided staple guyana unemployed glider shay ##tow carmine troll intervene squash superstar ##uce cylindrical len roadway researched handy ##rium ##jana meta lao declares ##rring ##tadt ##elin ##kova willem shrubs napoleonic realms skater qi volkswagen ##ł tad hara archaeologist awkwardly eerie ##kind wiley ##heimer ##24 titus organizers cfl crusaders lama usb vent enraged thankful occupants maximilian ##gaard possessing textbooks ##oran collaborator quaker ##ulo avalanche mono silky straits isaiah mustang surged resolutions potomac descend cl kilograms plato strains saturdays ##olin bernstein ##ype holstein ponytail ##watch belize conversely heroine perpetual ##ylus charcoal piedmont glee negotiating backdrop prologue ##jah ##mmy pasadena climbs ramos sunni ##holm ##tner ##tri anand deficiency hertfordshire stout ##avi aperture orioles ##irs doncaster intrigued bombed coating otis ##mat cocktail ##jit ##eto amir arousal sar ##proof ##act ##ories dixie pots ##bow whereabouts 159 ##fted drains bullying cottages scripture coherent fore poe appetite ##uration sampled ##ators ##dp derrick rotor jays peacock installment ##rro advisors ##coming rodeo scotch ##mot ##db ##fen ##vant ensued rodrigo dictatorship martyrs twenties ##н towed incidence marta rainforest sai scaled ##cles oceanic qualifiers symphonic mcbride dislike generalized aubrey colonization ##iation ##lion ##ssing disliked lublin salesman ##ulates spherical whatsoever sweating avalon contention punt severity alderman atari ##dina ##grant ##rop scarf seville vertices annexation fairfield fascination inspiring launches palatinate regretted ##rca feral ##iom elk nap olsen reddy yong ##leader ##iae garment transports feng gracie outrage viceroy insides ##esis breakup grady organizer softer grimaced 222 murals galicia arranging vectors ##rsten bas ##sb ##cens sloan ##eka bitten ara fender nausea bumped kris banquet comrades detector persisted ##llan adjustment endowed cinemas ##shot sellers ##uman peek epa kindly neglect simpsons talon mausoleum runaway hangul lookout ##cic rewards coughed acquainted chloride ##ald quicker accordion neolithic ##qa artemis coefficient lenny pandora tx ##xed ecstasy litter segunda chairperson gemma hiss rumor vow nasal antioch compensate patiently transformers ##eded judo morrow penis posthumous philips bandits husbands denote flaming ##any ##phones langley yorker 1760 walters ##uo ##kle gubernatorial fatty samsung leroy outlaw ##nine unpublished poole jakob ##ᵢ ##ₙ crete distorted superiority ##dhi intercept crust mig claus crashes positioning 188 stallion 301 frontal armistice ##estinal elton aj encompassing camel commemorated malaria woodward calf cigar penetrate ##oso willard ##rno ##uche illustrate amusing convergence noteworthy ##lma ##rva journeys realise manfred ##sable 410 ##vocation hearings fiance ##posed educators provoked adjusting ##cturing modular stockton paterson vlad rejects electors selena maureen ##tres uber ##rce swirled ##num proportions nanny pawn naturalist parma apostles awoke ethel wen ##bey monsoon overview ##inating mccain rendition risky adorned ##ih equestrian germain nj conspicuous confirming ##yoshi shivering ##imeter milestone rumours flinched bounds smacked token ##bei lectured automobiles ##shore impacted ##iable nouns nero ##leaf ismail prostitute trams ##lace bridget sud stimulus impressions reins revolves ##oud ##gned giro honeymoon ##swell criterion ##sms ##uil libyan prefers ##osition 211 preview sucks accusation bursts metaphor diffusion tolerate faye betting cinematographer liturgical specials bitterly humboldt ##ckle flux rattled ##itzer archaeologists odor authorised marshes discretion ##ов alarmed archaic inverse ##leton explorers ##pine drummond tsunami woodlands ##minate ##tland booklet insanity owning insert crafted calculus ##tore receivers ##bt stung ##eca ##nched prevailing travellers eyeing lila graphs ##borne 178 julien ##won morale adaptive therapist erica cw libertarian bowman pitches vita ##ional crook ##ads ##entation caledonia mutiny ##sible 1840s automation ##ß flock ##pia ironic pathology ##imus remarried ##22 joker withstand energies ##att shropshire hostages madeleine tentatively conflicting mateo recipes euros ol mercenaries nico ##ndon albuquerque augmented mythical bel freud ##child cough ##lica 365 freddy lillian genetically nuremberg calder 209 bonn outdoors paste suns urgency vin restraint tyson ##cera ##selle barrage bethlehem kahn ##par mounts nippon barony happier ryu makeshift sheldon blushed castillo barking listener taped bethel fluent headlines pornography rum disclosure sighing mace doubling gunther manly ##plex rt interventions physiological forwards emerges ##tooth ##gny compliment rib recession visibly barge faults connector exquisite prefect ##rlin patio ##cured elevators brandt italics pena 173 wasp satin ea botswana graceful respectable ##jima ##rter ##oic franciscan generates ##dl alfredo disgusting ##olate ##iously sherwood warns cod promo cheryl sino ##ة ##escu twitch ##zhi brownish thom ortiz ##dron densely ##beat carmel reinforce ##bana 187 anastasia downhill vertex contaminated remembrance harmonic homework ##sol fiancee gears olds angelica loft ramsay quiz colliery sevens ##cape autism ##hil walkway ##boats ruben abnormal ounce khmer ##bbe zachary bedside morphology punching ##olar sparrow convinces ##35 hewitt queer remastered rods mabel solemn notified lyricist symmetric ##xide 174 encore passports wildcats ##uni baja ##pac mildly ##ease bleed commodity mounds glossy orchestras ##omo damian prelude ambitions ##vet awhile remotely ##aud asserts imply ##iques distinctly modelling remedy ##dded windshield dani xiao ##endra audible powerplant 1300 invalid elemental acquisitions ##hala immaculate libby plata smuggling ventilation denoted minh ##morphism 430 differed dion kelley lore mocking sabbath spikes hygiene drown runoff stylized tally liberated aux interpreter righteous aba siren reaper pearce millie ##cier ##yra gaius ##iso captures ##ttering dorm claudio ##sic benches knighted blackness ##ored discount fumble oxidation routed ##ς novak perpendicular spoiled fracture splits ##urt pads topology ##cats axes fortunate offenders protestants esteem 221 broadband convened frankly hound prototypes isil facilitated keel ##sher sahara awaited bubba orb prosecutors 186 hem 520 ##xing relaxing remnant romney sorted slalom stefano ulrich ##active exemption folder pauses foliage hitchcock epithet 204 criticisms ##aca ballistic brody hinduism chaotic youths equals ##pala pts thicker analogous capitalist improvised overseeing sinatra ascended beverage ##tl straightforward ##kon curran ##west bois 325 induce surveying emperors sax unpopular ##kk cartoonist fused ##mble unto ##yuki localities ##cko ##ln darlington slain academie lobbying sediment puzzles ##grass defiance dickens manifest tongues alumnus arbor coincide 184 appalachian mustafa examiner cabaret traumatic yves bracelet draining heroin magnum baths odessa consonants mitsubishi ##gua kellan vaudeville ##fr joked null straps probation ##ław ceded interfaces ##pas ##zawa blinding viet 224 rothschild museo 640 huddersfield ##vr tactic ##storm brackets dazed incorrectly ##vu reg glazed fearful manifold benefited irony ##sun stumbling ##rte willingness balkans mei wraps ##aba injected ##lea gu syed harmless ##hammer bray takeoff poppy timor cardboard astronaut purdue weeping southbound cursing stalls diagonal ##neer lamar bryce comte weekdays harrington ##uba negatively ##see lays grouping ##cken ##henko affirmed halle modernist ##lai hodges smelling aristocratic baptized dismiss justification oilers ##now coupling qin snack healer ##qing gardener layla battled formulated stephenson gravitational ##gill ##jun 1768 granny coordinating suites ##cd ##ioned monarchs ##cote ##hips sep blended apr barrister deposition fia mina policemen paranoid ##pressed churchyard covert crumpled creep abandoning tr transmit conceal barr understands readiness spire ##cology ##enia ##erry 610 startling unlock vida bowled slots ##nat ##islav spaced trusting admire rig ##ink slack ##70 mv 207 casualty ##wei classmates ##odes ##rar ##rked amherst furnished evolve foundry menace mead ##lein flu wesleyan ##kled monterey webber ##vos wil ##mith ##на bartholomew justices restrained ##cke amenities 191 mediated sewage trenches ml mainz ##thus 1800s ##cula ##inski caine bonding 213 converts spheres superseded marianne crypt sweaty ensign historia ##br spruce ##post ##ask forks thoughtfully yukon pamphlet ames ##uter karma ##yya bryn negotiation sighs incapable ##mbre ##ntial actresses taft ##mill luce prevailed ##amine 1773 motionless envoy testify investing sculpted instructors provence kali cullen horseback ##while goodwin ##jos gaa norte ##ldon modify wavelength abd 214 skinned sprinter forecast scheduling marries squared tentative ##chman boer ##isch bolts swap fisherman assyrian impatiently guthrie martins murdoch 194 tanya nicely dolly lacy med ##45 syn decks fashionable millionaire ##ust surfing ##ml ##ision heaved tammy consulate attendees routinely 197 fuse saxophonist backseat malaya ##lord scowl tau ##ishly 193 sighted steaming ##rks 303 911 ##holes ##hong ching ##wife bless conserved jurassic stacey unix zion chunk rigorous blaine 198 peabody slayer dismay brewers nz ##jer det ##glia glover postwar int penetration sylvester imitation vertically airlift heiress knoxville viva ##uin 390 macon ##rim ##fighter ##gonal janice ##orescence ##wari marius belongings leicestershire 196 blanco inverted preseason sanity sobbing ##due ##elt ##dled collingwood regeneration flickering shortest ##mount ##osi feminism ##lat sherlock cabinets fumbled northbound precedent snaps ##mme researching ##akes guillaume insights manipulated vapor neighbour sap gangster frey f1 stalking scarcely callie barnett tendencies audi doomed assessing slung panchayat ambiguous bartlett ##etto distributing violating wolverhampton ##hetic swami histoire ##urus liable pounder groin hussain larsen popping surprises ##atter vie curt ##station mute relocate musicals authorization richter ##sef immortality tna bombings ##press deteriorated yiddish ##acious robbed colchester cs pmid ao verified balancing apostle swayed recognizable oxfordshire retention nottinghamshire contender judd invitational shrimp uhf ##icient cleaner longitudinal tanker ##mur acronym broker koppen sundance suppliers ##gil 4000 clipped fuels petite ##anne landslide helene diversion populous landowners auspices melville quantitative ##xes ferries nicky ##llus doo haunting roche carver downed unavailable ##pathy approximation hiroshima ##hue garfield valle comparatively keyboardist traveler ##eit congestion calculating subsidiaries ##bate serb modernization fairies deepened ville averages ##lore inflammatory tonga ##itch co₂ squads ##hea gigantic serum enjoyment retailer verona 35th cis ##phobic magna technicians ##vati arithmetic ##sport levin ##dation amtrak chow sienna ##eyer backstage entrepreneurship ##otic learnt tao ##udy worcestershire formulation baggage hesitant bali sabotage ##kari barren enhancing murmur pl freshly putnam syntax aces medicines resentment bandwidth ##sier grins chili guido ##sei framing implying gareth lissa genevieve pertaining admissions geo thorpe proliferation sato bela analyzing parting ##gor awakened ##isman huddled secrecy ##kling hush gentry 540 dungeons ##ego coasts ##utz sacrificed ##chule landowner mutually prevalence programmer adolescent disrupted seaside gee trusts vamp georgie ##nesian ##iol schedules sindh ##market etched hm sparse bey beaux scratching gliding unidentified 216 collaborating gems jesuits oro accumulation shaping mbe anal ##xin 231 enthusiasts newscast ##egan janata dewey parkinson 179 ankara biennial towering dd inconsistent 950 ##chet thriving terminate cabins furiously eats advocating donkey marley muster phyllis leiden ##user grassland glittering iucn loneliness 217 memorandum armenians ##ddle popularized rhodesia 60s lame ##illon sans bikini header orbits ##xx ##finger ##ulator sharif spines biotechnology strolled naughty yates ##wire fremantle milo ##mour abducted removes ##atin humming wonderland ##chrome ##ester hume pivotal ##rates armand grams believers elector rte apron bis scraped ##yria endorsement initials ##llation eps dotted hints buzzing emigration nearer ##tom indicators ##ulu coarse neutron protectorate ##uze directional exploits pains loire 1830s proponents guggenheim rabbits ritchie 305 hectare inputs hutton ##raz verify ##ako boilers longitude ##lev skeletal yer emilia citrus compromised ##gau pokemon prescription paragraph eduard cadillac attire categorized kenyan weddings charley ##bourg entertain monmouth ##lles nutrients davey mesh incentive practised ecosystems kemp subdued overheard ##rya bodily maxim ##nius apprenticeship ursula ##fight lodged rug silesian unconstitutional patel inspected coyote unbeaten ##hak 34th disruption convict parcel ##cl ##nham collier implicated mallory ##iac ##lab susannah winkler ##rber shia phelps sediments graphical robotic ##sner adulthood mart smoked ##isto kathryn clarified ##aran divides convictions oppression pausing burying ##mt federico mathias eileen ##tana kite hunched ##acies 189 ##atz disadvantage liza kinetic greedy paradox yokohama dowager trunks ventured ##gement gupta vilnius olaf ##thest crimean hopper ##ej progressively arturo mouthed arrondissement ##fusion rubin simulcast oceania ##orum ##stra ##rred busiest intensely navigator cary ##vine ##hini ##bies fife rowe rowland posing insurgents shafts lawsuits activate conor inward culturally garlic 265 ##eering eclectic ##hui ##kee ##nl furrowed vargas meteorological rendezvous ##aus culinary commencement ##dition quota ##notes mommy salaries overlapping mule ##iology ##mology sums wentworth ##isk ##zione mainline subgroup ##illy hack plaintiff verdi bulb differentiation engagements multinational supplemented bertrand caller regis ##naire ##sler ##arts ##imated blossom propagation kilometer viaduct vineyards ##uate beckett optimization golfer songwriters seminal semitic thud volatile evolving ridley ##wley trivial distributions scandinavia jiang ##ject wrestled insistence ##dio emphasizes napkin ##ods adjunct rhyme ##ricted ##eti hopeless surrounds tremble 32nd smoky ##ntly oils medicinal padded steer wilkes 219 255 concessions hue uniquely blinded landon yahoo ##lane hendrix commemorating dex specify chicks ##ggio intercity 1400 morley ##torm highlighting ##oting pang oblique stalled ##liner flirting newborn 1769 bishopric shaved 232 currie ##ush dharma spartan ##ooped favorites smug novella sirens abusive creations espana ##lage paradigm semiconductor sheen ##rdo ##yen ##zak nrl renew ##pose ##tur adjutant marches norma ##enity ineffective weimar grunt ##gat lordship plotting expenditure infringement lbs refrain av mimi mistakenly postmaster 1771 ##bara ras motorsports tito 199 subjective ##zza bully stew ##kaya prescott 1a ##raphic ##zam bids styling paranormal reeve sneaking exploding katz akbar migrant syllables indefinitely ##ogical destroys replaces applause ##phine pest ##fide 218 articulated bertie ##thing ##cars ##ptic courtroom crowley aesthetics cummings tehsil hormones titanic dangerously ##ibe stadion jaenelle auguste ciudad ##chu mysore partisans ##sio lucan philipp ##aly debating henley interiors ##rano ##tious homecoming beyonce usher henrietta prepares weeds ##oman ely plucked ##pire ##dable luxurious ##aq artifact password pasture juno maddy minsk ##dder ##ologies ##rone assessments martian royalist 1765 examines ##mani ##rge nino 223 parry scooped relativity ##eli ##uting ##cao congregational noisy traverse ##agawa strikeouts nickelodeon obituary transylvania binds depictions polk trolley ##yed ##lard breeders ##under dryly hokkaido 1762 strengths stacks bonaparte connectivity neared prostitutes stamped anaheim gutierrez sinai ##zzling bram fresno madhya ##86 proton ##lena ##llum ##phon reelected wanda ##anus ##lb ample distinguishing ##yler grasping sermons tomato bland stimulation avenues ##eux spreads scarlett fern pentagon assert baird chesapeake ir calmed distortion fatalities ##olis correctional pricing ##astic ##gina prom dammit ying collaborate ##chia welterweight 33rd pointer substitution bonded umpire communicating multitude paddle ##obe federally intimacy ##insky betray ssr ##lett ##lean ##lves ##therapy airbus ##tery functioned ud bearer biomedical netflix ##hire ##nca condom brink ik ##nical macy ##bet flap gma experimented jelly lavender ##icles ##ulia munro ##mian ##tial rye ##rle 60th gigs hottest rotated predictions fuji bu ##erence ##omi barangay ##fulness ##sas clocks ##rwood ##liness cereal roe wight decker uttered babu onion xml forcibly ##df petra sarcasm hartley peeled storytelling ##42 ##xley ##ysis ##ffa fibre kiel auditor fig harald greenville ##berries geographically nell quartz ##athic cemeteries ##lr crossings nah holloway reptiles chun sichuan snowy 660 corrections ##ivo zheng ambassadors blacksmith fielded fluids hardcover turnover medications melvin academies ##erton ro roach absorbing spaniards colton ##founded outsider espionage kelsey 245 edible ##ulf dora establishes ##sham ##tries contracting ##tania cinematic costello nesting ##uron connolly duff ##nology mma ##mata fergus sexes gi optics spectator woodstock banning ##hee ##fle differentiate outfielder refinery 226 312 gerhard horde lair drastically ##udi landfall ##cheng motorsport odi ##achi predominant quay skins ##ental edna harshly complementary murdering ##aves wreckage ##90 ono outstretched lennox munitions galen reconcile 470 scalp bicycles gillespie questionable rosenberg guillermo hostel jarvis kabul volvo opium yd ##twined abuses decca outpost ##cino sensible neutrality ##64 ponce anchorage atkins turrets inadvertently disagree libre vodka reassuring weighs ##yal glide jumper ceilings repertory outs stain ##bial envy ##ucible smashing heightened policing hyun mixes lai prima ##ples celeste ##bina lucrative intervened kc manually ##rned stature staffed bun bastards nairobi priced ##auer thatcher ##kia tripped comune ##ogan ##pled brasil incentives emanuel hereford musica ##kim benedictine biennale ##lani eureka gardiner rb knocks sha ##ael ##elled ##onate efficacy ventura masonic sanford maize leverage ##feit capacities santana ##aur novelty vanilla ##cter ##tour benin ##oir ##rain neptune drafting tallinn ##cable humiliation ##boarding schleswig fabian bernardo liturgy spectacle sweeney pont routledge ##tment cosmos ut hilt sleek universally ##eville ##gawa typed ##dry favors allegheny glaciers ##rly recalling aziz ##log parasite requiem auf ##berto ##llin illumination ##breaker ##issa festivities bows govern vibe vp 333 sprawled larson pilgrim bwf leaping ##rts ##ssel alexei greyhound hoarse ##dler ##oration seneca ##cule gaping ##ulously ##pura cinnamon ##gens ##rricular craven fantasies houghton engined reigned dictator supervising ##oris bogota commentaries unnatural fingernails spirituality tighten ##tm canadiens protesting intentional cheers sparta ##ytic ##iere ##zine widen belgarath controllers dodd iaaf navarre ##ication defect squire steiner whisky ##mins 560 inevitably tome ##gold chew ##uid ##lid elastic ##aby streaked alliances jailed regal ##ined ##phy czechoslovak narration absently ##uld bluegrass guangdong quran criticizing hose hari ##liest ##owa skier streaks deploy ##lom raft bose dialed huff ##eira haifa simplest bursting endings ib sultanate ##titled franks whitman ensures sven ##ggs collaborators forster organising ui banished napier injustice teller layered thump ##otti roc battleships evidenced fugitive sadie robotics ##roud equatorial geologist ##iza yielding ##bron ##sr internationale mecca ##diment sbs skyline toad uploaded reflective undrafted lal leafs bayern ##dai lakshmi shortlisted ##stick ##wicz camouflage donate af christi lau ##acio disclosed nemesis 1761 assemble straining northamptonshire tal ##asi bernardino premature heidi 42nd coefficients galactic reproduce buzzed sensations zionist monsieur myrtle ##eme archery strangled musically viewpoint antiquities bei trailers seahawks cured pee preferring tasmanian lange sul ##mail ##working colder overland lucivar massey gatherings haitian ##smith disapproval flaws ##cco ##enbach 1766 npr ##icular boroughs creole forums techno 1755 dent abdominal streetcar ##eson ##stream procurement gemini predictable ##tya acheron christoph feeder fronts vendor bernhard jammu tumors slang ##uber goaltender twists curving manson vuelta mer peanut confessions pouch unpredictable allowance theodor vascular ##factory bala authenticity metabolic coughing nanjing ##cea pembroke ##bard splendid 36th ff hourly ##ahu elmer handel ##ivate awarding thrusting dl experimentation ##hesion ##46 caressed entertained steak ##rangle biologist orphans baroness oyster stepfather ##dridge mirage reefs speeding ##31 barons 1764 227 inhabit preached repealed ##tral honoring boogie captives administer johanna ##imate gel suspiciously 1767 sobs ##dington backbone hayward garry ##folding ##nesia maxi ##oof ##ppe ellison galileo ##stand crimea frenzy amour bumper matrices natalia baking garth palestinians ##grove smack conveyed ensembles gardening ##manship ##rup ##stituting 1640 harvesting topography jing shifters dormitory ##carriage ##lston ist skulls ##stadt dolores jewellery sarawak ##wai ##zier fences christy confinement tumbling credibility fir stench ##bria ##plication ##nged ##sam virtues ##belt marjorie pba ##eem ##made celebrates schooner agitated barley fulfilling anthropologist ##pro restrict novi regulating ##nent padres ##rani ##hesive loyola tabitha milky olson proprietor crambidae guarantees intercollegiate ljubljana hilda ##sko ignorant hooded ##lts sardinia ##lidae ##vation frontman privileged witchcraft ##gp jammed laude poking ##than bracket amazement yunnan ##erus maharaja linnaeus 264 commissioning milano peacefully ##logies akira rani regulator ##36 grasses ##rance luzon crows compiler gretchen seaman edouard tab buccaneers ellington hamlets whig socialists ##anto directorial easton mythological ##kr ##vary rhineland semantic taut dune inventions succeeds ##iter replication branched ##pired jul prosecuted kangaroo penetrated ##avian middlesbrough doses bleak madam predatory relentless ##vili reluctance ##vir hailey crore silvery 1759 monstrous swimmers transmissions hawthorn informing ##eral toilets caracas crouch kb ##sett 295 cartel hadley ##aling alexia yvonne ##biology cinderella eton superb blizzard stabbing industrialist maximus ##gm ##orus groves maud clade oversized comedic ##bella rosen nomadic fulham montane beverages galaxies redundant swarm ##rot ##folia ##llis buckinghamshire fen bearings bahadur ##rom gilles phased dynamite faber benoit vip ##ount ##wd booking fractured tailored anya spices westwood cairns auditions inflammation steamed ##rocity ##acion ##urne skyla thereof watford torment archdeacon transforms lulu demeanor fucked serge ##sor mckenna minas entertainer ##icide caress originate residue ##sty 1740 ##ilised ##org beech ##wana subsidies ##ghton emptied gladstone ru firefighters voodoo ##rcle het nightingale tamara edmond ingredient weaknesses silhouette 285 compatibility withdrawing hampson ##mona anguish giggling ##mber bookstore ##jiang southernmost tilting ##vance bai economical rf briefcase dreadful hinted projections shattering totaling ##rogate analogue indicted periodical fullback ##dman haynes ##tenberg ##ffs ##ishment 1745 thirst stumble penang vigorous ##ddling ##kor ##lium octave ##ove ##enstein ##inen ##ones siberian ##uti cbn repeal swaying ##vington khalid tanaka unicorn otago plastered lobe riddle ##rella perch ##ishing croydon filtered graeme tripoli ##ossa crocodile ##chers sufi mined ##tung inferno lsu ##phi swelled utilizes £2 cale periodicals styx hike informally coop lund ##tidae ala hen qui transformations disposed sheath chickens ##cade fitzroy sas silesia unacceptable odisha 1650 sabrina pe spokane ratios athena massage shen dilemma ##drum ##riz ##hul corona doubtful niall ##pha ##bino fines cite acknowledging bangor ballard bathurst ##resh huron mustered alzheimer garments kinase tyre warship ##cp flashback pulmonary braun cheat kamal cyclists constructions grenades ndp traveller excuses stomped signalling trimmed futsal mosques relevance ##wine wta ##23 ##vah ##lter hoc ##riding optimistic ##´s deco sim interacting rejecting moniker waterways ##ieri ##oku mayors gdansk outnumbered pearls ##ended ##hampton fairs totals dominating 262 notions stairway compiling pursed commodities grease yeast ##jong carthage griffiths residual amc contraction laird sapphire ##marine ##ivated amalgamation dissolve inclination lyle packaged altitudes suez canons graded lurched narrowing boasts guise wed enrico ##ovsky rower scarred bree cub iberian protagonists bargaining proposing trainers voyages vans fishes ##aea ##ivist ##verance encryption artworks kazan sabre cleopatra hepburn rotting supremacy mecklenburg ##brate burrows hazards outgoing flair organizes ##ctions scorpion ##usions boo 234 chevalier dunedin slapping ##34 ineligible pensions ##38 ##omic manufactures emails bismarck 238 weakening blackish ding mcgee quo ##rling northernmost xx manpower greed sampson clicking ##ange ##horpe ##inations ##roving torre ##eptive ##moral symbolism 38th asshole meritorious outfits splashed biographies sprung astros ##tale 302 737 filly raoul nw tokugawa linden clubhouse ##apa tracts romano ##pio putin tags ##note chained dickson gunshot moe gunn rashid ##tails zipper ##bas ##nea contrasted ##ply ##udes plum pharaoh ##pile aw comedies ingrid sandwiches subdivisions 1100 mariana nokia kamen hz delaney veto herring ##words possessive outlines ##roup siemens stairwell rc gallantry messiah palais yells 233 zeppelin ##dm bolivar ##cede smackdown mckinley ##mora ##yt muted geologic finely unitary avatar hamas maynard rees bog contrasting ##rut liv chico disposition pixel ##erate becca dmitry yeshiva narratives ##lva ##ulton mercenary sharpe tempered navigate stealth amassed keynes ##lini untouched ##rrie havoc lithium ##fighting abyss graf southward wolverine balloons implements ngos transitions ##icum ambushed concacaf dormant economists ##dim costing csi rana universite boulders verity ##llon collin mellon misses cypress fluorescent lifeless spence ##ulla crewe shepard pak revelations ##م jolly gibbons paw ##dro ##quel freeing ##test shack fries palatine ##51 ##hiko accompaniment cruising recycled ##aver erwin sorting synthesizers dyke realities sg strides enslaved wetland ##ghan competence gunpowder grassy maroon reactors objection ##oms carlson gearbox macintosh radios shelton ##sho clergyman prakash 254 mongols trophies oricon 228 stimuli twenty20 cantonese cortes mirrored ##saurus bhp cristina melancholy ##lating enjoyable nuevo ##wny downfall schumacher ##ind banging lausanne rumbled paramilitary reflex ax amplitude migratory ##gall ##ups midi barnard lastly sherry ##hp ##nall keystone ##kra carleton slippery ##53 coloring foe socket otter ##rgos mats ##tose consultants bafta bison topping ##km 490 primal abandonment transplant atoll hideous mort pained reproduced tae howling ##turn unlawful billionaire hotter poised lansing ##chang dinamo retro messing nfc domesday ##mina blitz timed ##athing ##kley ascending gesturing ##izations signaled tis chinatown mermaid savanna jameson ##aint catalina ##pet ##hers cochrane cy chatting ##kus alerted computation mused noelle majestic mohawk campo octagonal ##sant ##hend 241 aspiring ##mart comprehend iona paralyzed shimmering swindon rhone ##eley reputed configurations pitchfork agitation francais gillian lipstick ##ilo outsiders pontifical resisting bitterness sewer rockies ##edd ##ucher misleading 1756 exiting galloway ##nging risked ##heart 246 commemoration schultz ##rka integrating ##rsa poses shrieked ##weiler guineas gladys jerking owls goldsmith nightly penetrating ##unced lia ##33 ignited betsy ##aring ##thorpe follower vigorously ##rave coded kiran knit zoology tbilisi ##28 ##bered repository govt deciduous dino growling ##bba enhancement unleashed chanting pussy biochemistry ##eric kettle repression toxicity nrhp ##arth ##kko ##bush ernesto commended outspoken 242 mca parchment sms kristen ##aton bisexual raked glamour navajo a2 conditioned showcased ##hma spacious youthful ##esa usl appliances junta brest layne conglomerate enchanted chao loosened picasso circulating inspect montevideo ##centric ##kti piazza spurred ##aith bari freedoms poultry stamford lieu ##ect indigo sarcastic bahia stump attach dvds frankenstein lille approx scriptures pollen ##script nmi overseen ##ivism tides proponent newmarket inherit milling ##erland centralized ##rou distributors credentials drawers abbreviation ##lco ##xon downing uncomfortably ripe ##oes erase franchises ##ever populace ##bery ##khar decomposition pleas ##tet daryl sabah ##stle ##wide fearless genie lesions annette ##ogist oboe appendix nair dripped petitioned maclean mosquito parrot rpg hampered 1648 operatic reservoirs ##tham irrelevant jolt summarized ##fp medallion ##taff ##− clawed harlow narrower goddard marcia bodied fremont suarez altering tempest mussolini porn ##isms sweetly oversees walkers solitude grimly shrines hk ich supervisors hostess dietrich legitimacy brushes expressive ##yp dissipated ##rse localized systemic ##nikov gettysburg ##js ##uaries dialogues muttering 251 housekeeper sicilian discouraged ##frey beamed kaladin halftime kidnap ##amo ##llet 1754 synonymous depleted instituto insulin reprised ##opsis clashed ##ctric interrupting radcliffe insisting medici 1715 ejected playfully turbulent ##47 starvation ##rini shipment rebellious petersen verification merits ##rified cakes ##charged 1757 milford shortages spying fidelity ##aker emitted storylines harvested seismic ##iform cheung kilda theoretically barbie lynx ##rgy ##tius goblin mata poisonous ##nburg reactive residues obedience ##евич conjecture ##rac 401 hating sixties kicker moaning motown ##bha emancipation neoclassical ##hering consoles ebert professorship ##tures sustaining assaults obeyed affluent incurred tornadoes ##eber ##zow emphasizing highlanders cheated helmets ##ctus internship terence bony executions legislators berries peninsular tinged ##aco 1689 amplifier corvette ribbons lavish pennant ##lander worthless ##chfield ##forms mariano pyrenees expenditures ##icides chesterfield mandir tailor 39th sergey nestled willed aristocracy devotees goodnight raaf rumored weaponry remy appropriations harcourt burr riaa ##lence limitation unnoticed guo soaking swamps ##tica collapsing tatiana descriptive brigham psalm ##chment maddox ##lization patti caliph ##aja akron injuring serra ##ganj basins ##sari astonished launcher ##church hilary wilkins sewing ##sf stinging ##fia ##ncia underwood startup ##ition compilations vibrations embankment jurist ##nity bard juventus groundwater kern palaces helium boca cramped marissa soto ##worm jae princely ##ggy faso bazaar warmly ##voking 229 pairing ##lite ##grate ##nets wien freaked ulysses rebirth ##alia ##rent mummy guzman jimenez stilled ##nitz trajectory tha woken archival professions ##pts ##pta hilly shadowy shrink ##bolt norwood glued migrate stereotypes devoid ##pheus 625 evacuate horrors infancy gotham knowles optic downloaded sachs kingsley parramatta darryl mor ##onale shady commence confesses kan ##meter ##placed marlborough roundabout regents frigates io ##imating gothenburg revoked carvings clockwise convertible intruder ##sche banged ##ogo vicky bourgeois ##mony dupont footing ##gum pd ##real buckle yun penthouse sane 720 serviced stakeholders neumann bb ##eers comb ##gam catchment pinning rallies typing ##elles forefront freiburg sweetie giacomo widowed goodwill worshipped aspirations midday ##vat fishery ##trick bournemouth turk 243 hearth ethanol guadalajara murmurs sl ##uge afforded scripted ##hta wah ##jn coroner translucent 252 memorials puck progresses clumsy ##race 315 candace recounted ##27 ##slin ##uve filtering ##mac howl strata heron leveled ##ays dubious ##oja ##т ##wheel citations exhibiting ##laya ##mics ##pods turkic ##lberg injunction ##ennial ##mit antibodies ##44 organise ##rigues cardiovascular cushion inverness ##zquez dia cocoa sibling ##tman ##roid expanse feasible tunisian algiers ##relli rus bloomberg dso westphalia bro tacoma 281 downloads ##ours konrad duran ##hdi continuum jett compares legislator secession ##nable ##gues ##zuka translating reacher ##gley ##ła aleppo ##agi tc orchards trapping linguist versatile drumming postage calhoun superiors ##mx barefoot leary ##cis ignacio alfa kaplan ##rogen bratislava mori ##vot disturb haas 313 cartridges gilmore radiated salford tunic hades ##ulsive archeological delilah magistrates auditioned brewster charters empowerment blogs cappella dynasties iroquois whipping ##krishna raceway truths myra weaken judah mcgregor ##horse mic refueling 37th burnley bosses markus premio query ##gga dunbar ##economic darkest lyndon sealing commendation reappeared ##mun addicted ezio slaughtered satisfactory shuffle ##eves ##thic ##uj fortification warrington ##otto resurrected fargo mane ##utable ##lei ##space foreword ox ##aris ##vern abrams hua ##mento sakura ##alo uv sentimental ##skaya midfield ##eses sturdy scrolls macleod ##kyu entropy ##lance mitochondrial cicero excelled thinner convoys perceive ##oslav ##urable systematically grind burkina 287 ##tagram ops ##aman guantanamo ##cloth ##tite forcefully wavy ##jou pointless ##linger ##tze layton portico superficial clerical outlaws ##hism burials muir ##inn creditors hauling rattle ##leg calais monde archers reclaimed dwell wexford hellenic falsely remorse ##tek dough furnishings ##uttered gabon neurological novice ##igraphy contemplated pulpit nightstand saratoga ##istan documenting pulsing taluk ##firmed busted marital ##rien disagreements wasps ##yes hodge mcdonnell mimic fran pendant dhabi musa ##nington congratulations argent darrell concussion losers regrets thessaloniki reversal donaldson hardwood thence achilles ritter ##eran demonic jurgen prophets goethe eki classmate buff ##cking yank irrational ##inging perished seductive qur sourced ##crat ##typic mustard ravine barre horizontally characterization phylogenetic boise ##dit ##runner ##tower brutally intercourse seduce ##bbing fay ferris ogden amar nik unarmed ##inator evaluating kyrgyzstan sweetness ##lford ##oki mccormick meiji notoriety stimulate disrupt figuring instructional mcgrath ##zoo groundbreaking ##lto flinch khorasan agrarian bengals mixer radiating ##sov ingram pitchers nad tariff ##cript tata ##codes ##emi ##ungen appellate lehigh ##bled ##giri brawl duct texans ##ciation ##ropolis skipper speculative vomit doctrines stresses 253 davy graders whitehead jozef timely cumulative haryana paints appropriately boon cactus ##ales ##pid dow legions ##pit perceptions 1730 picturesque ##yse periphery rune wr ##aha celtics sentencing whoa ##erin confirms variance 425 moines mathews spade rave m1 fronted fx blending alleging reared ##gl 237 ##paper grassroots eroded ##free ##physical directs ordeal ##sław accelerate hacker rooftop ##inia lev buys cebu devote ##lce specialising ##ulsion choreographed repetition warehouses ##ryl paisley tuscany analogy sorcerer hash huts shards descends exclude nix chaplin gaga ito vane ##drich causeway misconduct limo orchestrated glands jana ##kot u2 ##mple ##sons branching contrasts scoop longed ##virus chattanooga ##75 syrup cornerstone ##tized ##mind ##iaceae careless precedence frescoes ##uet chilled consult modelled snatch peat ##thermal caucasian humane relaxation spins temperance ##lbert occupations lambda hybrids moons mp3 ##oese 247 rolf societal yerevan ness ##ssler befriended mechanized nominate trough boasted cues seater ##hom bends ##tangle conductors emptiness ##lmer eurasian adriatic tian ##cie anxiously lark propellers chichester jock ev 2a ##holding credible recounts tori loyalist abduction ##hoot ##redo nepali ##mite ventral tempting ##ango ##crats steered ##wice javelin dipping laborers prentice looming titanium ##ː badges emir tensor ##ntation egyptians rash denies hawthorne lombard showers wehrmacht dietary trojan ##reus welles executing horseshoe lifeboat ##lak elsa infirmary nearing roberta boyer mutter trillion joanne ##fine ##oked sinks vortex uruguayan clasp sirius ##block accelerator prohibit sunken byu chronological diplomats ochreous 510 symmetrical 1644 maia ##tology salts reigns atrocities ##ия hess bared issn ##vyn cater saturated ##cycle ##isse sable voyager dyer yusuf ##inge fountains wolff ##39 ##nni engraving rollins atheist ominous ##ault herr chariot martina strung ##fell ##farlane horrific sahib gazes saetan erased ptolemy ##olic flushing lauderdale analytic ##ices 530 navarro beak gorilla herrera broom guadalupe raiding sykes 311 bsc deliveries 1720 invasions carmichael tajikistan thematic ecumenical sentiments onstage ##rians ##brand ##sume catastrophic flanks molten ##arns waller aimee terminating ##icing alternately ##oche nehru printers outraged ##eving empires template banners repetitive za ##oise vegetarian ##tell guiana opt cavendish lucknow synthesized ##hani ##mada finalized ##ctable fictitious mayoral unreliable ##enham embracing peppers rbis ##chio ##neo inhibition slashed togo orderly embroidered safari salty 236 barron benito totaled ##dak pubs simulated caden devin tolkien momma welding sesame ##ept gottingen hardness 630 shaman temeraire 620 adequately pediatric ##kit ck assertion radicals composure cadence seafood beaufort lazarus mani warily cunning kurdistan 249 cantata ##kir ares ##41 ##clusive nape townland geared insulted flutter boating violate draper dumping malmo ##hh ##romatic firearm alta bono obscured ##clave exceeds panorama unbelievable ##train preschool ##essed disconnected installing rescuing secretaries accessibility ##castle ##drive ##ifice ##film bouts slug waterway mindanao ##buro ##ratic halves ##ل calming liter maternity adorable bragg electrification mcc ##dote roxy schizophrenia ##body munoz kaye whaling 239 mil tingling tolerant ##ago unconventional volcanoes ##finder deportivo ##llie robson kaufman neuroscience wai deportation masovian scraping converse ##bh hacking bulge ##oun administratively yao 580 amp mammoth booster claremont hooper nomenclature pursuits mclaughlin melinda ##sul catfish barclay substrates taxa zee originals kimberly packets padma ##ality borrowing ostensibly solvent ##bri ##genesis ##mist lukas shreveport veracruz ##ь ##lou ##wives cheney tt anatolia hobbs ##zyn cyclic radiant alistair greenish siena dat independents ##bation conform pieter hyper applicant bradshaw spores telangana vinci inexpensive nuclei 322 jang nme soho spd ##ign cradled receptionist pow ##43 ##rika fascism ##ifer experimenting ##ading ##iec ##region 345 jocelyn maris stair nocturnal toro constabulary elgin ##kker msc ##giving ##schen ##rase doherty doping sarcastically batter maneuvers ##cano ##apple ##gai ##git intrinsic ##nst ##stor 1753 showtime cafes gasps lviv ushered ##thed fours restart astonishment transmitting flyer shrugs ##sau intriguing cones dictated mushrooms medial ##kovsky ##elman escorting gaped ##26 godfather ##door ##sell djs recaptured timetable vila 1710 3a aerodrome mortals scientology ##orne angelina mag convection unpaid insertion intermittent lego ##nated endeavor kota pereira ##lz 304 bwv glamorgan insults agatha fey ##cend fleetwood mahogany protruding steamship zeta ##arty mcguire suspense ##sphere advising urges ##wala hurriedly meteor gilded inline arroyo stalker ##oge excitedly revered ##cure earle introductory ##break ##ilde mutants puff pulses reinforcement ##haling curses lizards stalk correlated ##fixed fallout macquarie ##unas bearded denton heaving 802 ##ocation winery assign dortmund ##lkirk everest invariant charismatic susie ##elling bled lesley telegram sumner bk ##ogen ##к wilcox needy colbert duval ##iferous ##mbled allotted attends imperative ##hita replacements hawker ##inda insurgency ##zee ##eke casts ##yla 680 ives transitioned ##pack ##powering authoritative baylor flex cringed plaintiffs woodrow ##skie drastic ape aroma unfolded commotion nt preoccupied theta routines lasers privatization wand domino ek clenching nsa strategically showered bile handkerchief pere storing christophe insulting 316 nakamura romani asiatic magdalena palma cruises stripping 405 konstantin soaring ##berman colloquially forerunner havilland incarcerated parasites sincerity ##utus disks plank saigon ##ining corbin homo ornaments powerhouse ##tlement chong fastened feasibility idf morphological usable ##nish ##zuki aqueduct jaguars keepers ##flies aleksandr faust assigns ewing bacterium hurled tricky hungarians integers wallis 321 yamaha ##isha hushed oblivion aviator evangelist friars ##eller monograph ode ##nary airplanes labourers charms ##nee 1661 hagen tnt rudder fiesta transcript dorothea ska inhibitor maccabi retorted raining encompassed clauses menacing 1642 lineman ##gist vamps ##ape ##dick gloom ##rera dealings easing seekers ##nut ##pment helens unmanned ##anu ##isson basics ##amy ##ckman adjustments 1688 brutality horne ##zell sui ##55 ##mable aggregator ##thal rhino ##drick ##vira counters zoom ##01 ##rting mn montenegrin packard ##unciation ##♭ ##kki reclaim scholastic thugs pulsed ##icia syriac quan saddam banda kobe blaming buddies dissent ##lusion ##usia corbett jaya delle erratic lexie ##hesis 435 amiga hermes ##pressing ##leen chapels gospels jamal ##uating compute revolving warp ##sso ##thes armory ##eras ##gol antrim loki ##kow ##asian ##good ##zano braid handwriting subdistrict funky pantheon ##iculate concurrency estimation improper juliana ##his newcomers johnstone staten communicated ##oco ##alle sausage stormy ##stered ##tters superfamily ##grade acidic collateral tabloid ##oped ##rza bladder austen ##ellant mcgraw ##hay hannibal mein aquino lucifer wo badger boar cher christensen greenberg interruption ##kken jem 244 mocked bottoms cambridgeshire ##lide sprawling ##bbly eastwood ghent synth ##buck advisers ##bah nominally hapoel qu daggers estranged fabricated towels vinnie wcw misunderstanding anglia nothin unmistakable ##dust ##lova chilly marquette truss ##edge ##erine reece ##lty ##chemist ##connected 272 308 41st bash raion waterfalls ##ump ##main labyrinth queue theorist ##istle bharatiya flexed soundtracks rooney leftist patrolling wharton plainly alleviate eastman schuster topographic engages immensely unbearable fairchild 1620 dona lurking parisian oliveira ia indictment hahn bangladeshi ##aster vivo ##uming ##ential antonia expects indoors kildare harlan ##logue ##ogenic ##sities forgiven ##wat childish tavi ##mide ##orra plausible grimm successively scooted ##bola ##dget ##rith spartans emery flatly azure epilogue ##wark flourish ##iny ##tracted ##overs ##oshi bestseller distressed receipt spitting hermit topological ##cot drilled subunit francs ##layer eel ##fk ##itas octopus footprint petitions ufo ##say ##foil interfering leaking palo ##metry thistle valiant ##pic narayan mcpherson ##fast gonzales ##ym ##enne dustin novgorod solos ##zman doin ##raph ##patient ##meyer soluble ashland cuffs carole pendleton whistling vassal ##river deviation revisited constituents rallied rotate loomed ##eil ##nting amateurs augsburg auschwitz crowns skeletons ##cona bonnet 257 dummy globalization simeon sleeper mandal differentiated ##crow ##mare milne bundled exasperated talmud owes segregated ##feng ##uary dentist piracy props ##rang devlin ##torium malicious paws ##laid dependency ##ergy ##fers ##enna 258 pistons rourke jed grammatical tres maha wig 512 ghostly jayne ##achal ##creen ##ilis ##lins ##rence designate ##with arrogance cambodian clones showdown throttle twain ##ception lobes metz nagoya 335 braking ##furt 385 roaming ##minster amin crippled ##37 ##llary indifferent hoffmann idols intimidating 1751 261 influenza memo onions 1748 bandage consciously ##landa ##rage clandestine observes swiped tangle ##ener ##jected ##trum ##bill ##lta hugs congresses josiah spirited ##dek humanist managerial filmmaking inmate rhymes debuting grimsby ur ##laze duplicate vigor ##tf republished bolshevik refurbishment antibiotics martini methane newscasts royale horizons levant iain visas ##ischen paler ##around manifestation snuck alf chop futile pedestal rehab ##kat bmg kerman res fairbanks jarrett abstraction saharan ##zek 1746 procedural clearer kincaid sash luciano ##ffey crunch helmut ##vara revolutionaries ##tute creamy leach ##mmon 1747 permitting nes plight wendell ##lese contra ts clancy ipa mach staples autopsy disturbances nueva karin pontiac ##uding proxy venerable haunt leto bergman expands ##helm wal ##pipe canning celine cords obesity ##enary intrusion planner ##phate reasoned sequencing 307 harrow ##chon ##dora marred mcintyre repay tarzan darting 248 harrisburg margarita repulsed ##hur ##lding belinda hamburger novo compliant runways bingham registrar skyscraper ic cuthbert improvisation livelihood ##corp ##elial admiring ##dened sporadic believer casablanca popcorn ##29 asha shovel ##bek ##dice coiled tangible ##dez casper elsie resin tenderness rectory ##ivision avail sonar ##mori boutique ##dier guerre bathed upbringing vaulted sandals blessings ##naut ##utnant 1680 306 foxes pia corrosion hesitantly confederates crystalline footprints shapiro tirana valentin drones 45th microscope shipments texted inquisition wry guernsey unauthorized resigning 760 ripple schubert stu reassure felony ##ardo brittle koreans ##havan ##ives dun implicit tyres ##aldi ##lth magnolia ##ehan ##puri ##poulos aggressively fei gr familiarity ##poo indicative ##trust fundamentally jimmie overrun 395 anchors moans ##opus britannia armagh ##ggle purposely seizing ##vao bewildered mundane avoidance cosmopolitan geometridae quartermaster caf 415 chatter engulfed gleam purge ##icate juliette jurisprudence guerra revisions ##bn casimir brew ##jm 1749 clapton cloudy conde hermitage 278 simulations torches vincenzo matteo ##rill hidalgo booming westbound accomplishment tentacles unaffected ##sius annabelle flopped sloping ##litz dreamer interceptor vu ##loh consecration copying messaging breaker climates hospitalized 1752 torino afternoons winfield witnessing ##teacher breakers choirs sawmill coldly ##ege sipping haste uninhabited conical bibliography pamphlets severn edict ##oca deux illnesses grips ##pl rehearsals sis thinkers tame ##keepers 1690 acacia reformer ##osed ##rys shuffling ##iring ##shima eastbound ionic rhea flees littered ##oum rocker vomiting groaning champ overwhelmingly civilizations paces sloop adoptive ##tish skaters ##vres aiding mango ##joy nikola shriek ##ignon pharmaceuticals ##mg tuna calvert gustavo stocked yearbook ##urai ##mana computed subsp riff hanoi kelvin hamid moors pastures summons jihad nectar ##ctors bayou untitled pleasing vastly republics intellect ##η ##ulio ##tou crumbling stylistic sb ##ی consolation frequented h₂o walden widows ##iens 404 ##ignment chunks improves 288 grit recited ##dev snarl sociological ##arte ##gul inquired ##held bruise clube consultancy homogeneous hornets multiplication pasta prick savior ##grin ##kou ##phile yoon ##gara grimes vanishing cheering reacting bn distillery ##quisite ##vity coe dockyard massif ##jord escorts voss ##valent byte chopped hawke illusions workings floats ##koto ##vac kv annapolis madden ##onus alvaro noctuidae ##cum ##scopic avenge steamboat forte illustrates erika ##trip 570 dew nationalities bran manifested thirsty diversified muscled reborn ##standing arson ##lessness ##dran ##logram ##boys ##kushima ##vious willoughby ##phobia 286 alsace dashboard yuki ##chai granville myspace publicized tricked ##gang adjective ##ater relic reorganisation enthusiastically indications saxe ##lassified consolidate iec padua helplessly ramps renaming regulars pedestrians accents convicts inaccurate lowers mana ##pati barrie bjp outta someplace berwick flanking invoked marrow sparsely excerpts clothed rei ##ginal wept ##straße ##vish alexa excel ##ptive membranes aquitaine creeks cutler sheppard implementations ns ##dur fragrance budge concordia magnesium marcelo ##antes gladly vibrating ##rral ##ggles montrose ##omba lew seamus 1630 cocky ##ament ##uen bjorn ##rrick fielder fluttering ##lase methyl kimberley mcdowell reductions barbed ##jic ##tonic aeronautical condensed distracting ##promising huffed ##cala ##sle claudius invincible missy pious balthazar ci ##lang butte combo orson ##dication myriad 1707 silenced ##fed ##rh coco netball yourselves ##oza clarify heller peg durban etudes offender roast blackmail curvature ##woods vile 309 illicit suriname ##linson overture 1685 bubbling gymnast tucking ##mming ##ouin maldives ##bala gurney ##dda ##eased ##oides backside pinto jars racehorse tending ##rdial baronetcy wiener duly ##rke barbarian cupping flawed ##thesis bertha pleistocene puddle swearing ##nob ##tically fleeting prostate amulet educating ##mined ##iti ##tler 75th jens respondents analytics cavaliers papacy raju ##iente ##ulum ##tip funnel 271 disneyland ##lley sociologist ##iam 2500 faulkner louvre menon ##dson 276 ##ower afterlife mannheim peptide referees comedians meaningless ##anger ##laise fabrics hurley renal sleeps ##bour ##icle breakout kristin roadside animator clover disdain unsafe redesign ##urity firth barnsley portage reset narrows 268 commandos expansive speechless tubular ##lux essendon eyelashes smashwords ##yad ##bang ##claim craved sprinted chet somme astor wrocław orton 266 bane ##erving ##uing mischief ##amps ##sund scaling terre ##xious impairment offenses undermine moi soy contiguous arcadia inuit seam ##tops macbeth rebelled ##icative ##iot 590 elaborated frs uniformed ##dberg 259 powerless priscilla stimulated 980 qc arboretum frustrating trieste bullock ##nified enriched glistening intern ##adia locus nouvelle ollie ike lash starboard ee tapestry headlined hove rigged ##vite pollock ##yme thrive clustered cas roi gleamed olympiad ##lino pressured regimes ##hosis ##lick ripley ##ophone kickoff gallon rockwell ##arable crusader glue revolutions scrambling 1714 grover ##jure englishman aztec 263 contemplating coven ipad preach triumphant tufts ##esian rotational ##phus 328 falkland ##brates strewn clarissa rejoin environmentally glint banded drenched moat albanians johor rr maestro malley nouveau shaded taxonomy v6 adhere bunk airfields ##ritan 1741 encompass remington tran ##erative amelie mazda friar morals passions ##zai breadth vis ##hae argus burnham caressing insider rudd ##imov ##mini ##rso italianate murderous textual wainwright armada bam weave timer ##taken ##nh fra ##crest ardent salazar taps tunis ##ntino allegro gland philanthropic ##chester implication ##optera esq judas noticeably wynn ##dara inched indexed crises villiers bandit royalties patterned cupboard interspersed accessory isla kendrick entourage stitches ##esthesia headwaters ##ior interlude distraught draught 1727 ##basket biased sy transient triad subgenus adapting kidd shortstop ##umatic dimly spiked mcleod reprint nellie pretoria windmill ##cek singled ##mps 273 reunite ##orous 747 bankers outlying ##omp ##ports ##tream apologies cosmetics patsy ##deh ##ocks ##yson bender nantes serene ##nad lucha mmm 323 ##cius ##gli cmll coinage nestor juarez ##rook smeared sprayed twitching sterile irina embodied juveniles enveloped miscellaneous cancers dq gulped luisa crested swat donegal ref ##anov ##acker hearst mercantile ##lika doorbell ua vicki ##alla ##som bilbao psychologists stryker sw horsemen turkmenistan wits ##national anson mathew screenings ##umb rihanna ##agne ##nessy aisles ##iani ##osphere hines kenton saskatoon tasha truncated ##champ ##itan mildred advises fredrik interpreting inhibitors ##athi spectroscopy ##hab ##kong karim panda ##oia ##nail ##vc conqueror kgb leukemia ##dity arrivals cheered pisa phosphorus shielded ##riated mammal unitarian urgently chopin sanitary ##mission spicy drugged hinges ##tort tipping trier impoverished westchester ##caster 267 epoch nonstop ##gman ##khov aromatic centrally cerro ##tively ##vio billions modulation sedimentary 283 facilitating outrageous goldstein ##eak ##kt ld maitland penultimate pollard ##dance fleets spaceship vertebrae ##nig alcoholism als recital ##bham ##ference ##omics m2 ##bm trois ##tropical ##в commemorates ##meric marge ##raction 1643 670 cosmetic ravaged ##ige catastrophe eng ##shida albrecht arterial bellamy decor harmon ##rde bulbs synchronized vito easiest shetland shielding wnba ##glers ##ssar ##riam brianna cumbria ##aceous ##rard cores thayer ##nsk brood hilltop luminous carts keynote larkin logos ##cta ##ا ##mund ##quay lilith tinted 277 wrestle mobilization ##uses sequential siam bloomfield takahashi 274 ##ieving presenters ringo blazed witty ##oven ##ignant devastation haydn harmed newt therese ##peed gershwin molina rabbis sudanese 001 innate restarted ##sack ##fus slices wb ##shah enroll hypothetical hysterical 1743 fabio indefinite warped ##hg exchanging 525 unsuitable ##sboro gallo 1603 bret cobalt homemade ##hunter mx operatives ##dhar terraces durable latch pens whorls ##ctuated ##eaux billing ligament succumbed ##gly regulators spawn ##brick ##stead filmfare rochelle ##nzo 1725 circumstance saber supplements ##nsky ##tson crowe wellesley carrot ##9th ##movable primate drury sincerely topical ##mad ##rao callahan kyiv smarter tits undo ##yeh announcements anthologies barrio nebula ##islaus ##shaft ##tyn bodyguards 2021 assassinate barns emmett scully ##mah ##yd ##eland ##tino ##itarian demoted gorman lashed prized adventist writ ##gui alla invertebrates ##ausen 1641 amman 1742 align healy redistribution ##gf ##rize insulation ##drop adherents hezbollah vitro ferns yanking 269 php registering uppsala cheerleading confines mischievous tully ##ross 49th docked roam stipulated pumpkin ##bry prompt ##ezer blindly shuddering craftsmen frail scented katharine scramble shaggy sponge helix zaragoza 279 ##52 43rd backlash fontaine seizures posse cowan nonfiction telenovela wwii hammered undone ##gpur encircled irs ##ivation artefacts oneself searing smallpox ##belle ##osaurus shandong breached upland blushing rankin infinitely psyche tolerated docking evicted ##col unmarked ##lving gnome lettering litres musique ##oint benevolent ##jal blackened ##anna mccall racers tingle ##ocene ##orestation introductions radically 292 ##hiff ##باد 1610 1739 munchen plead ##nka condo scissors ##sight ##tens apprehension ##cey ##yin hallmark watering formulas sequels ##llas aggravated bae commencing ##building enfield prohibits marne vedic civilized euclidean jagger beforehand blasts dumont ##arney ##nem 740 conversions hierarchical rios simulator ##dya ##lellan hedges oleg thrusts shadowed darby maximize 1744 gregorian ##nded ##routed sham unspecified ##hog emory factual ##smo ##tp fooled ##rger ortega wellness marlon ##oton ##urance casket keating ley enclave ##ayan char influencing jia ##chenko 412 ammonia erebidae incompatible violins cornered ##arat grooves astronauts columbian rampant fabrication kyushu mahmud vanish ##dern mesopotamia ##lete ict ##rgen caspian kenji pitted ##vered 999 grimace roanoke tchaikovsky twinned ##analysis ##awan xinjiang arias clemson kazakh sizable 1662 ##khand ##vard plunge tatum vittorio ##nden cholera ##dana ##oper bracing indifference projectile superliga ##chee realises upgrading 299 porte retribution ##vies nk stil ##resses ama bureaucracy blackberry bosch testosterone collapses greer ##pathic ioc fifties malls ##erved bao baskets adolescents siegfried ##osity ##tosis mantra detecting existent fledgling ##cchi dissatisfied gan telecommunication mingled sobbed 6000 controversies outdated taxis ##raus fright slams ##lham ##fect ##tten detectors fetal tanned ##uw fray goth olympian skipping mandates scratches sheng unspoken hyundai tracey hotspur restrictive ##buch americana mundo ##bari burroughs diva vulcan ##6th distinctions thumping ##ngen mikey sheds fide rescues springsteen vested valuation ##ece ##ely pinnacle rake sylvie ##edo almond quivering ##irus alteration faltered ##wad 51st hydra ticked ##kato recommends ##dicated antigua arjun stagecoach wilfred trickle pronouns ##pon aryan nighttime ##anian gall pea stitch ##hei leung milos ##dini eritrea nexus starved snowfall kant parasitic cot discus hana strikers appleton kitchens ##erina ##partisan ##itha ##vius disclose metis ##channel 1701 tesla ##vera fitch 1735 blooded ##tila decimal ##tang ##bai cyclones eun bottled peas pensacola basha bolivian crabs boil lanterns partridge roofed 1645 necks ##phila opined patting ##kla ##lland chuckles volta whereupon ##nche devout euroleague suicidal ##dee inherently involuntary knitting nasser ##hide puppets colourful courageous southend stills miraculous hodgson richer rochdale ethernet greta uniting prism umm ##haya ##itical ##utation deterioration pointe prowess ##ropriation lids scranton billings subcontinent ##koff ##scope brute kellogg psalms degraded ##vez stanisław ##ructured ferreira pun astonishing gunnar ##yat arya prc gottfried ##tight excursion ##ographer dina ##quil ##nare huffington illustrious wilbur gundam verandah ##zard naacp ##odle constructive fjord kade ##naud generosity thrilling baseline cayman frankish plastics accommodations zoological ##fting cedric qb motorized ##dome ##otted squealed tackled canucks budgets situ asthma dail gabled grasslands whimpered writhing judgments ##65 minnie pv ##carbon bananas grille domes monique odin maguire markham tierney ##estra ##chua libel poke speedy atrium laval notwithstanding ##edly fai kala ##sur robb ##sma listings luz supplementary tianjin ##acing enzo jd ric scanner croats transcribed ##49 arden cv ##hair ##raphy ##lver ##uy 357 seventies staggering alam horticultural hs regression timbers blasting ##ounded montagu manipulating ##cit catalytic 1550 troopers ##meo condemnation fitzpatrick ##oire ##roved inexperienced 1670 castes ##lative outing 314 dubois flicking quarrel ste learners 1625 iq whistled ##class 282 classify tariffs temperament 355 folly liszt ##yles immersed jordanian ceasefire apparel extras maru fished ##bio harta stockport assortment craftsman paralysis transmitters ##cola blindness ##wk fatally proficiency solemnly ##orno repairing amore groceries ultraviolet ##chase schoolhouse ##tua resurgence nailed ##otype ##× ruse saliva diagrams ##tructing albans rann thirties 1b antennas hilarious cougars paddington stats ##eger breakaway ipod reza authorship prohibiting scoffed ##etz ##ttle conscription defected trondheim ##fires ivanov keenan ##adan ##ciful ##fb ##slow locating ##ials ##tford cadiz basalt blankly interned rags rattling ##tick carpathian reassured sync bum guildford iss staunch ##onga astronomers sera sofie emergencies susquehanna ##heard duc mastery vh1 williamsburg bayer buckled craving ##khan ##rdes bloomington ##write alton barbecue ##bians justine ##hri ##ndt delightful smartphone newtown photon retrieval peugeot hissing ##monium ##orough flavors lighted relaunched tainted ##games ##lysis anarchy microscopic hopping adept evade evie ##beau inhibit sinn adjustable hurst intuition wilton cisco 44th lawful lowlands stockings thierry ##dalen ##hila ##nai fates prank tb maison lobbied provocative 1724 4a utopia ##qual carbonate gujarati purcell ##rford curtiss ##mei overgrown arenas mediation swallows ##rnik respectful turnbull ##hedron ##hope alyssa ozone ##ʻi ami gestapo johansson snooker canteen cuff declines empathy stigma ##ags ##iner ##raine taxpayers gui volga ##wright ##copic lifespan overcame tattooed enactment giggles ##ador ##camp barrington bribe obligatory orbiting peng ##enas elusive sucker ##vating cong hardship empowered anticipating estrada cryptic greasy detainees planck sudbury plaid dod marriott kayla ##ears ##vb ##zd mortally ##hein cognition radha 319 liechtenstein meade richly argyle harpsichord liberalism trumpets lauded tyrant salsa tiled lear promoters reused slicing trident ##chuk ##gami ##lka cantor checkpoint ##points gaul leger mammalian ##tov ##aar ##schaft doha frenchman nirvana ##vino delgado headlining ##eron ##iography jug tko 1649 naga intersections ##jia benfica nawab ##suka ashford gulp ##deck ##vill ##rug brentford frazier pleasures dunne potsdam shenzhen dentistry ##tec flanagan ##dorff ##hear chorale dinah prem quezon ##rogated relinquished sutra terri ##pani flaps ##rissa poly ##rnet homme aback ##eki linger womb ##kson ##lewood doorstep orthodoxy threaded westfield ##rval dioceses fridays subsided ##gata loyalists ##biotic ##ettes letterman lunatic prelate tenderly invariably souza thug winslow ##otide furlongs gogh jeopardy ##runa pegasus ##umble humiliated standalone tagged ##roller freshmen klan ##bright attaining initiating transatlantic logged viz ##uance 1723 combatants intervening stephane chieftain despised grazed 317 cdc galveston godzilla macro simulate ##planes parades ##esses 960 ##ductive ##unes equator overdose ##cans ##hosh ##lifting joshi epstein sonora treacherous aquatics manchu responsive ##sation supervisory ##christ ##llins ##ibar ##balance ##uso kimball karlsruhe mab ##emy ignores phonetic reuters spaghetti 820 almighty danzig rumbling tombstone designations lured outset ##felt supermarkets ##wt grupo kei kraft susanna ##blood comprehension genealogy ##aghan ##verted redding ##ythe 1722 bowing ##pore ##roi lest sharpened fulbright valkyrie sikhs ##unds swans bouquet merritt ##tage ##venting commuted redhead clerks leasing cesare dea hazy ##vances fledged greenfield servicemen ##gical armando blackout dt sagged downloadable intra potion pods ##4th ##mism xp attendants gambia stale ##ntine plump asteroids rediscovered buds flea hive ##neas 1737 classifications debuts ##eles olympus scala ##eurs ##gno ##mute hummed sigismund visuals wiggled await pilasters clench sulfate ##ances bellevue enigma trainee snort ##sw clouded denim ##rank ##rder churning hartman lodges riches sima ##missible accountable socrates regulates mueller ##cr 1702 avoids solids himalayas nutrient pup ##jevic squat fades nec ##lates ##pina ##rona ##ου privateer tequila ##gative ##mpton apt hornet immortals ##dou asturias cleansing dario ##rries ##anta etymology servicing zhejiang ##venor ##nx horned erasmus rayon relocating £10 ##bags escalated promenade stubble 2010s artisans axial liquids mora sho yoo ##tsky bundles oldies ##nally notification bastion ##ths sparkle ##lved 1728 leash pathogen highs ##hmi immature 880 gonzaga ignatius mansions monterrey sweets bryson ##loe polled regatta brightest pei rosy squid hatfield payroll addict meath cornerback heaviest lodging ##mage capcom rippled ##sily barnet mayhem ymca snuggled rousseau ##cute blanchard 284 fragmented leighton chromosomes risking ##md ##strel ##utter corinne coyotes cynical hiroshi yeomanry ##ractive ebook grading mandela plume agustin magdalene ##rkin bea femme trafford ##coll ##lun ##tance 52nd fourier upton ##mental camilla gust iihf islamabad longevity ##kala feldman netting ##rization endeavour foraging mfa orr ##open greyish contradiction graz ##ruff handicapped marlene tweed oaxaca spp campos miocene pri configured cooks pluto cozy pornographic ##entes 70th fairness glided jonny lynne rounding sired ##emon ##nist remade uncover ##mack complied lei newsweek ##jured ##parts ##enting ##pg 293 finer guerrillas athenian deng disused stepmother accuse gingerly seduction 521 confronting ##walker ##going gora nostalgia sabres virginity wrenched ##minated syndication wielding eyre ##56 ##gnon ##igny behaved taxpayer sweeps ##growth childless gallant ##ywood amplified geraldine scrape ##ffi babylonian fresco ##rdan ##kney ##position 1718 restricting tack fukuoka osborn selector partnering ##dlow 318 gnu kia tak whitley gables ##54 ##mania mri softness immersion ##bots ##evsky 1713 chilling insignificant pcs ##uis elites lina purported supplemental teaming ##americana ##dding ##inton proficient rouen ##nage ##rret niccolo selects ##bread fluffy 1621 gruff knotted mukherjee polgara thrash nicholls secluded smoothing thru corsica loaf whitaker inquiries ##rrier ##kam indochina 289 marlins myles peking ##tea extracts pastry superhuman connacht vogel ##ditional ##het ##udged ##lash gloss quarries refit teaser ##alic ##gaon 20s materialized sling camped pickering tung tracker pursuant ##cide cranes soc ##cini ##typical ##viere anhalt overboard workout chores fares orphaned stains ##logie fenton surpassing joyah triggers ##itte grandmaster ##lass ##lists clapping fraudulent ledger nagasaki ##cor ##nosis ##tsa eucalyptus tun ##icio ##rney ##tara dax heroism ina wrexham onboard unsigned ##dates moshe galley winnie droplets exiles praises watered noodles ##aia fein adi leland multicultural stink bingo comets erskine modernized canned constraint domestically chemotherapy featherweight stifled ##mum darkly irresistible refreshing hasty isolate ##oys kitchener planners ##wehr cages yarn implant toulon elects childbirth yue ##lind ##lone cn rightful sportsman junctions remodeled specifies ##rgh 291 ##oons complimented ##urgent lister ot ##logic bequeathed cheekbones fontana gabby ##dial amadeus corrugated maverick resented triangles ##hered ##usly nazareth tyrol 1675 assent poorer sectional aegean ##cous 296 nylon ghanaian ##egorical ##weig cushions forbid fusiliers obstruction somerville ##scia dime earrings elliptical leyte oder polymers timmy atm midtown piloted settles continual externally mayfield ##uh enrichment henson keane persians 1733 benji braden pep 324 ##efe contenders pepsi valet ##isches 298 ##asse ##earing goofy stroll ##amen authoritarian occurrences adversary ahmedabad tangent toppled dorchester 1672 modernism marxism islamist charlemagne exponential racks unicode brunette mbc pic skirmish ##bund ##lad ##powered ##yst hoisted messina shatter ##ctum jedi vantage ##music ##neil clemens mahmoud corrupted authentication lowry nils ##washed omnibus wounding jillian ##itors ##opped serialized narcotics handheld ##arm ##plicity intersecting stimulating ##onis crate fellowships hemingway casinos climatic fordham copeland drip beatty leaflets robber brothel madeira ##hedral sphinx ultrasound ##vana valor forbade leonid villas ##aldo duane marquez ##cytes disadvantaged forearms kawasaki reacts consular lax uncles uphold ##hopper concepcion dorsey lass ##izan arching passageway 1708 researches tia internationals ##graphs ##opers distinguishes javanese divert ##uven plotted ##listic ##rwin ##erik ##tify affirmative signifies validation ##bson kari felicity georgina zulu ##eros ##rained ##rath overcoming ##dot argyll ##rbin 1734 chiba ratification windy earls parapet ##marks hunan pristine astrid punta ##gart brodie ##kota ##oder malaga minerva rouse ##phonic bellowed pagoda portals reclamation ##gur ##odies ##⁄₄ parentheses quoting allergic palette showcases benefactor heartland nonlinear ##tness bladed cheerfully scans ##ety ##hone 1666 girlfriends pedersen hiram sous ##liche ##nator 1683 ##nery ##orio ##umen bobo primaries smiley ##cb unearthed uniformly fis metadata 1635 ind ##oted recoil ##titles ##tura ##ια 406 hilbert jamestown mcmillan tulane seychelles ##frid antics coli fated stucco ##grants 1654 bulky accolades arrays caledonian carnage optimism puebla ##tative ##cave enforcing rotherham seo dunlop aeronautics chimed incline zoning archduke hellenistic ##oses ##sions candi thong ##ople magnate rustic ##rsk projective slant ##offs danes hollis vocalists ##ammed congenital contend gesellschaft ##ocating ##pressive douglass quieter ##cm ##kshi howled salim spontaneously townsville buena southport ##bold kato 1638 faerie stiffly ##vus ##rled 297 flawless realising taboo ##7th bytes straightening 356 jena ##hid ##rmin cartwright berber bertram soloists 411 noses 417 coping fission hardin inca ##cen 1717 mobilized vhf ##raf biscuits curate ##85 ##anial 331 gaunt neighbourhoods 1540 ##abas blanca bypassed sockets behold coincidentally ##bane nara shave splinter terrific ##arion ##erian commonplace juris redwood waistband boxed caitlin fingerprints jennie naturalized ##ired balfour craters jody bungalow hugely quilt glitter pigeons undertaker bulging constrained goo ##sil ##akh assimilation reworked ##person persuasion ##pants felicia ##cliff ##ulent 1732 explodes ##dun ##inium ##zic lyman vulture hog overlook begs northwards ow spoil ##urer fatima favorably accumulate sargent sorority corresponded dispersal kochi toned ##imi ##lita internacional newfound ##agger ##lynn ##rigue booths peanuts ##eborg medicare muriel nur ##uram crates millennia pajamas worsened ##breakers jimi vanuatu yawned ##udeau carousel ##hony hurdle ##ccus ##mounted ##pod rv ##eche airship ambiguity compulsion recapture ##claiming arthritis ##osomal 1667 asserting ngc sniffing dade discontent glendale ported ##amina defamation rammed ##scent fling livingstone ##fleet 875 ##ppy apocalyptic comrade lcd ##lowe cessna eine persecuted subsistence demi hoop reliefs 710 coptic progressing stemmed perpetrators 1665 priestess ##nio dobson ebony rooster itf tortricidae ##bbon ##jian cleanup ##jean ##øy 1721 eighties taxonomic holiness ##hearted ##spar antilles showcasing stabilized ##nb gia mascara michelangelo dawned ##uria ##vinsky extinguished fitz grotesque £100 ##fera ##loid ##mous barges neue throbbed cipher johnnie ##a1 ##mpt outburst ##swick spearheaded administrations c1 heartbreak pixels pleasantly ##enay lombardy plush ##nsed bobbie ##hly reapers tremor xiang minogue substantive hitch barak ##wyl kwan ##encia 910 obscene elegance indus surfer bribery conserve ##hyllum ##masters horatio ##fat apes rebound psychotic ##pour iteration ##mium ##vani botanic horribly antiques dispose paxton ##hli ##wg timeless 1704 disregard engraver hounds ##bau ##version looted uno facilitates groans masjid rutland antibody disqualification decatur footballers quake slacks 48th rein scribe stabilize commits exemplary tho ##hort ##chison pantry traversed ##hiti disrepair identifiable vibrated baccalaureate ##nnis csa interviewing ##iensis ##raße greaves wealthiest 343 classed jogged £5 ##58 ##atal illuminating knicks respecting ##uno scrubbed ##iji ##dles kruger moods growls raider silvia chefs kam vr cree percival ##terol gunter counterattack defiant henan ze ##rasia ##riety equivalence submissions ##fra ##thor bautista mechanically ##heater cornice herbal templar ##mering outputs ruining ligand renumbered extravagant mika blockbuster eta insurrection ##ilia darkening ferocious pianos strife kinship ##aer melee ##anor ##iste ##may ##oue decidedly weep ##jad ##missive ##ppel 354 puget unease ##gnant 1629 hammering kassel ob wessex ##lga bromwich egan paranoia utilization ##atable ##idad contradictory provoke ##ols ##ouring ##tangled knesset ##very ##lette plumbing ##sden ##¹ greensboro occult sniff 338 zev beaming gamer haggard mahal ##olt ##pins mendes utmost briefing gunnery ##gut ##pher ##zh ##rok 1679 khalifa sonya ##boot principals urbana wiring ##liffe ##minating ##rrado dahl nyu skepticism np townspeople ithaca lobster somethin ##fur ##arina ##−1 freighter zimmerman biceps contractual ##herton amend hurrying subconscious ##anal 336 meng clermont spawning ##eia ##lub dignitaries impetus snacks spotting twigs ##bilis ##cz ##ouk libertadores nic skylar ##aina ##firm gustave asean ##anum dieter legislatures flirt bromley trolls umar ##bbies ##tyle blah parc bridgeport crank negligence ##nction 46th constantin molded bandages seriousness 00pm siegel carpets compartments upbeat statehood ##dner ##edging marko 730 platt ##hane paving ##iy 1738 abbess impatience limousine nbl ##talk 441 lucille mojo nightfall robbers ##nais karel brisk calves replicate ascribed telescopes ##olf intimidated ##reen ballast specialization ##sit aerodynamic caliphate rainer visionary ##arded epsilon ##aday ##onte aggregation auditory boosted reunification kathmandu loco robyn 402 acknowledges appointing humanoid newell redeveloped restraints ##tained barbarians chopper 1609 italiana ##lez ##lho investigates wrestlemania ##anies ##bib 690 ##falls creaked dragoons gravely minions stupidity volley ##harat ##week musik ##eries ##uously fungal massimo semantics malvern ##ahl ##pee discourage embryo imperialism 1910s profoundly ##ddled jiangsu sparkled stat ##holz sweatshirt tobin ##iction sneered ##cheon ##oit brit causal smyth ##neuve diffuse perrin silvio ##ipes ##recht detonated iqbal selma ##nism ##zumi roasted ##riders tay ##ados ##mament ##mut ##rud 840 completes nipples cfa flavour hirsch ##laus calderon sneakers moravian ##ksha 1622 rq 294 ##imeters bodo ##isance ##pre ##ronia anatomical excerpt ##lke dh kunst ##tablished ##scoe biomass panted unharmed gael housemates montpellier ##59 coa rodents tonic hickory singleton ##taro 451 1719 aldo breaststroke dempsey och rocco ##cuit merton dissemination midsummer serials ##idi haji polynomials ##rdon gs enoch prematurely shutter taunton £3 ##grating ##inates archangel harassed ##asco 326 archway dazzling ##ecin 1736 sumo wat ##kovich 1086 honneur ##ently ##nostic ##ttal ##idon 1605 403 1716 blogger rents ##gnan hires ##ikh ##dant howie ##rons handler retracted shocks 1632 arun duluth kepler trumpeter ##lary peeking seasoned trooper ##mara laszlo ##iciencies ##rti heterosexual ##inatory ##ssion indira jogging ##inga ##lism beit dissatisfaction malice ##ately nedra peeling ##rgeon 47th stadiums 475 vertigo ##ains iced restroom ##plify ##tub illustrating pear ##chner ##sibility inorganic rappers receipts watery ##kura lucinda ##oulos reintroduced ##8th ##tched gracefully saxons nutritional wastewater rained favourites bedrock fisted hallways likeness upscale ##lateral 1580 blinds prequel ##pps ##tama deter humiliating restraining tn vents 1659 laundering recess rosary tractors coulter federer ##ifiers ##plin persistence ##quitable geschichte pendulum quakers ##beam bassett pictorial buffet koln ##sitor drills reciprocal shooters ##57 ##cton ##tees converge pip dmitri donnelly yamamoto aqua azores demographics hypnotic spitfire suspend wryly roderick ##rran sebastien ##asurable mavericks ##fles ##200 himalayan prodigy ##iance transvaal demonstrators handcuffs dodged mcnamara sublime 1726 crazed ##efined ##till ivo pondered reconciled shrill sava ##duk bal cad heresy jaipur goran ##nished 341 lux shelly whitehall ##hre israelis peacekeeping ##wled 1703 demetrius ousted ##arians ##zos beale anwar backstroke raged shrinking cremated ##yck benign towing wadi darmstadt landfill parana soothe colleen sidewalks mayfair tumble hepatitis ferrer superstructure ##gingly ##urse ##wee anthropological translators ##mies closeness hooves ##pw mondays ##roll ##vita landscaping ##urized purification sock thorns thwarted jalan tiberius ##taka saline ##rito confidently khyber sculptors ##ij brahms hammersmith inspectors battista fivb fragmentation hackney ##uls arresting exercising antoinette bedfordshire ##zily dyed ##hema 1656 racetrack variability ##tique 1655 austrians deteriorating madman theorists aix lehman weathered 1731 decreed eruptions 1729 flaw quinlan sorbonne flutes nunez 1711 adored downwards fable rasped 1712 moritz mouthful renegade shivers stunts dysfunction restrain translit 327 pancakes ##avio ##cision ##tray 351 vial ##lden bain ##maid ##oxide chihuahua malacca vimes ##rba ##rnier 1664 donnie plaques ##ually 337 bangs floppy huntsville loretta nikolay ##otte eater handgun ubiquitous ##hett eras zodiac 1634 ##omorphic 1820s ##zog cochran ##bula ##lithic warring ##rada dalai excused blazers mcconnell reeling bot este ##abi geese hoax taxon ##bla guitarists ##icon condemning hunts inversion moffat taekwondo ##lvis 1624 stammered ##rest ##rzy sousa fundraiser marylebone navigable uptown cabbage daniela salman shitty whimper ##kian ##utive programmers protections rm ##rmi ##rued forceful ##enes fuss ##tao ##wash brat oppressive reykjavik spartak ticking ##inkles ##kiewicz adolph horst maui protege straighten cpc landau concourse clements resultant ##ando imaginative joo reactivated ##rem ##ffled ##uising consultative ##guide flop kaitlyn mergers parenting somber ##vron supervise vidhan ##imum courtship exemplified harmonies medallist refining ##rrow ##ка amara ##hum 780 goalscorer sited overshadowed rohan displeasure secretive multiplied osman ##orth engravings padre ##kali ##veda miniatures mis ##yala clap pali rook ##cana 1692 57th antennae astro oskar 1628 bulldog crotch hackett yucatan ##sure amplifiers brno ferrara migrating ##gree thanking turing ##eza mccann ting andersson onslaught gaines ganga incense standardization ##mation sentai scuba stuffing turquoise waivers alloys ##vitt regaining vaults ##clops ##gizing digger furry memorabilia probing ##iad payton rec deutschland filippo opaque seamen zenith afrikaans ##filtration disciplined inspirational ##merie banco confuse grafton tod ##dgets championed simi anomaly biplane ##ceptive electrode ##para 1697 cleavage crossbow swirl informant ##lars ##osta afi bonfire spec ##oux lakeside slump ##culus ##lais ##qvist ##rrigan 1016 facades borg inwardly cervical xl pointedly 050 stabilization ##odon chests 1699 hacked ctv orthogonal suzy ##lastic gaulle jacobite rearview ##cam ##erted ashby ##drik ##igate ##mise ##zbek affectionately canine disperse latham ##istles ##ivar spielberg ##orin ##idium ezekiel cid ##sg durga middletown ##cina customized frontiers harden ##etano ##zzy 1604 bolsheviks ##66 coloration yoko ##bedo briefs slabs debra liquidation plumage ##oin blossoms dementia subsidy 1611 proctor relational jerseys parochial ter ##ici esa peshawar cavalier loren cpi idiots shamrock 1646 dutton malabar mustache ##endez ##ocytes referencing terminates marche yarmouth ##sop acton mated seton subtly baptised beige extremes jolted kristina telecast ##actic safeguard waldo ##baldi ##bular endeavors sloppy subterranean ##ensburg ##itung delicately pigment tq ##scu 1626 ##ound collisions coveted herds ##personal ##meister ##nberger chopra ##ricting abnormalities defective galician lucie ##dilly alligator likened ##genase burundi clears complexion derelict deafening diablo fingered champaign dogg enlist isotope labeling mrna ##erre brilliance marvelous ##ayo 1652 crawley ether footed dwellers deserts hamish rubs warlock skimmed ##lizer 870 buick embark heraldic irregularities ##ajan kiara ##kulam ##ieg antigen kowalski ##lge oakley visitation ##mbit vt ##suit 1570 murderers ##miento ##rites chimneys ##sling condemn custer exchequer havre ##ghi fluctuations ##rations dfb hendricks vaccines ##tarian nietzsche biking juicy ##duced brooding scrolling selangor ##ragan 352 annum boomed seminole sugarcane ##dna departmental dismissing innsbruck arteries ashok batavia daze kun overtook ##rga ##tlan beheaded gaddafi holm electronically faulty galilee fractures kobayashi ##lized gunmen magma aramaic mala eastenders inference messengers bf ##qu 407 bathrooms ##vere 1658 flashbacks ideally misunderstood ##jali ##weather mendez ##grounds 505 uncanny ##iii 1709 friendships ##nbc sacrament accommodated reiterated logistical pebbles thumped ##escence administering decrees drafts ##flight ##cased ##tula futuristic picket intimidation winthrop ##fahan interfered 339 afar francoise morally uta cochin croft dwarfs ##bruck ##dents ##nami biker ##hner ##meral nano ##isen ##ometric ##pres ##ан brightened meek parcels securely gunners ##jhl ##zko agile hysteria ##lten ##rcus bukit champs chevy cuckoo leith sadler theologians welded ##section 1663 jj plurality xander ##rooms ##formed shredded temps intimately pau tormented ##lok ##stellar 1618 charred ems essen ##mmel alarms spraying ascot blooms twinkle ##abia ##apes internment obsidian ##chaft snoop ##dav ##ooping malibu ##tension quiver ##itia hays mcintosh travers walsall ##ffie 1623 beverley schwarz plunging structurally m3 rosenthal vikram ##tsk 770 ghz ##onda ##tiv chalmers groningen pew reckon unicef ##rvis 55th ##gni 1651 sulawesi avila cai metaphysical screwing turbulence ##mberg augusto samba 56th baffled momentary toxin ##urian ##wani aachen condoms dali steppe ##3d ##app ##oed ##year adolescence dauphin electrically inaccessible microscopy nikita ##ega atv ##cel ##enter ##oles ##oteric ##ы accountants punishments wrongly bribes adventurous clinch flinders southland ##hem ##kata gough ##ciency lads soared ##ה undergoes deformation outlawed rubbish ##arus ##mussen ##nidae ##rzburg arcs ##ingdon ##tituted 1695 wheelbase wheeling bombardier campground zebra ##lices ##oj ##bain lullaby ##ecure donetsk wylie grenada ##arding ##ης squinting eireann opposes ##andra maximal runes ##broken ##cuting ##iface ##ror ##rosis additive britney adultery triggering ##drome detrimental aarhus containment jc swapped vichy ##ioms madly ##oric ##rag brant ##ckey ##trix 1560 1612 broughton rustling ##stems ##uder asbestos mentoring ##nivorous finley leaps ##isan apical pry slits substitutes ##dict intuitive fantasia insistent unreasonable ##igen ##vna domed hannover margot ponder ##zziness impromptu jian lc rampage stemming ##eft andrey gerais whichever amnesia appropriated anzac clicks modifying ultimatum cambrian maids verve yellowstone ##mbs conservatoire ##scribe adherence dinners spectra imperfect mysteriously sidekick tatar tuba ##aks ##ifolia distrust ##athan ##zle c2 ronin zac ##pse celaena instrumentalist scents skopje ##mbling comical compensated vidal condor intersect jingle wavelengths ##urrent mcqueen ##izzly carp weasel 422 kanye militias postdoctoral eugen gunslinger ##ɛ faux hospice ##for appalled derivation dwarves ##elis dilapidated ##folk astoria philology ##lwyn ##otho ##saka inducing philanthropy ##bf ##itative geek markedly sql ##yce bessie indices rn ##flict 495 frowns resolving weightlifting tugs cleric contentious 1653 mania rms ##miya ##reate ##ruck ##tucket bien eels marek ##ayton ##cence discreet unofficially ##ife leaks ##bber 1705 332 dung compressor hillsborough pandit shillings distal ##skin 381 ##tat ##you nosed ##nir mangrove undeveloped ##idia textures ##inho ##500 ##rise ae irritating nay amazingly bancroft apologetic compassionate kata symphonies ##lovic airspace ##lch 930 gifford precautions fulfillment sevilla vulgar martinique ##urities looting piccolo tidy ##dermott quadrant armchair incomes mathematicians stampede nilsson ##inking ##scan foo quarterfinal ##ostal shang shouldered squirrels ##owe 344 vinegar ##bner ##rchy ##systems delaying ##trics ars dwyer rhapsody sponsoring ##gration bipolar cinder starters ##olio ##urst 421 signage ##nty aground figurative mons acquaintances duets erroneously soyuz elliptic recreated ##cultural ##quette ##ssed ##tma ##zcz moderator scares ##itaire ##stones ##udence juniper sighting ##just ##nsen britten calabria ry bop cramer forsyth stillness ##л airmen gathers unfit ##umber ##upt taunting ##rip seeker streamlined ##bution holster schumann tread vox ##gano ##onzo strive dil reforming covent newbury predicting ##orro decorate tre ##puted andover ie asahi dept dunkirk gills ##tori buren huskies ##stis ##stov abstracts bets loosen ##opa 1682 yearning ##glio ##sir berman effortlessly enamel napoli persist ##peration ##uez attache elisa b1 invitations ##kic accelerating reindeer boardwalk clutches nelly polka starbucks ##kei adamant huey lough unbroken adventurer embroidery inspecting stanza ##ducted naia taluka ##pone ##roids chases deprivation florian ##jing ##ppet earthly ##lib ##ssee colossal foreigner vet freaks patrice rosewood triassic upstate ##pkins dominates ata chants ks vo ##400 ##bley ##raya ##rmed 555 agra infiltrate ##ailing ##ilation ##tzer ##uppe ##werk binoculars enthusiast fujian squeak ##avs abolitionist almeida boredom hampstead marsden rations ##ands inflated 334 bonuses rosalie patna ##rco 329 detachments penitentiary 54th flourishing woolf ##dion ##etched papyrus ##lster ##nsor ##toy bobbed dismounted endelle inhuman motorola tbs wince wreath ##ticus hideout inspections sanjay disgrace infused pudding stalks ##urbed arsenic leases ##hyl ##rrard collarbone ##waite ##wil dowry ##bant ##edance genealogical nitrate salamanca scandals thyroid necessitated ##! ##" ### ##$ ##% ##& ##' ##( ##) ##* ##+ ##, ##- ##. ##/ ##: ##; ##< ##= ##> ##? ##@ ##[ ##\ ##] ##^ ##_ ##` ##{ ##| ##} ##~ ##¡ ##¢ ##£ ##¤ ##¥ ##¦ ##§ ##¨ ##© ##ª ##« ##¬ ##® ##± ##´ ##µ ##¶ ##· ##º ##» ##¼ ##¾ ##¿ ##æ ##ð ##÷ ##þ ##đ ##ħ ##ŋ ##œ ##ƒ ##ɐ ##ɑ ##ɒ ##ɔ ##ɕ ##ə ##ɡ ##ɣ ##ɨ ##ɪ ##ɫ ##ɬ ##ɯ ##ɲ ##ɴ ##ɹ ##ɾ ##ʀ ##ʁ ##ʂ ##ʃ ##ʉ ##ʊ ##ʋ ##ʌ ##ʎ ##ʐ ##ʑ ##ʒ ##ʔ ##ʰ ##ʲ ##ʳ ##ʷ ##ʸ ##ʻ ##ʼ ##ʾ ##ʿ ##ˈ ##ˡ ##ˢ ##ˣ ##ˤ ##β ##γ ##δ ##ε ##ζ ##θ ##κ ##λ ##μ ##ξ ##ο ##π ##ρ ##σ ##τ ##υ ##φ ##χ ##ψ ##ω ##б ##г ##д ##ж ##з ##м ##п ##с ##у ##ф ##х ##ц ##ч ##ш ##щ ##ъ ##э ##ю ##ђ ##є ##і ##ј ##љ ##њ ##ћ ##ӏ ##ա ##բ ##գ ##դ ##ե ##թ ##ի ##լ ##կ ##հ ##մ ##յ ##ն ##ո ##պ ##ս ##վ ##տ ##ր ##ւ ##ք ##־ ##א ##ב ##ג ##ד ##ו ##ז ##ח ##ט ##י ##ך ##כ ##ל ##ם ##מ ##ן ##נ ##ס ##ע ##ף ##פ ##ץ ##צ ##ק ##ר ##ש ##ת ##، ##ء ##ب ##ت ##ث ##ج ##ح ##خ ##ذ ##ز ##س ##ش ##ص ##ض ##ط ##ظ ##ع ##غ ##ـ ##ف ##ق ##ك ##و ##ى ##ٹ ##پ ##چ ##ک ##گ ##ں ##ھ ##ہ ##ے ##अ ##आ ##उ ##ए ##क ##ख ##ग ##च ##ज ##ट ##ड ##ण ##त ##थ ##द ##ध ##न ##प ##ब ##भ ##म ##य ##र ##ल ##व ##श ##ष ##स ##ह ##ा ##ि ##ी ##ो ##। ##॥ ##ং ##অ ##আ ##ই ##উ ##এ ##ও ##ক ##খ ##গ ##চ ##ছ ##জ ##ট ##ড ##ণ ##ত ##থ ##দ ##ধ ##ন ##প ##ব ##ভ ##ম ##য ##র ##ল ##শ ##ষ ##স ##হ ##া ##ি ##ী ##ে ##க ##ச ##ட ##த ##ந ##ன ##ப ##ம ##ய ##ர ##ல ##ள ##வ ##ா ##ி ##ு ##ே ##ை ##ನ ##ರ ##ಾ ##ක ##ය ##ර ##ල ##ව ##ා ##ก ##ง ##ต ##ท ##น ##พ ##ม ##ย ##ร ##ล ##ว ##ส ##อ ##า ##เ ##་ ##། ##ག ##ང ##ད ##ན ##པ ##བ ##མ ##འ ##ར ##ལ ##ས ##မ ##ა ##ბ ##გ ##დ ##ე ##ვ ##თ ##ი ##კ ##ლ ##მ ##ნ ##ო ##რ ##ს ##ტ ##უ ##ᄀ ##ᄂ ##ᄃ ##ᄅ ##ᄆ ##ᄇ ##ᄉ ##ᄊ ##ᄋ ##ᄌ ##ᄎ ##ᄏ ##ᄐ ##ᄑ ##ᄒ ##ᅡ ##ᅢ ##ᅥ ##ᅦ ##ᅧ ##ᅩ ##ᅪ ##ᅭ ##ᅮ ##ᅯ ##ᅲ ##ᅳ ##ᅴ ##ᅵ ##ᆨ ##ᆫ ##ᆯ ##ᆷ ##ᆸ ##ᆼ ##ᴬ ##ᴮ ##ᴰ ##ᴵ ##ᴺ ##ᵀ ##ᵃ ##ᵇ ##ᵈ ##ᵉ ##ᵍ ##ᵏ ##ᵐ ##ᵒ ##ᵖ ##ᵗ ##ᵘ ##ᵣ ##ᵤ ##ᵥ ##ᶜ ##ᶠ ##‐ ##‑ ##‒ ##– ##— ##― ##‖ ##‘ ##’ ##‚ ##“ ##” ##„ ##† ##‡ ##• ##… ##‰ ##′ ##″ ##› ##‿ ##⁄ ##⁰ ##ⁱ ##⁴ ##⁵ ##⁶ ##⁷ ##⁸ ##⁹ ##⁻ ##ⁿ ##₅ ##₆ ##₇ ##₈ ##₉ ##₊ ##₍ ##₎ ##ₐ ##ₑ ##ₒ ##ₓ ##ₕ ##ₖ ##ₗ ##ₘ ##ₚ ##ₛ ##ₜ ##₤ ##₩ ##€ ##₱ ##₹ ##ℓ ##№ ##ℝ ##™ ##⅓ ##⅔ ##← ##↑ ##→ ##↓ ##↔ ##↦ ##⇄ ##⇌ ##⇒ ##∂ ##∅ ##∆ ##∇ ##∈ ##∗ ##∘ ##√ ##∞ ##∧ ##∨ ##∩ ##∪ ##≈ ##≡ ##≤ ##≥ ##⊂ ##⊆ ##⊕ ##⊗ ##⋅ ##─ ##│ ##■ ##▪ ##● ##★ ##☆ ##☉ ##♠ ##♣ ##♥ ##♦ ##♯ ##⟨ ##⟩ ##ⱼ ##⺩ ##⺼ ##⽥ ##、 ##。 ##〈 ##〉 ##《 ##》 ##「 ##」 ##『 ##』 ##〜 ##あ ##い ##う ##え ##お ##か ##き ##く ##け ##こ ##さ ##し ##す ##せ ##そ ##た ##ち ##っ ##つ ##て ##と ##な ##に ##ぬ ##ね ##の ##は ##ひ ##ふ ##へ ##ほ ##ま ##み ##む ##め ##も ##や ##ゆ ##よ ##ら ##り ##る ##れ ##ろ ##を ##ん ##ァ ##ア ##ィ ##イ ##ウ ##ェ ##エ ##オ ##カ ##キ ##ク ##ケ ##コ ##サ ##シ ##ス ##セ ##タ ##チ ##ッ ##ツ ##テ ##ト ##ナ ##ニ ##ノ ##ハ ##ヒ ##フ ##ヘ ##ホ ##マ ##ミ ##ム ##メ ##モ ##ャ ##ュ ##ョ ##ラ ##リ ##ル ##レ ##ロ ##ワ ##ン ##・ ##ー ##一 ##三 ##上 ##下 ##不 ##世 ##中 ##主 ##久 ##之 ##也 ##事 ##二 ##五 ##井 ##京 ##人 ##亻 ##仁 ##介 ##代 ##仮 ##伊 ##会 ##佐 ##侍 ##保 ##信 ##健 ##元 ##光 ##八 ##公 ##内 ##出 ##分 ##前 ##劉 ##力 ##加 ##勝 ##北 ##区 ##十 ##千 ##南 ##博 ##原 ##口 ##古 ##史 ##司 ##合 ##吉 ##同 ##名 ##和 ##囗 ##四 ##国 ##國 ##土 ##地 ##坂 ##城 ##堂 ##場 ##士 ##夏 ##外 ##大 ##天 ##太 ##夫 ##奈 ##女 ##子 ##学 ##宀 ##宇 ##安 ##宗 ##定 ##宣 ##宮 ##家 ##宿 ##寺 ##將 ##小 ##尚 ##山 ##岡 ##島 ##崎 ##川 ##州 ##巿 ##帝 ##平 ##年 ##幸 ##广 ##弘 ##張 ##彳 ##後 ##御 ##德 ##心 ##忄 ##志 ##忠 ##愛 ##成 ##我 ##戦 ##戸 ##手 ##扌 ##政 ##文 ##新 ##方 ##日 ##明 ##星 ##春 ##昭 ##智 ##曲 ##書 ##月 ##有 ##朝 ##木 ##本 ##李 ##村 ##東 ##松 ##林 ##森 ##楊 ##樹 ##橋 ##歌 ##止 ##正 ##武 ##比 ##氏 ##民 ##水 ##氵 ##氷 ##永 ##江 ##沢 ##河 ##治 ##法 ##海 ##清 ##漢 ##瀬 ##火 ##版 ##犬 ##王 ##生 ##田 ##男 ##疒 ##発 ##白 ##的 ##皇 ##目 ##相 ##省 ##真 ##石 ##示 ##社 ##神 ##福 ##禾 ##秀 ##秋 ##空 ##立 ##章 ##竹 ##糹 ##美 ##義 ##耳 ##良 ##艹 ##花 ##英 ##華 ##葉 ##藤 ##行 ##街 ##西 ##見 ##訁 ##語 ##谷 ##貝 ##貴 ##車 ##軍 ##辶 ##道 ##郎 ##郡 ##部 ##都 ##里 ##野 ##金 ##鈴 ##镇 ##長 ##門 ##間 ##阝 ##阿 ##陳 ##陽 ##雄 ##青 ##面 ##風 ##食 ##香 ##馬 ##高 ##龍 ##龸 ##fi ##fl ##！ ##（ ##） ##， ##－ ##． ##／ ##： ##？ ##～ ================================================ FILE: transtab/trainer.py ================================================ import os import pdb import math import time import json import torch from torch import nn from torch.utils.data import Dataset, DataLoader import numpy as np import pandas as pd from tqdm.autonotebook import trange from transtab import constants from transtab.evaluator import predict, get_eval_metric_fn, EarlyStopping from transtab.modeling_transtab import TransTabFeatureExtractor from transtab.trainer_utils import SupervisedTrainCollator, TrainDataset from transtab.trainer_utils import get_parameter_names from transtab.trainer_utils import get_scheduler import logging logger = logging.getLogger(__name__) class Trainer: def __init__(self, model, train_set_list, test_set_list=None, collate_fn=None, output_dir='./ckpt', num_epoch=10, batch_size=64, lr=1e-4, weight_decay=0, patience=5, eval_batch_size=256, warmup_ratio=None, warmup_steps=None, balance_sample=False, load_best_at_last=True, ignore_duplicate_cols=False, eval_metric='auc', eval_less_is_better=False, num_workers=0, **kwargs, ): '''args: train_set_list: a list of training sets [(x_1,y_1),(x_2,y_2),...] test_set_list: a list of tuples of test set (x, y), same as train_set_list. if set None, do not do evaluation and early stopping patience: the max number of early stop patience num_workers: how many workers used to process dataloader. recommend to be 0 if training data smaller than 10000. eval_less_is_better: if the set eval_metric is the less the better. For val_loss, it should be set True. ''' self.model = model if isinstance(train_set_list, tuple): train_set_list = [train_set_list] if isinstance(test_set_list, tuple): test_set_list = [test_set_list] self.train_set_list = train_set_list self.test_set_list = test_set_list self.collate_fn = collate_fn if collate_fn is None: self.collate_fn = SupervisedTrainCollator( categorical_columns=model.categorical_columns, numerical_columns=model.numerical_columns, binary_columns=model.binary_columns, ignore_duplicate_cols=ignore_duplicate_cols, ) self.trainloader_list = [ self._build_dataloader(trainset, batch_size, collator=self.collate_fn, num_workers=num_workers) for trainset in train_set_list ] if test_set_list is not None: self.testloader_list = [ self._build_dataloader(testset, eval_batch_size, collator=self.collate_fn, num_workers=num_workers, shuffle=False) for testset in test_set_list ] else: self.testloader_list = None self.test_set_list = test_set_list self.output_dir = output_dir self.early_stopping = EarlyStopping(output_dir=output_dir, patience=patience, verbose=False, less_is_better=eval_less_is_better) self.args = { 'lr':lr, 'weight_decay':weight_decay, 'batch_size':batch_size, 'num_epoch':num_epoch, 'eval_batch_size':eval_batch_size, 'warmup_ratio': warmup_ratio, 'warmup_steps': warmup_steps, 'num_training_steps': self.get_num_train_steps(train_set_list, num_epoch, batch_size), 'eval_metric': get_eval_metric_fn(eval_metric), 'eval_metric_name': eval_metric, } self.args['steps_per_epoch'] = int(self.args['num_training_steps'] / (num_epoch*len(self.train_set_list))) if not os.path.exists(output_dir): os.makedirs(output_dir) self.optimizer = None self.lr_scheduler = None self.balance_sample = balance_sample self.load_best_at_last = load_best_at_last def train(self): args = self.args self.create_optimizer() if args['warmup_ratio'] is not None or args['warmup_steps'] is not None: num_train_steps = args['num_training_steps'] logger.info(f'set warmup training in initial {num_train_steps} steps') self.create_scheduler(num_train_steps, self.optimizer) start_time = time.time() for epoch in trange(args['num_epoch'], desc='Epoch'): ite = 0 train_loss_all = 0 for dataindex in range(len(self.trainloader_list)): for data in self.trainloader_list[dataindex]: self.optimizer.zero_grad() logits, loss = self.model(data[0], data[1]) loss.backward() self.optimizer.step() train_loss_all += loss.item() ite += 1 if self.lr_scheduler is not None: self.lr_scheduler.step() if self.test_set_list is not None: eval_res_list = self.evaluate() eval_res = np.mean(eval_res_list) print('epoch: {}, test {}: {:.6f}'.format(epoch, self.args['eval_metric_name'], eval_res)) self.early_stopping(-eval_res, self.model) if self.early_stopping.early_stop: print('early stopped') break print('epoch: {}, train loss: {:.4f}, lr: {:.6f}, spent: {:.1f} secs'.format(epoch, train_loss_all, self.optimizer.param_groups[0]['lr'], time.time()-start_time)) if os.path.exists(self.output_dir): if self.test_set_list is not None: # load checkpoints logger.info(f'load best at last from {self.output_dir}') state_dict = torch.load(os.path.join(self.output_dir, constants.WEIGHTS_NAME), map_location='cpu') self.model.load_state_dict(state_dict) self.save_model(self.output_dir) logger.info('training complete, cost {:.1f} secs.'.format(time.time()-start_time)) def evaluate(self): # evaluate in each epoch self.model.eval() eval_res_list = [] for dataindex in range(len(self.testloader_list)): y_test, pred_list, loss_list = [], [], [] for data in self.testloader_list[dataindex]: if data[1] is not None: label = data[1] if isinstance(label, pd.Series): label = label.values y_test.append(label) with torch.no_grad(): logits, loss = self.model(data[0], data[1]) if loss is not None: loss_list.append(loss.item()) if logits is not None: if logits.shape[-1] == 1: # binary classification pred_list.append(logits.sigmoid().detach().cpu().numpy()) else: # multi-class classification pred_list.append(torch.softmax(logits,-1).detach().cpu().numpy()) if len(pred_list)>0: pred_all = np.concatenate(pred_list, 0) if logits.shape[-1] == 1: pred_all = pred_all.flatten() if self.args['eval_metric_name'] == 'val_loss': eval_res = np.mean(loss_list) else: y_test = np.concatenate(y_test, 0) eval_res = self.args['eval_metric'](y_test, pred_all) eval_res_list.append(eval_res) return eval_res_list def train_no_dataloader(self, resume_from_checkpoint = None, ): resume_from_checkpoint = None if not resume_from_checkpoint else resume_from_checkpoint args = self.args self.create_optimizer() if args['warmup_ratio'] is not None or args['warmup_steps'] is not None: print('set warmup training.') self.create_scheduler(args['num_training_steps'], self.optimizer) for epoch in range(args['num_epoch']): ite = 0 # go through all train sets for train_set in self.train_set_list: x_train, y_train = train_set train_loss_all = 0 for i in range(0, len(x_train), args['batch_size']): self.model.train() if self.balance_sample: bs_x_train_pos = x_train.loc[y_train==1].sample(int(args['batch_size']/2)) bs_y_train_pos = y_train.loc[bs_x_train_pos.index] bs_x_train_neg = x_train.loc[y_train==0].sample(int(args['batch_size']/2)) bs_y_train_neg = y_train.loc[bs_x_train_neg.index] bs_x_train = pd.concat([bs_x_train_pos, bs_x_train_neg], axis=0) bs_y_train = pd.concat([bs_y_train_pos, bs_y_train_neg], axis=0) else: bs_x_train = x_train.iloc[i:i+args['batch_size']] bs_y_train = y_train.loc[bs_x_train.index] self.optimizer.zero_grad() logits, loss = self.model(bs_x_train, bs_y_train) loss.backward() self.optimizer.step() train_loss_all += loss.item() ite += 1 if self.lr_scheduler is not None: self.lr_scheduler.step() if self.test_set is not None: # evaluate in each epoch self.model.eval() x_test, y_test = self.test_set pred_all = predict(self.model, x_test, self.args['eval_batch_size']) eval_res = self.args['eval_metric'](y_test, pred_all) print('epoch: {}, test {}: {}'.format(epoch, self.args['eval_metric_name'], eval_res)) self.early_stopping(-eval_res, self.model) if self.early_stopping.early_stop: print('early stopped') break print('epoch: {}, train loss: {}, lr: {:.6f}'.format(epoch, train_loss_all, self.optimizer.param_groups[0]['lr'])) if os.path.exists(self.output_dir): if self.test_set is not None: # load checkpoints print('load best at last from', self.output_dir) state_dict = torch.load(os.path.join(self.output_dir, constants.WEIGHTS_NAME), map_location='cpu') self.model.load_state_dict(state_dict) self.save_model(self.output_dir) def save_model(self, output_dir=None): if output_dir is None: print('no path assigned for save mode, default saved to ./ckpt/model.pt !') output_dir = self.output_dir if not os.path.exists(output_dir): os.makedirs(output_dir, exist_ok=True) logger.info(f'saving model checkpoint to {output_dir}') self.model.save(output_dir) self.collate_fn.save(output_dir) if self.optimizer is not None: torch.save(self.optimizer.state_dict(), os.path.join(output_dir, constants.OPTIMIZER_NAME)) if self.lr_scheduler is not None: torch.save(self.lr_scheduler.state_dict(), os.path.join(output_dir, constants.SCHEDULER_NAME)) if self.args is not None: train_args = {} for k,v in self.args.items(): if isinstance(v, int) or isinstance(v, str) or isinstance(v, float): train_args[k] = v with open(os.path.join(output_dir, constants.TRAINING_ARGS_NAME), 'w', encoding='utf-8') as f: f.write(json.dumps(train_args)) def create_optimizer(self): if self.optimizer is None: decay_parameters = get_parameter_names(self.model, [nn.LayerNorm]) decay_parameters = [name for name in decay_parameters if "bias" not in name] optimizer_grouped_parameters = [ { "params": [p for n, p in self.model.named_parameters() if n in decay_parameters], "weight_decay": self.args['weight_decay'], }, { "params": [p for n, p in self.model.named_parameters() if n not in decay_parameters], "weight_decay": 0.0, }, ] self.optimizer = torch.optim.Adam(optimizer_grouped_parameters, lr=self.args['lr']) def create_scheduler(self, num_training_steps, optimizer): self.lr_scheduler = get_scheduler( 'cosine', optimizer = optimizer, num_warmup_steps=self.get_warmup_steps(num_training_steps), num_training_steps=num_training_steps, ) return self.lr_scheduler def get_num_train_steps(self, train_set_list, num_epoch, batch_size): total_step = 0 for trainset in train_set_list: x_train, _ = trainset total_step += np.ceil(len(x_train) / batch_size) total_step *= num_epoch return total_step def get_warmup_steps(self, num_training_steps): """ Get number of steps used for a linear warmup. """ warmup_steps = ( self.args['warmup_steps'] if self.args['warmup_steps'] is not None else math.ceil(num_training_steps * self.args['warmup_ratio']) ) return warmup_steps def _build_dataloader(self, trainset, batch_size, collator, num_workers=8, shuffle=True): trainloader = DataLoader( TrainDataset(trainset), collate_fn=collator, batch_size=batch_size, shuffle=shuffle, num_workers=num_workers, pin_memory=True, drop_last=False, ) return trainloader ================================================ FILE: transtab/trainer_utils.py ================================================ import pdb import os import random import math import numpy as np import pandas as pd import torch from torch.utils.data import Dataset, DataLoader from transformers.optimization import ( get_linear_schedule_with_warmup, get_cosine_schedule_with_warmup, get_cosine_with_hard_restarts_schedule_with_warmup, get_polynomial_decay_schedule_with_warmup, get_constant_schedule, get_constant_schedule_with_warmup ) from transtab.modeling_transtab import TransTabFeatureExtractor TYPE_TO_SCHEDULER_FUNCTION = { 'linear': get_linear_schedule_with_warmup, 'cosine': get_cosine_schedule_with_warmup, 'cosine_with_restarts': get_cosine_with_hard_restarts_schedule_with_warmup, 'polynomial': get_polynomial_decay_schedule_with_warmup, 'constant': get_constant_schedule, 'constant_with_warmup': get_constant_schedule_with_warmup, } class TrainDataset(Dataset): def __init__(self, trainset): self.x, self.y = trainset def __len__(self): return len(self.x) def __getitem__(self, index): x = self.x.iloc[index-1:index] if self.y is not None: y = self.y.iloc[index-1:index] else: y = None return x, y class TrainCollator: '''A base class for all collate function used for TransTab training. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, ignore_duplicate_cols=False, **kwargs, ): self.feature_extractor=TransTabFeatureExtractor( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, disable_tokenizer_parallel=True, ignore_duplicate_cols=ignore_duplicate_cols, ) def save(self, path): self.feature_extractor.save(path) def __call__(self, data): raise NotImplementedError class SupervisedTrainCollator(TrainCollator): def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, ignore_duplicate_cols=False, **kwargs, ): super().__init__( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, ignore_duplicate_cols=ignore_duplicate_cols, ) def __call__(self, data): x = pd.concat([row[0] for row in data]) y = pd.concat([row[1] for row in data]) inputs = self.feature_extractor(x) return inputs, y class TransTabCollatorForCL(TrainCollator): '''support positive pair sampling for contrastive learning of transtab model. ''' def __init__(self, categorical_columns=None, numerical_columns=None, binary_columns=None, overlap_ratio=0.5, num_partition=3, ignore_duplicate_cols=False, **kwargs) -> None: super().__init__( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, ignore_duplicate_cols=ignore_duplicate_cols, ) assert num_partition > 0, f'number of contrastive subsets must be greater than 0, got {num_partition}' assert isinstance(num_partition,int), f'number of constrative subsets must be int, got {type(num_partition)}' assert overlap_ratio >= 0 and overlap_ratio < 1, f'overlap_ratio must be in [0, 1), got {overlap_ratio}' self.overlap_ratio=overlap_ratio self.num_partition=num_partition def __call__(self, data): ''' Take a list of subsets (views) from the original tests. ''' # 1. build positive pairs # 2. encode each pair using feature extractor df_x = pd.concat([row[0] for row in data]) df_y = pd.concat([row[1] for row in data]) if self.num_partition > 1: sub_x_list = self._build_positive_pairs(df_x, self.num_partition) else: sub_x_list = self._build_positive_pairs_single_view(df_x) input_x_list = [] for sub_x in sub_x_list: inputs = self.feature_extractor(sub_x) input_x_list.append(inputs) res = {'input_sub_x':input_x_list} return res, df_y def _build_positive_pairs(self, x, n): '''build multi-view of each sample by spliting columns ''' x_cols = x.columns.tolist() sub_col_list = np.array_split(np.array(x_cols), n) len_cols = len(sub_col_list[0]) overlap = int(math.ceil(len_cols * (self.overlap_ratio))) sub_x_list = [] for i, sub_col in enumerate(sub_col_list): if overlap > 0 and i < n-1: sub_col = np.concatenate([sub_col, sub_col_list[i+1][:overlap]]) elif overlap >0 and i == n-1: sub_col = np.concatenate([sub_col, sub_col_list[i-1][-overlap:]]) # np.random.shuffle(sub_col) sub_x = x.copy()[sub_col] sub_x_list.append(sub_x) return sub_x_list def _build_positive_pairs_single_view(self, x): x_cols = x.columns.tolist() sub_x_list = [x] n_corrupt = int(len(x_cols)*0.5) corrupt_cols = x_cols[:n_corrupt] x_corrupt = x.copy()[corrupt_cols] np.random.shuffle(x_corrupt.values) sub_x_list.append(pd.concat([x.copy().drop(corrupt_cols,axis=1), x_corrupt], axis=1)) return sub_x_list def get_parameter_names(model, forbidden_layer_types): """ Returns the names of the model parameters that are not inside a forbidden layer. """ result = [] for name, child in model.named_children(): result += [ f"{name}.{n}" for n in get_parameter_names(child, forbidden_layer_types) if not isinstance(child, tuple(forbidden_layer_types)) ] # Add model specific parameters (defined with nn.Parameter) since they are not in any child. result += list(model._parameters.keys()) return result def random_seed(seed): os.environ['PYTHONHASHSEED'] = str(seed) random.seed(seed) np.random.seed(seed) torch.manual_seed(seed) def get_scheduler( name, optimizer, num_warmup_steps = None, num_training_steps = None, ): ''' Unified API to get any scheduler from its name. Parameters ---------- name: str The name of the scheduler to use. optimizer: torch.optim.Optimizer The optimizer that will be used during training. num_warmup_steps: int The number of warmup steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. num_training_steps: int The number of training steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. ''' name = name.lower() schedule_func = TYPE_TO_SCHEDULER_FUNCTION[name] if name == 'constant': return schedule_func(optimizer) if num_warmup_steps is None: raise ValueError(f"{name} requires `num_warmup_steps`, please provide that argument.") if name == 'constant_with_warmup': return schedule_func(optimizer, num_warmup_steps=num_warmup_steps) if num_training_steps is None: raise ValueError(f"{name} requires `num_training_steps`, please provide that argument.") return schedule_func(optimizer, num_warmup_steps=num_warmup_steps, num_training_steps=num_training_steps) ================================================ FILE: transtab/transtab.py ================================================ import pdb import os from transtab import constants from transtab.modeling_transtab import TransTabClassifier, TransTabRegressor, TransTabFeatureExtractor, TransTabFeatureProcessor from transtab.modeling_transtab import TransTabForCL from transtab.modeling_transtab import TransTabInputEncoder, TransTabModel from transtab.dataset import load_data from transtab.evaluator import predict, evaluate from transtab.trainer import Trainer from transtab.trainer_utils import TransTabCollatorForCL from transtab.trainer_utils import random_seed def build_classifier( categorical_columns=None, numerical_columns=None, binary_columns=None, feature_extractor=None, num_class=2, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, activation='relu', device='cuda:0', checkpoint=None, **kwargs) -> TransTabClassifier: '''Build a :class:`transtab.modeling_transtab.TransTabClassifier`. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. num_class: int number of output classes to be predicted. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. checkpoint: str the directory to load the pretrained TransTab model. Returns ------- A TransTabClassifier model. ''' model = TransTabClassifier( categorical_columns = categorical_columns, numerical_columns = numerical_columns, binary_columns = binary_columns, feature_extractor = feature_extractor, num_class=num_class, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, device=device, **kwargs, ) if checkpoint is not None: model.load(checkpoint) return model def build_regressor( categorical_columns=None, numerical_columns=None, binary_columns=None, feature_extractor=None, num_class=1, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, activation='relu', device='cuda:0', checkpoint=None, **kwargs) -> TransTabRegressor: '''Build a :class:`transtab.modeling_transtab.TransTabRegressor`. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. num_class: int number of output classes to be predicted. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. checkpoint: str the directory to load the pretrained TransTab model. Returns ------- A TransTabClassifier model. ''' model = TransTabRegressor( categorical_columns = categorical_columns, numerical_columns = numerical_columns, binary_columns = binary_columns, feature_extractor = feature_extractor, num_class=num_class, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, device=device, **kwargs, ) if checkpoint is not None: model.load(checkpoint) return model def build_extractor( categorical_columns=None, numerical_columns=None, binary_columns=None, ignore_duplicate_cols=False, disable_tokenizer_parallel=False, checkpoint=None, **kwargs,) -> TransTabFeatureExtractor: '''Build a feature extractor for TransTab model. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). ignore_duplicate_cols: bool if there is one column assigned to more than one type, e.g., the feature age is both nominated as categorical and binary columns, the model will raise errors. set True to avoid this error as the model will ignore this duplicate feature. disable_tokenizer_parallel: bool if the returned feature extractor is leveraged by the collate function for a dataloader, try to set this False in case the dataloader raises errors because the dataloader builds multiple workers and the tokenizer builds multiple workers at the same time. checkpoint: str the directory of the predefined TransTabFeatureExtractor. Returns ------- A TransTabFeatureExtractor module. ''' feature_extractor = TransTabFeatureExtractor( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, disable_tokenizer_parallel=disable_tokenizer_parallel, ignore_duplicate_cols=ignore_duplicate_cols, ) if checkpoint is not None: extractor_path = os.path.join(checkpoint, constants.EXTRACTOR_STATE_DIR) if os.path.exists(extractor_path): feature_extractor.load(extractor_path) else: feature_extractor.load(checkpoint) return feature_extractor def build_encoder( categorical_columns=None, numerical_columns=None, binary_columns=None, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, activation='relu', device='cuda:0', checkpoint=None, **kwargs, ): ''' Build a feature encoder that maps inputs tabular samples to embeddings. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. If set zero, only use the embedding layer to get token-level embeddings. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. Ignored if `num_layer=0` is zero. hidden_dropout_prob: float the dropout ratio in the transformer encoder. Ignored if `num_layer=0` is zero. ffn_dim: int the dimension of feed-forward layer in the transformer layer. Ignored if `num_layer=0` is zero. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. Ignored if `num_layer=0` is zero. device: str the device, ``"cpu"`` or ``"cuda:0"``. checkpoint: str the directory to load the pretrained TransTab model. ''' if num_layer == 0: feature_extractor = TransTabFeatureExtractor( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, ) feature_processor = TransTabFeatureProcessor( vocab_size=feature_extractor.vocab_size, pad_token_id=feature_extractor.pad_token_id, hidden_dim=hidden_dim, hidden_dropout_prob=hidden_dropout_prob, device=device, ) enc = TransTabInputEncoder(feature_extractor, feature_processor) enc.load(checkpoint) else: enc = TransTabModel( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, ffn_dim=ffn_dim, activation=activation, device=device, ) if checkpoint is not None: enc.load(checkpoint) return enc def build_contrastive_learner( categorical_columns=None, numerical_columns=None, binary_columns=None, projection_dim=128, num_partition=3, overlap_ratio=0.5, supervised=True, hidden_dim=128, num_layer=2, num_attention_head=8, hidden_dropout_prob=0, ffn_dim=256, activation='relu', device='cuda:0', checkpoint=None, ignore_duplicate_cols=True, **kwargs, ): '''Build a contrastive learner for pretraining based on TransTab. If no cat/num/bin specified, the model takes ALL as categorical columns, which may undermine the performance significantly. If there is one column assigned to more than one type, e.g., the feature age is both nominated as categorical and binary columns, the model will raise errors. set ``ignore_duplicate_cols=True`` to avoid this error as the model will ignore this duplicate feature. Parameters ---------- categorical_columns: list a list of categorical feature names. numerical_columns: list a list of numerical feature names. binary_columns: list a list of binary feature names, accept binary indicators like (yes,no); (true,false); (0,1). feature_extractor: TransTabFeatureExtractor a feature extractor to tokenize the input tables. if not passed the model will build itself. hidden_dim: int the dimension of hidden embeddings. num_layer: int the number of transformer layers used in the encoder. num_attention_head: int the numebr of heads of multihead self-attention layer in the transformers. hidden_dropout_prob: float the dropout ratio in the transformer encoder. ffn_dim: int the dimension of feed-forward layer in the transformer layer. projection_dim: int the dimension of projection head on the top of encoder. overlap_ratio: float the overlap ratio of columns of different partitions when doing subsetting. num_partition: int the number of partitions made for vertical-partition contrastive learning. supervised: bool whether or not to take supervised VPCL, otherwise take self-supervised VPCL. temperature: float temperature used to compute logits for contrastive learning. base_temperature: float base temperature used to normalize the temperature. activation: str the name of used activation functions, support ``"relu"``, ``"gelu"``, ``"selu"``, ``"leakyrelu"``. device: str the device, ``"cpu"`` or ``"cuda:0"``. checkpoint: str the directory of the pretrained transtab model. ignore_duplicate_cols: bool if there is one column assigned to more than one type, e.g., the feature age is both nominated as categorical and binary columns, the model will raise errors. set True to avoid this error as the model will ignore this duplicate feature. Returns ------- A TransTabForCL model. ''' model = TransTabForCL( categorical_columns = categorical_columns, numerical_columns = numerical_columns, binary_columns = binary_columns, num_partition= num_partition, hidden_dim=hidden_dim, num_layer=num_layer, num_attention_head=num_attention_head, hidden_dropout_prob=hidden_dropout_prob, supervised=supervised, ffn_dim=ffn_dim, projection_dim=projection_dim, overlap_ratio=overlap_ratio, activation=activation, device=device, ) if checkpoint is not None: model.load(checkpoint) # build collate function for contrastive learning collate_fn = TransTabCollatorForCL( categorical_columns=categorical_columns, numerical_columns=numerical_columns, binary_columns=binary_columns, overlap_ratio=overlap_ratio, num_partition=num_partition, ignore_duplicate_cols=ignore_duplicate_cols ) if checkpoint is not None: collate_fn.feature_extractor.load(os.path.join(checkpoint, constants.EXTRACTOR_STATE_DIR)) return model, collate_fn def train(model, trainset, valset=None, num_epoch=10, batch_size=64, eval_batch_size=256, lr=1e-4, weight_decay=0, patience=5, warmup_ratio=None, warmup_steps=None, eval_metric='auc', output_dir='./ckpt', collate_fn=None, num_workers=0, balance_sample=False, load_best_at_last=True, ignore_duplicate_cols=False, eval_less_is_better=False, **kwargs, ): ''' The shared train function for all TransTabModel based models. Parameters ---------- model: TransTabModel and its subclass A subclass of the base model. Should be able to output logits and loss in forward, e.g., ``logit, loss = model(x, y)``. trainset: list or tuple a list of trainsets, or a single trainset consisting of (x, y). x: pd.DataFrame or dict, y: pd.Series. valset: list or tuple a list of valsets, or a single valset of consisting of (x, y). num_epoch: int number of training epochs. batch_size: int training batch size. eval_batch_size: int evaluation batch size. lr: float training learning rate. weight_decay: float training weight decay. patience: int early stopping patience, only valid when ``valset`` is given. warmup_ratio: float the portion of training steps for learning rate warmup, if `warmup_steps` is set, it will be ignored. warmup_steps: int the number of training steps for learning rate warmup. eval_metric: str the evaluation metric during training for early stopping, can be ``"acc"``, ``"auc"``, ``"mse"``, ``"val_loss"``. output_dir: str the output training model weights and feature extractor configurations. collate_fn: function specify training collate function if it is not standard supervised learning, e.g., contrastive learning. num_workers: int the number of workers for the dataloader. balance_sample: bool balance_sample: whether or not do bootstrapping to maintain in batch samples are in balanced classes, only support binary classification. load_best_at_last: bool whether or not load the best checkpoint after the training completes. ignore_duplicate_cols: bool whether or not ignore the contradictory of cat/num/bin cols eval_less_is_better: bool if the set eval_metric is the less the better. For val_loss, it should be set True. Returns ------- None ''' if isinstance(trainset, tuple): trainset = [trainset] train_args = { 'num_epoch': num_epoch, 'batch_size': batch_size, 'eval_batch_size': eval_batch_size, 'lr': lr, 'weight_decay':weight_decay, 'patience':patience, 'warmup_ratio':warmup_ratio, 'warmup_steps':warmup_steps, 'eval_metric':eval_metric, 'output_dir':output_dir, 'collate_fn':collate_fn, 'num_workers':num_workers, 'balance_sample':balance_sample, 'load_best_at_last':load_best_at_last, 'ignore_duplicate_cols':ignore_duplicate_cols, 'eval_less_is_better':eval_less_is_better, } trainer = Trainer( model, trainset, valset, **train_args, ) trainer.train()