\n \n \n \n
\n
\n\n\n\nWe can examine the associated metadata for the same year.\n\n.. code:: python3\n\n metadata[5]\n\n\n\n\n.. raw:: html\n\n | \n | 104-1st-1 | \n104-1st-10 | \n104-1st-100 | \n104-1st-101 | \n104-1st-102 | \n104-1st-103 | \n104-1st-104 | \n104-1st-105 | \n104-1st-106 | \n104-1st-107 | \n... | \n104-1st-90 | \n104-1st-91 | \n104-1st-92 | \n104-1st-93 | \n104-1st-94 | \n104-1st-95 | \n104-1st-96 | \n104-1st-97 | \n104-1st-98 | \n104-1st-99 | \n
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| legislator | \n\n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n | \n |
| Abercrombie | \n0.0 | \n1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n... | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n
| Ackerman | \n0.0 | \n1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n... | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n
| Allard | \n0.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n... | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n1.0 | \n0.0 | \n-1.0 | \n
| Andrews | \n0.0 | \n1.0 | \n0.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n0.0 | \n0.0 | \n0.0 | \n... | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n
| Archer | \n0.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n... | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n0.0 | \n
| ... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n... | \n
| Young (AK) | \n0.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n... | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n
| Young (FL) | \n0.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n... | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n
| Zeliff | \n0.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n... | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n
| Zimmer | \n0.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n-1.0 | \n... | \n-1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n-1.0 | \n
| de la Garza | \n0.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n... | \n0.0 | \n-1.0 | \n-1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n1.0 | \n-1.0 | \n1.0 | \n
438 rows × 885 columns
\n\n \n \n \n
\n
\n\n\n\nYou may note that sometimes representatives names list a state in\nparenthesis afterwards. This is to provide disambiguation for\nrepresentatives that happen to have the last name. This actually\ncomplicates matters for us since the disambiguation is only applied in\nthose cases where there is a name collision in that sitting of congress.\nThat means that for several years a representative may have simply their\nlast name, but then switch to being disambiguated, before potentially\nswitching back again. This would make it much harder to consistently\ntreck representatives over their entire career in congress. To fix this\nup we’ll simply re-index by a unique representative ID that has their\nlast name, party, and state all listed over all the voting dataframes.\nWe’ll need a function to generate those from the metadata, and then\nwe’ll need to apply it to all the records. Importantly we’ll have to\nfinesse those situations where representatives are listed twice (under\nun-ambiguous and disambiguated names) with some groupby tricks.\n\n.. code:: python3\n\n def unique_legislator(row):\n name, state, party = row.legislator, row.state, row.party\n # Strip of disambiguating state designators\n if re.search(r'(\\w+) \\([A-Z]{2}\\)', name) is not None:\n name = name[:-5]\n return f\"{name} ({party}, {state})\"\n\n.. code:: python3\n\n for i, _ in enumerate(votes):\n votes[i].index = pd.Index(metadata[i].apply(unique_legislator, axis=1), name=\"legislator_index\")\n votes[i] = votes[i].groupby(level=0).sum()\n metadata[i].index = pd.Index(metadata[i].apply(unique_legislator, axis=1), name=\"legislator_index\")\n metadata[i] = metadata[i].groupby(level=0).first()\n\nNow that we have the data at least a little wrangled into order there is\nthe question of ensuring some degree of continuity fore representatives.\nTo make this a little easier we’ll use voting records over *four year\nspans* instead of over single years. Equally importantly we’ll do this\nin a sliding window fashion so that we consider the record for 1990-1994\nand then the record for 1991-1995 and so on. By overlapping the windows\nin this way we can ensure a little greater continuity of political\nstance through the years. To make this happen we just have to merge data\nframes in a sliding set of pairs, and then merge the pairs via the same\napproach:\n\n.. code:: python3\n\n votes = [\n pd.merge(\n v1, v2, how=\"outer\", on=\"legislator_index\"\n ).fillna(0.0).sort_index()\n for v1, v2 in zip(votes[:-1], votes[1:])\n ] + votes[-1:]\n \n metadata = [\n pd.concat([m1, m2]).groupby(\"legislator_index\").first().sort_index()\n for m1, m2 in zip(metadata[:-1], metadata[1:])\n ] + metadata[-1:]\n\nThat’s the pairs of years; now we merge these pairwise to get sets of\nfour years worth of votes.\n\n.. code:: python3\n\n votes = [\n pd.merge(\n v1, v2, how=\"outer\", on=\"legislator_index\"\n ).fillna(0.0).sort_index()\n for v1, v2 in zip(votes[:-1], votes[1:])\n ] + votes[-1:]\n \n metadata = [\n pd.concat([m1, m2]).groupby(level=0).first().sort_index()\n for m1, m2 in zip(metadata[:-1], metadata[1:])\n ] + metadata[-1:]\n\nApplying AlignedUMAP\n--------------------\n\nTo make use of AlignedUMAP we need to generate relations between\nconsecutive dataset slices. In this case that means we need to have a\nrelation describing row from one four year slice corresponds to a row\nfrom the following four year slice for the same representative. For\nAlignedUMAP to work this should be formatted as a list of dictionaries;\neach dictionary gives a mapping from indices of one slice to indices of\nthe next. Importantly this mapping can be partial – it only has to\nrelate indices for which there is a match between the two slices.\n\nThe vote dataframes that we are using for slices are already indexed\nwith unique identifiers for representatives, so to make relations we\nsimply have to match them up, creating a dictionary of indices from one\nto the other. In practice we can do this relatively efficiently by using\npandas to merge dataframes on the pandas indexes of the two vote\ndataframes with the data being simply the numeric indices of the rows.\nThe resulting dictionary is then just the dictionary of pairs given by\nthe inner join.\n\n.. code:: python3\n\n def make_relation(from_df, to_df):\n left = pd.DataFrame(data=np.arange(len(from_df)), index=from_df.index)\n right = pd.DataFrame(data=np.arange(len(to_df)), index=to_df.index)\n merge = pd.merge(left, right, left_index=True, right_index=True)\n return dict(merge.values)\n\nWith a function for relation creation in place we simply need to apply\nit to each consecutive pair of vote dataframes.\n\n.. code:: python3\n\n relations = [make_relation(x,y) for x, y in zip(votes[:-1], votes[1:])]\n\nIf you are still unsure of what these relations are it might be\nbeneficial to look at a few of the dictionaries, along with the\ncorresponding pairs of vote dataframes. Here is (part of) the first\nrelation dictionary:\n\n.. code:: python3\n\n relations[0]\n\n\n\n\n.. parsed-literal::\n\n {0: 0,\n 1: 1,\n 3: 2,\n 4: 3,\n 5: 4,\n 6: 5,\n 7: 6,\n 8: 7,\n 9: 8,\n 10: 9,\n 11: 10,\n 12: 11,\n 13: 12,\n 14: 13,\n 15: 14,\n ...\n 475: 547,\n 476: 549,\n 477: 550,\n 478: 552,\n 479: 553,\n 480: 554,\n 481: 555,\n 482: 556,\n 483: 557,\n 484: 559}\n\n\n\nNow we are finally in a position to run AlignedUMAP. Most of the\nstandard UMAP parameters are available for use, including choosing a\nmetric and a number of neighbors. Here we will also make use of the\nextra AlignedUMAP parameters ``alignment_regularisation`` and\n``alignment_window_size``. The first is a value that weights how\nimportant retaining alignment is. Typically the value is much smaller\nthan this (the default is 0.01), but given the relatively high\nvolatility in voting records we are going to increase it here. The\nsecond parameter, ``alignment_window_size`` determines how far out on\neither side AlignedUMAP will look when aligning embeddings – even though\nthe relations are specified only between consecutive slices it will\nchain them together to construct relations reaching further. In this\ncase we’ll have it look as far out as 5 slices either side.\n\n.. code:: python3\n\n %%time\n aligned_mapper = umap.aligned_umap.AlignedUMAP(\n metric=\"cosine\",\n n_neighbors=20,\n alignment_regularisation=0.1, \n alignment_window_size=5,\n n_epochs=200,\n random_state=42,\n ).fit(votes, relations=relations)\n embeddings = aligned_mapper.embeddings_\n\n\n.. parsed-literal::\n\n CPU times: user 6min 7s, sys: 30.6 s, total: 6min 37s\n Wall time: 5min 57s\n\n\nVisualizing the Results\n-----------------------\n\nNow we need to plot the data somehow. To make the visualization\ninteresting it would be beneficial to have some colour variation –\nideally showing a different view of the relative political stance. For\nthat we want to attempt to get an idea of the position of each candidate\nfrom an alternative source. To do this we can try to extract the vote\nmargin that the representative won by. The catch here is that while the\nelection data can be collected and processed, the names don’t match\nperfectly as they come from a different source. That means we need to do\nour best to get a name match for each candidate. We’ll use fuzzy string\nmatching restricted to the relevant year and state to try to get a good\nmatch. A notebook providing details for obtaining and processing the\nelection winners data can be found\n`here | \n | legislator | \nstate | \nparty | \n
|---|---|---|---|
| 0 | \nAbercrombie | \nHI | \nD | \n
| 1 | \nAckerman | \nNY | \nD | \n
| 2 | \nAllard | \nCO | \nR | \n
| 3 | \nAndrews | \nNJ | \nD | \n
| 4 | \nArcher | \nTX | \nR | \n
| ... | \n... | \n... | \n... | \n
| 430 | \nYoung (AK) | \nAK | \nR | \n
| 431 | \nYoung (FL) | \nFL | \nR | \n
| 432 | \nZeliff | \nNH | \nR | \n
| 433 | \nZimmer | \nNJ | \nR | \n
| 89 | \nde la Garza | \nTX | \nD | \n
438 rows × 3 columns
\n\n \n \n \n
\n
\n\n\n\nNow we need to simply go through the metadata and fill it out with the\nextra information we can glean from the election winners data. Since we\ncan’t do exact name matching (the data for both is somewhat messy when\nit comes to text fields like names) we can’t simply perform a join, but\nmust instead process things year by year and representative by\nrepresentative, finding the best string match on name that we can for\nthe given year and state election. In practice we are undoubtedly going\nto get some of these wrong, and if the goal was a rigorous analysis\nbased on this data a lot more care would need to be taken. Since this is\njust a demonstration and we’ll only be using this extra information as a\ncolour channel in plots we can excuise a few errors here and there from\nin-exact data processing.\n\n.. code:: python3\n\n n_name_misses = 0\n for year, df in enumerate(metadata, 1990):\n df[\"partisan_lean\"] = 0.5\n df[\"district\"] = np.full(len(df), -1, dtype=np.int8)\n for idx, (loc, row) in enumerate(df.iterrows()):\n name, state, party = row.legislator, row.state, row.party\n # Strip of disambiguating state designators\n if re.search(r'(\\w+) \\([A-Z]{2}\\)', name) is not None:\n name = name[:-5]\n # Get a party designator matching the election_winners data\n party = \"republican\" if party == \"R\" else \"democrat\"\n # Restrict to the right state and time-frame\n state_election_winners = election_winners[(election_winners.state == state) \n & (election_winners.year <= year + 4)\n & (election_winners.year >= year - 4)]\n # Try to match a name; and fail \"gracefully\"\n try:\n matched_name = process.extractOne(\n name, \n state_election_winners.winner.tolist(),\n scorer=fuzz.partial_token_sort_ratio,\n score_cutoff=50,\n )\n except:\n matched_name = None\n \n # If we got a unique match, get the election data\n if matched_name is not None:\n winner = state_election_winners[state_election_winners.winner == matched_name[0]]\n else:\n winner = []\n \n # We either have none, one, or *several* match elections. Take a best guess.\n if len(winner) < 1:\n df.loc[loc, [\"partisan_lean\"]] = 0.25 if party == \"republican\" else 0.75\n n_name_misses += 1\n elif len(winner) > 1:\n df.iloc[idx, 4] = int(winner.district.values[-1])\n df.iloc[idx, 3] = float(winner.winning_ratio.values[-1])\n else:\n df.iloc[idx, 4] = int(winner.district.values)\n df.iloc[idx, 3] = float(winner.winning_ratio.values[0])\n \n print(f\"Failed to match a name {n_name_misses} times\")\n\n\n.. parsed-literal::\n\n Failed to match a name 100 times\n\n\nNow that we have the relative partisan leanings based on district\nelection margins we can color the plot. We can obviously label the plot\nwith the representatives names. The last remaining catch (when using\nmatplotlib for the plotting) is the get the plot bounds (since we will\nbe placing text markers directly into the plot, and thus not\nautogenerating bounds). This is a simple enough matter of computing some\nbounds as an adjustment a little outside the data limits.\n\n.. code:: python3\n\n def axis_bounds(embedding):\n left = embedding.T[0].min()\n right = embedding.T[0].max()\n bottom = embedding.T[1].min()\n top = embedding.T[1].max()\n width = right - left\n height = top - bottom\n adj_h = width * 0.1\n adj_v = height * 0.05\n return [left - adj_h, right + adj_h, bottom - adj_v, top + adj_v]\n\nNow for the plot. Let’s pick a random time slice (you are welcome to try\nothers) and draw the representatives names in their embedded locations\nfor that slice, coloured by their relative election victory margin.\n\n.. code:: python3\n\n fig, ax = plt.subplots(figsize=(12,12))\n e = 5\n ax.axis(axis_bounds(embeddings[e]))\n ax.set_aspect('equal')\n for i in range(embeddings[e].shape[0]):\n ax.text(embeddings[e][i, 0],\n embeddings[e][i, 1],\n metadata[e].index.values[i],\n color=plt.cm.RdBu(np.float32(metadata[e][\"partisan_lean\"].values[i])), \n fontsize=8,\n horizontalalignment='center',\n verticalalignment='center',\n )\n\n\n\n\n.. image:: images/aligned_umap_politics_demo_31_0.png\n\n\nThis gives a good idea of the layout in a single time slices, and by\nplotting different time slices we can get some idea of how things have\nevolved. We can go further, however, by plotting a representative as\ncurve through time as their relative political position in congress\nchanges. For that we will need a 3D plot – we need both the UMAP x and y\ncoordinates, as well as a third coordinate giving the year. I found this\neasiest to do in plotly, so let’s import that. To make nice smooth\ncurves through time we will also import the ``scipy.interpolate`` module\nwhich will let is interpolate a smooth curve from the discrete positions\nthat a representatives appears in over time.\n\n.. code:: python3\n\n import plotly.graph_objects as go\n import scipy.interpolate\n\nWrangling the data into shape for this is the next step; first let’s get\neverything in a single dataframe that we can extract relevant data from\non an as-needed basis.\n\n.. code:: python3\n\n df = pd.DataFrame(np.vstack(embeddings), columns=('x', 'y'))\n df['z'] = np.concatenate([[year] * len(embeddings[i]) for i, year in enumerate(range(1990, 2021))])\n df['representative_id'] = np.concatenate([v.index for v in votes])\n df['partisan_lean'] = np.concatenate([m[\"partisan_lean\"].values for m in metadata])\n\nNext we’ll need that interpolation of the curve for a given\nrepresentative. We’ll write a function to handle that as there is a\nlittle bit of case-based logic that makes it non-trivial. We are going\nto get handed year data and want to interpolate the UMAP x and y\ncoordinates for a single representative.\n\nThe first major catch is that many representatives don’t have a single\ncontiguous block of years for which they were in congress: they were\nelected for several years, missed re-election, and then came back to\ncongress several years later (possibly in another district). Each such\nblock of contiguous years needs to be a separate path, and we shouldn’t\nconnect them. We therefore need some logic to find the contiguous blocks\nand generate smooth paths for each of them.\n\nAnother catch is that some representatives have only been in office for\na year or two (special elections and so forth) and we can’t do a cubic\nspline interpolation for that; we can devolve to linear interpolation or\nquadratic splines for those cases, so simply add the point itself for\nthe odd single year cases.\n\nWith those issues in hand we can then simply use the scipy ``interp1d``\nfunction to generate smooth curves through the points.\n\n.. code:: python3\n\n INTERP_KIND = {2:\"linear\", 3:\"quadratic\", 4:\"cubic\"}\n \n def interpolate_paths(z, x, y, c, rep_id):\n consecutive_year_blocks = np.where(np.diff(z) != 1)[0] + 1\n z_blocks = np.split(z, consecutive_year_blocks) \n x_blocks = np.split(x, consecutive_year_blocks)\n y_blocks = np.split(y, consecutive_year_blocks) \n c_blocks = np.split(c, consecutive_year_blocks)\n \n paths = []\n \n for block_idx, zs in enumerate(z_blocks):\n \n text = f\"{rep_id} -- partisan_lean: {np.mean(c_blocks[block_idx]):.2f}\"\n \n if len(zs) > 1:\n kind = INTERP_KIND.get(len(zs), \"cubic\")\n else:\n paths.append(\n (zs, x_blocks[block_idx], y_blocks[block_idx], c_blocks[block_idx], text)\n )\n continue\n \n z = np.linspace(np.min(zs), np.max(zs), 100)\n x = scipy.interpolate.interp1d(zs, x_blocks[block_idx], kind=kind)(z)\n y = scipy.interpolate.interp1d(zs, y_blocks[block_idx], kind=kind)(z)\n c = scipy.interpolate.interp1d(zs, c_blocks[block_idx], kind=\"linear\")(z)\n \n paths.append((z, x, y, c, text))\n \n return paths\n\nAnd now we can use plotly to draw the resulting curves. For plotly we\nuse the ``Scatter3D`` method, which supports a “lines” mode that can\ndraw curves in 3D space. We can colour the curves by the partisan lean\nscore we derived from the election data – in fact the colour can vary\nthrough the trace as the election margins vary. Since this is a plotly\nplot it is interactive, so you can rotate it around and view it from all\nangles.\n\nUnfortunately the interactive plotly plot does not embed into the documentation\nwell, so we present here a static image. If you run this yourself, however,\nyou will get the interactive version.\n\n.. code:: python3\n\n traces = []\n for rep in df.representative_id.unique():\n z = df.z[df.representative_id == rep].values\n x = df.x[df.representative_id == rep].values\n y = df.y[df.representative_id == rep].values\n c = df.partisan_lean[df.representative_id == rep]\n \n for z, x, y, c, text in interpolate_paths(z, x, y, c, rep):\n trace = go.Scatter3d(\n x=x, y=z, z=y, \n mode=\"lines\",\n hovertext=text,\n hoverinfo=\"text\",\n line=dict(\n color=c,\n cmin=0.0,\n cmid=0.5,\n cmax=1.0,\n cauto=False,\n colorscale=\"RdBu\",\n colorbar=dict(),\n width=2.5,\n ),\n opacity=1.0,\n )\n traces.append(trace)\n \n fig = go.Figure(data=traces)\n fig.update_layout(\n width=800,\n height=600,\n scene=dict(\n aspectratio = dict( x=0.5, y=1.25, z=0.5 ),\n yaxis_title=\"Year\",\n xaxis_title=\"UMAP-X\",\n zaxis_title=\"UMAP-Y\",\n ),\n scene_camera=dict(eye=dict( x=0.5, y=0.8, z=0.75 )),\n autosize=False,\n showlegend=False,\n )\n fig_widget = go.FigureWidget(fig)\n fig_widget\n\n\n.. image:: images/aligned_umap_politics_demo_spaghetti.png\n\n..\n .. raw:: html\n :file: aligned_umap_plotly_plot.html\n\nThis concludes our exploration for now.\n\n\n"
},
{
"path": "doc/api.rst",
"content": "UMAP API Guide\n==============\n\nUMAP has only two classes, :class:`UMAP`, and :class:`ParametricUMAP`, which inherits from it.\n\nUMAP\n----\n\n.. autoclass:: umap.umap_.UMAP\n :members:\n\nParametricUMAP\n----\n\n.. autoclass:: umap.parametric_umap.ParametricUMAP\n :members:\n\nA number of internal functions can also be accessed separately for more fine tuned work.\n\nUseful Functions\n----------------\n\n.. automodule:: umap.umap_\n :members:\n :exclude-members: UMAP\n\n"
},
{
"path": "doc/basic_usage.rst",
"content": "How to Use UMAP\n===============\n\nUMAP is a general purpose manifold learning and dimension reduction\nalgorithm. It is designed to be compatible with\n`scikit-learn | \n | year | \nstate | \ndistrict | \nwinner | \nparty | \nwinning_ratio | \n
|---|---|---|---|---|---|---|
| 0 | \n1976 | \nAK | \n0 | \nDon Young | \nrepublican | \n0.289986 | \n
| 0 | \n1976 | \nAL | \n1 | \nJack Edwards | \nrepublican | \n0.374808 | \n
| 0 | \n1976 | \nAL | \n2 | \nWilliam L. \\\\\"Bill\\\"\\\" Dickinson\" | \nrepublican | \n0.423953 | \n
| 0 | \n1976 | \nAL | \n3 | \nBill Nichols | \ndemocrat | \n1.000000 | \n
| 0 | \n1976 | \nAL | \n4 | \nTom Bevill | \ndemocrat | \n0.803825 | \n
\n \n \n \n
\n
\n\n\n\nSince this is for demonstration purposes we will get rid of the NAs in\nthe data; in a real world setting one would wish to take more care with\nproper handling of missing data.\n\n.. code:: python3\n\n penguins = penguins.dropna()\n penguins.species.value_counts()\n\n\n\n\n.. parsed-literal::\n\n Adelie 146\n Gentoo 119\n Chinstrap 68\n Name: species, dtype: int64\n\n\n.. image:: https://github.com/allisonhorst/palmerpenguins/blob/c19a904462482430170bfe2c718775ddb7dbb885/man/figures/culmen_depth.png?raw=true\n :width: 300px\n :align: center\n :alt: Diagram of culmen measurements on a penguin\n\nSee the `github repostiory | \n | species | \nisland | \nbill_length_mm | \nbill_depth_mm | \nflipper_length_mm | \nbody_mass_g | \nsex | \nyear | \n
|---|---|---|---|---|---|---|---|---|
| 0 | \nAdelie | \nTorgersen | \n39.1 | \n18.7 | \n181.0 | \n3750.0 | \nmale | \n2007 | \n
| 1 | \nAdelie | \nTorgersen | \n39.5 | \n17.4 | \n186.0 | \n3800.0 | \nfemale | \n2007 | \n
| 2 | \nAdelie | \nTorgersen | \n40.3 | \n18.0 | \n195.0 | \n3250.0 | \nfemale | \n2007 | \n
| 3 | \nAdelie | \nTorgersen | \nNaN | \nNaN | \nNaN | \nNaN | \nNaN | \n2007 | \n
| 4 | \nAdelie | \nTorgersen | \n36.7 | \n19.3 | \n193.0 | \n3450.0 | \nfemale | \n2007 | \n
\n
\n \"\"\"))\n \n plot_figure.scatter(\n 'x',\n 'y',\n source=datasource,\n color=dict(field='digit', transform=color_mapping),\n line_alpha=0.6,\n fill_alpha=0.6,\n size=4\n )\n show(plot_figure)\n\n\n\n.. raw:: html\n :file: basic_usage_bokeh_example.html\n\nAs can be seen, the nines that blend between the ones and the sevens are\nodd looking nines (that aren't very rounded) and do, indeed, interpolate\nsurprisingly well between ones with hats and crossed sevens. In contrast\nthe small disjoint cluster of ones at the bottom of the plot is made up\nof ones with feet (a horizontal line at the base of the one) which are,\nindeed, quite distinct from the general mass of ones.\n\nThis concludes our introduction to basic UMAP usage -- hopefully this\nhas given you the tools to get started for yourself. Further tutorials,\ncovering UMAP parameters and more advanced usage are also available when\nyou wish to dive deeper.\n\n--------------\n\n.. raw:: html\n\n \n 
\n
\n \n Digit:\n @digit\n
\n \n\nPenguin data information\n\n.. raw:: html\n\n
\n\nPeguin data are from:\n\n**Gorman KB, Williams TD, Fraser WR** (2014) Ecological Sexual\nDimorphism and Environmental Variability within a Community of Antarctic\nPenguins (Genus *Pygoscelis*). PLoS ONE 9(3): e90081.\ndoi:10.1371/journal.pone.0090081\n\nSee the full paper\n`HERE\n\nOriginal data access and use\n\n.. raw:: html\n\n
\n\nFrom Gorman et al.: \"Data reported here are publicly available within\nthe PAL-LTER data system (datasets #219, 220, and 221):\nhttp://oceaninformatics.ucsd.edu/datazoo/data/pallter/datasets. These\ndata are additionally archived within the United States (US) LTER\nNetwork's Information System Data Portal: https://portal.lternet.edu/.\nIndividuals interested in using these data are therefore expected to\nfollow the US LTER Network's Data Access Policy, Requirements and Use\nAgreement: https://lternet.edu/data-access-policy/.\"\n\nAnyone interested in publishing the data should contact `Dr. Kristen\nGorman\n
\n\"\"\"\n )\n)\n\nplot_figure.circle(\n \"x\",\n \"y\",\n source=datasource,\n color=dict(field=\"digit\", transform=color_mapping),\n line_alpha=0.6,\n fill_alpha=0.6,\n size=4,\n)\nshow(plot_figure)\n"
},
{
"path": "doc/clustering.rst",
"content": "Using UMAP for Clustering\n=========================\n\nUMAP can be used as an effective preprocessing step to boost the\nperformance of density based clustering. This is somewhat controversial,\nand should be attempted with care. For a good discussion of some of the\nissues involved in this, please see the various answers `in this\nstackoverflow\nthread \n \n
\n \n Digit:\n @digit\n
\n\n \n \n \n
\n
\n\n\n\nFor our purposes let’s take “price” as a “target” variable (as is often\nthe case when the dataset is used in machine learning contexts). What we\nwould like to do is provide a UMAP embedding of the data using the\nremaining features. This is tricky since we can’t exactly use a\neuclidean metric over the whole thing. What we can do, however, is split\nthe data into two distinct types: the purely numeric features relating\nto size and weight, and the categorical features of color, cut and\nclarity. Let’s pull each of those feature sets out so we can work with\nthem independently.\n\n.. code:: python3\n\n numeric = diamonds[[\"carat\", \"table\", \"x\", \"y\", \"z\"]].copy()\n ordinal = diamonds[[\"cut\", \"color\", \"clarity\"]].copy()\n\nNow we have a new problem: the numeric features are not at all on the\nsame scales, so any sort of standard distance metric across them will be\ndominated by those features with the largest ranges. We can correct for\nthat by performing feature scaling. To do that we’ll make use of\nsklearn’s ``RobustScaler`` which uses robust statistics (such as the\nmedian and interquartile range) to center and rescale the data feature\nby feature. If we look at the results on the first five rows we see that\nthe different features are all now reasonably comparable, and it is\nreasonable to apply something like euclidean distance across them.\n\n.. code:: python3\n\n scaled_numeric = RobustScaler().fit_transform(numeric)\n scaled_numeric[:5]\n\n\n\n\n.. parsed-literal::\n\n array([[-0.734375 , -0.66666667, -0.95628415, -0.95054945, -0.97345133],\n [-0.765625 , 1.33333333, -0.98907104, -1.02747253, -1.07964602],\n [-0.734375 , 2.66666667, -0.90163934, -0.9010989 , -1.07964602],\n [-0.640625 , 0.33333333, -0.81967213, -0.81318681, -0.79646018],\n [-0.609375 , 0.33333333, -0.7431694 , -0.74725275, -0.69026549]])\n\n\n\nWhat is the best way to handle the categorical features? If they are\npurely categorical it would make sense to one-hot encode the categories\nand use “dice” distance between them. A downside of that is that, with\nso few categories, it is a very coarse metric which will fail to provide\nmuch differentiation. For the diamonds dataset, however, the categories\ncome with a strict order: Ideal cut is better than Premium cut, which is\nbetter than Very Good cut and so on. Color grades work similarly, and\nthere is a distinct grading scheme for clarity as well. We can use an\nordinal encoding on these categories. Now, while the *ranges* of values\nmay vary, the differences between them are all comparable – a difference\nof 1 for each grade level. That means we don’t need to rescale this data\nafter the ordinal coding.\n\n.. code:: python3\n\n ordinal[\"cut\"] = ordinal.cut.map({\"Fair\":0, \"Good\":1, \"Very Good\":2, \"Premium\":3, \"Ideal\":4})\n ordinal[\"color\"] = ordinal.color.map({\"D\":0, \"E\":1, \"F\":2, \"G\":3, \"H\":4, \"I\":5, \"J\":6})\n ordinal[\"clarity\"] = ordinal.clarity.map({\"I1\":0, \"SI2\":1, \"SI1\":2, \"VS2\":3, \"VS1\":4, \"VVS2\":5, \"VVS1\":6, \"IF\":7})\n\n.. code:: python3\n\n ordinal\n\n\n\n\n.. raw:: html\n\n | \n | carat | \ncut | \ncolor | \nclarity | \ndepth | \ntable | \nprice | \nx | \ny | \nz | \n
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n0.23 | \nIdeal | \nE | \nSI2 | \n61.5 | \n55.0 | \n326 | \n3.95 | \n3.98 | \n2.43 | \n
| 1 | \n0.21 | \nPremium | \nE | \nSI1 | \n59.8 | \n61.0 | \n326 | \n3.89 | \n3.84 | \n2.31 | \n
| 2 | \n0.23 | \nGood | \nE | \nVS1 | \n56.9 | \n65.0 | \n327 | \n4.05 | \n4.07 | \n2.31 | \n
| 3 | \n0.29 | \nPremium | \nI | \nVS2 | \n62.4 | \n58.0 | \n334 | \n4.20 | \n4.23 | \n2.63 | \n
| 4 | \n0.31 | \nGood | \nJ | \nSI2 | \n63.3 | \n58.0 | \n335 | \n4.34 | \n4.35 | \n2.75 | \n
\n \n \n \n
\n
\n\n\n\nAs noted we can use euclidean as a sensible distance on the rescaled\nnumeric data. On the other hand since the different ordinal categories\nare entirelty independent of each other, and we have a strict ordinal\ncodin, the socalled “manhattan” metric makes more sense here – it is\nsimply the sum of the absolute differences in each category. As before\nwe can now train UMAP models on each dataset – this time, however, since\nthe datasets are different we need different metrics and even different\nvalues of ``n_neighbors``.\n\n.. code:: python3\n\n numeric_mapper = umap.UMAP(n_neighbors=15, random_state=42).fit(scaled_numeric)\n ordinal_mapper = umap.UMAP(metric=\"manhattan\", n_neighbors=150, random_state=42).fit(ordinal.values)\n\nWe can look at the results of each of these independent views of the\ndataset reduced to 2D using UMAP. Let’s first look at the numeric data\non size and weight of the diamonds. We can colour by the price to get\nsome idea of how the dataset fits together.\n\n.. code:: python3\n\n umap.plot.points(numeric_mapper, values=diamonds[\"price\"], cmap=\"viridis\")\n\n.. image:: images/composing_models_36_1.png\n\n\nWe see that while the data generally correlates somewhat with the price\nof the diamonds there are distinctly different threads in the data,\npresumably corresponding to different styles of cut, and how that\nresults in different sizing of diamonds in the various dimensions,\ndepending on the weight.\n\nIn contrast we ca look at the ordinal data. In this case we’ll colour it\nby the different categories as well as by price.\n\n.. code:: python3\n\n fig, ax = umap.plot.plt.subplots(2, 2, figsize=(12,12))\n umap.plot.points(ordinal_mapper, labels=diamonds[\"color\"], ax=ax[0,0])\n umap.plot.points(ordinal_mapper, labels=diamonds[\"clarity\"], ax=ax[0,1])\n umap.plot.points(ordinal_mapper, labels=diamonds[\"cut\"], ax=ax[1,0])\n umap.plot.points(ordinal_mapper, values=diamonds[\"price\"], cmap=\"viridis\", ax=ax[1,1])\n\n.. image:: images/composing_models_38_1.png\n\n\nAs you can see this is a markedly different result! The ordinal data has\na relatively coarse metric, since the different categories can only take\non a small range of discrete values. This means that, with respect to\nthe trio of color, cut, and clarity, diamonds are largely either almost\nidentical, or quite distinct. The result is very tight groupings which\nhave very high density. You can see a gradient of color from left to\nright in the plot; colouring by cut or clarity show different\nstratifications. The combination of these very distinct statifications\nresults in this highly clustered embedding. It is exactly for this\nreason that we need such a high ``n_neighbors`` value: the very local\nstructure of the data is merely clusters of identical categories; we\nneed to see wider to learn more structure.\n\nGiven these radically different views of the data, what do we get if we\ntry to integrate them together? As before we can use the intersection\nand union operators to simply combine the models. As noted before this\nis a somewhat time-consuming operation as a new 2D representation for\nthe combined models needs to be optimized.\n\n.. code:: python3\n\n intersection_mapper = numeric_mapper * ordinal_mapper\n union_mapper = numeric_mapper + ordinal_mapper\n\nLet’s start by looking at the intersection; here we are only really\ndecreasing connectivity since edges are assigned the probability of\nexisting in *both* data views (before re-asserting local connectivity\nand uniform distribution assumptions).\n\n.. code:: python3\n\n umap.plot.points(intersection_mapper, values=diamonds[\"price\"], cmap=\"viridis\")\n\n.. image:: images/composing_models_42_1.png\n\n\nWhat we get most closely represents the numeric data view. Why is this?\nBecause the categorical data view has points either connected with\ncertainty (because they are, or are nearly, identical) or very loosely.\nThe points connected with near certainty are very dense clusters –\nalmost points in the plot – and mostly what we are doing with the\nintersection is breaking up those clusters with the more fine-grained\nand variable connectivity provided by the numerical data. At th esame\ntime we have shifted the result significantly from the numerical data\nview on its own; the categorical information has made each cluster more\nuniform (rather than being a gradient) in its price.\n\nGiven this result, what would you expect of the union?\n\n.. code:: python3\n\n umap.plot.points(union_mapper, labels=diamonds[\"color\"])\n\n.. image:: images/composing_models_44_1.png\n\n\nWhat we get in practice looks a lot more like the categorical view of\nthe data. This time we are only *increasing* the connectivity (prior to\nre-asserting local connectivity and uniform distribution assumptions);\nthus we retain most of the structure of the high-connectivity\ncategorical view. Note, however, that we have created more connected and\ncoherent clusters in the center of the plot, showing a range of diamond\ncolors, and the introduction of the numerical size and weight\ninformation has induced a rearrangement of the individual clusters\naround the fringes.\n\nWe can go a step further and experiment with the contrast composition\nmethod.\n\n.. code:: python3\n\n contrast_mapper = numeric_mapper - ordinal_mapper\n\n.. code:: python3\n\n umap.plot.points(contrast_mapper, values=diamonds[\"price\"], cmap=\"viridis\")\n\n.. image:: images/composing_models_47_1.png\n\n\nHere we see that we’ve retained a lot of the structure of the numeric\ndata view, but have refined and broken it down further into clear\nclusters with price gradients running through each of them.\n\nTo further demonstrate the power of this approach we can go a step\nfurther and intersect a higher ``n_neighbors`` based embedding of the\nnumeric data view with our existing union of numeric and categorical\ndata – providing a model that is a composition of three simpler models.\n\n.. code:: python3\n\n intersect_union_mapper = umap.UMAP(random_state=42, n_neighbors=60).fit(numeric) * union_mapper\n\n.. code:: python3\n\n umap.plot.points(intersect_union_mapper, values=diamonds[\"price\"], cmap=\"viridis\")\n\n.. image:: images/composing_models_50_1.png\n\nHere the greater global structure from the larger ``n_neighbors`` value\nglues together longer strands and we get an interesting result out. In\nthis case it is not necessarily particularly informative, but it is\nincluded as a demonstration that even composed models can be composed\nwith each other, stacking together potentially many different views.\n"
},
{
"path": "doc/conf.py",
"content": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n#\n# umap documentation build configuration file, created by\n# sphinx-quickstart on Fri Jun 8 10:09:40 2018.\n#\n# This file is execfile()d with the current directory set to its\n# containing dir.\n#\n# Note that not all possible configuration values are present in this\n# autogenerated file.\n#\n# All configuration values have a default; values that are commented out\n# serve to show the default.\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n#\nimport os\nimport sys\n\nsys.path.insert(0, os.path.abspath(\".\"))\nsys.path.insert(0, os.path.abspath(\"..\"))\n\n\n# -- General configuration ------------------------------------------------\n\n# If your documentation needs a minimal Sphinx version, state it here.\n#\n# needs_sphinx = '1.0'\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\n \"sphinx.ext.autodoc\",\n \"sphinx.ext.intersphinx\",\n \"sphinx.ext.mathjax\",\n \"sphinx.ext.viewcode\",\n # 'bokeh.sphinxext.bokeh_plot',\n \"sphinx_gallery.gen_gallery\",\n \"numpydoc\",\n]\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = [\"_templates\"]\n\n# The suffix(es) of source filenames.\n# You can specify multiple suffix as a list of string:\n#\n# source_suffix = ['.rst', '.md']\nsource_suffix = \".rst\"\n\n# The master toctree document.\nmaster_doc = \"index\"\n\n# General information about the project.\nproject = \"umap\"\ncopyright = \"2018, Leland McInnes\"\nauthor = \"Leland McInnes\"\n\n# The version info for the project you're documenting, acts as replacement for\n# |version| and |release|, also used in various other places throughout the\n# built documents.\n#\n# The short X.Y version.\nversion = \"0.5\"\n# The full version, including alpha/beta/rc tags.\nrelease = \"0.5.8\"\n\n# The language for content autogenerated by Sphinx. Refer to documentation\n# for a list of supported languages.\n#\n# This is also used if you do content translation via gettext catalogs.\n# Usually you set \"language\" from the command line for these cases.\nlanguage = \"en\"\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This patterns also effect to html_static_path and html_extra_path\nexclude_patterns = [\"_build\", \"Thumbs.db\", \".DS_Store\"]\n\n# The name of the Pygments (syntax highlighting) style to use.\npygments_style = \"sphinx\"\n\n# If true, `todo` and `todoList` produce output, else they produce nothing.\ntodo_include_todos = False\n\n\n# -- Options for HTML output ----------------------------------------------\n\n# The theme to use for HTML and HTML Help pages. See the documentation for\n# a list of builtin themes.\n#\n# html_theme = 'alabaster'\nhtml_theme = \"sphinx_rtd_theme\"\n\n\n# Theme options are theme-specific and customize the look and feel of a theme\n# further. For a list of options available for each theme, see the\n# documentation.\n#\nhtml_theme_options = {\n \"navigation_depth\": 3,\n \"logo_only\": True,\n}\n\nhtml_logo = \"logo.png\"\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = [\"_static\"]\n\n\n# Custom sidebar templates, must be a dictionary that maps document names\n# to template names.\n#\n# This is required for the alabaster theme\n# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars\n# html_sidebars = {\n# '**': [\n# 'relations.html', # needs 'show_related': True theme option to display\n# 'searchbox.html',\n# ]\n# }\n\nhtml_sidebars = {\n \"**\": [\n \"globaltoc.html\",\n \"relations.html\", # needs 'show_related': True theme option to display\n \"searchbox.html\",\n ]\n}\n\n\n# -- Options for HTMLHelp output ------------------------------------------\n\n# Output file base name for HTML help builder.\nhtmlhelp_basename = \"umapdoc\"\n\n\n# -- Options for LaTeX output ---------------------------------------------\n\nlatex_elements = {\n # The paper size ('letterpaper' or 'a4paper').\n #\n # 'papersize': 'letterpaper',\n # The font size ('10pt', '11pt' or '12pt').\n #\n # 'pointsize': '10pt',\n # Additional stuff for the LaTeX preamble.\n #\n # 'preamble': '',\n # Latex figure (float) alignment\n #\n # 'figure_align': 'htbp',\n}\n\n# Grouping the document tree into LaTeX files. List of tuples\n# (source start file, target name, title,\n# author, documentclass [howto, manual, or own class]).\nlatex_documents = [\n (master_doc, \"umap.tex\", \"umap Documentation\", \"Leland McInnes\", \"manual\"),\n]\n\n\n# -- Options for manual page output ---------------------------------------\n\n# One entry per manual page. List of tuples\n# (source start file, name, description, authors, manual section).\nman_pages = [(master_doc, \"umap\", \"umap Documentation\", [author], 1)]\n\n\n# -- Options for Texinfo output -------------------------------------------\n\n# Grouping the document tree into Texinfo files. List of tuples\n# (source start file, target name, title, author,\n# dir menu entry, description, category)\ntexinfo_documents = [\n (\n master_doc,\n \"umap\",\n \"umap Documentation\",\n author,\n \"umap\",\n \"One line description of project.\",\n \"Miscellaneous\",\n ),\n]\n\n# Example configuration for intersphinx: refer to the Python standard library.\nintersphinx_mapping = {\n \"python\": (\"https://docs.python.org/{.major}\".format(sys.version_info), None),\n \"numpy\": (\"https://docs.scipy.org/doc/numpy/\", None),\n \"scipy\": (\"https://docs.scipy.org/doc/scipy/reference\", None),\n \"matplotlib\": (\"https://matplotlib.org/\", None),\n \"pandas\": (\"https://pandas.pydata.org/pandas-docs/stable/\", None),\n \"sklearn\": (\"http://scikit-learn.org/stable/\", None),\n \"bokeh\": (\"http://bokeh.pydata.org/en/latest/\", None),\n}\n\n# -- Options for sphinx-gallery ---------------------------------------------\n\nsphinx_gallery_conf = {\n # path to your examples scripts\n \"examples_dirs\": \"../examples\",\n \"ignore_pattern\": r\"(.*torus.*|inverse_transform.*)\\.py\",\n # path where to save gallery generated examples\n \"gallery_dirs\": \"auto_examples\",\n \"plot_gallery\": False, # Turn off running the examples for now\n \"reference_url\": {\n # The commented-out URLs are supposed to have search.js files under them.\n \"umap\": None,\n # \"python\": \"https://docs.python.org/{.major}\".format(sys.version_info),\n # Numpy actually DOES have a search.js file but does return valid JSON\n # (missing double quotes on some property names)\n # \"numpy\": \"https://docs.scipy.org/doc/numpy/\",\n # scipt ends up with a 404 for a different URL\n # \"scipy\": \"https://docs.scipy.org/doc/scipy/reference\",\n \"matplotlib\": \"https://matplotlib.org/\",\n # \"pandas\": \"https://pandas.pydata.org/pandas-docs/stable/\",\n # \"sklearn\": \"http://scikit-learn.org/stable/\",\n \"bokeh\": \"http://bokeh.pydata.org/en/latest/\",\n },\n}\n\n\ndef setup(app):\n app.add_js_file(\n \"https://cdnjs.cloudflare.com/ajax/libs/require.js/2.1.10/require.min.js\"\n )\n app.add_js_file(\"https://cdn.plot.ly/plotly-latest.min.js\")\n\n\n# This prevents complaints autosummary stub files not being found for each method on\n# the class. See https://stackoverflow.com/q/65198998\nnumpydoc_show_class_members = False\n"
},
{
"path": "doc/densmap_demo.rst",
"content": "Better Preserving Local Density with DensMAP\n============================================\n\nA notable assumption in UMAP is that the data is uniformly distributed\non some manifold and that it is ultimately this manifold that we would\nlike to present. This is highly effective for many use cases, but it can\nbe the case that one would like to preserve more information about the\nrelative local density of data. A recent paper presented a technique\ncalled\n`DensMAP | \n | cut | \ncolor | \nclarity | \n
|---|---|---|---|
| 0 | \n4 | \n1 | \n1 | \n
| 1 | \n3 | \n1 | \n2 | \n
| 2 | \n1 | \n1 | \n4 | \n
| 3 | \n3 | \n5 | \n3 | \n
| 4 | \n1 | \n6 | \n1 | \n
| ... | \n... | \n... | \n... | \n
| 53935 | \n4 | \n0 | \n2 | \n
| 53936 | \n1 | \n0 | \n2 | \n
| 53937 | \n2 | \n0 | \n2 | \n
| 53938 | \n3 | \n4 | \n1 | \n
| 53939 | \n4 | \n0 | \n1 | \n
53940 rows × 3 columns
\n- \n\n.. raw:: html\n\n
- \n\nThe random seed when computing the k-nn.\n\n.. raw:: html\n\n \n\n.. raw:: html\n\n
- \n\nThe n_neighbors value when computing the k-nn.\n\n.. raw:: html\n\n \n\n.. raw:: html\n\n
- \n\nThe random seed when running UMAP.\n\n.. raw:: html\n\n \n\n.. raw:: html\n\n
\n \n
\n \n \n \n"
},
{
"path": "examples/digits/digits.py",
"content": "from bokeh.plotting import figure, output_file, show\nfrom bokeh.models import CategoricalColorMapper, ColumnDataSource\nfrom bokeh.palettes import Category10\n\nimport umap\nfrom sklearn.datasets import load_digits\n\ndigits = load_digits()\nembedding = umap.UMAP().fit_transform(digits.data)\n\noutput_file(\"digits.html\")\n\ntargets = [str(d) for d in digits.target_names]\n\nsource = ColumnDataSource(\n dict(\n x=[e[0] for e in embedding],\n y=[e[1] for e in embedding],\n label=[targets[d] for d in digits.target],\n )\n)\n\ncmap = CategoricalColorMapper(factors=targets, palette=Category10[10])\n\np = figure(title=\"test umap\")\np.circle(\n x=\"x\",\n y=\"y\",\n source=source,\n color={\"field\": \"label\", \"transform\": cmap},\n legend=\"label\",\n)\n\nshow(p)\n"
},
{
"path": "examples/galaxy10sdss.py",
"content": "\"\"\"\nUMAP on the Galaxy10SDSS dataset\n---------------------------------------------------------\n\nThis is an example of using UMAP on the Galaxy10SDSS\ndataset. The goal of this example is largely to\ndemonstrate the use of supervised learning as an\neffective tool for visualizing and reducing complex data.\nIn addition, hdbscan is used to classify the processed\ndata.\n\"\"\"\n\nimport numpy as np\nimport h5py\nimport matplotlib.pyplot as plt\nimport umap\nimport os\n\n# from sklearn.model_selection import train_test_split\nimport math\nimport requests\n\n# libraries for clustering\nimport hdbscan\nimport sklearn.cluster as cluster\nfrom sklearn.metrics import adjusted_rand_score, adjusted_mutual_info_score\n\nif not os.path.isfile(\"Galaxy10.h5\"):\n url = \"http://astro.utoronto.ca/~bovy/Galaxy10/Galaxy10.h5\"\n r = requests.get(url, allow_redirects=True)\n open(\"Galaxy10.h5\", \"wb\").write(r.content)\n\n# To get the images and labels from file\nwith h5py.File(\"Galaxy10.h5\", \"r\") as F:\n images = np.array(F[\"images\"])\n labels = np.array(F[\"ans\"])\n\nX_train = np.empty([math.floor(len(labels) / 100), 14283], dtype=np.float64)\ny_train = np.empty([math.floor(len(labels) / 100)], dtype=np.float64)\nX_test = X_train\ny_test = y_train\n# Get a subset of the data\nfor i in range(math.floor(len(labels) / 100)):\n X_train[i, :] = np.array(np.ndarray.flatten(images[i, :, :, :]), dtype=np.float64)\n y_train[i] = labels[i]\n X_test[i, :] = np.array(\n np.ndarray.flatten(images[i + math.floor(len(labels) / 100), :, :, :]),\n dtype=np.float64,\n )\n y_test[i] = labels[i + math.floor(len(labels) / 100)]\n\n# Plot distribution\nclasses, frequency = np.unique(y_train, return_counts=True)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nplt.bar(classes, frequency)\nplt.xlabel(\"Class\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Data Subset\")\nplt.savefig(\"galaxy10_subset.svg\")\n# 2D Embedding\n## UMAP\nreducer = umap.UMAP(\n n_components=2, n_neighbors=20, random_state=42, transform_seed=42, verbose=False\n)\nreducer.fit(X_train)\n\ngalaxy10_umap = reducer.transform(X_train)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nplt.scatter(\n galaxy10_umap[:, 0],\n galaxy10_umap[:, 1],\n c=y_train,\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_train,\n)\nplt.colorbar(boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_2D_umap.svg\")\n### UMAP - Supervised\nreducer = umap.UMAP(\n n_components=2, n_neighbors=15, random_state=42, transform_seed=42, verbose=False\n)\nreducer.fit(X_train, y_train)\n\ngalaxy10_umap_supervised = reducer.transform(X_train)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nplt.scatter(\n galaxy10_umap_supervised[:, 0],\n galaxy10_umap_supervised[:, 1],\n c=y_train,\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_train,\n)\nplt.colorbar(boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_2D_umap_supervised.svg\")\n### UMAP - Supervised prediction\ngalaxy10_umap_supervised_prediction = reducer.transform(X_test)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nplt.scatter(\n galaxy10_umap_supervised_prediction[:, 0],\n galaxy10_umap_supervised_prediction[:, 1],\n c=y_test,\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_test,\n)\nplt.colorbar(boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_2D_umap_supervised_prediction.svg\")\n\n# cluster the data\nlabels = hdbscan.HDBSCAN(\n min_samples=10,\n min_cluster_size=10,\n).fit_predict(galaxy10_umap_supervised_prediction)\nclustered = labels >= 0\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nplt.scatter(\n galaxy10_umap_supervised_prediction[~clustered, 0],\n galaxy10_umap_supervised_prediction[~clustered, 1],\n color=(0.5, 0.5, 0.5),\n alpha=0.5,\n)\nplt.scatter(\n galaxy10_umap_supervised_prediction[clustered, 0],\n galaxy10_umap_supervised_prediction[clustered, 1],\n c=y_test[clustered],\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_test[clustered],\n)\nplt.colorbar(boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_2D_umap_supervised_prediction_clustered.svg\")\n\n# Print out information on quality of clustering\nprint(\"2D Supervised Embedding with Clustering\")\nprint(adjusted_rand_score(y_test, labels), adjusted_mutual_info_score(y_test, labels))\n\nprint(\n adjusted_rand_score(y_test[clustered], labels[clustered]),\n adjusted_mutual_info_score(y_test[clustered], labels[clustered]),\n)\n\nprint(np.sum(clustered) / y_test.shape[0])\n\n# 3D Embedding\n## UMAP\nreducer = umap.UMAP(\n n_components=3, n_neighbors=20, random_state=42, transform_seed=42, verbose=False\n)\nreducer.fit(X_train)\ngalaxy10_umap = reducer.transform(X_train)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nax = fig.add_subplot(projection=\"3d\")\np = ax.scatter(\n galaxy10_umap[:, 0],\n galaxy10_umap[:, 1],\n galaxy10_umap[:, 2],\n c=y_train,\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_train,\n)\nfig.colorbar(p, ax=ax, boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_3D_umap.svg\")\n## UMAP - Supervised\nreducer = umap.UMAP(\n n_components=3, n_neighbors=20, random_state=42, transform_seed=42, verbose=False\n)\nreducer.fit(X_train, y_train)\ngalaxy10_umap_supervised = reducer.transform(X_train)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nax = fig.add_subplot(projection=\"3d\")\np = ax.scatter(\n galaxy10_umap_supervised[:, 0],\n galaxy10_umap_supervised[:, 1],\n galaxy10_umap_supervised[:, 2],\n c=y_train,\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_train,\n)\nfig.colorbar(p, ax=ax, boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_3D_umap_supervised.svg\")\n## UMAP - Supervised prediction\ngalaxy10_umap_supervised_prediction = reducer.transform(X_test)\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nax = fig.add_subplot(projection=\"3d\")\np = ax.scatter(\n galaxy10_umap_supervised_prediction[:, 0],\n galaxy10_umap_supervised_prediction[:, 1],\n galaxy10_umap_supervised_prediction[:, 2],\n c=y_test,\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_test,\n)\nfig.colorbar(p, ax=ax, boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_3D_umap_supervised_prediction.svg\")\n\n# cluster the data\nlabels = hdbscan.HDBSCAN(\n min_samples=10,\n min_cluster_size=10,\n).fit_predict(galaxy10_umap_supervised_prediction)\nclustered = labels >= 0\nfig = plt.figure(1, figsize=(4, 4))\nplt.clf()\nax = fig.add_subplot(projection=\"3d\")\nq = ax.scatter(\n galaxy10_umap_supervised_prediction[~clustered, 0],\n galaxy10_umap_supervised_prediction[~clustered, 1],\n galaxy10_umap_supervised_prediction[~clustered, 2],\n color=(0.5, 0.5, 0.5),\n alpha=0.5,\n)\np = ax.scatter(\n galaxy10_umap_supervised_prediction[clustered, 0],\n galaxy10_umap_supervised_prediction[clustered, 1],\n galaxy10_umap_supervised_prediction[clustered, 2],\n c=y_test[clustered],\n cmap=plt.cm.nipy_spectral,\n edgecolor=\"k\",\n label=y_test[clustered],\n)\nfig.colorbar(p, ax=ax, boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.savefig(\"galaxy10_3D_umap_supervised_prediction_clustered.svg\")\n\n# Print out information on quality of clustering\nprint(\"3D Supervised Embedding with Clustering\")\nprint(adjusted_rand_score(y_test, labels), adjusted_mutual_info_score(y_test, labels))\n\nprint(\n adjusted_rand_score(y_test[clustered], labels[clustered]),\n adjusted_mutual_info_score(y_test[clustered], labels[clustered]),\n)\n\nprint(np.sum(clustered) / y_test.shape[0])\n# Dimensions 4 to 25\nfor dimensions in range(4, 26):\n reducer = umap.UMAP(\n n_components=dimensions,\n n_neighbors=20,\n random_state=42,\n transform_seed=42,\n verbose=False,\n )\n reducer.fit(X_train, y_train)\n galaxy10_umap_supervised = reducer.transform(X_train)\n # UMAP - Supervised prediction\n galaxy10_umap_supervised_prediction = reducer.transform(X_test)\n # cluster the data\n labels = hdbscan.HDBSCAN(\n min_samples=10,\n min_cluster_size=10,\n ).fit_predict(galaxy10_umap_supervised_prediction)\n clustered = labels >= 0\n # Print out information on quality of clustering\n print(str(dimensions) + \"D Supervised Embedding with Clustering\")\n print(\n adjusted_rand_score(y_test, labels), adjusted_mutual_info_score(y_test, labels)\n )\n print(\n adjusted_rand_score(y_test[clustered], labels[clustered]),\n adjusted_mutual_info_score(y_test[clustered], labels[clustered]),\n )\n print(np.sum(clustered) / y_test.shape[0])\n"
},
{
"path": "examples/inverse_transform_example.py",
"content": "#!/usr/bin/env python\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom sklearn.datasets import fetch_openml\n\nimport umap\n\nmnist = fetch_openml(\"Fashion-MNIST\", version=1)\n\n\ntrans = umap.UMAP(\n n_neighbors=10,\n random_state=42,\n metric=\"euclidean\",\n output_metric=\"euclidean\",\n init=\"spectral\",\n verbose=True,\n).fit(mnist.data)\n\ncorners = np.array(\n [\n [-5.1, 2.9],\n [-1.9, 6.4],\n [-5.4, -6.3],\n [8.3, 4.0],\n ] # 7 # 4 # 1 # 0\n)\n\ntest_pts = np.array(\n [\n (corners[0] * (1 - x) + corners[1] * x) * (1 - y)\n + (corners[2] * (1 - x) + corners[3] * x) * y\n for y in np.linspace(0, 1, 10)\n for x in np.linspace(0, 1, 10)\n ]\n)\n\ninv_transformed_points = trans.inverse_transform(test_pts)\n\nplt.scatter(\n trans.embedding_[:, 0],\n trans.embedding_[:, 1],\n c=mnist.target,\n cmap=\"Spectral\",\n s=0.25,\n)\nplt.colorbar(boundaries=np.arange(11) - 0.5).set_ticks(np.arange(10))\nplt.scatter(test_pts[:, 0], test_pts[:, 1], marker=\"x\", c=\"k\")\n\nfig, ax = plt.subplots(10, 10)\nfor i in range(10):\n for j in range(10):\n ax[i, j].imshow(\n inv_transformed_points[i * 10 + j].reshape(28, 28), origin=\"upper\"\n )\n ax[i, j].get_xaxis().set_visible(False)\n ax[i, j].get_yaxis().set_visible(False)\n\nplt.show()\n"
},
{
"path": "examples/iris/iris.html",
"content": "\n\n\n \n \n \n \n
\n \n \n \n"
},
{
"path": "examples/iris/iris.py",
"content": "from bokeh.plotting import figure, output_file, show\nfrom bokeh.models import CategoricalColorMapper, ColumnDataSource\nfrom bokeh.palettes import Category10\n\nimport umap\nfrom sklearn.datasets import load_iris\n\niris = load_iris()\nembedding = umap.UMAP(\n n_neighbors=50, learning_rate=0.5, init=\"random\", min_dist=0.001\n).fit_transform(iris.data)\n\noutput_file(\"iris.html\")\n\n\ntargets = [str(d) for d in iris.target_names]\n\nsource = ColumnDataSource(\n dict(\n x=[e[0] for e in embedding],\n y=[e[1] for e in embedding],\n label=[targets[d] for d in iris.target],\n )\n)\n\ncmap = CategoricalColorMapper(factors=targets, palette=Category10[10])\n\np = figure(title=\"Test UMAP on Iris dataset\")\np.circle(\n x=\"x\",\n y=\"y\",\n source=source,\n color={\"field\": \"label\", \"transform\": cmap},\n legend=\"label\",\n)\n\nshow(p)\n"
},
{
"path": "examples/mnist_torus_sphere_example.py",
"content": "#!/usr/bin/env python\n\nimport matplotlib.pyplot as plt\nimport numba\nimport numpy as np\nfrom mayavi import mlab\nfrom sklearn.datasets import load_digits\nfrom sklearn.model_selection import train_test_split\n\nimport umap\n\ndigits = load_digits()\nX_train, X_test, y_train, y_test = train_test_split(\n digits.data, digits.target, stratify=digits.target, random_state=42\n)\n\ntarget_spaces = [\"plane\", \"torus\", \"sphere\"]\n\nif \"plane\" in target_spaces:\n # embed onto a plane\n\n trans = umap.UMAP(\n n_neighbors=10,\n random_state=42,\n metric=\"euclidean\",\n output_metric=\"euclidean\",\n init=\"spectral\",\n verbose=True,\n ).fit(X_train)\n\n plt.scatter(\n trans.embedding_[:, 0], trans.embedding_[:, 1], c=y_train, cmap=\"Spectral\"\n )\n plt.show()\n\nif \"torus\" in target_spaces:\n # embed onto a torus\n # note: this is a topological torus, not a geometric torus. Think\n # Pacman, not donut.\n\n @numba.njit(fastmath=True)\n def torus_euclidean_grad(x, y, torus_dimensions=(2 * np.pi, 2 * np.pi)):\n \"\"\"Standard euclidean distance.\n\n ..math::\n D(x, y) = \\sqrt{\\sum_i (x_i - y_i)^2}\n \"\"\"\n distance_sqr = 0.0\n g = np.zeros_like(x)\n for i in range(x.shape[0]):\n a = abs(x[i] - y[i])\n if 2 * a < torus_dimensions[i]:\n distance_sqr += a**2\n g[i] = x[i] - y[i]\n else:\n distance_sqr += (torus_dimensions[i] - a) ** 2\n g[i] = (x[i] - y[i]) * (a - torus_dimensions[i]) / a\n distance = np.sqrt(distance_sqr)\n return distance, g / (1e-6 + distance)\n\n trans = umap.UMAP(\n n_neighbors=10,\n random_state=42,\n metric=\"euclidean\",\n output_metric=torus_euclidean_grad,\n init=\"spectral\",\n min_dist=0.15, # requires adjustment since the torus has limited space\n verbose=True,\n ).fit(X_train)\n\n mlab.clf()\n x, y, z = np.mgrid[-3:3:50j, -3:3:50j, -3:3:50j]\n\n # Plot a torus\n R = 2\n r = 1\n values = (R - np.sqrt(x**2 + y**2)) ** 2 + z**2 - r**2\n mlab.contour3d(x, y, z, values, color=(1.0, 1.0, 1.0), contours=[0])\n\n # torus angles -> 3D\n x = (R + r * np.cos(trans.embedding_[:, 0])) * np.cos(trans.embedding_[:, 1])\n y = (R + r * np.cos(trans.embedding_[:, 0])) * np.sin(trans.embedding_[:, 1])\n z = r * np.sin(trans.embedding_[:, 0])\n\n pts = mlab.points3d(\n x, y, z, y_train, colormap=\"spectral\", scale_mode=\"none\", scale_factor=0.1\n )\n\n mlab.show()\n\nif \"sphere\" in target_spaces:\n # embed onto a sphere\n trans = umap.UMAP(\n n_neighbors=10,\n random_state=42,\n metric=\"euclidean\",\n output_metric=\"haversine\",\n init=\"spectral\",\n min_dist=0.15, # requires adjustment since the sphere has limited space\n verbose=True,\n ).fit(X_train)\n\n mlab.clf()\n x, y, z = np.mgrid[-3:3:50j, -3:3:50j, -3:3:50j]\n\n # Plot a sphere\n r = 3\n values = x**2 + y**2 + z**2 - r**2\n mlab.contour3d(x, y, z, values, color=(1.0, 1.0, 1.0), contours=[0])\n\n # latitude, longitude -> 3D\n x = r * np.sin(trans.embedding_[:, 0]) * np.cos(trans.embedding_[:, 1])\n y = r * np.sin(trans.embedding_[:, 0]) * np.sin(trans.embedding_[:, 1])\n z = r * np.cos(trans.embedding_[:, 0])\n\n pts = mlab.points3d(\n x, y, z, y_train, colormap=\"spectral\", scale_mode=\"none\", scale_factor=0.2\n )\n\n mlab.show()\n"
},
{
"path": "examples/mnist_transform_new_data.py",
"content": "#!/usr/bin/env python\n\n\"\"\"\nUMAP on the MNIST Digits dataset\n--------------------------------\n\nA simple example demonstrating how to use UMAP on a larger\ndataset such as MNIST. We first pull the MNIST dataset and\nthen use UMAP to reduce it to only 2-dimensions for\neasy visualisation.\n\nNote that UMAP manages to both group the individual digit\nclasses, but also to retain the overall global structure\namong the different digit classes -- keeping 1 far from\n0, and grouping triplets of 3,5,8 and 4,7,9 which can\nblend into one another in some cases.\n\"\"\"\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.datasets import fetch_openml\nfrom sklearn.model_selection import train_test_split\n\nimport umap\n\nsns.set(context=\"paper\", style=\"white\")\n\nmnist = fetch_openml(\"mnist_784\", version=1)\nX_train, X_test, y_train, y_test = train_test_split(\n mnist.data, mnist.target, stratify=mnist.target, random_state=42\n)\n\nreducer = umap.UMAP(random_state=42)\nembedding_train = reducer.fit_transform(X_train)\nembedding_test = reducer.transform(X_test)\n\nfig, ax = plt.subplots(1, 2, sharex=True, sharey=True, figsize=(12, 10))\nax[0].scatter(\n embedding_train[:, 0], embedding_train[:, 1], c=y_train, cmap=\"Spectral\" # , s=0.1\n)\nax[1].scatter(\n embedding_test[:, 0], embedding_test[:, 1], c=y_test, cmap=\"Spectral\" # , s=0.1\n)\nplt.setp(ax[0], xticks=[], yticks=[])\nplt.setp(ax[1], xticks=[], yticks=[])\nplt.suptitle(\"MNIST data embedded into two dimensions by UMAP\", fontsize=18)\nax[0].set_title(\"Training Set\", fontsize=12)\nax[1].set_title(\"Test Set\", fontsize=12)\nplt.show()\n"
},
{
"path": "examples/plot_algorithm_comparison.py",
"content": "\"\"\"\nComparison of Dimension Reduction Techniques\n--------------------------------------------\n\nA comparison of several different dimension reduction\ntechniques on a variety of toy datasets. The datasets\nare all toy datasets, but should provide a representative\nrange of the strengths and weaknesses of the different\nalgorithms.\n\nThe time to perform the dimension reduction with each\nalgorithm and each dataset is recorded in the lower\nright of each plot.\n\nThings to note about the datasets:\n\n- Blobs: A set of five gaussian blobs in 10 dimensional\n space. This should be a prototypical example\n of something that should clearly separate\n even in a reduced dimension space.\n- Iris: a classic small dataset with one distinct class\n and two classes that are not clearly separated.\n- Digits: handwritten digits -- ideally different digit\n classes should form distinct groups. Due to\n the nature of handwriting digits may have several\n forms (crossed or uncrossed sevens, capped or\n straight line oes, etc.)\n- Wine: wine characteristics ideally used for a toy\n regression. Ultimately the data is essentially\n one dimensional in nature.\n- Swiss Roll: data is essentially a rectangle, but\n has been \"rolled up\" like a swiss roll\n in three dimensional space. Ideally a\n dimension reduction technique should\n be able to \"unroll\" it. The data\n has been coloured according to one dimension\n of the rectangle, so should form\n a rectangle of smooth color variation.\n- Sphere: the two dimensional surface of a three\n dimensional sphere. This cannot be represented\n accurately in two dimensions without tearing.\n The sphere has been coloured with hue around\n the equator and black to white from the south\n to north pole.\n\"\"\"\n\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport time\n\nfrom sklearn import datasets, decomposition, manifold, preprocessing\nfrom colorsys import hsv_to_rgb\n\nimport umap\n\nsns.set(context=\"paper\", style=\"white\")\n\nblobs, blob_labels = datasets.make_blobs(\n n_samples=500, n_features=10, centers=5, random_state=42\n)\niris = datasets.load_iris()\ndigits = datasets.load_digits(n_class=10)\nwine = datasets.load_wine()\nswissroll, swissroll_labels = datasets.make_swiss_roll(\n n_samples=1000, noise=0.1, random_state=42\n)\nsphere = np.random.normal(size=(600, 3))\nsphere = preprocessing.normalize(sphere)\nsphere_hsv = np.array(\n [\n (\n (np.arctan2(c[1], c[0]) + np.pi) / (2 * np.pi),\n np.abs(c[2]),\n min((c[2] + 1.1), 1.0),\n )\n for c in sphere\n ]\n)\nsphere_colors = np.array([hsv_to_rgb(*c) for c in sphere_hsv])\n\nreducers = [\n (manifold.TSNE, {\"perplexity\": 50}),\n # (manifold.LocallyLinearEmbedding, {'n_neighbors':10, 'method':'hessian'}),\n (manifold.Isomap, {\"n_neighbors\": 30}),\n (manifold.MDS, {}),\n (decomposition.PCA, {}),\n (umap.UMAP, {\"n_neighbors\": 30, \"min_dist\": 0.3}),\n]\n\ntest_data = [\n (blobs, blob_labels),\n (iris.data, iris.target),\n (digits.data, digits.target),\n (wine.data, wine.target),\n (swissroll, swissroll_labels),\n (sphere, sphere_colors),\n]\ndataset_names = [\"Blobs\", \"Iris\", \"Digits\", \"Wine\", \"Swiss Roll\", \"Sphere\"]\n\nn_rows = len(test_data)\nn_cols = len(reducers)\nax_index = 1\nax_list = []\n\n# plt.figure(figsize=(9 * 2 + 3, 12.5))\nplt.figure(figsize=(10, 8))\nplt.subplots_adjust(\n left=0.02, right=0.98, bottom=0.001, top=0.96, wspace=0.05, hspace=0.01\n)\nfor data, labels in test_data:\n for reducer, args in reducers:\n start_time = time.time()\n embedding = reducer(n_components=2, **args).fit_transform(data)\n elapsed_time = time.time() - start_time\n ax = plt.subplot(n_rows, n_cols, ax_index)\n if isinstance(labels[0], tuple):\n ax.scatter(*embedding.T, s=10, c=labels, alpha=0.5)\n else:\n ax.scatter(*embedding.T, s=10, c=labels, cmap=\"Spectral\", alpha=0.5)\n ax.text(\n 0.99,\n 0.01,\n \"{:.2f} s\".format(elapsed_time),\n transform=ax.transAxes,\n size=14,\n horizontalalignment=\"right\",\n )\n ax_list.append(ax)\n ax_index += 1\nplt.setp(ax_list, xticks=[], yticks=[])\n\nfor i in np.arange(n_rows) * n_cols:\n ax_list[i].set_ylabel(dataset_names[i // n_cols], size=16)\nfor i in range(n_cols):\n ax_list[i].set_xlabel(repr(reducers[i][0]()).split(\"(\")[0], size=16)\n ax_list[i].xaxis.set_label_position(\"top\")\n\nplt.tight_layout()\nplt.show()\n"
},
{
"path": "examples/plot_fashion-mnist_example.py",
"content": "\"\"\"\nUMAP on the Fashion MNIST Digits dataset using Datashader\n---------------------------------------------------------\n\nThis is a simple example of using UMAP on the Fashion-MNIST\ndataset. The goal of this example is largely to demonstrate\nthe use of datashader as an effective tool for visualising\nUMAP results. In particular datashader allows visualisation\nof very large datasets where overplotting can be a serious\nproblem. It supports coloring by categorical variables\n(as shown in this example), or by continuous variables,\nor by density (as is common in datashader examples).\n\"\"\"\n\nimport umap\nimport numpy as np\nimport pandas as pd\nimport requests\nimport os\nimport datashader as ds\nimport datashader.utils as utils\nimport datashader.transfer_functions as tf\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nsns.set(context=\"paper\", style=\"white\")\n\nif not os.path.isfile(\"fashion-mnist.csv\"):\n csv_data = requests.get(\"https://www.openml.org/data/get_csv/18238735/phpnBqZGZ\")\n with open(\"fashion-mnist.csv\", \"w\") as f:\n f.write(csv_data.text)\nsource_df = pd.read_csv(\"fashion-mnist.csv\")\n\ndata = source_df.iloc[:, :784].values.astype(np.float32)\ntarget = source_df[\"class\"].values\n\npal = [\n \"#9e0142\",\n \"#d8434e\",\n \"#f67a49\",\n \"#fdbf6f\",\n \"#feeda1\",\n \"#f1f9a9\",\n \"#bfe5a0\",\n \"#74c7a5\",\n \"#378ebb\",\n \"#5e4fa2\",\n]\ncolor_key = {str(d): c for d, c in enumerate(pal)}\n\nreducer = umap.UMAP(random_state=42)\nembedding = reducer.fit_transform(data)\n\ndf = pd.DataFrame(embedding, columns=(\"x\", \"y\"))\ndf[\"class\"] = pd.Series([str(x) for x in target], dtype=\"category\")\n\ncvs = ds.Canvas(plot_width=400, plot_height=400)\nagg = cvs.points(df, \"x\", \"y\", ds.count_cat(\"class\"))\nimg = tf.shade(agg, color_key=color_key, how=\"eq_hist\")\n\nutils.export_image(img, filename=\"fashion-mnist\", background=\"black\")\n\nimage = plt.imread(\"fashion-mnist.png\")\nfig, ax = plt.subplots(figsize=(6, 6))\nplt.imshow(image)\nplt.setp(ax, xticks=[], yticks=[])\nplt.title(\n \"Fashion MNIST data embedded\\n\"\n \"into two dimensions by UMAP\\n\"\n \"visualised with Datashader\",\n fontsize=12,\n)\n\nplt.show()\n"
},
{
"path": "examples/plot_feature_extraction_classification.py",
"content": "\"\"\"\nUMAP as a Feature Extraction Technique for Classification\n---------------------------------------------------------\n\nThe following script shows how UMAP can be used as a feature extraction\ntechnique to improve the accuracy on a classification task. It also shows\nhow UMAP can be integrated in standard scikit-learn pipelines.\n\nThe first step is to create a dataset for a classification task, which is\nperformed with the function ``sklearn.datasets.make_classification``. The\ndataset is then split into a training set and a test set using the\n``sklearn.model_selection.train_test_split`` function.\n\nSecond, a linear SVM is fitted on the training set. To choose the best\nhyperparameters automatically, a gridsearch is performed on the training set.\nThe performance of the model is then evaluated on the test set with the\naccuracy metric.\n\n Third, the previous step is repeated with a slight modification: UMAP is\n used as a feature extraction technique. This small change results in a\n substantial improvement compared to the model where raw data is used.\n\"\"\"\n\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import train_test_split, GridSearchCV\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.svm import LinearSVC\n\nfrom umap import UMAP\n\n# Make a toy dataset\nX, y = make_classification(\n n_samples=1000,\n n_features=300,\n n_informative=250,\n n_redundant=0,\n n_repeated=0,\n n_classes=2,\n random_state=1212,\n)\n\n# Split the dataset into a training set and a test set\nX_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=0.2, random_state=42\n)\n\n# Classification with a linear SVM\nsvc = LinearSVC(dual=False, random_state=123)\nparams_grid = {\"C\": [10**k for k in range(-3, 4)]}\nclf = GridSearchCV(svc, params_grid)\nclf.fit(X_train, y_train)\nprint(\n \"Accuracy on the test set with raw data: {:.3f}\".format(clf.score(X_test, y_test))\n)\n\n# Transformation with UMAP followed by classification with a linear SVM\numap = UMAP(random_state=456)\npipeline = Pipeline([(\"umap\", umap), (\"svc\", svc)])\nparams_grid_pipeline = {\n \"umap__n_neighbors\": [5, 20],\n \"umap__n_components\": [15, 25, 50],\n \"svc__C\": [10**k for k in range(-3, 4)],\n}\n\n\nclf_pipeline = GridSearchCV(pipeline, params_grid_pipeline)\nclf_pipeline.fit(X_train, y_train)\nprint(\n \"Accuracy on the test set with UMAP transformation: {:.3f}\".format(\n clf_pipeline.score(X_test, y_test)\n )\n)\n"
},
{
"path": "examples/plot_mnist_example.py",
"content": "\"\"\"\nUMAP on the MNIST Digits dataset\n--------------------------------\n\nA simple example demonstrating how to use UMAP on a larger\ndataset such as MNIST. We first pull the MNIST dataset and\nthen use UMAP to reduce it to only 2-dimensions for\neasy visualisation.\n\nNote that UMAP manages to both group the individual digit\nclasses, but also to retain the overall global structure\namong the different digit classes -- keeping 1 far from\n0, and grouping triplets of 3,5,8 and 4,7,9 which can\nblend into one another in some cases.\n\"\"\"\n\nimport umap\nfrom sklearn.datasets import fetch_openml\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nsns.set(context=\"paper\", style=\"white\")\n\nmnist = fetch_openml(\"mnist_784\", version=1)\n\nreducer = umap.UMAP(random_state=42)\nembedding = reducer.fit_transform(mnist.data)\n\nfig, ax = plt.subplots(figsize=(12, 10))\ncolor = mnist.target.astype(int)\nplt.scatter(embedding[:, 0], embedding[:, 1], c=color, cmap=\"Spectral\", s=0.1)\nplt.setp(ax, xticks=[], yticks=[])\nplt.title(\"MNIST data embedded into two dimensions by UMAP\", fontsize=18)\n\nplt.show()\n"
},
{
"path": "notebooks/AnimatingUMAP.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Making Animations of UMAP Hyper-parameters\\n\",\n \"\\n\",\n \"Sometimes one of the best ways to see the effects of hyperparameters is simply to visualise what happens as they change. We can do that in practice with UMAP by simply creating an animation that transitions between embeddings generated with variations of hyperparameters. To do this we'll make use of matplotlib and its animation capabilities. Jake Vanderplas has [a great tutorial](https://jakevdp.github.io/blog/2012/08/18/matplotlib-animation-tutorial/) if you want to know more about creating animations with matplotlib.\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"**Note:**\\n\",\n \"This is a self contained example of how to use UMAP and the impact of individual hyper-parameters. To make sure everything works correctly please use `conda`.\\n\",\n \"For install and usage details see [here](https://docs.conda.io/en/latest/miniconda.html)\\n\",\n \"\\n\",\n \"To create animations we need `ffmpeg`. It can be installed with `conda`.\\n\",\n \"\\n\",\n \"If you already have `ffmpeg` installed on your machine and you know what you are doing you do not need conda. It is only used to install `ffmpeg`.\\n\",\n \"\\n\",\n \"=> Remove the next two cells if you are not using `conda`.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"conda 4.7.12\\n\"\n ]\n }\n ],\n \"source\": [\n \"!conda --version\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Collecting package metadata (current_repodata.json): ...working... done\\n\",\n \"Solving environment: ...working... done\\n\",\n \"\\n\",\n \"## Package Plan ##\\n\",\n \"\\n\",\n \" environment location: C:\\\\Progams\\\\Miniconda\\\\envs\\\\tf\\n\",\n \"\\n\",\n \" added / updated specs:\\n\",\n \" - ffmpeg\\n\",\n \"\\n\",\n \"\\n\",\n \"The following packages will be downloaded:\\n\",\n \"\\n\",\n \" package | build\\n\",\n \" ---------------------------|-----------------\\n\",\n \" ca-certificates-2019.9.11 | hecc5488_0 181 KB conda-forge\\n\",\n \" certifi-2019.9.11 | py37_0 155 KB\\n\",\n \" ffmpeg-4.2 | h6538335_0 23.4 MB conda-forge\\n\",\n \" openssl-1.1.1c | hfa6e2cd_0 4.7 MB conda-forge\\n\",\n \" ------------------------------------------------------------\\n\",\n \" Total: 28.5 MB\\n\",\n \"\\n\",\n \"The following NEW packages will be INSTALLED:\\n\",\n \"\\n\",\n \" ffmpeg conda-forge/win-64::ffmpeg-4.2-h6538335_0\\n\",\n \"\\n\",\n \"The following packages will be UPDATED:\\n\",\n \"\\n\",\n \" ca-certificates pkgs/main::ca-certificates-2019.5.15-1 --> conda-forge::ca-certificates-2019.9.11-hecc5488_0\\n\",\n \" certifi 2019.6.16-py37_1 --> 2019.9.11-py37_0\\n\",\n \"\\n\",\n \"The following packages will be SUPERSEDED by a higher-priority channel:\\n\",\n \"\\n\",\n \" openssl pkgs/main::openssl-1.1.1c-he774522_1 --> conda-forge::openssl-1.1.1c-hfa6e2cd_0\\n\",\n \"\\n\",\n \"\\n\",\n \"\\n\",\n \"Downloading and Extracting Packages\\n\",\n \"\\n\",\n \"openssl-1.1.1c | 4.7 MB | | 0% \\n\",\n \"openssl-1.1.1c | 4.7 MB | | 0% \\n\",\n \"openssl-1.1.1c | 4.7 MB | #5 | 15% \\n\",\n \"openssl-1.1.1c | 4.7 MB | ### | 30% \\n\",\n \"openssl-1.1.1c | 4.7 MB | ####5 | 46% \\n\",\n \"openssl-1.1.1c | 4.7 MB | ###### | 61% \\n\",\n \"openssl-1.1.1c | 4.7 MB | #######5 | 75% \\n\",\n \"openssl-1.1.1c | 4.7 MB | ######### | 90% \\n\",\n \"openssl-1.1.1c | 4.7 MB | ########## | 100% \\n\",\n \"\\n\",\n \"ca-certificates-2019 | 181 KB | | 0% \\n\",\n \"ca-certificates-2019 | 181 KB | ########## | 100% \\n\",\n \"\\n\",\n \"certifi-2019.9.11 | 155 KB | | 0% \\n\",\n \"certifi-2019.9.11 | 155 KB | ########## | 100% \\n\",\n \"\\n\",\n \"ffmpeg-4.2 | 23.4 MB | | 0% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | 2 | 2% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | 5 | 5% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | 8 | 8% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #1 | 11% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #4 | 14% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #7 | 17% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ## | 20% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ##3 | 23% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ##6 | 26% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ##9 | 29% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ###2 | 32% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ###5 | 35% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ###8 | 38% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ####1 | 41% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ####4 | 44% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ####7 | 47% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ##### | 50% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #####3 | 53% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #####6 | 56% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #####9 | 59% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ######2 | 62% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ######5 | 65% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ######8 | 68% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #######1 | 71% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #######4 | 74% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #######7 | 77% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ######## | 81% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ########3 | 84% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ########6 | 87% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ########9 | 90% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #########2 | 93% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #########5 | 96% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | #########8 | 99% \\n\",\n \"ffmpeg-4.2 | 23.4 MB | ########## | 100% \\n\",\n \"Preparing transaction: ...working... done\\n\",\n \"Verifying transaction: ...working... done\\n\",\n \"Executing transaction: ...working... done\\n\"\n ]\n }\n ],\n \"source\": [\n \"!conda install -c conda-forge ffmpeg -y\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Python 3.7.3\\n\"\n ]\n }\n ],\n \"source\": [\n \"!python --version\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"To start we'll need some basic libraries. First ``numpy`` will be needed for basic array manipulation. Since we will be visualising the results we will need ``matplotlib`` and ``seaborn``. Finally we will need ``umap`` for doing the dimension reduction itself.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Requirement already satisfied: numpy in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (1.17.1)\\n\",\n \"Requirement already satisfied: matplotlib in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (3.1.1)\\n\",\n \"Requirement already satisfied: seaborn in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (0.9.0)\\n\",\n \"Requirement already satisfied: umap-learn in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (0.3.10)\\n\",\n \"Requirement already satisfied: kiwisolver>=1.0.1 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from matplotlib) (1.1.0)\\n\",\n \"Requirement already satisfied: cycler>=0.10 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from matplotlib) (0.10.0)\\n\",\n \"Requirement already satisfied: python-dateutil>=2.1 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from matplotlib) (2.8.0)\\n\",\n \"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from matplotlib) (2.4.2)\\n\",\n \"Requirement already satisfied: pandas>=0.15.2 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from seaborn) (0.25.1)\\n\",\n \"Requirement already satisfied: scipy>=0.14.0 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from seaborn) (1.3.1)\\n\",\n \"Requirement already satisfied: scikit-learn>=0.16 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from umap-learn) (0.21.3)\\n\",\n \"Requirement already satisfied: numba>=0.37 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from umap-learn) (0.45.0)\\n\",\n \"Requirement already satisfied: setuptools in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from kiwisolver>=1.0.1->matplotlib) (41.0.1)\\n\",\n \"Requirement already satisfied: six in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from cycler>=0.10->matplotlib) (1.12.0)\\n\",\n \"Requirement already satisfied: pytz>=2017.2 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from pandas>=0.15.2->seaborn) (2019.2)\\n\",\n \"Requirement already satisfied: joblib>=0.11 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from scikit-learn>=0.16->umap-learn) (0.13.2)\\n\",\n \"Requirement already satisfied: llvmlite>=0.29.0dev0 in c:\\\\progams\\\\miniconda\\\\envs\\\\tf\\\\lib\\\\site-packages (from numba>=0.37->umap-learn) (0.29.0)\\n\"\n ]\n }\n ],\n \"source\": [\n \"!pip install numpy matplotlib seaborn umap-learn\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"To start let's load everything we'll need\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"%matplotlib inline\\n\",\n \"import numpy as np\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"from mpl_toolkits.axes_grid1 import make_axes_locatable\\n\",\n \"from matplotlib import animation\\n\",\n \"from IPython.display import HTML\\n\",\n \"import seaborn as sns\\n\",\n \"import itertools\\n\",\n \"sns.set(style='white', rc={'figure.figsize':(14, 12), 'animation.html': 'html5'})\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Ignore UserWarnings\\n\",\n \"import warnings\\n\",\n \"warnings.simplefilter('ignore', UserWarning)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from sklearn.datasets import load_digits\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from umap import UMAP\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"To try this out we'll needs a reasonably small dataset (so embedding runs don't take *too* long since we'll be doing a lot of them). For ease of reproducibility for everyone else I'll use the digits dataset from sklearn. If you want to try other datasets just drop them in here -- COIL20 might be interesting, or you might have your own data.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"array([[ 0., 0., 5., ..., 0., 0., 0.],\\n\",\n \" [ 0., 0., 0., ..., 10., 0., 0.],\\n\",\n \" [ 0., 0., 0., ..., 16., 9., 0.],\\n\",\n \" ...,\\n\",\n \" [ 0., 0., 1., ..., 6., 0., 0.],\\n\",\n \" [ 0., 0., 2., ..., 12., 0., 0.],\\n\",\n \" [ 0., 0., 10., ..., 12., 1., 0.]])\"\n ]\n },\n \"execution_count\": 9,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"digits = load_digits()\\n\",\n \"data = digits.data\\n\",\n \"data\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"We need to move the points in between the embeddings given by different parameter values. There are potentially fancy ways to do this (Something using rotation and reflection to get an initial alignment might be interesting), but we'll use straighforward linear interpolation between the two embeddings. To do this we'll need a simple function that can turn out intermediate embeddings for the in-between frames of the animation.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 10,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def tween(e1, e2, n_frames=20):\\n\",\n \" for i in range(5):\\n\",\n \" yield e1\\n\",\n \" for i in range(n_frames):\\n\",\n \" alpha = i / float(n_frames - 1)\\n\",\n \" yield (1 - alpha) * e1 + alpha * e2\\n\",\n \" for i in range(5):\\n\",\n \" yield(e2)\\n\",\n \" return\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now that we can fill in intermediate frame we just need to generate all the embeddings. We'll create a function that can take an argument and set of parameter values and then generate all the embeddings including the in-between frames.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 11,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def generate_frame_data(data, arg_name='n_neighbors', arg_list=[]):\\n\",\n \" result = []\\n\",\n \" es = []\\n\",\n \" for arg in arg_list:\\n\",\n \" kwargs = {arg_name:arg}\\n\",\n \" if len(es) > 0:\\n\",\n \" es.append(UMAP(init=es[-1], negative_sample_rate=3, **kwargs).fit_transform(data))\\n\",\n \" else:\\n\",\n \" es.append(UMAP(negative_sample_rate=3, **kwargs).fit_transform(data))\\n\",\n \" \\n\",\n \" for e1, e2 in zip(es[:-1], es[1:]):\\n\",\n \" result.extend(list(tween(e1, e2)))\\n\",\n \" \\n\",\n \" return result\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Next we just need to create a function to actually generate the animation given a list of embeddings (one for each frame). This is really just a matter of workign through the details of how matplotlib generates animations -- I would refer you again to Jake's tutorial if you are interested in the detailed mechanics of this.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 12,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def create_animation(frame_data, arg_name='n_neighbors', arg_list=[]):\\n\",\n \" fig, ax = plt.subplots()\\n\",\n \" all_data = np.vstack(frame_data)\\n\",\n \" frame_bounds = (all_data[:, 0].min() * 1.1, \\n\",\n \" all_data[:, 0].max() * 1.1,\\n\",\n \" all_data[:, 1].min() * 1.1, \\n\",\n \" all_data[:, 1].max() * 1.1)\\n\",\n \" ax.set_xlim(frame_bounds[0], frame_bounds[1])\\n\",\n \" ax.set_ylim(frame_bounds[2], frame_bounds[3])\\n\",\n \" points = ax.scatter(frame_data[0][:, 0], frame_data[0][:, 1], \\n\",\n \" s=5, c=digits.target, cmap='Spectral', animated=True)\\n\",\n \" title = ax.set_title('', fontsize=24)\\n\",\n \" ax.set_xticks([])\\n\",\n \" ax.set_yticks([])\\n\",\n \"\\n\",\n \" cbar = fig.colorbar(\\n\",\n \" points,\\n\",\n \" cax=make_axes_locatable(ax).append_axes(\\\"right\\\", size=\\\"5%\\\", pad=0.05),\\n\",\n \" orientation=\\\"vertical\\\",\\n\",\n \" values=np.arange(10),\\n\",\n \" boundaries=np.arange(11)-0.5,\\n\",\n \" ticks=np.arange(10),\\n\",\n \" drawedges=True,\\n\",\n \" )\\n\",\n \" cbar.ax.yaxis.set_ticklabels(np.arange(10), fontsize=18)\\n\",\n \"\\n\",\n \" def init():\\n\",\n \" points.set_offsets(frame_data[0])\\n\",\n \" arg = arg_list[0]\\n\",\n \" arg_str = f'{arg:.3f}' if isinstance(arg, float) else f'{arg}'\\n\",\n \" title.set_text(f'UMAP with {arg_name}={arg_str}')\\n\",\n \" return (points,)\\n\",\n \"\\n\",\n \" def animate(i):\\n\",\n \" points.set_offsets(frame_data[i])\\n\",\n \" if (i + 15) % 30 == 0:\\n\",\n \" arg = arg_list[(i + 15) // 30]\\n\",\n \" arg_str = f'{arg:.3f}' if isinstance(arg, float) else f'{arg}'\\n\",\n \" title.set_text(f'UMAP with {arg_name}={arg_str}')\\n\",\n \" return (points,)\\n\",\n \"\\n\",\n \" anim = animation.FuncAnimation(fig, animate, init_func=init, frames=len(frame_data), interval=20, blit=True)\\n\",\n \" plt.close()\\n\",\n \" return anim\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Finally a little bit of glue to make it all go together.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 13,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def animate_param(data, arg_name='n_neighbors', arg_list=[]):\\n\",\n \" frame_data = generate_frame_data(data, arg_name, arg_list)\\n\",\n \" return create_animation(frame_data, arg_name, arg_list)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"Now we can create an animation. It will be embedded as an HTML5 video into this notebook.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 14,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/html\": [\n \"\"\n ],\n \"text/plain\": [\n \"\\\\n\\\"+\\n\",\n \" \\\"
\\\"}};\\n\",\n \"\\n\",\n \" function display_loaded() {\\n\",\n \" var el = document.getElementById(\\\"1001\\\");\\n\",\n \" if (el != null) {\\n\",\n \" el.textContent = \\\"BokehJS is loading...\\\";\\n\",\n \" }\\n\",\n \" if (root.Bokeh !== undefined) {\\n\",\n \" if (el != null) {\\n\",\n \" el.textContent = \\\"BokehJS \\\" + root.Bokeh.version + \\\" successfully loaded.\\\";\\n\",\n \" }\\n\",\n \" } else if (Date.now() < root._bokeh_timeout) {\\n\",\n \" setTimeout(display_loaded, 100)\\n\",\n \" }\\n\",\n \" }\\n\",\n \"\\n\",\n \"\\n\",\n \" function run_callbacks() {\\n\",\n \" try {\\n\",\n \" root._bokeh_onload_callbacks.forEach(function(callback) {\\n\",\n \" if (callback != null)\\n\",\n \" callback();\\n\",\n \" });\\n\",\n \" } finally {\\n\",\n \" delete root._bokeh_onload_callbacks\\n\",\n \" }\\n\",\n \" console.debug(\\\"Bokeh: all callbacks have finished\\\");\\n\",\n \" }\\n\",\n \"\\n\",\n \" function load_libs(css_urls, js_urls, callback) {\\n\",\n \" if (css_urls == null) css_urls = [];\\n\",\n \" if (js_urls == null) js_urls = [];\\n\",\n \"\\n\",\n \" root._bokeh_onload_callbacks.push(callback);\\n\",\n \" if (root._bokeh_is_loading > 0) {\\n\",\n \" console.debug(\\\"Bokeh: BokehJS is being loaded, scheduling callback at\\\", now());\\n\",\n \" return null;\\n\",\n \" }\\n\",\n \" if (js_urls == null || js_urls.length === 0) {\\n\",\n \" run_callbacks();\\n\",\n \" return null;\\n\",\n \" }\\n\",\n \" console.debug(\\\"Bokeh: BokehJS not loaded, scheduling load and callback at\\\", now());\\n\",\n \" root._bokeh_is_loading = css_urls.length + js_urls.length;\\n\",\n \"\\n\",\n \" function on_load() {\\n\",\n \" root._bokeh_is_loading--;\\n\",\n \" if (root._bokeh_is_loading === 0) {\\n\",\n \" console.debug(\\\"Bokeh: all BokehJS libraries/stylesheets loaded\\\");\\n\",\n \" run_callbacks()\\n\",\n \" }\\n\",\n \" }\\n\",\n \"\\n\",\n \" function on_error() {\\n\",\n \" console.error(\\\"failed to load \\\" + url);\\n\",\n \" }\\n\",\n \"\\n\",\n \" for (var i = 0; i < css_urls.length; i++) {\\n\",\n \" var url = css_urls[i];\\n\",\n \" const element = document.createElement(\\\"link\\\");\\n\",\n \" element.onload = on_load;\\n\",\n \" element.onerror = on_error;\\n\",\n \" element.rel = \\\"stylesheet\\\";\\n\",\n \" element.type = \\\"text/css\\\";\\n\",\n \" element.href = url;\\n\",\n \" console.debug(\\\"Bokeh: injecting link tag for BokehJS stylesheet: \\\", url);\\n\",\n \" document.body.appendChild(element);\\n\",\n \" }\\n\",\n \"\\n\",\n \" for (var i = 0; i < js_urls.length; i++) {\\n\",\n \" var url = js_urls[i];\\n\",\n \" var element = document.createElement('script');\\n\",\n \" element.onload = on_load;\\n\",\n \" element.onerror = on_error;\\n\",\n \" element.async = false;\\n\",\n \" element.src = url;\\n\",\n \" console.debug(\\\"Bokeh: injecting script tag for BokehJS library: \\\", url);\\n\",\n \" document.head.appendChild(element);\\n\",\n \" }\\n\",\n \" };var element = document.getElementById(\\\"1001\\\");\\n\",\n \" if (element == null) {\\n\",\n \" console.error(\\\"Bokeh: ERROR: autoload.js configured with elementid '1001' but no matching script tag was found. \\\")\\n\",\n \" return false;\\n\",\n \" }\\n\",\n \"\\n\",\n \" function inject_raw_css(css) {\\n\",\n \" const element = document.createElement(\\\"style\\\");\\n\",\n \" element.appendChild(document.createTextNode(css));\\n\",\n \" document.body.appendChild(element);\\n\",\n \" }\\n\",\n \"\\n\",\n \" \\n\",\n \" var js_urls = [];\\n\",\n \" var css_urls = [];\\n\",\n \" \\n\",\n \"\\n\",\n \" var inline_js = [\\n\",\n \" function(Bokeh) {\\n\",\n \" /* BEGIN bokeh.min.js */\\n\",\n \" /*!\\n\",\n \" * Copyright (c) 2012 - 2019, Anaconda, Inc., and Bokeh Contributors\\n\",\n \" * All rights reserved.\\n\",\n \" * \\n\",\n \" * Redistribution and use in source and binary forms, with or without modification,\\n\",\n \" * are permitted provided that the following conditions are met:\\n\",\n \" * \\n\",\n \" * Redistributions of source code must retain the above copyright notice,\\n\",\n \" * this list of conditions and the following disclaimer.\\n\",\n \" * \\n\",\n \" * Redistributions in binary form must reproduce the above copyright notice,\\n\",\n \" * this list of conditions and the following disclaimer in the documentation\\n\",\n \" * and/or other materials provided with the distribution.\\n\",\n \" * \\n\",\n \" * Neither the name of Anaconda nor the names of any contributors\\n\",\n \" * may be used to endorse or promote products derived from this software\\n\",\n \" * without specific prior written permission.\\n\",\n \" * \\n\",\n \" * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \\\"AS IS\\\"\\n\",\n \" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\\n\",\n \" * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\\n\",\n \" * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE\\n\",\n \" * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\\n\",\n \" * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\\n\",\n \" * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\\n\",\n \" * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\\n\",\n \" * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\\n\",\n \" * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF\\n\",\n \" * THE POSSIBILITY OF SUCH DAMAGE.\\n\",\n \" */\\n\",\n \" (function(root, factory) {\\n\",\n \" root[\\\"Bokeh\\\"] = factory();\\n\",\n \" })(this, function() {\\n\",\n \" var define;\\n\",\n \" var parent_require = typeof require === \\\"function\\\" && require\\n\",\n \" return (function(modules, entry, aliases, externals) {\\n\",\n \" if (aliases === undefined) aliases = {};\\n\",\n \" if (externals === undefined) externals = {};\\n\",\n \"\\n\",\n \" var cache = {};\\n\",\n \"\\n\",\n \" var normalize = function(name) {\\n\",\n \" if (typeof name === \\\"number\\\")\\n\",\n \" return name;\\n\",\n \"\\n\",\n \" if (name === \\\"bokehjs\\\")\\n\",\n \" return entry;\\n\",\n \"\\n\",\n \" var prefix = \\\"@bokehjs/\\\"\\n\",\n \" if (name.slice(0, prefix.length) === prefix)\\n\",\n \" name = name.slice(prefix.length)\\n\",\n \"\\n\",\n \" var alias = aliases[name]\\n\",\n \" if (alias != null)\\n\",\n \" return alias;\\n\",\n \"\\n\",\n \" var trailing = name.length > 0 && name[name.lenght-1] === \\\"/\\\";\\n\",\n \" var index = aliases[name + (trailing ? \\\"\\\" : \\\"/\\\") + \\\"index\\\"];\\n\",\n \" if (index != null)\\n\",\n \" return index;\\n\",\n \"\\n\",\n \" return name;\\n\",\n \" }\\n\",\n \"\\n\",\n \" var require = function(name) {\\n\",\n \" var mod = cache[name];\\n\",\n \" if (!mod) {\\n\",\n \" var id = normalize(name);\\n\",\n \"\\n\",\n \" mod = cache[id];\\n\",\n \" if (!mod) {\\n\",\n \" if (!modules[id]) {\\n\",\n \" if (parent_require && externals[id]) {\\n\",\n \" try {\\n\",\n \" mod = {exports: parent_require(id)};\\n\",\n \" cache[id] = cache[name] = mod;\\n\",\n \" return mod.exports;\\n\",\n \" } catch (e) {}\\n\",\n \" }\\n\",\n \"\\n\",\n \" var err = new Error(\\\"Cannot find module '\\\" + name + \\\"'\\\");\\n\",\n \" err.code = 'MODULE_NOT_FOUND';\\n\",\n \" throw err;\\n\",\n \" }\\n\",\n \"\\n\",\n \" mod = {exports: {}};\\n\",\n \" cache[id] = cache[name] = mod;\\n\",\n \" modules[id].call(mod.exports, require, mod, mod.exports);\\n\",\n \" } else\\n\",\n \" cache[name] = mod;\\n\",\n \" }\\n\",\n \"\\n\",\n \" return mod.exports;\\n\",\n \" }\\n\",\n \"\\n\",\n \" var main = require(entry);\\n\",\n \" main.require = require;\\n\",\n \"\\n\",\n \" main.register_plugin = function(plugin_modules, plugin_entry, plugin_aliases, plugin_externals) {\\n\",\n \" if (plugin_aliases === undefined) plugin_aliases = {};\\n\",\n \" if (plugin_externals === undefined) plugin_externals = {};\\n\",\n \"\\n\",\n \" for (var name in plugin_modules) {\\n\",\n \" modules[name] = plugin_modules[name];\\n\",\n \" }\\n\",\n \"\\n\",\n \" for (var name in plugin_aliases) {\\n\",\n \" aliases[name] = plugin_aliases[name];\\n\",\n \" }\\n\",\n \"\\n\",\n \" for (var name in plugin_externals) {\\n\",\n \" externals[name] = plugin_externals[name];\\n\",\n \" }\\n\",\n \"\\n\",\n \" var plugin = require(plugin_entry);\\n\",\n \"\\n\",\n \" for (var name in plugin) {\\n\",\n \" main[name] = plugin[name];\\n\",\n \" }\\n\",\n \"\\n\",\n \" return plugin;\\n\",\n \" }\\n\",\n \"\\n\",\n \" return main;\\n\",\n \" })\\n\",\n \" ([\\n\",\n \" function _(n,o,r){n(1),function(n){for(var o in n)r.hasOwnProperty(o)||(r[o]=n[o])}(n(102))},\\n\",\n \" function _(n,c,f){n(2),n(11),n(14),n(21),n(49),n(52),n(87),n(94),n(100)},\\n\",\n \" function _(e,n,a){e(3)()||Object.defineProperty(Object,\\\"assign\\\",{value:e(4),configurable:!0,enumerable:!1,writable:!0})},\\n\",\n \" function _(r,t,o){t.exports=function(){var r,t=Object.assign;return\\\"function\\\"==typeof t&&(t(r={foo:\\\"raz\\\"},{bar:\\\"dwa\\\"},{trzy:\\\"trzy\\\"}),r.foo+r.bar+r.trzy===\\\"razdwatrzy\\\")}},\\n\",\n \" function _(t,r,n){var o=t(5),c=t(10),a=Math.max;r.exports=function(t,r){var n,f,h,i=a(arguments.length,2);for(t=Object(c(t)),h=function(o){try{t[o]=r[o]}catch(t){n||(n=t)}},f=1;f\\\\n\\\"+\\n\",\n \" \\\"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\\\n\\\"+\\n\",\n \" \\\"may be due to a slow or bad network connection. Possible fixes:\\\\n\\\"+\\n\",\n \" \\\"
\\\\n\\\"+\\n\",\n \" \\\"- \\\\n\\\"+\\n\",\n \" \\\"
- re-rerun `output_notebook()` to attempt to load from CDN again, or \\\\n\\\"+\\n\",\n \" \\\"
- use INLINE resources instead, as so: \\\\n\\\"+\\n\",\n \" \\\"
\\\\n\\\"+\\n\",\n \" \\\"from bokeh.resources import INLINE\\\\n\\\"+\\n\",\n \" \\\"output_notebook(resources=INLINE)\\\\n\\\"+\\n\",\n \" \\\"\\\\n\\\"+\\n\",\n \" \\\"