Repository: google-research/talk-like-a-graph
Branch: main
Commit: 36af51e19ef7
Files: 20
Total size: 340.3 KB
Directory structure:
gitextract_nuchrmnk/
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── __init__.py
├── talk_like_a_graph/
│ ├── README.md
│ ├── __init__.py
│ ├── graph_generators.py
│ ├── graph_generators_runner.py
│ ├── graph_generators_test.py
│ ├── graph_metrics.py
│ ├── graph_metrics_test.py
│ ├── graph_tasks.py
│ ├── graph_tasks_generator.py
│ ├── graph_tasks_utils.py
│ ├── graph_text_encoders.py
│ ├── graph_text_encoders_test.py
│ └── name_dictionaries.py
└── tutorial/
├── KDD-Tutorial-1-Talk-Like-a-Graph.ipynb
├── KDD-Tutorial-2-Let-Your-Graph-Do-The-Talking.ipynb
└── README.md
================================================
FILE CONTENTS
================================================
================================================
FILE: CONTRIBUTING.md
================================================
# How to Contribute
## Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution,
this simply gives us permission to use and redistribute your contributions as
part of the project. Head over to to see
your current agreements on file or to sign a new one.
You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Community Guidelines
This project follows [Google's Open Source Community
Guidelines](https://opensource.google/conduct/).
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
# Encoding Graphs and Structured Data in Language Models
This repository contains code for
[Talk like a Graph: Encoding Graphs for Large Language Models](https://arxiv.org/abs/2310.04560)
and
[Let Your Graph Do the Talking: Encoding Structured Data for LLMs](https://arxiv.org/abs/2402.05862).
## Cite us
If you use this package for published work, please cite the following.
For the "Talk like a Graph" project:
```
@inproceedigs{fatemi2024talk,
title={Talk like a Graph: Encoding Graphs for Large Language Models},
author={Bahare Fatemi and Jonathan Halcrow and Bryan Perozzi},
booktitle={International Conference on Learning Representations (ICLR)},
year={2024}
}
```
For Graph Token (a.k.a "Let Your Graph Do the Talking"):
```
@misc{perozzi2024letgraphtalkingencoding,
title={Let Your Graph Do the Talking: Encoding Structured Data for LLMs},
author={Bryan Perozzi and Bahare Fatemi and Dustin Zelle and Anton Tsitsulin and Mehran Kazemi and Rami Al-Rfou and Jonathan Halcrow},
year={2024},
eprint={2402.05862},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2402.05862},
}
```
## Disclaimer
This is not an official Google product.
================================================
FILE: __init__.py
================================================
================================================
FILE: talk_like_a_graph/README.md
================================================
# Using Large Language Models to Solve Graph Problems
This repository contains the code to generate graph reasoning problems with
different graph generator algorithms and graph encoding methods, as well as
different prompting techniques.
The graph tasks are `edge existence`, `node degree`, `node count`, `edge count`,
`connected nodes`, `disconnected nodes`, `cycle check`, `reachability`,
`shortest path`, `maximum flow`, `node classification`, and `triangle counting`.
The datasets used here are proposed in our paper:
[Talk like a Graph: Encoding Graphs for Large Language Models](https://arxiv.org/abs/2310.04560).
### Generating graphs
```sh
./graphqa/graph_generator.sh
```
### Generating files for tasks
```sh
./graphqa/task_generator.sh
```
## Contact us
For questions or comments about the implementation, please contact
baharef@google.com.
## Cite us
If you use this package for published work, please cite the following:
```
@inproceedigs{fatemi2024talk,
title={Talk like a Graph: Encoding Graphs for Large Language Models},
author={Bahare Fatemi and Jonathan Halcrow and Bryan Perozzi},
booktitle={International Conference on Learning Representations (ICLR)},
year={2024}
}
```
## Disclaimer
This is not an official Google product.
# Placeholder for internal data notes.
================================================
FILE: talk_like_a_graph/__init__.py
================================================
================================================
FILE: talk_like_a_graph/graph_generators.py
================================================
r"""Random graph generation."""
import random
import networkx as nx
import numpy as np
_NUMBER_OF_NODES_RANGE = {
"small": np.arange(5, 10),
"medium": np.arange(10, 15),
"large": np.arange(15, 20),
}
_NUMBER_OF_COMMUNITIES_RANGE = {
"small": np.arange(2, 4),
"medium": np.arange(2, 8),
"large": np.arange(2, 10),
}
def generate_graphs(
number_of_graphs: int,
algorithm: str,
directed: bool,
random_seed: int = 1234,
er_min_sparsity: float = 0.0,
er_max_sparsity: float = 1.0,
) -> list[nx.Graph]:
"""Generating multiple graphs using the provided algorithms.
Args:
number_of_graphs: number of graphs to generate
algorithm: the random graph generator algorithm
directed: whether to generate directed or undirected graphs.
random_seed: the random seed to generate graphs with.
er_min_sparsity: minimum sparsity of er graphs.
er_max_sparsity: maximum sparsity of er graphs.
Returns:
generated_graphs: a list of nx graphs.
Raises:
NotImplementedError: if the algorithm is not yet implemented.
"""
random.seed(random_seed)
np.random.seed(random_seed)
generated_graphs = []
graph_sizes = random.choices(
list(_NUMBER_OF_NODES_RANGE.keys()), k=number_of_graphs
)
random_state = np.random.RandomState(random_seed)
if algorithm == "er":
for i in range(number_of_graphs):
sparsity = random.uniform(er_min_sparsity, er_max_sparsity)
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
generated_graphs.append(
nx.erdos_renyi_graph(
number_of_nodes, sparsity, seed=random_state, directed=directed
)
)
elif algorithm == "ba":
for i in range(number_of_graphs):
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
m = random.randint(1, number_of_nodes - 1)
generated_graph = nx.barabasi_albert_graph(
number_of_nodes, m, seed=random_state
)
if directed:
generated_graphs.append(randomize_directions(generated_graph))
else:
generated_graphs.append(generated_graph)
elif algorithm == "sbm":
for i in range(number_of_graphs):
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
number_of_communities = random.choice(
_NUMBER_OF_COMMUNITIES_RANGE[graph_sizes[i]]
)
# sizes forms number of nodes in communities.
sizes = []
for _ in range(number_of_communities - 1):
sizes.append(
random.randint(
1,
max(
1,
number_of_nodes - sum(sizes) - (number_of_communities - 1),
),
)
)
sizes.append(number_of_nodes - sum(sizes))
# p forms probabilities of communities connecting each other.
p = np.random.uniform(size=(number_of_communities, number_of_communities))
if random.uniform(0, 1) < 0.5:
p = np.maximum(p, p.transpose())
else:
p = np.minimum(p, p.transpose())
sbm_graph = nx.stochastic_block_model(
sizes, p, seed=random_state, directed=directed
)
# sbm graph generator automatically adds dictionary attributes.
sbm_graph = remove_graph_data(sbm_graph)
generated_graphs.append(sbm_graph)
elif algorithm == "sfn":
for i in range(number_of_graphs):
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
generated_graph = nx.scale_free_graph(number_of_nodes, seed=random_state)
# sfn graphs are by defaukt directed.
if not directed:
generated_graphs.append(remove_directions(generated_graph))
else:
generated_graphs.append(generated_graph)
elif algorithm == "complete":
for i in range(number_of_graphs):
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
create_using = nx.DiGraph if directed else nx.Graph
generated_graphs.append(
nx.complete_graph(number_of_nodes, create_using=create_using)
)
elif algorithm == "star":
for i in range(number_of_graphs):
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
# number_of_nodes for star is the input + a center node.
generated_graph = nx.star_graph(number_of_nodes - 1)
if directed:
generated_graphs.append(randomize_directions(generated_graph))
else:
generated_graphs.append(generated_graph)
elif algorithm == "path":
for i in range(number_of_graphs):
number_of_nodes = random.choice(_NUMBER_OF_NODES_RANGE[graph_sizes[i]])
create_using = nx.DiGraph if directed else nx.Graph
generated_graphs.append(
nx.path_graph(number_of_nodes, create_using=create_using)
)
else:
raise NotImplementedError()
return generated_graphs
def remove_graph_data(graph: nx.Graph) -> nx.Graph:
# GraphML writer does not support dictionary data for nodes or graphs.
for ind in range((graph.number_of_nodes())):
graph.nodes[ind].pop("block", None)
graph_data_keys = list(graph.graph.keys())
for _, node in enumerate(graph_data_keys):
graph.graph.pop(node, None)
return graph
def randomize_directions(graph: nx.Graph) -> nx.DiGraph:
# Converting the undirected graph to a directed graph.
directed_graph = graph.to_directed()
# For each edge, randomly choose a direction.
edges = list(graph.edges())
for u, v in edges:
if random.random() < 0.5:
directed_graph.remove_edge(u, v)
else:
directed_graph.remove_edge(v, u)
return directed_graph
def remove_directions(graph: nx.Graph) -> nx.Graph:
# Converting the direted graph to an undirected one by removing directions.
undirected_graph = nx.Graph()
undirected_graph.add_nodes_from(graph.nodes())
# Add edges between nodes, ignoring directions.
for u, v in graph.edges():
undirected_graph.add_edge(u, v)
return undirected_graph
================================================
FILE: talk_like_a_graph/graph_generators_runner.py
================================================
r"""Random graph generation.
This code generates random graph using different algorithms.
# Placeholder for Google-internal comments.
"""
from collections.abc import Sequence
import os
from absl import app
from absl import flags
import networkx as nx
# Internal import.
from . import graph_generators
_ALGORITHM = flags.DEFINE_string(
"algorithm",
None,
"The graph generating algorithm to use.",
required=True,
)
_NUMBER_OF_GRAPHS = flags.DEFINE_integer(
"number_of_graphs",
None,
"The number of graphs to generate.",
required=True,
)
_DIRECTED = flags.DEFINE_bool(
"directed", False, "Whether to generate directed graphs."
)
_OUTPUT_PATH = flags.DEFINE_string(
"output_path", None, "The output path to write the graphs.", required=True
)
_SPLIT = flags.DEFINE_string(
"split", None, "The dataset split to generate.", required=True
)
_MIN_SPARSITY = flags.DEFINE_float("min_sparsity", 0.0, "The minimum sparsity.")
_MAX_SPARSITY = flags.DEFINE_float("max_sparsity", 1.0, "The maximum sparsity.")
def write_graphs(graphs: list[nx.Graph], output_dir: str) -> None:
if not os.path.isdir(output_dir):
os.makedirs(output_dir)
for ind, graph in enumerate(graphs):
nx.write_graphml(
graph,
os.Open(
os.path.join(output_dir, str(ind) + ".graphml"),
"wb",
),
)
def main(argv: Sequence[str]) -> None:
if len(argv) > 1:
raise app.UsageError("Too many command-line arguments.")
if _SPLIT.value == "train":
random_seed = 9876
elif _SPLIT.value == "test":
random_seed = 1234
elif _SPLIT.value == "validation":
random_seed = 5432
else:
raise NotImplementedError()
generated_graphs = graph_generators.generate_graphs(
number_of_graphs=_NUMBER_OF_GRAPHS.value,
algorithm=_ALGORITHM.value,
directed=_DIRECTED.value,
random_seed=random_seed,
er_min_sparsity=_MIN_SPARSITY.value,
er_max_sparsity=_MAX_SPARSITY.value,
)
write_graphs(
graphs=generated_graphs,
output_dir=os.path.join(
_OUTPUT_PATH.value,
"directed" if _DIRECTED.value else "undirected",
_ALGORITHM.value,
_SPLIT.value,
),
)
if __name__ == "__main__":
app.run(main)
================================================
FILE: talk_like_a_graph/graph_generators_test.py
================================================
from absl.testing import parameterized
from . import graph_generators
from absl.testing import absltest
class GraphGenerationTest(absltest.TestCase, parameterized.TestCase):
@parameterized.named_parameters(
dict(
testcase_name='er_undirected_1',
algorithm='er',
directed=False,
k=1,
),
dict(
testcase_name='er_directed_1',
algorithm='er',
directed=True,
k=1,
),
dict(
testcase_name='ba_undirected_5',
algorithm='ba',
directed=False,
k=5,
),
dict(
testcase_name='ba_directed_5',
algorithm='ba',
directed=True,
k=5,
),
)
def test_number_of_graphs(self, algorithm, directed, k):
generated_graph = graph_generators.generate_graphs(k, algorithm, directed)
self.assertLen(generated_graph, k)
@parameterized.named_parameters(
dict(
testcase_name='er_undirected',
algorithm='er',
directed=False,
),
dict(
testcase_name='er_directed',
algorithm='er',
directed=True,
),
dict(
testcase_name='ba_undirected',
algorithm='ba',
directed=False,
),
dict(
testcase_name='ba_directed',
algorithm='ba',
directed=True,
),
dict(
testcase_name='sbm_undirected',
algorithm='sbm',
directed=False,
),
dict(
testcase_name='sbm_directed',
algorithm='sbm',
directed=True,
),
dict(
testcase_name='sfn_undirected',
algorithm='sfn',
directed=False,
),
dict(
testcase_name='sfn_directed',
algorithm='sfn',
directed=True,
),
dict(
testcase_name='complete_undirected',
algorithm='complete',
directed=False,
),
dict(
testcase_name='complete_directed',
algorithm='complete',
directed=True,
),
dict(
testcase_name='star_undirected',
algorithm='star',
directed=False,
),
dict(
testcase_name='star_directed',
algorithm='star',
directed=True,
),
dict(
testcase_name='path_undirected',
algorithm='path',
directed=False,
),
dict(
testcase_name='path_directed',
algorithm='path',
directed=True,
),
)
def test_directions(self, algorithm, directed):
generated_graph = graph_generators.generate_graphs(1, algorithm, directed)
self.assertEqual(generated_graph[0].is_directed(), directed)
if __name__ == '__main__':
googletest.main()
================================================
FILE: talk_like_a_graph/graph_metrics.py
================================================
"""Metrics for seqio tasks over graph data.
This module contains definitions of metric_fns to be used for scoring
graph tasks from nlgraph and graphqa.
"""
from typing import Mapping, Sequence
def yes_no_accuracy(
targets: Sequence[str], predictions: Sequence[str]
) -> Mapping[str, float]:
"""Assesses the accuracy of LLM outputs on Yes/No tasks.
Targets must contain either the word 'yes' or the word 'no' but not both.
Predictions are binarized by checking for 'yes' or 'no' in the first line.
Args:
targets: The expected output strings.
predictions: The LLM outputs.
Returns:
Returns a dict of the following metrics:
yes_no_accuracy: The % where the target and prediction match.
yes_no_ambiguous: The % where the prediction contained yes and no
yes_no_indeterminate: The % where the prediction contained neither yes nor
no
Raises:
ValueError: If a target string contains 'yes' and 'no'
"""
num_correct = 0
num_ambiguous = 0
num_indeterminate = 0
for target, prediction in zip(targets, predictions):
normalized_target = target.lower()
binarized_target = 'yes' in normalized_target
print(binarized_target)
if binarized_target and 'no' in normalized_target:
raise ValueError(f'Ambiguous target string, {target}')
if not binarized_target and 'no' not in normalized_target:
raise ValueError(f'Indeterminate target string, {target}')
normalized_prediction = prediction.splitlines()
if not normalized_prediction:
normalized_prediction = ''
else:
normalized_prediction = normalized_prediction[0]
normalized_prediction = normalized_prediction.lower()
binarized_prediction = 'yes' in normalized_prediction.lower()
print(normalized_prediction)
if binarized_prediction and 'no' in normalized_prediction:
num_ambiguous += 1
continue
if not binarized_prediction and 'no' not in normalized_prediction:
num_indeterminate += 1
continue
if binarized_prediction == binarized_target:
num_correct += 1
return {
'yes_no_accuracy': num_correct / len(targets),
'yes_no_ambiguous': num_ambiguous / len(targets),
'yes_no_indeterminate': num_indeterminate / len(targets),
}
================================================
FILE: talk_like_a_graph/graph_metrics_test.py
================================================
from . import graph_metrics
from absl.testing import absltest
class GraphTasksTest(absltest.TestCase):
def test_yes_no_correct(self):
result = graph_metrics.yes_no_accuracy(
targets=["Yes", "The answer is yes.", "No", "The answer is no.", "No"],
predictions=[
"yes",
"Yes\nNo",
"No\nYes",
"That's gonna be no from me.",
"yes",
],
)
self.assertEqual(result["yes_no_ambiguous"], 0)
self.assertEqual(result["yes_no_indeterminate"], 0)
self.assertAlmostEqual(result["yes_no_accuracy"], 0.8)
def test_yes_no_ambiguous(self):
result = graph_metrics.yes_no_accuracy(
targets=["Yes", "The answer is yes.", "No", "The answer is no.", "No"],
predictions=[
"yes",
"Yes No",
"No Yes",
"That's gonna be no from me.",
"yes",
],
)
self.assertEqual(result["yes_no_ambiguous"], 0.4)
self.assertEqual(result["yes_no_indeterminate"], 0)
self.assertAlmostEqual(result["yes_no_accuracy"], 0.4)
def test_yes_no_indeterminate(self):
result = graph_metrics.yes_no_accuracy(
targets=["Yes", "The answer is yes.", "No", "The answer is no.", "No"],
predictions=[
"yes",
"\n No",
"",
"That's gonna be no from me.",
"yes",
],
)
self.assertEqual(result["yes_no_ambiguous"], 0)
self.assertEqual(result["yes_no_indeterminate"], 0.4)
self.assertAlmostEqual(result["yes_no_accuracy"], 0.4)
def test_yes_no_accuracy_raises_on_ambiguous_target(self):
with self.assertRaises(ValueError):
graph_metrics.yes_no_accuracy(
targets=["Yes but maybe no"],
predictions=["yes"],
)
def test_yes_no_accuracy_raises_on_indeterminate_target(self):
with self.assertRaises(ValueError):
graph_metrics.yes_no_accuracy(
targets=["Hmm?"],
predictions=["yes"],
)
if __name__ == "__main__":
googletest.main()
================================================
FILE: talk_like_a_graph/graph_tasks.py
================================================
"""The graph tasks to be tried with LLMs."""
import random
import networkx as nx
import numpy as np
from . import graph_text_encoders
class GraphTask:
"""The parent class for all the graph tasks."""
def __init__(self):
self.name = 'default'
self.maximum_nnodes_cot_graph = 10
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
raise NotImplementedError()
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
):
raise NotImplementedError()
class CycleCheck(GraphTask):
"""The graph task to check if there is at least one cycle or not."""
def __init__(self):
super().__init__()
self.name = 'cycle_check'
self._task_description = 'Q: Is there a cycle in this graph?\nA: '
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
for ind, graph in enumerate(graphs):
question = (
graph_text_encoders.encode_graph(graph, encoding_method)
+ self._task_description
)
try:
nx.find_cycle(graph)
answer = 'Yes, there is a cycle.'
except nx.NetworkXNoCycle:
answer = 'No, there is no cycle.'
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': self._task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
"""Create a few shot example w or w/o cot for the graph graph."""
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = (
graph_text_encoders.encode_graph(graph, encoding_method)
+ self._task_description
)
try:
cycle = nx.find_cycle(graph)
cycle_text = ''
answer = 'Yes, there is a cycle. '
if cot:
for pair in cycle:
cycle_text += (
name_dict[pair[0]]
+ ' is connected to '
+ name_dict[pair[1]]
+ ', '
)
cycle_cot = 'The cycle is: %s.' % cycle_text[:-2]
answer += cycle_cot
except nx.NetworkXNoCycle:
answer = 'No, there is no cycle.'
return question + answer
def choose_few_shot_examples(
self,
few_shots_dict: dict[tuple[str, str], list[str]],
encoding_method: str,
k: int = 2,
) -> str:
"""Choose few shot examples for each algorithm."""
pos_cycle_algorithms = ['er', 'ba', 'sbm', 'sfn', 'complete']
neg_cycle_algorithms = ['star', 'path']
few_shots_str = ''
# choose k-1 shots for pos algorithms and one negative.
positive_algorithms = random.choices(pos_cycle_algorithms, k=k - 1)
for positive_algorithm in positive_algorithms:
example_list = few_shots_dict[(positive_algorithm, encoding_method)]
few_shots_str += 'Example: ' + random.choice(example_list) + '\n'
negative_algorithm = random.choice(neg_cycle_algorithms)
example_list = few_shots_dict[(negative_algorithm, encoding_method)]
few_shots_str += 'Example: ' + random.choice(example_list) + '\n'
return few_shots_str
class EdgeExistence(GraphTask):
"""The graph task to check if an edge exist in a graph or not."""
def __init__(self):
super().__init__()
self.name = 'edge_existence'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
task_description = 'Q: Is node %s connected to node %s?\nA: ' % (
name_dict[source],
name_dict[target],
)
question += task_description
if ((source, target) in graph.edges()) or (
(target, source) in graph.edges()
):
answer = 'Yes.'
else:
answer = 'No.'
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source, target],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += 'Q: Is node %s connected to node %s?\nA: ' % (
name_dict[source],
name_dict[target],
)
if ((source, target) in graph.edges()) or (
(target, source) in graph.edges()
):
answer = 'Yes.'
if cot:
answer += (
' Because, there is an edge from %s to %s in the graph description.'
% (name_dict[source], name_dict[target])
)
else:
answer = 'No.'
if cot:
answer += (
' Because, there is no edge from %s to %s in the graph description.'
% (name_dict[source], name_dict[target])
)
return question + answer
class NodeCount(GraphTask):
"""The graph task for finding number of nodes in a graph."""
def __init__(self):
super().__init__()
self.name = 'node_count'
self._task_description = 'Q: How many nodes are in this graph?\nA: '
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
for ind, graph in enumerate(graphs):
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += self._task_description
answer = ' %d.' % len(graph.nodes())
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': self._task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [],
}
return examples_dict
def get_nodes_string(self, name_dict: dict[int, str], nnodes: int) -> str:
node_string = ''
for i in range(nnodes - 1):
node_string += name_dict[i] + ', '
node_string += 'and ' + name_dict[nnodes - 1]
return node_string
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += self._task_description
answer = '%d.' % len(graph.nodes())
if cot:
answer += ' The nodes are %s.' % self.get_nodes_string(
name_dict, len(graph.nodes())
)
return question + answer
class NodeDegree(GraphTask):
"""The graph task for finding degree of a node in a graph."""
def __init__(self):
super().__init__()
self.name = 'node_degree'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
question = graph_text_encoders.encode_graph(graph, encoding_method)
source_node = random.sample(list(graph.nodes()), k=1)[0]
task_description = (
'Q: What is the degree of node %s?\nA: ' % name_dict[source_node]
)
question += task_description
answer = '%d.' % graph.degree[source_node]
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source_node],
}
return examples_dict
def get_edge_string(
self, name_dict: dict[int, str], graph: nx.Graph, source_node: int
) -> str:
"""Gets a string identifying the edges a given node is connected to."""
edge_string = ''
target_edges = graph.edges(source_node)
target_nodes = []
for edge in target_edges:
target_nodes.append(edge[1])
if target_nodes:
for i in range(len(target_nodes) - 1):
edge_string += name_dict[target_nodes[i]] + ', '
edge_string += 'and ' + name_dict[target_nodes[-1]]
else:
edge_string = 'no nodes'
return edge_string
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = graph_text_encoders.encode_graph(graph, encoding_method)
source_node = random.sample(list(graph.nodes()), k=1)[0]
question += (
'Q: What is the degree of node %s?\nA: ' % name_dict[source_node]
)
answer = '%d.' % graph.degree[source_node]
if cot:
answer += ' This is because %s is connected to %s.' % (
name_dict[source_node],
self.get_edge_string(name_dict, graph, source_node),
)
return question + answer
class EdgeCount(GraphTask):
"""The graph task for finding number of edges in a graph."""
def __init__(self):
super().__init__()
self.name = 'edge_count'
self._task_description = 'Q: How many edges are in this graph?\nA: '
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
for ind, graph in enumerate(graphs):
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += self._task_description
answer = ' %d.' % len(graph.edges())
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': self._task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [],
}
return examples_dict
def get_edges_string(
self, name_dict: dict[int, str], edges: list[tuple[int, int]]
) -> str:
edges_string = ''
for edge in edges:
edges_string += (
'(' + name_dict[edge[0]] + ', ' + name_dict[edge[1]] + '), '
)
return edges_string.strip()[:-1]
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += self._task_description
answer = '%d.' % len(graph.edges())
if cot:
answer += ' The edges are %s.' % self.get_edges_string(
name_dict, list(graph.edges())
)
return question + answer
class ConnectedNodes(GraphTask):
"""The graph task for finding connected nodes to a given node in a graph."""
def __init__(self):
super().__init__()
self.name = 'connected_nodes'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
question = graph_text_encoders.encode_graph(graph, encoding_method)
source_node = random.sample(list(graph.nodes()), k=1)[0]
task_description = (
'Q: List all the nodes connected to %s in alphabetical order.\nA: '
% name_dict[source_node]
)
question += task_description
outgoing_edges = list(graph.edges(source_node))
if outgoing_edges:
answer = self.get_connected_nodes(outgoing_edges, name_dict) + '.'
else:
answer = ' No nodes.'
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source_node],
}
return examples_dict
def get_connected_nodes(
self, edges: list[tuple[int, int]], name_dict: dict[int, str]
) -> str:
"""Gets a string including all the nodes that are connected to source."""
connected_nodes = []
for edge in edges:
connected_nodes.append(name_dict[edge[1]])
connected_nodes_string = ''
if connected_nodes:
try:
int(connected_nodes[0])
connected_nodes_string = ', '.join(map(str, connected_nodes))
except ValueError:
# Check if these are not integers, sort
connected_nodes_string = ', '.join(map(str, sorted(connected_nodes)))
return connected_nodes_string
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = graph_text_encoders.encode_graph(graph, encoding_method)
source_node = random.sample(list(graph.nodes()), k=1)[0]
question += (
'Q: List all the nodes connected to %s in alphabetical order.\nA: '
% name_dict[source_node]
)
outgoing_edges = list(graph.edges(source_node))
answer = ''
if outgoing_edges:
answer = self.get_connected_nodes(outgoing_edges, name_dict) + '.'
if cot:
answer += ' This is because there is an edge from %s to %s.' % (
name_dict[source_node],
answer,
)
else:
answer = 'No nodes.'
if cot:
answer += (
' This is because %s is not connected to any node.'
% name_dict[source_node]
)
return question + answer
class DisconnectedNodes(GraphTask):
"""The task for finding disconnected nodes for a given node in a graph."""
def __init__(self):
super().__init__()
self.name = 'disconnected_nodes'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
question = graph_text_encoders.encode_graph(graph, encoding_method)
source_node = random.sample(list(graph.nodes()), k=1)[0]
task_description = (
'Q: List all the nodes that are not connected to %s in alphabetical'
' order.\nA: '
% name_dict[source_node]
)
question += task_description
outgoing_edges = list(graph.edges(source_node))
answer = self.get_disconnected_nodes(
source_node, outgoing_edges, name_dict, list(graph.nodes())
)
if not answer:
answer = 'No nodes'
answer += '.'
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source_node],
}
return examples_dict
def get_disconnected_nodes(
self,
source: int,
edges: list[tuple[int, int]],
name_dict: dict[int, str],
all_nodes: list[int],
) -> str:
"""Gets a string with all the nodes that are not connected to source."""
for edge in edges:
if edge[1] in all_nodes:
all_nodes.remove(edge[1])
if source in all_nodes:
all_nodes.remove(source)
all_nodes_names = []
for node in all_nodes:
all_nodes_names.append(name_dict[node])
# sorted operation should be different for integers vs strings.
if all_nodes_names:
try:
int(all_nodes_names[0])
for ind, value in enumerate(all_nodes_names):
all_nodes_names[ind] = int(value)
all_nodes_names = sorted(all_nodes_names)
for ind, value in enumerate(all_nodes_names):
all_nodes_names[ind] = str(value)
except ValueError:
pass
return ', '.join(map(str, sorted(all_nodes_names)))
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = graph_text_encoders.encode_graph(graph, encoding_method)
source_node = random.sample(list(graph.nodes()), k=1)[0]
question += (
'Q: List all the nodes that are not connected to %s in alphabetical'
' order.\nA: '
% name_dict[source_node]
)
outgoing_edges = list(graph.edges(source_node))
answer = ''
disconnected_nodes_string = self.get_disconnected_nodes(
source_node, outgoing_edges, name_dict, list(graph.nodes())
)
if outgoing_edges:
if not disconnected_nodes_string:
disconnected_nodes_string = 'No nodes'
answer = disconnected_nodes_string + '.'
if cot:
answer += ' This is because there is not an edge from %s to %s.' % (
name_dict[source_node],
answer,
)
else:
answer = ' No nodes.'
if cot:
answer += (
' This is because %s is connected to all nodes.'
% name_dict[source_node]
)
return question + answer
class Reachability(GraphTask):
"""The graph task to check if there is a path from a source to target."""
def __init__(self):
super().__init__()
self.name = 'reachability'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
task_description = 'Q: Is there a path from node %s to node %s?\nA: ' % (
name_dict[source],
name_dict[target],
)
question += task_description
if nx.has_path(graph, source, target):
answer = 'Yes.'
else:
answer = 'No.'
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source, target],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += 'Q: Is there a path from node %s to node %s?\nA: ' % (
name_dict[source],
name_dict[target],
)
if nx.has_path(graph, source, target):
answer = 'Yes.'
if cot:
path = nx.shortest_path(graph, source, target)
explanation = ' Because'
for i in range(len(path) - 1):
# The only edge or the non-last edges in the path.
if len(path) == 2 or i < len(path) - 2:
sep = ','
# The last edge in a path with more than one edge.
else:
sep = ', and'
explanation += '%s there is an edge from node %d to node %d' % (
sep,
path[i],
path[i + 1],
)
explanation += ' .'
answer += explanation
else:
answer = 'No.'
if cot:
answer += (
' Because, there is no path connecting node %s to node %s based on'
' the graph description.' % (name_dict[source], name_dict[target])
)
return question + answer
class ShortestPath(GraphTask):
"""The graph task to check if there is a path from a source to target."""
def __init__(self):
super().__init__()
self.name = 'shortest_path'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
task_description = (
'Q: What is the length of the shortest path from node %s to node'
' %s?\nA: '
% (
name_dict[source],
name_dict[target],
)
)
question += task_description
try:
path = nx.shortest_path(graph, source, target)
answer = str(len(path) - 1) + '.'
except nx.NetworkXNoPath:
answer = 'There is no path from node %s to node %s.' % (
name_dict[source],
name_dict[target],
)
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source, target],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += (
'Q: What is the length of the shortest path from node %s to node'
' %s?\nA: '
% (
name_dict[source],
name_dict[target],
)
)
if nx.has_path(graph, source, target):
path = nx.shortest_path(graph, source, target)
answer = str(len(path) - 1) + '.'
if cot:
explanation = ' Because'
for i in range(len(path) - 1):
# The only edge or the non-last edges in the path.
if len(path) == 2 or i < len(path) - 2:
sep = ','
# The last edge in a path with more than one edge.
else:
sep = ', and'
explanation += '%s there is an edge from node %d to node %d' % (
sep,
path[i],
path[i + 1],
)
explanation += ' .'
answer += explanation
else:
answer = 'There is no path from node %s to node %s.' % (
name_dict[source],
name_dict[target],
)
if cot:
answer += (
' Because, there is no path connecting node %s to node %s based on'
' the graph description.' % (name_dict[source], name_dict[target])
)
return question + answer
class TriangleCounting(GraphTask):
"""The graph task to count the number of triangles in a graph."""
def __init__(self):
super().__init__()
self.name = 'triangle_counting'
self._task_description = 'Q: How many triangles are in this graph?\nA: '
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
for ind, graph in enumerate(graphs):
question = (
graph_text_encoders.encode_graph(graph, encoding_method)
+ self._task_description
)
ntriangles = int(np.sum(list(nx.triangles(graph).values())) / 3)
answer = '%i.' % ntriangles
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': self._task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
"""Create a few shot example w or w/o cot for the graph graph."""
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = (
graph_text_encoders.encode_graph(graph, encoding_method)
+ self._task_description
)
triangles_dict = nx.triangles(graph)
ntriangles = int(np.sum(list(triangles_dict.values())) / 3)
if ntriangles > 0:
answer = '%i.' % ntriangles
if cot:
ntriangles_cot = ''
for key, value in triangles_dict.items():
if value > 0:
if value == 1:
ntriangles_cot += (
'There is %i triangle including node %s as a vertex.\n'
% (value, name_dict[key])
)
else:
ntriangles_cot += (
'There are %i triangles including node %s as a vertex.\n'
% (value, name_dict[key])
)
ntriangles_cot += (
'Summing the number of triangles for all nodes and dividing them by'
' three gives us %i triangles in total.' % ntriangles
)
answer += ntriangles_cot
else:
answer = '0.'
if cot:
ntriangles_cot = 'No three nodes form a triangle of edges.'
answer += ntriangles_cot
return question + answer
class MaximumFlow(GraphTask):
"""The graph task to compute the maximum flow from a source to a target."""
def __init__(self):
super().__init__()
self.name = 'maximum_flow'
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
graph = add_edge_weight(graph)
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
task_description = (
'Q: What is the maximum capacity of the flow from node %s to node'
' %s?\nA: ' % (name_dict[source], name_dict[target])
)
question += task_description
maximum_flow_value = nx.maximum_flow(
graph, source, target, capacity='weight'
)[0]
answer = str(maximum_flow_value) + '.'
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(len(graph.nodes())),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
'node_ids': [source, target],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
graph = add_edge_weight(graph)
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
source, target = random.sample(list(graph.nodes()), k=2)
question = graph_text_encoders.encode_graph(graph, encoding_method)
question += (
'Q: What is the maximum capacity of the flow from node %s to'
' node %s?\nA: ' % (name_dict[source], name_dict[target])
)
flow_value, flow_dict = nx.maximum_flow(
graph, source, target, capacity='weight'
)
answer = str(flow_value) + '.'
if flow_value > 0:
if cot:
explanation = ' This is because of the following edges: '
for edge, capacity in flow_dict.items():
for key, value in capacity.items():
if value > 0:
explanation += (
'the edge from node %i to node %i with capacity %i, '
% (
edge,
key,
value,
)
)
explanation = explanation.strip()[:-1] + '.'
answer += explanation
else:
if cot:
answer += (
' Because, there is no path connecting node %s to node %s based on'
' the graph description.' % (name_dict[source], name_dict[target])
)
return question + answer
def has_edge_weights(graph):
for _, _, data in graph.edges(data=True):
if 'weight' not in data:
return False
return True
def add_edge_weight(graph):
if has_edge_weights(graph):
return graph
else:
for edge in graph.edges():
graph[edge[0]][edge[1]]['weight'] = random.randint(1, 10)
return graph
class NodeClassification(GraphTask):
"""The graph task to classify a given node in the graph."""
def __init__(self):
super().__init__()
self.name = 'node_classification'
self.classes = [
'soccer',
'baseball',
'tennis',
'golf',
'football',
'surfing',
]
def prepare_examples_dict(
self,
graphs: list[nx.Graph],
generator_algorithms: list[str],
encoding_method: str,
) -> dict[int, dict[str, str | list[int]]]:
classes = random.sample(list(self.classes), k=2)
examples_dict = {}
name_dict = graph_text_encoders.get_tlag_node_encoder(None, encoding_method)
for ind, graph in enumerate(graphs):
question = graph_text_encoders.encode_graph(graph, encoding_method)
nnodes = len(graph.nodes())
# Sampling nnodes // 2 + 1 nodes.
sampled_nodes = random.sample(
list(graph.nodes(data=True)), k=nnodes // 2 + 1
)
# Adding the class of half of the nodes.
for node_data in sampled_nodes[:-1]:
node_class = classes[node_data[1]['block']]
question += (
'Node ' + name_dict[node_data[0]] + ' likes ' + node_class + '.\n'
)
# Reserving the last sampled node for the question.
task_description = 'Q: Does node %s like %s or %s?\nA: ' % (
name_dict[sampled_nodes[-1][0]],
classes[0],
classes[1],
)
question += task_description
answer = classes[sampled_nodes[-1][1]['block']]
examples_dict[ind] = {
'question': question,
'answer': answer,
'nnodes': str(nnodes),
'nedges': str(len(graph.edges())),
'task_description': task_description,
'graph': graph,
'algorithm': generator_algorithms[ind],
# id of the last samples node
'node_ids': [sampled_nodes[-1][0]],
}
return examples_dict
def create_few_shot_example(
self, graph: nx.Graph, encoding_method: str, cot: bool
) -> str:
classes = random.sample(list(self.classes), k=2)
name_dict = graph_text_encoders.get_tlag_node_encoder(
graph, encoding_method
)
question = graph_text_encoders.encode_graph(graph, encoding_method)
nnodes = len(graph.nodes())
sampled_nodes = random.sample(
list(graph.nodes(data=True)), k=nnodes // 2 + 1
)
for node_data in sampled_nodes[:-1]:
node_class = classes[node_data[1]['block']]
question += (
'Node ' + name_dict[node_data[0]] + ' likes ' + node_class + '.\n'
)
task_description = 'Q: Does node %s like %s or %s?\nA: ' % (
name_dict[sampled_nodes[-1][0]],
classes[0],
classes[1],
)
question += task_description
answer = classes[sampled_nodes[-1][1]['block']]
if cot:
explanation = (
' This is because most of the nodes that are connected to node %s'
' likes %s.'
% (sampled_nodes[-1][0], classes[sampled_nodes[-1][1]['block']])
)
answer += explanation
return question + answer
================================================
FILE: talk_like_a_graph/graph_tasks_generator.py
================================================
r"""The graph tasks to be tried with LLMs..
This code loads graphs and creates graph tasks and output them as tf examples in
a recordio file in the task directory provided.
# Placeholder for Google-internal comments.
"""
from collections.abc import Sequence
import os
import random
from absl import app
from absl import flags
import networkx as nx
import numpy as np
from . import graph_tasks
from . import graph_tasks_utils as utils
_TASK_DIR = flags.DEFINE_string(
'task_dir', None, 'The directory to write tasks.', required=True
)
_GRAPHS_DIR = flags.DEFINE_string(
'graphs_dir', None, 'The directory containing the graphs.', required=True
)
_RANDOM_SEED = flags.DEFINE_integer(
'random_seed',
None,
'The random seed to use for task generation.',
required=True,
)
def zero_shot(
task: graph_tasks.GraphTask,
graphs: list[nx.Graph],
algorithms: list[str],
text_encoders: list[str],
cot: bool,
random_seed: int,
split: str,
) -> None:
"""Creating zero-shot or zero-cot examples for the given task.
Args:
task: the corresponding graph task.
graphs: the list of graphs to use for the task.
algorithms: the algorithm used to generate the graphs.
text_encoders: the encoders to use in the tasks.
cot: whether to apply cot or not.
random_seed: the random seed to use in the process.
split: whether we are creating a train or test split.
"""
random.seed(random_seed)
zero_shot_examples = utils.create_zero_shot_task(
task, graphs, algorithms, text_encoders, cot=cot
)
file_name = task.name + ('_zero_cot_' if cot else '_zero_shot_')
file_name += split + '.recordio'
utils.write_examples(
zero_shot_examples,
os.path.join(_TASK_DIR.value, file_name),
)
def few_shot(
task: graph_tasks.GraphTask,
graphs: list[nx.Graph],
few_shot_graphs: list[nx.Graph],
algorithms: list[str],
text_encoders: list[str],
cot: bool,
bag: bool,
random_seed: int,
) -> None:
"""Creating few-shot, cot, or cot-bag examples for the given task.
Args:
task: the corresponding graph task.
graphs: the list of graphs to use for the task.
few_shot_graphs: the list of graphs to generate few shot examples for.
algorithms: the algorithm used to generate the graphs.
text_encoders: the encoders to use in the tasks.
cot: whether to apply cot or not.
bag: whether to apply build-a-graph method or not.
random_seed: the random seed to use in the process.
"""
random.seed(random_seed)
few_shot_examples = utils.create_few_shot_task(
task,
graphs,
algorithms,
few_shot_graphs,
text_encoders,
cot=cot,
bag=bag,
random_seed=random_seed,
)
file_name = task.name
if cot and bag:
file_name += '_few_shot_cot_bag_test.recordio'
elif cot:
file_name += '_few_shot_cot_test.recordio'
else:
file_name += '_few_shot_test.recordio'
utils.write_examples(
few_shot_examples,
os.path.join(_TASK_DIR.value, file_name),
)
def generate_random_sbm_graph(random_state: np.random.RandomState):
# Sampling a small number as the probability of the two nodes in different
# communities being connected.
small_number = random.uniform(0, 0.05)
# Sampling a large number as probability of the nodes in one community
# being connected.
large_number = random.uniform(0.6, 0.8)
number_of_nodes = random.choice(np.arange(5, 20))
sizes = [number_of_nodes // 2, number_of_nodes // 2]
probs = [[large_number, small_number], [small_number, large_number]]
return nx.stochastic_block_model(sizes, probs, seed=random_state)
def main(argv: Sequence[str]) -> None:
if len(argv) > 1:
raise app.UsageError('Too many command-line arguments.')
algorithms = ['er']
directions = ['undirected']
text_encoders = ['adjacency']
# Loading the graphs.
graphs = []
generator_algorithms = []
for algorithm in algorithms:
for direction in directions:
loaded_graphs = utils.load_graphs(
_GRAPHS_DIR.value,
algorithm,
'train',
direction,
)
graphs += loaded_graphs
generator_algorithms += [algorithm] * len(loaded_graphs)
# Defining a task on the graphs
task = graph_tasks.ShortestPath()
if isinstance(task, graph_tasks.NodeClassification):
# The node classification task requires SBM graphs. As it's not possible to
# write graphs with data (e.g., blocks data as in SBM graphs), we regenerate
# graphs.
random_state = np.random.RandomState(_RANDOM_SEED.value)
print('Generating sbm graphs')
graphs = [
generate_random_sbm_graph(random_state) for _ in range(len(graphs))
]
zero_shot(
task,
graphs,
generator_algorithms,
text_encoders,
cot=False,
random_seed=_RANDOM_SEED.value,
split='test',
)
zero_shot(
task,
graphs,
generator_algorithms,
text_encoders,
cot=True,
random_seed=_RANDOM_SEED.value,
split='test',
)
# Loading few-shot graphs.
few_shot_graphs = []
for algorithm in algorithms:
for direction in directions:
few_shot_graphs += utils.load_graphs(
_GRAPHS_DIR.value,
algorithm,
'train',
direction,
)
if isinstance(task, graph_tasks.NodeClassification):
# The node classification task requires SBM graphs. As it's not possible to
# write graphs with data (e.g., blocks data as in SBM graphs), we regenerate
# graphs.
random_state = np.random.RandomState(_RANDOM_SEED.value + 1)
print('Generating few shot sbm graphs')
few_shot_graphs = [
generate_random_sbm_graph(random_state)
for _ in range(len(few_shot_graphs))
]
few_shot(
task,
graphs,
few_shot_graphs,
generator_algorithms,
text_encoders,
cot=False,
bag=False,
random_seed=_RANDOM_SEED.value,
)
few_shot(
task,
graphs,
few_shot_graphs,
generator_algorithms,
text_encoders,
cot=True,
bag=False,
random_seed=_RANDOM_SEED.value,
)
few_shot(
task,
graphs,
few_shot_graphs,
generator_algorithms,
text_encoders,
cot=True,
bag=True,
random_seed=_RANDOM_SEED.value,
)
if __name__ == '__main__':
app.run(main)
================================================
FILE: talk_like_a_graph/graph_tasks_utils.py
================================================
"""The graph tasks to be tried with LLMs."""
import os
import random
import networkx as nx
import numpy as np
import seqio
import tensorflow as tf
import tensorflow_gnn as tfgnn
# Google-internal import(s).
# Internal import.
from . import graph_tasks
from tensorflow.core.example import example_pb2
from tensorflow.core.example import feature_pb2
def laplacian_pos_embedding(graph: nx.Graph, units: int = 4) -> nx.Graph:
"""Adds the laplacian positional encoding."""
m = nx.normalized_laplacian_matrix(
graph, nodelist=sorted(graph.nodes), weight=None
).astype(np.float32)
u, _, _ = np.linalg.svd(m.todense(), compute_uv=True)
if units > u.shape[1]:
u = np.pad(u, ((0, 0), (0, units - u.shape[1])))
nx.set_node_attributes(
graph, dict(zip(sorted(graph.nodes), u[:, :units])), name='lpe'
)
return graph
def to_tfgnn(graph: nx.Graph, node_ids: list[int]) -> tfgnn.GraphTensor:
"""Convert a given nx graph to a tfgnn graph."""
if graph.edges(data=True):
s, t, w = zip(*[
(s, t, (d['weight'] if d and 'weight' in d else None))
for s, t, d in graph.edges(data=True)
])
else:
s, t, w = (), (), ()
# tfgnn assumes graphs are directed. Adding the rev edges for an undirected
# graph.
if not graph.is_directed():
s, t, w = s + t, t + s, w + w
graph = laplacian_pos_embedding(graph, units=4)
features = set(k for n in graph.nodes for k in graph.nodes[n].keys()) # pylint: disable=g-complex-comprehension
node_features = {
f: tf.convert_to_tensor([graph.nodes[n][f] for n in graph.nodes])
for f in features
}
# If all edges have a non-trivial weight, then we record the weights.
if all(w):
edge_features = {'weights': tf.convert_to_tensor(w, dtype=tf.int32)}
gt = tfgnn.homogeneous(
tf.convert_to_tensor(s, dtype=tf.int32),
tf.convert_to_tensor(t, dtype=tf.int32),
node_features=node_features,
edge_features=edge_features,
)
else:
gt = tfgnn.homogeneous(
tf.convert_to_tensor(s, dtype=tf.int32),
tf.convert_to_tensor(t, dtype=tf.int32),
node_features=node_features,
)
if not node_ids:
# No node is mentioned in the task description.
return gt
node_sets = {
**gt.node_sets,
'_readout': tfgnn.NodeSet.from_fields(sizes=[1]),
}
if len(node_ids) == 1:
# Tasks requiring only one node id e.g., computing node degree.
edge_sets = {
**gt.edge_sets,
'_readout/node': tfgnn.EdgeSet.from_fields(
sizes=[1],
adjacency=tfgnn.Adjacency.from_indices(
source=('nodes', node_ids),
target=('_readout', [0]),
),
),
}
elif len(node_ids) == 2:
# Tasks requiring two nodes e.g., shortest path from one node to the other.
edge_sets = {
**gt.edge_sets,
'_readout/source': tfgnn.EdgeSet.from_fields(
sizes=[1],
adjacency=tfgnn.Adjacency.from_indices(
source=('nodes', node_ids[:1]),
target=('_readout', [0]),
),
),
'_readout/target': tfgnn.EdgeSet.from_fields(
sizes=[1],
adjacency=tfgnn.Adjacency.from_indices(
source=('nodes', node_ids[1:]),
target=('_readout', [0]),
),
),
}
else:
# Raising an error if more than two nodes are mentiones.
raise ValueError(f'Invalid number of integers: {len(node_ids)}')
return tfgnn.GraphTensor.from_pieces(
context=gt.context, node_sets=node_sets, edge_sets=edge_sets
)
def create_example_feature(
key: int,
question: str,
answer: str,
algorithm: str,
encoding_method: str,
nnodes: str,
nedges: str,
task_description: str,
graph: nx.Graph,
node_ids: list[int],
) -> example_pb2.Example:
"""Create a tensorflow example from a datapoint."""
key_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[str(key).encode()])
)
question_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[question.encode()])
)
answer_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[answer.encode()])
)
algorithm_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[algorithm.encode()])
)
encoding_method_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[encoding_method.encode()])
)
nnodes_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[nnodes.encode()])
)
nedges_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[nedges.encode()])
)
task_description_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[task_description.encode()])
)
gt = to_tfgnn(graph, node_ids)
graph_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(
value=[tfgnn.write_example(gt).SerializeToString()]
)
)
directed_feature = feature_pb2.Feature(
bytes_list=tf.train.BytesList(value=[str(graph.is_directed()).encode()])
)
example_feats = tf.train.Features(
feature={
'id': key_feature,
'question': question_feature,
'answer': answer_feature,
'algorithm': algorithm_feature,
'text_encoding': encoding_method_feature,
'nnodes': nnodes_feature,
'nedges': nedges_feature,
'task_description': task_description_feature,
'graph': graph_feature,
'directed': directed_feature,
}
)
return example_pb2.Example(features=example_feats)
def load_graphs(
base_path: str,
algorithm: str,
split: str,
direction: str,
max_nnodes: int = 20,
) -> list[nx.Graph]:
"""Load a list of graphs from a given algorithm and split."""
graphs_path = os.path.join(
base_path,
direction,
algorithm,
split,
)
loaded_graphs = []
all_files = gfile.ListDir(graphs_path)
for file in all_files:
if file.endswith('.graphml'):
path = os.path.join(graphs_path, file)
graph = nx.read_graphml(os.Open(path, 'rb'), node_type=int)
if graph.number_of_nodes() <= max_nnodes:
loaded_graphs.append(graph)
return loaded_graphs
def prepare_examples(
examples_dict: dict[int, dict[str, str | list[int]]],
encoding_method: str,
) -> list[example_pb2.Example]:
"""Create a list of tf.train.Example from a dict of examples."""
examples = []
for key, value in examples_dict.items():
(
question,
answer,
nnodes,
nedges,
task_description,
graph,
algorithm,
node_ids,
) = (
value['question'],
value['answer'],
value['nnodes'],
value['nedges'],
value['task_description'],
value['graph'],
value['algorithm'],
value['node_ids'],
)
examples.append(
create_example_feature(
key,
question,
answer,
algorithm,
encoding_method,
nnodes,
nedges,
task_description,
graph,
node_ids,
)
)
return examples
def create_zero_shot_task(
task: graph_tasks.GraphTask,
graphs: list[nx.Graph],
generator_algorithms: list[str],
text_encoders: list[str],
cot: bool = False,
) -> list[example_pb2.Example]:
"""Create a recordio file with zero-shot examples for the task."""
examples = []
for encoding_method in text_encoders:
examples_dict = task.prepare_examples_dict(
graphs, generator_algorithms, encoding_method
)
if cot:
for key in examples_dict.keys():
examples_dict[key]['question'] += "Let's think step by step. "
examples += prepare_examples(examples_dict, encoding_method)
return examples
def write_examples(examples: list[example_pb2.Example], output_path: str):
with recordio.RecordWriter(output_path) as output_file:
for example in examples:
output_file.WriteRecord(example.SerializeToString())
def prepare_few_shots(
task: graph_tasks.GraphTask,
graphs: list[nx.Graph],
text_encoders: list[str],
cot: bool,
) -> dict[str, list[str]]:
"""Create a dict of few-shot examples with their cot for the task."""
few_shots_examples_dict = {}
for encoding_method in text_encoders:
if encoding_method not in few_shots_examples_dict:
few_shots_examples_dict[(encoding_method)] = []
for graph in graphs:
few_shots_examples_dict[(encoding_method)].append(
task.create_few_shot_example(graph, encoding_method, cot)
)
return few_shots_examples_dict
def choose_few_shot_examples(
few_shots_dict: dict[str, list[str]],
encoding_method: str,
k: int = 2,
) -> str:
"""Choose few shot examples for each algorithm."""
few_shots_str = ''
for _ in range(k):
example_list = few_shots_dict[encoding_method]
few_shots_str += 'Example: ' + random.choice(example_list) + '\n'
return few_shots_str
def create_few_shot_task(
task: graph_tasks.GraphTask,
graphs: list[nx.Graph],
generator_algorithms: list[str],
few_shots_graphs: list[nx.Graph],
text_encoders: list[str],
cot: bool,
bag: bool,
random_seed: int,
) -> list[example_pb2.Example]:
"""Create a recordio file with few-shot examples for the task."""
# LINT.IfChange
vocab_path = None
# LINT.ThenChange(//research/graph/llm/graphqa/copy.bara.sky)
# Loading the palm tokenizer to calculate number of tokens in the sequence.
sp_vocab = seqio.SentencePieceVocabulary(vocab_path)
number_of_tokens = {}
examples = []
print('prepare few shot task', 'cot', cot, 'bag', bag)
few_shots_examples_dict = prepare_few_shots(
task,
few_shots_graphs,
text_encoders,
cot,
)
for encoding_method in text_encoders:
random.seed(random_seed)
examples_dict = task.prepare_examples_dict(
graphs, generator_algorithms, encoding_method
)
for key in examples_dict.keys():
few_shots_examples = choose_few_shot_examples(
few_shots_examples_dict,
encoding_method,
)
examples_dict[key]['question'] = (
few_shots_examples + 'Example: ' + examples_dict[key]['question']
)
if bag:
examples_dict[key]['question'] = examples_dict[key]['question'].replace(
'\nQ: ',
"\nLet's construct the graph with the nodes and edges first.\nQ: ",
) # pytype: disable=attribute-error
if encoding_method not in number_of_tokens:
number_of_tokens[encoding_method] = []
number_of_tokens[encoding_method].append(
len(sp_vocab.encode(examples_dict[key]['question']))
)
examples += prepare_examples(examples_dict, encoding_method)
# Printing maximum number of tokens in the sequence.
for key, value in number_of_tokens.items():
print(key, np.max(value))
return examples
================================================
FILE: talk_like_a_graph/graph_text_encoders.py
================================================
"""Library for encoding graphs in text."""
import networkx as nx
from . import name_dictionaries
def create_node_string(name_dict, nnodes: int) -> str:
node_string = ""
sorted_keys = list(sorted(name_dict.keys()))
for i in sorted_keys[: nnodes - 1]:
node_string += name_dict[i] + ", "
node_string += "and " + name_dict[sorted_keys[nnodes - 1]]
return node_string
def nx_encoder(graph: nx.Graph, _: dict[int, str], edge_type="id") -> str:
"""Encoding a graph as entries of an adjacency matrix."""
if graph.is_directed():
output = (
"In a directed graph, (s,p,o) means that there is an edge from node s"
" to node o of type p. "
)
else:
output = (
"In an undirected graph, (s,p,o) means that node s and node o are"
" connected with an undirected edge of type p. "
)
name_dict = {x: str(x) for x in graph.nodes()}
nodes_string = create_node_string(name_dict, nnodes=len(graph.nodes()))
output += "G describes a graph among nodes %s.\n" % nodes_string
if graph.edges():
output += "The edges in G are: "
for i, j in graph.edges():
edge_type = graph.get_edge_data(i, j)[edge_type]
if edge_type is None:
edge_type = "linked"
output += "(%s, %s, %s) " % (name_dict[i], edge_type, name_dict[j])
return output.strip() + ".\n"
def adjacency_encoder(graph: nx.Graph, name_dict: dict[int, str]) -> str:
"""Encoding a graph as entries of an adjacency matrix."""
if graph.is_directed():
output = (
"In a directed graph, (i,j) means that there is an edge from node i to"
" node j. "
)
else:
output = (
"In an undirected graph, (i,j) means that node i and node j are"
" connected with an undirected edge. "
)
nodes_string = create_node_string(name_dict, len(graph.nodes()))
output += "G describes a graph among nodes %s.\n" % nodes_string
if graph.edges():
output += "The edges in G are: "
for i, j in graph.edges():
output += "(%s, %s) " % (name_dict[i], name_dict[j])
return output.strip() + ".\n"
def friendship_encoder(graph: nx.Graph, name_dict: dict[int, str]) -> str:
"""Encoding a graph as a friendship graph."""
if graph.is_directed():
raise ValueError("Friendship encoder is not defined for directed graphs.")
nodes_string = create_node_string(name_dict, len(graph.nodes()))
output = (
"G describes a friendship graph among nodes %s.\n" % nodes_string.strip()
)
if graph.edges():
output += "We have the following edges in G:\n"
for i, j in graph.edges():
output += "%s and %s are friends.\n" % (name_dict[i], name_dict[j])
return output
def coauthorship_encoder(graph: nx.Graph, name_dict: dict[int, str]) -> str:
"""Encoding a graph as a coauthorship graph."""
if graph.is_directed():
raise ValueError("Coauthorship encoder is not defined for directed graphs.")
nodes_string = create_node_string(name_dict, len(graph.nodes()))
output = (
"G describes a coauthorship graph among nodes %s.\n"
% nodes_string.strip()
)
if graph.edges():
output += "In this coauthorship graph:\n"
for i, j in graph.edges():
output += "%s and %s wrote a paper together.\n" % (
name_dict[i],
name_dict[j],
)
return output.strip() + ".\n"
def incident_encoder(graph: nx.Graph, name_dict: dict[int, str]) -> str:
"""Encoding a graph with its incident lists."""
nodes_string = create_node_string(name_dict, len(graph.nodes()))
output = "G describes a graph among nodes %s.\n" % nodes_string
if graph.edges():
output += "In this graph:\n"
for source_node in graph.nodes():
target_nodes = graph.neighbors(source_node)
target_nodes_str = ""
nedges = 0
for target_node in target_nodes:
target_nodes_str += name_dict[target_node] + ", "
nedges += 1
if nedges > 1:
output += "Node %s is connected to nodes %s.\n" % (
source_node,
target_nodes_str[:-2],
)
elif nedges == 1:
output += "Node %d is connected to node %s.\n" % (
source_node,
target_nodes_str[:-2],
)
return output
def social_network_encoder(graph: nx.Graph, name_dict: dict[int, str]) -> str:
"""Encoding a graph as a social network graph."""
if graph.is_directed():
raise ValueError(
"Social network encoder is not defined for directed graphs."
)
nodes_string = create_node_string(name_dict, len(graph.nodes()))
output = (
"G describes a social network graph among nodes %s.\n"
% nodes_string.strip()
)
if graph.edges():
output += "We have the following edges in G:\n"
for i, j in graph.edges():
output += "%s and %s are connected.\n" % (name_dict[i], name_dict[j])
return output
def expert_encoder(graph: nx.Graph, name_dict: dict[int, str]) -> str:
nodes_string = create_node_string(name_dict, len(graph.nodes()))
output = (
"You are a graph analyst and you have been given a graph G among nodes"
" %s.\n"
% nodes_string.strip()
)
output += "G has the following undirected edges:\n" if graph.edges() else ""
for i, j in graph.edges():
output += "%s -> %s\n" % (name_dict[i], name_dict[j])
return output
def nodes_to_text(graph, encoding_type):
"""Get dictionary converting node ids to text."""
if encoding_type == "integer":
return name_dictionaries.create_name_dict(graph, "integer", nnodes=1000)
elif encoding_type == "popular":
return name_dictionaries.create_name_dict(graph, "popular")
elif encoding_type == "alphabet":
return name_dictionaries.create_name_dict(graph, "alphabet")
elif encoding_type == "got":
return name_dictionaries.create_name_dict(graph, "got")
elif encoding_type == "south_park":
return name_dictionaries.create_name_dict(graph, "south_park")
elif encoding_type == "politician":
return name_dictionaries.create_name_dict(graph, "politician")
elif encoding_type == "random":
return name_dictionaries.create_name_dict(
graph, "random_integer", nnodes=1000
)
elif encoding_type == "nx_node_name":
return name_dictionaries.create_name_dict(graph, "nx_node_name")
else:
raise ValueError("Unknown encoding type: %s" % encoding_type)
def get_tlag_node_encoder(graph, encoder_name):
"""Find the node encoder used in the 'Talk Like a Graph' paper."""
if encoder_name == "adjacency":
return nodes_to_text(graph, "integer")
elif encoder_name == "incident":
return nodes_to_text(graph, "integer")
elif encoder_name == "friendship":
return nodes_to_text(graph, "popular")
elif encoder_name == "south_park":
return nodes_to_text(graph, "south_park")
elif encoder_name == "got":
return nodes_to_text(graph, "got")
elif encoder_name == "politician":
return nodes_to_text(graph, "politician")
elif encoder_name == "social_network":
return nodes_to_text(graph, "popular")
elif encoder_name == "expert":
return nodes_to_text(graph, "expert")
elif encoder_name == "coauthorship":
return nodes_to_text(graph, "popular")
elif encoder_name == "random":
return nodes_to_text(graph, "random")
elif encoder_name == "nx_node_name":
return nodes_to_text(graph, "nx_node_name")
else:
raise ValueError("Unknown graph encoder strategy: %s" % encoder_name)
# A dictionary from edge encoder name to the corresponding function.
EDGE_ENCODER_FN = {
"adjacency": adjacency_encoder,
"incident": incident_encoder,
"friendship": friendship_encoder,
"south_park": friendship_encoder,
"got": friendship_encoder,
"politician": social_network_encoder,
"social_network": social_network_encoder,
"expert": expert_encoder,
"coauthorship": coauthorship_encoder,
"random": adjacency_encoder,
"nx_edge_encoder": nx_encoder,
}
def with_ids(graph: nx.Graph, node_encoder: str) -> nx.Graph:
nx.set_node_attributes(graph, nodes_to_text(graph, node_encoder), name="id")
return graph
def encode_graph(
graph: nx.Graph, graph_encoder=None, node_encoder=None, edge_encoder=None
) -> str:
r"""Encodes a graph as text.
This relies on choosing:
a node_encoder and an edge_encoder:
or
a graph_encoder (a predefined pair of node and edge encoding strategies).
Note that graph_encoders may assume that the graph has some properties
(e.g. integer keys).
Example usage:
.. code-block:: python
```
# Use a predefined graph encoder from the paper.
>>> G = nx.karate_club_graph()
>>> encode_graph(G, graph_encoder="adjacency")
'In an undirected graph, (i,j) means that node i and node j are
connected
with an undirected edge. G describes a graph among nodes 0, 1, 2, 3, 4, 5,
6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 30, 31, 32, and 33.\nThe edges in G are: (0, 1) (0, 2) (0, 3)
...'
# Use the node's name in the graph as the node identifier.
>>> G = nx.les_miserables_graph()
>>> encode_graph(G, node_encoder="nx_node_name", edge_encoder="friendship")
'G describes a friendship graph among nodes Anzelma, Babet, Bahorel,
Bamatabois, BaronessT, Blacheville, Bossuet, Boulatruelle, Brevet, ...
We have the following edges in G:
Napoleon and Myriel are friends. Myriel and MlleBaptistine are friends...'
# Use the `id` feature from the edges to describe the edge type.
>>> G = nx.karate_club_graph()
>>> encode_graph(G, node_encoder="nx_node_name", edge_encoder="nx_edge_id")
'In an undirected graph, (s,p,o) means that node s and node o are connected
with an undirected edge of type p. G describes a graph among nodes 0, 1, 2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
25, 26, 27, 28, 29, 30, 31, 32, and 33.
The edges in G are: (0, linked, 1) (0, linked, 2) (0, linked, 3) ...'
```
Args:
graph: the graph to be encoded.
graph_encoder: the name of the graph encoder to use.
node_encoder: the name of the node encoder to use.
edge_encoder: the name of the edge encoder to use.
Returns:
The encoded graph as a string.
"""
# Check that only one of graph_encoder or (node_encoder, edge_encoder) is set.
if graph_encoder and (node_encoder or edge_encoder):
raise ValueError(
"Only one of graph_encoder or (node_encoder, edge_encoder) can be set."
)
if graph_encoder:
if isinstance(graph_encoder, str):
node_encoder_dict = get_tlag_node_encoder(graph, graph_encoder)
return EDGE_ENCODER_FN[graph_encoder](graph, node_encoder_dict)
else:
return graph_encoder(graph)
else:
node_encoder_dict = nodes_to_text(graph, node_encoder)
return EDGE_ENCODER_FN[edge_encoder](graph, node_encoder_dict)
================================================
FILE: talk_like_a_graph/graph_text_encoders_test.py
================================================
"""Testing for graph_text_encoders.py."""
from absl.testing import parameterized
import networkx as nx
from . import graph_text_encoders
from absl.testing import absltest
_G = nx.Graph()
_G.add_node(0)
_G.add_node(1)
_G.add_node(2)
_G.add_node(3)
_G.add_edge(0, 1)
_G.add_edge(1, 2)
_G.add_edge(2, 3)
_G.add_edge(3, 0)
class GraphTextEncodersTest(absltest.TestCase, parameterized.TestCase):
@parameterized.named_parameters(
dict(
testcase_name='adjacency_integer',
encoding_method='adjacency',
expected_result=(
'In an undirected graph, (i,j) means that node i and node j are'
' connected with an undirected edge. G describes a graph among'
' nodes 0, 1, 2, and 3.\nThe edges in G are: (0, 1) (0, 3) (1, 2)'
' (2, 3).\n'
),
),
dict(
testcase_name='incident_integer',
encoding_method='incident',
expected_result=(
'G describes a graph among nodes 0, 1, 2, and 3.\nIn this'
' graph:\nNode 0 is connected to nodes 1, 3.\nNode 1 is connected'
' to nodes 0, 2.\nNode 2 is connected to nodes 1, 3.\nNode 3 is'
' connected to nodes 2, 0.\n'
),
),
dict(
testcase_name='friendship_per_line_popular',
encoding_method='friendship',
expected_result=(
'G describes a friendship graph among nodes James, Robert, John,'
' and Michael.\nWe have the following edges in G:\nJames and'
' Robert are friends.\nJames and Michael are friends.\nRobert and'
' John are friends.\nJohn and Michael are friends.\n'
),
),
dict(
testcase_name='social_network_politician',
encoding_method='politician',
expected_result=(
'G describes a social network graph among nodes Barack, Jimmy,'
' Arnold, and Bernie.\nWe have the following edges in'
' G:\nBarack and Jimmy are connected.\nBarack and Bernie are'
' connected.\nJimmy and Arnold are connected.\nArnold and'
' Bernie are connected.\n'
),
),
)
def test_encoders(self, encoding_method, expected_result):
self.assertEqual(
graph_text_encoders.encode_graph(_G, encoding_method),
expected_result,
)
if __name__ == '__main__':
googletest.main()
================================================
FILE: talk_like_a_graph/name_dictionaries.py
================================================
"""Creates a dictionary mapping integers to node names."""
import random
_RANDOM_SEED = 1234
random.seed(_RANDOM_SEED)
_INTEGER_NAMES = [str(x) for x in range(10000)]
_POPULAR_NAMES = [
"James",
"Robert",
"John",
"Michael",
"David",
"Mary",
"Patricia",
"Jennifer",
"Linda",
"Elizabeth",
"William",
"Richard",
"Joseph",
"Thomas",
"Christopher",
"Barbara",
"Susan",
"Jessica",
"Sarah",
"Karen",
"Daniel",
"Lisa",
"Matthew",
"Nancy",
"Anthony",
"Betty",
"Mark",
"Margaret",
"Donald",
"Sandra",
"Steven",
"Ashley",
"Paul",
"Kimberly",
"Andrew",
"Emily",
"Joshua",
"Donna",
"Kenneth",
"Michelle",
"Kevin",
"Carol",
"Brian",
"Amanda",
"George",
"Melissa",
"Edward",
"Deborah",
"Ronald",
"Stephanie",
"Timothy",
"Rebecca",
"Jason",
"Sharon",
"Jeffrey",
"Laura",
"Ryan",
"Cynthia",
"Jacob",
"Dorothy",
"Gary",
"Olivia",
"Nicholas",
"Emma",
"Eric",
"Sophia",
"Jonathan",
"Ava",
"Stephen",
"Isabella",
"Scott",
"Mia",
"Justin",
"Abigail",
"Brandon",
"Madison",
"Frank",
"Chloe",
"Benjamin",
"Victoria",
"Samuel",
"Lauren",
"Gregory",
"Hannah",
"Alexander",
"Grace",
"Frank",
"Alexis",
"Raymond",
"Alice",
"Patrick",
"Samantha",
"Jack",
"Natalie",
"Dennis",
"Anna",
"Jerry",
"Taylor",
"Tyler",
"Kayla",
"Henry",
"Hailey",
"Douglas",
"Jasmine",
"Peter",
"Nicole",
"Adam",
"Amy",
"Nathan",
"Christina",
"Zachary",
"Andrea",
"Jose",
"Leah",
"Walter",
"Angelina",
"Harold",
"Valerie",
"Kyle",
"Veronica",
"Ethan",
"Carl",
"Arthur",
"Roger",
"Noah",
]
_SOUTH_PARK_NAMES = [
"Eric",
"Kenny",
"Kyle",
"Stan",
"Tolkien",
"Heidi",
"Bebe",
"Liane",
"Sharon",
"Linda",
"Gerald",
"Veronica",
"Michael",
"Jimbo",
"Herbert",
"Malcolm",
"Gary",
"Steve",
"Chris",
"Wendy",
]
_GOT_NAMES = [
"Ned",
"Cat",
"Daenerys",
"Jon",
"Bran",
"Sansa",
"Arya",
"Cersei",
"Jaime",
"Petyr",
"Robert",
"Jorah",
"Viserys",
"Joffrey",
"Maester",
"Theon",
"Rodrik",
"Lysa",
"Stannis",
"Osha",
]
_POLITICIAN_NAMES = [
"Barack",
"Jimmy",
"Arnold",
"Bernie",
"Bill",
"Kamala",
"Hillary",
"Elizabeth",
"John",
"Ben",
"Joe",
"Alexandria",
"George",
"Nancy",
"Pete",
"Madeleine",
"Elijah",
"Gabrielle",
"Al",
]
_ALPHABET_NAMES = [
"A",
"B",
"C",
"D",
"E",
"F",
"G",
"H",
"I",
"J",
"K",
"L",
"M",
"N",
"O",
"P",
"Q",
"R",
"S",
"T",
"U",
"V",
"W",
"X",
"Y",
"Z",
"AA",
"BB",
"CC",
"DD",
"EE",
"FF",
"GG",
"HH",
"II",
"JJ",
"KK",
"LL",
"MM",
"NN",
"OO",
"PP",
"QQ",
"RR",
"SS",
"TT",
"UU",
"VV",
"WW",
"XX",
"YY",
"ZZ",
]
def create_name_dict(graph, name: str, nnodes: int = 20) -> dict[int, str]:
"""The runner function to map integers to node names.
Args:
graph: the graph to be encoded.
name: name of the approach for mapping.
nnodes: optionally provide nnodes in the graph to be encoded.
Returns:
A dictionary from integers to strings.
"""
if name == "alphabet":
names_list = _ALPHABET_NAMES
elif name == "integer":
names_list = _INTEGER_NAMES
elif name == "random_integer":
names_list = []
for _ in range(nnodes):
names_list.append(str(random.randint(0, 1000000)))
elif name == "popular":
names_list = _POPULAR_NAMES
elif name == "south_park":
names_list = _SOUTH_PARK_NAMES
elif name == "got":
names_list = _GOT_NAMES
elif name == "politician":
names_list = _POLITICIAN_NAMES
elif name == "nx_node_name":
return {x: str(x) for x in graph.nodes()}
else:
raise ValueError(f"Unknown approach: {name}")
name_dict = {}
for ind, value in enumerate(names_list):
name_dict[ind] = value
return name_dict
================================================
FILE: tutorial/KDD-Tutorial-1-Talk-Like-a-Graph.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "EMeYjNCilDE_"
},
"source": [
"This is a noteboook that illustrates how to encode structured data for use in Large Language Models. It is Part 1 of a two part tutorial from **KDD'24**.\n",
"\n",
"If you find this tutorial useful or want to know more, please consider our publication: \n",
"[**Talk like a Graph: Encoding Graphs for Large Language Models**](https://openreview.net/pdf?id=IuXR1CCrSi)\n",
"\n",
"```\n",
"@inproceedings{\n",
"fatemi2024talk,\n",
" title={Talk like a Graph: Encoding Graphs for Large Language Models},\n",
" author={Bahare Fatemi and Jonathan Halcrow and Bryan Perozzi},\n",
" booktitle={The Twelfth International Conference on Learning Representations},\n",
" year={2024},\n",
" url={https://openreview.net/forum?id=IuXR1CCrSi}\n",
"}\n",
"```\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5Pt0yWFcnIdy"
},
"source": [
"## Tutorial Part I: Text Encoding of Graph Information\n",
"\n",
"This first notebook focuses on using text serialization methods from **Talk Like a Graph** to perform graph reasoning tasks with off-the-shelf LLMs. This notebook focuses on using static **Gemini** models (which have free API quota), but the techniques (and code) here are generally applicable to any LLM.\n",
"\n",
"\n",
"Notebook Outline:\n",
"\n",
"1. Setup (Install Dependencies, get Google Cloud API key)\n",
"1. Dataset creation\n",
"1. Graph-to-Text conversion\n",
"1. Evaluation\n",
"1. Exercise: *Graph Encoding Challenge*\n",
"1. Exercise: DBLP Dataset\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SAgPqF6SANGs"
},
"source": [
"# Setup"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "N6cnEIIlXhCj"
},
"source": [
"## Get Google AI Studio Key\n",
"\n",
"Go to [Google AI Studio](https://aistudio.google.com/app/u/1/apikey) and click on the `Create API Key` button. Follow prompts in dialog to setup an API key with a new Google Cloud Project.\n",
"\n",
"Then add the API key to your colab secrets (using the side panel 🔑) as in the picture below:\n",
"\n",
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KUTLOFaZM4lj"
},
"outputs": [],
"source": [
"# @title Load API Key\n",
"from google.colab import userdata\n",
"API_KEY = userdata.get('gemini')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YFxTwdqup_z5"
},
"source": [
"## Install Dependencies\n",
"\n",
"*This will take a minute or two...*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Qdm_qp_EQJe8"
},
"outputs": [],
"source": [
"%%capture\n",
"!pip install xmltodict\n",
"!pip install networkx ipysigma"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MVTY3dUIMsvo"
},
"outputs": [],
"source": [
"from collections import OrderedDict\n",
"from dataclasses import dataclass\n",
"from functools import cache\n",
"import itertools\n",
"from typing import Callable\n",
"import networkx as nx\n",
"import requests\n",
"import xmltodict\n",
"from ipysigma import Sigma\n",
"import math\n",
"import base64\n",
"import json"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hHpmncOLgBMw"
},
"outputs": [],
"source": [
"from google.colab import output\n",
"output.enable_custom_widget_manager()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-VvPogUCwLUC"
},
"source": [
"### DBLP to graph API"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f1Sj1xEmi1fa"
},
"outputs": [],
"source": [
"@dataclass(frozen=True)\n",
"class Author:\n",
" id: str\n",
" name: str\n",
"\n",
"@dataclass(frozen=True)\n",
"class Paper:\n",
" id: str\n",
" title: str\n",
" date: str\n",
" author_ids: list[str]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mL2uT670P6I_"
},
"outputs": [],
"source": [
"@cache\n",
"def _load_page(author_id: str) -\u003e OrderedDict:\n",
" data = requests.get(f'https://dblp.org/pid/{author_id}.xml')\n",
" return xmltodict.parse(data.text)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "k3B1xBosM5iK"
},
"outputs": [],
"source": [
"def clean_name(name: str) -\u003e str:\n",
" return ''.join([x for x in name if str.isalpha(x) or x in (' ', '-')])\n",
"\n",
"def read_authors(xmldata: OrderedDict, author_pid: str) -\u003e list[Author]:\n",
" authors = [Author(id=author_pid, name=xmldata['dblpperson']['@name'])]\n",
" for item in xmldata['dblpperson']['coauthors']['co']:\n",
" na = item['na']\n",
" if isinstance(na, list): na = na[0]\n",
" authors.append(Author(id=na['@pid'], name=clean_name(na['#text'])))\n",
" return authors\n",
"\n",
"def read_papers(xmldata: OrderedDict) -\u003e list[Paper]:\n",
" papers = []\n",
" for article in xmldata['dblpperson']['r']:\n",
" if not isinstance(article, dict):\n",
" print('???', article)\n",
" continue\n",
" [article_key] = article.keys()\n",
" article = article[article_key]\n",
" if 'author' not in article:\n",
" print('!!!', article)\n",
" continue\n",
" if isinstance(article['author'], list):\n",
" author_ids = [x['@pid'] for x in article['author']]\n",
" else:\n",
" author_ids = [article['author']['@pid']]\n",
" papers.append(Paper(id=article['@key'], title=article['title'], date=article['@mdate'], author_ids=author_ids))\n",
" return papers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "47h3IJBbBBkj"
},
"outputs": [],
"source": [
"def dblp_data(author_id: str) -\u003e [list[Author], list[Paper]]:\n",
" xmldata = _load_page(author_id)\n",
" return (read_authors(xmldata, author_pid), read_papers(xmldata))\n",
"\n",
"\n",
"def _expand(id2author: list[str]) -\u003e list[str]:\n",
" result = []\n",
" for author_id in id2author:\n",
" authors, _ = dblp_data(author_id)\n",
" result.extend([x.id for x in authors])\n",
" return list(set(result))\n",
"\n",
"\n",
"def coauthorship_graph(\n",
" authors: dict[str, Author], papers: dict[str, Paper]\n",
") -\u003e nx.Graph:\n",
" g = nx.Graph()\n",
" for a in authors.keys():\n",
" g.add_node(a, name=authors[a].name, id=authors[a].id)\n",
" # g.add_nodes_from(authors.keys())\n",
" for paper in papers.values():\n",
" g.add_edges_from(itertools.combinations(paper.author_ids, 2))\n",
" return g\n",
"\n",
"\n",
"def author_paper_graph(\n",
" authors: dict[str, Author], papers: dict[str, Paper]\n",
") -\u003e nx.Graph:\n",
" g = nx.Graph()\n",
" g.add_nodes_from(authors.keys())\n",
" g.add_nodes_from(papers.keys())\n",
" for paper in papers.values():\n",
" g.add_edges_from([paper.id, author] for author in paper.author_ids)\n",
" return g\n",
"\n",
"\n",
"def dblp_graph(\n",
" root_id: str,\n",
" n_hops: int = 1,\n",
" graph_fn: Callable[\n",
" [dict[str, Author], dict[str, Paper]], nx.Graph\n",
" ] = coauthorship_graph,\n",
") -\u003e nx.Graph:\n",
" author_list = [root_id]\n",
" for _ in range(n_hops - 1):\n",
" author_list = _expand(author_list)\n",
" authors, papers = zip(*[dblp_data(x) for x in author_list])\n",
" authors = {author.id: author for author in itertools.chain(*authors)}\n",
" papers = {paper.id: paper for paper in itertools.chain(*papers)}\n",
" return graph_fn(authors, papers)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_og7v0X-lA5V"
},
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "njSRRJLoAapA"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "JUCaM8NDYoMx"
},
"source": [
"### Load Talk Like a Graph Library"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kXymy2QvxdNw"
},
"outputs": [],
"source": [
"!git clone https://github.com/google-research/talk-like-a-graph.git\n",
"import sys\n",
"sys.path.insert(0, \"/content/talk-like-a-graph\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Dmb35I-Oxoca"
},
"outputs": [],
"source": [
"from talk_like_a_graph.graph_generators import generate_graphs\n",
"from talk_like_a_graph.graph_tasks import CycleCheck, ShortestPath, NodeCount"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qmgx9ffcX_2W"
},
"outputs": [],
"source": [
"import google.generativeai as genai\n",
"import os\n",
"\n",
"genai.configure(api_key=API_KEY)\n",
"\n",
"### Talk like a Graph code\n",
"import random\n",
"import networkx as nx\n",
"import numpy as np\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I88orawwu4BS"
},
"source": [
"### Load Saved Data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "51XDyyJWu5fJ"
},
"outputs": [],
"source": [
"saved_coauthorship_graph = nx.node_link_graph(json.loads(base64.b64decode(b'''eyJkaXJlY3RlZCI6IGZhbHNlLCAibXVsdGlncmFwaCI6IGZhbHNlLCAiZ3JhcGgiOiB7fSwgIm5vZGVzIjogW3sibmFtZSI6ICJCcnlhbiBQZXJvenppIiwgImlkIjogIjkxLzEwODEzIn0sIHsibmFtZSI6ICJTYW1pIEFidS1FbC1IYWlqYSIsICJpZCI6ICIxMjcvNjYyMCJ9LCB7Im5hbWUiOiAiTGVtYW4gQWtvZ2x1IiwgImlkIjogIjAyLzY5NzkifSwgeyJuYW1lIjogIlJhbWkgQWwtUmZvdSIsICJpZCI6ICIxMjIvMzAzMyJ9LCB7Im5hbWUiOiAiQWxleGFuZGVyIEEgQWxlbWkiLCAiaWQiOiAiMTYwLzgxNTgifSwgeyJuYW1lIjogIk5hemFuaW4gQWxpcG91cmZhcmQiLCAiaWQiOiAiMjAwLzAxNzgifSwgeyJuYW1lIjogIkZpbGlwZSBNaWd1ZWwgR29uXHUwMGU3YWx2ZXMgZGUgQWxtZWlkYSIsICJpZCI6ICIyNzYvNTcwMiJ9LCB7Im5hbWUiOiAiS3lyaWFrb3MgQXhpb3RpcyIsICJpZCI6ICIxNzYvNTEzOSJ9LCB7Im5hbWUiOiAiS2VzaGF2IEJhbGFzdWJyYW1hbmlhbiIsICJpZCI6ICIyNTgvOTEyOCJ9LCB7Im5hbWUiOiAiTWF0dCBCYXJuZXMgIiwgImlkIjogIjE2MC85OTEyIn0sIHsibmFtZSI6ICJNb2hhbW1hZCBIb3NzZWluIEJhdGVuaSAiLCAiaWQiOiAiMjIvNDczOSJ9LCB7Im5hbWUiOiAiUGV0ZXIgVyBCYXR0YWdsaWEiLCAiaWQiOiAiNDEvMzQwMCJ9LCB7Im5hbWUiOiAiU3BlbmNlciBCZWVjaGVyIiwgImlkIjogIjE1NC83ODAzIn0sIHsibmFtZSI6ICJYdWUgQmVuIiwgImlkIjogIjE1OS85NTE3In0sIHsibmFtZSI6ICJNYXJ0aW4gQmxhaXMiLCAiaWQiOiAiMjI1LzQxMTUifSwgeyJuYW1lIjogIkFsZWtzYW5kYXIgQm9qY2hldnNraSIsICJpZCI6ICIyMDMvODExNCJ9LCB7Im5hbWUiOiAiTmVzbGloYW4gQnVsdXQiLCAiaWQiOiAiMTk0LzQ4MjQifSwgeyJuYW1lIjogIk1pY2hhZWwgQnVycm93cyIsICJpZCI6ICI3My82MTUifSwgeyJuYW1lIjogIkthaWRpIENhbyIsICJpZCI6ICIyMDMvODIwNyJ9LCB7Im5hbWUiOiAiSW5lcyBDaGFtaSIsICJpZCI6ICIxODQvMjAyMCJ9LCB7Im5hbWUiOiAiSGFvY2hlbiBDaGVuIiwgImlkIjogIjE3Ni8yNjIwIn0sIHsibmFtZSI6ICJNdWhhbyBDaGVuIiwgImlkIjogIjE3My8yNjA4In0sIHsibmFtZSI6ICJUaW5nIENoZW4iLCAiaWQiOiAiMTkvMTc2NiJ9LCB7Im5hbWUiOiAiWWFucWluZyBDaGVuIiwgImlkIjogIjY4LzE0MjUifSwgeyJuYW1lIjogIlpoaWthaSBDaGVuIiwgImlkIjogIjkyLzQwIn0sIHsibmFtZSI6ICJaaGl5dWFuIENoZW5nICIsICJpZCI6ICI1OC83MjExLTIifSwgeyJuYW1lIjogIkVkIEggQ2hpIiwgImlkIjogIjEzLzMxMCJ9LCB7Im5hbWUiOiAiSm9zaHVhIFYgRGlsbG9uIiwgImlkIjogIjc4Lzg3MDAifSwgeyJuYW1lIjogIkthaXplIERpbmciLCAiaWQiOiAiMjM0LzY4NzgifSwgeyJuYW1lIjogIkppYWxpbiBEb25nIiwgImlkIjogIjIxMS8zOTY4In0sIHsibmFtZSI6ICJBcm5vIEVpZ2Vud2lsbGlnIiwgImlkIjogImUvQXJub0VpZ2Vud2lsbGlnIn0sIHsibmFtZSI6ICJQZXRlciBFbmdsZXJ0IiwgImlkIjogIjEzMi85MTEzIn0sIHsibmFtZSI6ICJBbGVzc2FuZHJvIEVwYXN0byIsICJpZCI6ICI1OC83ODAyIn0sIHsibmFtZSI6ICJIb3NzZWluIEVzZmFuZGlhcmkiLCAiaWQiOiAiMTQ2Lzc3NDYifSwgeyJuYW1lIjogIkJhaGFyZSBGYXRlbWkiLCAiaWQiOiAiMTgyLzIzNTYifSwgeyJuYW1lIjogIk9sZWtzYW5kciBGZXJsdWRpbiIsICJpZCI6ICIzMjQvMjYxMyJ9LCB7Im5hbWUiOiAiSGVuZHJpayBGaWNodGVuYmVyZ2VyIiwgImlkIjogIjEzMy8yMDI5In0sIHsibmFtZSI6ICJFZHVhcmRvIEZsZXVyeSIsICJpZCI6ICIyMTEvODA3MyJ9LCB7Im5hbWUiOiAiS2ltb24gRm91bnRvdWxha2lzIiwgImlkIjogIjE0OS81Nzk5In0sIHsibmFtZSI6ICJBcmFtIEdhbHN0eWFuIiwgImlkIjogIjE2LzM0MTEifSwgeyJuYW1lIjogIkpvaGFubmVzIEdhc3RlaWdlciIsICJpZCI6ICIyMjgvNzg5NyJ9LCB7Im5hbWUiOiAiU3RlcGhhbiBHXHUwMGZjbm5lbWFubiIsICJpZCI6ICI0My8zMDExIn0sIHsibmFtZSI6ICJKYW1lcyBUIEhhbGJlcnQiLCAiaWQiOiAiMTMwLzYzMzAifSwgeyJuYW1lIjogIkpvbmF0aGFuIEhhbGNyb3ciLCAiaWQiOiAiMjcxLzA3NjkifSwgeyJuYW1lIjogIkV0aGFuIEhhbGwiLCAiaWQiOiAiMzc4LzQyOTIifSwgeyJuYW1lIjogIkppYXdlaSBIYW4gIiwgImlkIjogImgvSmlhd2VpSGFuIn0sIHsibmFtZSI6ICJIcmF5ciBIYXJ1dHl1bnlhbiIsICJpZCI6ICIxOTgvMTQ2NSJ9LCB7Im5hbWUiOiAiRGFrZSBIZSIsICJpZCI6ICI0Ni81NDY2In0sIHsibmFtZSI6ICJaaWhhbyBIZSIsICJpZCI6ICIyNDYvMjk2MSJ9LCB7Im5hbWUiOiAiTGljaGFuIEhvbmciLCAiaWQiOiAiODUvNDY5NyJ9LCB7Im5hbWUiOiAiWWlmYW4gSHUgIiwgImlkIjogIjkyLzQ5OC0xIn0sIHsibmFtZSI6ICJaaW5pdSBIdSIsICJpZCI6ICIxODAvNTQzNiJ9LCB7Im5hbWUiOiAiRGkgSHVhbmciLCAiaWQiOiAiNDUvNzgwIn0sIHsibmFtZSI6ICJZdXpob25nIEh1YW5nIiwgImlkIjogIjIxMy84MDgwIn0sIHsibmFtZSI6ICJZaXpodSBKaWFvIiwgImlkIjogIjI1MC85NzU3In0sIHsibmFtZSI6ICJXZWkgSmluICIsICJpZCI6ICI2Ni8yMTczLTkifSwgeyJuYW1lIjogIkFtb2wgS2Fwb29yIiwgImlkIjogIjIxNS81MDE0In0sIHsibmFtZSI6ICJNZWhyYW4gS2F6ZW1pIiwgImlkIjogIjE0OS8xMjgzIn0sIHsibmFtZSI6ICJWaXZlayBLdWxrYXJuaSIsICJpZCI6ICI5Mi85OTIyIn0sIHsibmFtZSI6ICJTaWx2aW8gTGF0dGFuemkiLCAiaWQiOiAiNDYvNjYxMSJ9LCB7Im5hbWUiOiAiSm9vbnNlb2sgTGVlIiwgImlkIjogIjc3LzEzMTkifSwgeyJuYW1lIjogIktyaXN0aW5hIExlcm1hbiIsICJpZCI6ICI5OS80MzMifSwgeyJuYW1lIjogIkp1cmUgTGVza292ZWMiLCAiaWQiOiAibC9KdXJlTGVza292ZWMifSwgeyJuYW1lIjogIkFuZHJldyBMZXZpbmUiLCAiaWQiOiAiMDYvMTA4MTQifSwgeyJuYW1lIjogIkJpbmdoZW5nIExpIiwgImlkIjogIjIzMS8xOTA0In0sIHsibmFtZSI6ICJXYWkgTG9rIFNpYm9uIExpIiwgImlkIjogIjI1OC81MDAzIn0sIHsibmFtZSI6ICJBbGJlcnQgSmlvbmdxaWFuIExpYW5nIiwgImlkIjogIjM0OC82NDk4In0sIHsibmFtZSI6ICJBbmRyXHUwMGU5IExpbmhhcmVzIiwgImlkIjogIjEzNy8zMjUyIn0sIHsibmFtZSI6ICJIdWFuIExpdSAiLCAiaWQiOiAiOTIvMzA5LTEifSwgeyJuYW1lIjogIkh1aSBMaXUgIiwgImlkIjogIjkzLzQwMTAtMzEifSwgeyJuYW1lIjogIkppbmd6aGUgTGl1IiwgImlkIjogIjMzOS83NTU4In0sIHsibmFtZSI6ICJMdXlhbmcgTGl1IiwgImlkIjogIjY5LzgxNjcifSwgeyJuYW1lIjogIk1pY2hhbCBMdWthc2lrIiwgImlkIjogIjcyLzExMzM4In0sIHsibmFtZSI6ICJLYXJpc2htYSBNYWxrYW4iLCAiaWQiOiAiMjYzLzk5NTkifSwgeyJuYW1lIjogIkhhaXRhbyBNYW8iLCAiaWQiOiAiMjIxLzY1MTYifSwgeyJuYW1lIjogIkVsYW4gTWFya293aXR6IiwgImlkIjogIjI4NC85NDAxIn0sIHsibmFtZSI6ICJCcmFuZG9uIEEgTWF5ZXIiLCAiaWQiOiAiMjcvMTEyNjEifSwgeyJuYW1lIjogIkNocmlzIE1jQ3ViYmluIiwgImlkIjogIjI5Ny8xODIxIn0sIHsibmFtZSI6ICJDaGFyaXRoIE1lbmRpcyIsICJpZCI6ICIxNjMvMzE3NSJ9LCB7Im5hbWUiOiAiVmFoYWIgUyBNaXJyb2tuaSIsICJpZCI6ICJtL1ZhaGFiU01pcnJva25pIn0sIHsibmFtZSI6ICJNZWhybm9vc2ggTWlydGFoZXJpIiwgImlkIjogIjIzNi80NDQ5In0sIHsibmFtZSI6ICJGcmVkIE1vcnN0YXR0ZXIiLCAiaWQiOiAiNTEvOTY4NyJ9LCB7Im5hbWUiOiAiQWxleGFuZHJ1IE1vc29pIiwgImlkIjogIjI3MS8xMDI1In0sIHsibmFtZSI6ICJFbW1hbnVlbCBNXHUwMGZjbGxlciIsICJpZCI6ICI1NS82ODk5In0sIHsibmFtZSI6ICJNYXJpbmEgTXVua2hvZXZhIiwgImlkIjogIjIxNS8zNjgyIn0sIHsibmFtZSI6ICJLZXZpbiBNdXJwaHkgIiwgImlkIjogIjI2LzI1OTkifSwgeyJuYW1lIjogIlNoYXduIE9CYW5pb24iLCAiaWQiOiAiMTI3Lzc0OTcifSwgeyJuYW1lIjogIkpvaG4gUGFsb3dpdGNoIiwgImlkIjogIjE3NS8xMjMxIn0sIHsibmFtZSI6ICJNaWhpciBQYXJhZGthciIsICJpZCI6ICIyMDUvMzkwOCJ9LCB7Im5hbWUiOiAiSmFuIFBmZWlmZXIiLCAiaWQiOiAiMTYzLzIzMzkifSwgeyJuYW1lIjogIk1hbmdwbyBNYW5ncG8gUGhvdGhpbGltdGhhbmEiLCAiaWQiOiAiMzY5Lzc2ODkifSwgeyJuYW1lIjogIlBoaXRjaGF5YSBNYW5ncG8gUGhvdGhpbGltdGhhbmEiLCAiaWQiOiAiMTI3LzMxMjgifSwgeyJuYW1lIjogIk5hdGFsaWEgUG9ub21hcmV2YSAiLCAiaWQiOiAiNzEvNjc2OC0xIn0sIHsibmFtZSI6ICJTdGVmYW4gUG9zdGF2YXJ1IiwgImlkIjogIjE5NC8xNTcwIn0sIHsibmFtZSI6ICJBYmR1bCBSYWhtYW4iLCAiaWQiOiAiODcvMTAxMDAifSwgeyJuYW1lIjogIkRlZXBhayBSYW1hY2hhbmRyYW4iLCAiaWQiOiAiODAvNzAzIn0sIHsibmFtZSI6ICJDaHJpc3RvcGhlciBSXHUwMGU5IiwgImlkIjogInIvQ2hyaXN0b3BoZXJSZSJ9LCB7Im5hbWUiOiAiQXJpYSBSZXphZWkiLCAiaWQiOiAiMTYwLzg2NzcifSwgeyJuYW1lIjogIkJlbmVkZWsgUm96ZW1iZXJjemtpIiwgImlkIjogIjIxNS8zNzQyIn0sIHsibmFtZSI6ICJTYW0gUnV0aCIsICJpZCI6ICIyNzEvMDYzNyJ9LCB7Im5hbWUiOiAiSmFjayBTYWFsd2VhY2h0ZXIiLCAiaWQiOiAiMTg0LzAwMzMifSwgeyJuYW1lIjogIlJ1c2xhbiBTYWxha2h1dGRpbm92IiwgImlkIjogIjYyLzU4ODQifSwgeyJuYW1lIjogIlBhdHJpY2lhIElnbGVzaWFzIFNcdTAwZTFuY2hleiIsICJpZCI6ICIxMzEvNzE2OSJ9LCB7Im5hbWUiOiAiQWx2YXJvIFNhbmNoZXotR29uemFsZXoiLCAiaWQiOiAiMjIyLzE4ODkifSwgeyJuYW1lIjogIkNsYXl0b24gU2FuZm9yZCIsICJpZCI6ICIyMzIvMTc5NyJ9LCB7Im5hbWUiOiAiTWljaGFlbCBTY2h1ZXBwZXJ0IiwgImlkIjogIjE2OS8xNzQyIn0sIHsibmFtZSI6ICJTdW5neW9uZyBTZW8iLCAiaWQiOiAiMTc4LzMyMDkifSwgeyJuYW1lIjogIkplbm5pZmVyIFNoZSIsICJpZCI6ICIyNDMvNTc2MCJ9LCB7Im5hbWUiOiAiU3RldmVuIFNraWVuYSIsICJpZCI6ICJzL1N0ZXZlblNraWVuYSJ9LCB7Im5hbWUiOiAiR3JlZyBWZXIgU3RlZWciLCAiaWQiOiAiODIvOTA1OCJ9LCB7Im5hbWUiOiAiS2V4dWFuIFN1biAiLCAiaWQiOiAiMTE1LzY0NjYtMiJ9LCB7Im5hbWUiOiAiWGlhb2ZlaSBTdW4iLCAiaWQiOiAiODcvNzI5NyJ9LCB7Im5hbWUiOiAiSmlsaWFuZyBUYW5nIiwgImlkIjogIjY0LzEwODEyIn0sIHsibmFtZSI6ICJNYXl1ciBUaGFrdXIiLCAiaWQiOiAiNTgvNTQ5MiJ9LCB7Im5hbWUiOiAiWWluZ3RhbyBUaWFuIiwgImlkIjogIjE4MC81MzM1In0sIHsibmFtZSI6ICJMb25nIFRyYW4tVGhhbmgiLCAiaWQiOiAiNDYvODMzMyJ9LCB7Im5hbWUiOiAiQW50b24gVHNpdHN1bGluIiwgImlkIjogIjIxNy8xNjY4In0sIHsibmFtZSI6ICJLZXZpbiBWaWxsZWxhIiwgImlkIjogIjMyNC8yNTY3In0sIHsibmFtZSI6ICJMaXNhIFdhbmciLCAiaWQiOiAiMTk4Lzc0MjMifSwgeyJuYW1lIjogIlJ1b3hpIFdhbmciLCAiaWQiOiAiNzIvMTEyNDQifSwgeyJuYW1lIjogIkRhdmlkIFdvbmciLCAiaWQiOiAiNDcvMzE4In0sIHsibmFtZSI6ICJZdWUgV3UgIiwgImlkIjogIjQxLzU5NzktMSJ9LCB7Im5hbWUiOiAiTGluIFlhbmcgIiwgImlkIjogIjE2Ni82MjY0In0sIHsibmFtZSI6ICJTaGVuZ2hhbyBZYW5nICIsICJpZCI6ICI0MS80NDgyLTIifSwgeyJuYW1lIjogIk11c3RhZmEgWWFzaXIiLCAiaWQiOiAiMzUyLzQyMzcifSwgeyJuYW1lIjogIkppbnllb25nIFlpbSIsICJpZCI6ICIxNzYvNDIwNSJ9LCB7Im5hbWUiOiAiTWluamkgWW9vbiIsICJpZCI6ICIyMDUvMjY1MSJ9LCB7Im5hbWUiOiAiU29uZyBZdSIsICJpZCI6ICI1NC8xMjE2In0sIHsibmFtZSI6ICJEdXN0aW4gWmVsbGUiLCAiaWQiOiAiMjM5LzYwNzAifSwgeyJuYW1lIjogIlBlaWxpbiBaaG9uZyIsICJpZCI6ICIxNDgvOTYzMiJ9LCB7Im5hbWUiOiAiWWFucWkgWmhvdSIsICJpZCI6ICIxNDIvMzIwMiJ9LCB7Im5hbWUiOiAiUWkgWmh1ICIsICJpZCI6ICI2Ni81OTIzLTgifV0sICJsaW5rcyI6IFt7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjI3MS8wNzY5In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE0OS8xMjgzIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxMjIvMzAzMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjExLzM5NjgifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE2Ni82MjY0In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMzIvMTc5NyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMzc4LzQyOTIifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjI2My85OTU5In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNzYvNDIwNSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTc1LzEyMzEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE3OC8zMjA5In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI5Mi80MCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjIxLzY1MTYifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjMzOS83NTU4In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI1NC8xMjE2In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMzEvMTkwNCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNjYvMjE3My05In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI5My80MDEwLTMxIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI2NC8xMDgxMiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNTUvNjg5OSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTQ2Lzc3NDYifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIyLzQ3MzkifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjgwLzcwMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjA1LzI2NTEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjQxLzU5NzktMSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNjIvNTg4NCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjEzMy8yMDI5In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxMjcvNjYyMCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjAzLzgyMDcifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjEyNy8zMTI4In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNDIvMzIwMiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTYzLzMxNzUifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogImwvSnVyZUxlc2tvdmVjIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIzNjkvNzY4OSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNzMvNjE1In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMzQvNjg3OCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMzQ4LzY0OTgifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE5LzE3NjYifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjcyLzExMjQ0In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI4NS80Njk3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxMy8zMTAifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjkyLzMwOS0xIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI1OC83MjExLTIifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIxNS8zNjgyIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI3OC84NzAwIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNzYvNTEzOSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTk0LzQ4MjQifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIyOC83ODk3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI2Ni81OTIzLTgifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjI1MC85NzU3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI3MS82NzY4LTEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogImgvSmlhd2VpSGFuIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIzNTIvNDIzNyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNDYvODMzMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTg0LzIwMjAifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogInIvQ2hyaXN0b3BoZXJSZSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjYvMjU5OSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNTgvNzgwMiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTQ4Lzk2MzIifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE4MC81NDM2In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMTUvMzc0MiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMzI0LzI2MTMifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogImUvQXJub0VpZ2Vud2lsbGlnIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMjUvNDExNSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTYzLzIzMzkifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyNTgvNTAwMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNDEvMzQwMCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjc2LzU3MDIifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjQ2LzY2MTEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjEzNy8zMjUyIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNDkvNTc5OSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNDYvNTQ2NiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNDEvNDQ4Mi0yIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyODQvOTQwMSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjU4LzkxMjgifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIzNi80NDQ5In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI4Mi85MDU4In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNi8zNDExIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxMzIvOTExMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjE1LzUwMTQifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIwMy84MTE0In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI3Mi8xMTMzOCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNDMvMzAxMSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjcxLzEwMjUifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjI3MS8wNjM3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI0NS83ODAifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjI0Ni8yOTYxIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIyMTMvODA4MCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTE1LzY0NjYtMiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiOTkvNDMzIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI1MS85Njg3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNTkvOTUxNyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNjkvODE2NyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTYwLzk5MTIifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjEyNy83NDk3In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxOTQvMTU3MCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTgwLzUzMzUifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE3Ni8yNjIwIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNzMvMjYwOCJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiODcvNzI5NyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAicy9TdGV2ZW5Ta2llbmEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjIwMC8wMTc4In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxOTgvMTQ2NSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNzcvMTMxOSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMDIvNjk3OSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiOTIvNDk4LTEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE2MC84MTU4In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI2OC8xNDI1In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI5Mi85OTIyIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxNjAvODY3NyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjExLzgwNzMifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjE2OS8xNzQyIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxODQvMDAzMyJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiNTgvNTQ5MiJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMjk3LzE4MjEifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjEzMC82MzMwIn0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICIxMzEvNzE2OSJ9LCB7InNvdXJjZSI6ICI5MS8xMDgxMyIsICJ0YXJnZXQiOiAiMTU0Lzc4MDMifSwgeyJzb3VyY2UiOiAiOTEvMTA4MTMiLCAidGFyZ2V0IjogIjA2LzEwODE0In0sIHsic291cmNlIjogIjkxLzEwODEzIiwgInRhcmdldCI6ICI4Ny8xMDEwMCJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjIwMy84MjA3In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxMjcvMzEyOCJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjE0Mi8zMjAyIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxNjMvMzE3NSJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAibC9KdXJlTGVza292ZWMifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjM2OS83Njg5In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxODIvMjM1NiJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiNzMvNjE1In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICI3OC84NzAwIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxNzYvNTEzOSJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMTk0LzQ4MjQifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjIyOC83ODk3In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIyMi80NzM5In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxNDkvMTI4MyJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjcxLzA3NjkifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjE4NC8yMDIwIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICJyL0NocmlzdG9waGVyUmUifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjI2LzI1OTkifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjMyNC8yNjEzIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICJlL0Fybm9FaWdlbndpbGxpZyJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjI1LzQxMTUifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjU4LzUwMDMifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjQxLzM0MDAifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjI3Ni81NzAyIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICI0Ni82NjExIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxMzcvMzI1MiJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIyODQvOTQwMSJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjU4LzkxMjgifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjIzNi80NDQ5In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICI4Mi85MDU4In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxNi8zNDExIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICI0NS83ODAifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjI0Ni8yOTYxIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIyMTMvODA4MCJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMTE1LzY0NjYtMiJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiOTkvNDMzIn0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICI1MS85Njg3In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMjAwLzAxNzgifSwgeyJzb3VyY2UiOiAiMTI3LzY2MjAiLCAidGFyZ2V0IjogIjE5OC8xNDY1In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICI3Ny8xMzE5In0sIHsic291cmNlIjogIjEyNy82NjIwIiwgInRhcmdldCI6ICIxMjIvMzAzMyJ9LCB7InNvdXJjZSI6ICIxMjcvNjYyMCIsICJ0YXJnZXQiOiAiMTYwLzgxNTgifSwgeyJzb3VyY2UiOiAiMDIvNjk3OSIsICJ0YXJnZXQiOiAiMTYwLzg2NzcifSwgeyJzb3VyY2UiOiAiMDIvNjk3OSIsICJ0YXJnZXQiOiAiMTMxLzcxNjkifSwgeyJzb3VyY2UiOiAiMDIvNjk3OSIsICJ0YXJnZXQiOiAiNTUvNjg5OSJ9LCB7InNvdXJjZSI6ICIxMjIvMzAzMyIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiMTIyLzMwMzMiLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjEyMi8zMDMzIiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIxMjIvMzAzMyIsICJ0YXJnZXQiOiAiMTQ5LzEyODMifSwgeyJzb3VyY2UiOiAiMTIyLzMwMzMiLCAidGFyZ2V0IjogIjI3MS8wNzY5In0sIHsic291cmNlIjogIjEyMi8zMDMzIiwgInRhcmdldCI6ICIxNjAvODE1OCJ9LCB7InNvdXJjZSI6ICIxMjIvMzAzMyIsICJ0YXJnZXQiOiAiMTc2LzI2MjAifSwgeyJzb3VyY2UiOiAiMTIyLzMwMzMiLCAidGFyZ2V0IjogInMvU3RldmVuU2tpZW5hIn0sIHsic291cmNlIjogIjEyMi8zMDMzIiwgInRhcmdldCI6ICI5Mi85OTIyIn0sIHsic291cmNlIjogIjEyMi8zMDMzIiwgInRhcmdldCI6ICI2OC8xNDI1In0sIHsic291cmNlIjogIjIwMC8wMTc4IiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICIyMDAvMDE3OCIsICJ0YXJnZXQiOiAiOTkvNDMzIn0sIHsic291cmNlIjogIjIwMC8wMTc4IiwgInRhcmdldCI6ICIxOTgvMTQ2NSJ9LCB7InNvdXJjZSI6ICIyMDAvMDE3OCIsICJ0YXJnZXQiOiAiODIvOTA1OCJ9LCB7InNvdXJjZSI6ICIyMDAvMDE3OCIsICJ0YXJnZXQiOiAiMTYvMzQxMSJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiMzI0LzI2MTMifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogImUvQXJub0VpZ2Vud2lsbGlnIn0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICIyMjUvNDExNSJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiMjU4LzUwMDMifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjQxLzM0MDAifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjE5NC80ODI0In0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICIyNzEvMDc2OSJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiNDYvNjYxMSJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiMTM3LzMyNTIifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjI3LzExMjYxIn0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICJtL1ZhaGFiU01pcnJva25pIn0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiMjA1LzM5MDgifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjI0My81NzYwIn0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIyNzYvNTcwMiIsICJ0YXJnZXQiOiAiMzI0LzI1NjcifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjE5OC83NDIzIn0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICI0Ny8zMTgifSwgeyJzb3VyY2UiOiAiMjc2LzU3MDIiLCAidGFyZ2V0IjogIjE5NC8xNTcwIn0sIHsic291cmNlIjogIjI3Ni81NzAyIiwgInRhcmdldCI6ICIxODAvNTMzNSJ9LCB7InNvdXJjZSI6ICIxNzYvNTEzOSIsICJ0YXJnZXQiOiAiNzgvODcwMCJ9LCB7InNvdXJjZSI6ICIxNzYvNTEzOSIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiMTc2LzUxMzkiLCAidGFyZ2V0IjogIjE5NC80ODI0In0sIHsic291cmNlIjogIjE3Ni81MTM5IiwgInRhcmdldCI6ICIyMjgvNzg5NyJ9LCB7InNvdXJjZSI6ICIxNzYvNTEzOSIsICJ0YXJnZXQiOiAiMjIvNDczOSJ9LCB7InNvdXJjZSI6ICIyNTgvOTEyOCIsICJ0YXJnZXQiOiAiMjg0Lzk0MDEifSwgeyJzb3VyY2UiOiAiMjU4LzkxMjgiLCAidGFyZ2V0IjogIjIzNi80NDQ5In0sIHsic291cmNlIjogIjI1OC85MTI4IiwgInRhcmdldCI6ICI4Mi85MDU4In0sIHsic291cmNlIjogIjI1OC85MTI4IiwgInRhcmdldCI6ICIxNi8zNDExIn0sIHsic291cmNlIjogIjE2MC85OTEyIiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICIxNjAvOTkxMiIsICJ0YXJnZXQiOiAiMTU5Lzk1MTcifSwgeyJzb3VyY2UiOiAiMTYwLzk5MTIiLCAidGFyZ2V0IjogIjY5LzgxNjcifSwgeyJzb3VyY2UiOiAiMTYwLzk5MTIiLCAidGFyZ2V0IjogIjIyNS80MTE1In0sIHsic291cmNlIjogIjE2MC85OTEyIiwgInRhcmdldCI6ICIxMjcvNzQ5NyJ9LCB7InNvdXJjZSI6ICIyMi80NzM5IiwgInRhcmdldCI6ICIxNDkvMTI4MyJ9LCB7InNvdXJjZSI6ICIyMi80NzM5IiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIyMi80NzM5IiwgInRhcmdldCI6ICIxNDYvNzc0NiJ9LCB7InNvdXJjZSI6ICIyMi80NzM5IiwgInRhcmdldCI6ICI4MC83MDMifSwgeyJzb3VyY2UiOiAiMjIvNDczOSIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIyMi80NzM5IiwgInRhcmdldCI6ICI3OC84NzAwIn0sIHsic291cmNlIjogIjIyLzQ3MzkiLCAidGFyZ2V0IjogIjE4Mi8yMzU2In0sIHsic291cmNlIjogIjIyLzQ3MzkiLCAidGFyZ2V0IjogIjE5NC80ODI0In0sIHsic291cmNlIjogIjIyLzQ3MzkiLCAidGFyZ2V0IjogIjIyOC83ODk3In0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjMyNC8yNjEzIn0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogImUvQXJub0VpZ2Vud2lsbGlnIn0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjIyNS80MTE1In0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjI1OC81MDAzIn0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjE5NC80ODI0In0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjI3MS8wNzY5In0sIHsic291cmNlIjogIjQxLzM0MDAiLCAidGFyZ2V0IjogIjQ2LzY2MTEifSwgeyJzb3VyY2UiOiAiNDEvMzQwMCIsICJ0YXJnZXQiOiAiMTM3LzMyNTIifSwgeyJzb3VyY2UiOiAiNDEvMzQwMCIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiNDEvMzQwMCIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICIyNDMvNTc2MCJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICIzMjQvMjU2NyJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICI0MS8zNDAwIiwgInRhcmdldCI6ICI0Ny8zMTgifSwgeyJzb3VyY2UiOiAiMTU0Lzc4MDMiLCAidGFyZ2V0IjogIjI5Ny8xODIxIn0sIHsic291cmNlIjogIjE1NC83ODAzIiwgInRhcmdldCI6ICIxMzAvNjMzMCJ9LCB7InNvdXJjZSI6ICIxNTkvOTUxNyIsICJ0YXJnZXQiOiAiMjE1LzUwMTQifSwgeyJzb3VyY2UiOiAiMTU5Lzk1MTciLCAidGFyZ2V0IjogIjY5LzgxNjcifSwgeyJzb3VyY2UiOiAiMTU5Lzk1MTciLCAidGFyZ2V0IjogIjIyNS80MTE1In0sIHsic291cmNlIjogIjE1OS85NTE3IiwgInRhcmdldCI6ICIxMjcvNzQ5NyJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMzI0LzI2MTMifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogImUvQXJub0VpZ2Vud2lsbGlnIn0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMTYzLzIzMzkifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICIyNTgvNTAwMyJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMTk0LzQ4MjQifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjI3MS8wNzY5In0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICI0Ni82NjExIn0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICIxMzcvMzI1MiJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICIzMjQvMjU2NyJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMTk4Lzc0MjMifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjQ3LzMxOCJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMjE1LzM3NDIifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjEzMi85MTEzIn0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMjAzLzgxMTQifSwgeyJzb3VyY2UiOiAiMjI1LzQxMTUiLCAidGFyZ2V0IjogIjIyOC83ODk3In0sIHsic291cmNlIjogIjIyNS80MTE1IiwgInRhcmdldCI6ICI3Mi8xMTMzOCJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiNDMvMzAxMSJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiNjkvODE2NyJ9LCB7InNvdXJjZSI6ICIyMjUvNDExNSIsICJ0YXJnZXQiOiAiMTI3Lzc0OTcifSwgeyJzb3VyY2UiOiAiMjAzLzgxMTQiLCAidGFyZ2V0IjogIjIyOC83ODk3In0sIHsic291cmNlIjogIjIwMy84MTE0IiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICIyMDMvODExNCIsICJ0YXJnZXQiOiAiMjE1LzM3NDIifSwgeyJzb3VyY2UiOiAiMjAzLzgxMTQiLCAidGFyZ2V0IjogIjcyLzExMzM4In0sIHsic291cmNlIjogIjIwMy84MTE0IiwgInRhcmdldCI6ICI0My8zMDExIn0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICI3OC84NzAwIn0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICIxODIvMjM1NiJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAiMjI4Lzc4OTcifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICIxNDkvMTI4MyJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjI3MS8wNzY5In0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICIzMjQvMjYxMyJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAiZS9Bcm5vRWlnZW53aWxsaWcifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAiMjU4LzUwMDMifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjQ2LzY2MTEifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjEzNy8zMjUyIn0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICIyNy8xMTI2MSJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAiMTc1LzEyMzEifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjIwNS8zOTA4In0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICIyNDMvNTc2MCJ9LCB7InNvdXJjZSI6ICIxOTQvNDgyNCIsICJ0YXJnZXQiOiAiMzI0LzI1NjcifSwgeyJzb3VyY2UiOiAiMTk0LzQ4MjQiLCAidGFyZ2V0IjogIjE5OC83NDIzIn0sIHsic291cmNlIjogIjE5NC80ODI0IiwgInRhcmdldCI6ICI0Ny8zMTgifSwgeyJzb3VyY2UiOiAiNzMvNjE1IiwgInRhcmdldCI6ICIzNjkvNzY4OSJ9LCB7InNvdXJjZSI6ICI3My82MTUiLCAidGFyZ2V0IjogIjIwMy84MjA3In0sIHsic291cmNlIjogIjczLzYxNSIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiNzMvNjE1IiwgInRhcmdldCI6ICIxNjMvMzE3NSJ9LCB7InNvdXJjZSI6ICIyMDMvODIwNyIsICJ0YXJnZXQiOiAiMTI3LzMxMjgifSwgeyJzb3VyY2UiOiAiMjAzLzgyMDciLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjIwMy84MjA3IiwgInRhcmdldCI6ICIxNDIvMzIwMiJ9LCB7InNvdXJjZSI6ICIyMDMvODIwNyIsICJ0YXJnZXQiOiAiMTYzLzMxNzUifSwgeyJzb3VyY2UiOiAiMjAzLzgyMDciLCAidGFyZ2V0IjogImwvSnVyZUxlc2tvdmVjIn0sIHsic291cmNlIjogIjIwMy84MjA3IiwgInRhcmdldCI6ICIzNjkvNzY4OSJ9LCB7InNvdXJjZSI6ICIyMDMvODIwNyIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiMTg0LzIwMjAiLCAidGFyZ2V0IjogInIvQ2hyaXN0b3BoZXJSZSJ9LCB7InNvdXJjZSI6ICIxODQvMjAyMCIsICJ0YXJnZXQiOiAiMjYvMjU5OSJ9LCB7InNvdXJjZSI6ICIxNzYvMjYyMCIsICJ0YXJnZXQiOiAiMTgwLzUzMzUifSwgeyJzb3VyY2UiOiAiMTc2LzI2MjAiLCAidGFyZ2V0IjogIjE3My8yNjA4In0sIHsic291cmNlIjogIjE3Ni8yNjIwIiwgInRhcmdldCI6ICI4Ny83Mjk3In0sIHsic291cmNlIjogIjE3Ni8yNjIwIiwgInRhcmdldCI6ICJzL1N0ZXZlblNraWVuYSJ9LCB7InNvdXJjZSI6ICIxNzYvMjYyMCIsICJ0YXJnZXQiOiAiOTIvNDk4LTEifSwgeyJzb3VyY2UiOiAiMTc2LzI2MjAiLCAidGFyZ2V0IjogIjkyLzk5MjIifSwgeyJzb3VyY2UiOiAiMTczLzI2MDgiLCAidGFyZ2V0IjogIjE4MC81MzM1In0sIHsic291cmNlIjogIjE3My8yNjA4IiwgInRhcmdldCI6ICI4Ny83Mjk3In0sIHsic291cmNlIjogIjE3My8yNjA4IiwgInRhcmdldCI6ICJzL1N0ZXZlblNraWVuYSJ9LCB7InNvdXJjZSI6ICIxOS8xNzY2IiwgInRhcmdldCI6ICIyMzQvNjg3OCJ9LCB7InNvdXJjZSI6ICIxOS8xNzY2IiwgInRhcmdldCI6ICIzNDgvNjQ5OCJ9LCB7InNvdXJjZSI6ICIxOS8xNzY2IiwgInRhcmdldCI6ICI3Mi8xMTI0NCJ9LCB7InNvdXJjZSI6ICIxOS8xNzY2IiwgInRhcmdldCI6ICI4NS80Njk3In0sIHsic291cmNlIjogIjE5LzE3NjYiLCAidGFyZ2V0IjogIjEzLzMxMCJ9LCB7InNvdXJjZSI6ICIxOS8xNzY2IiwgInRhcmdldCI6ICI5Mi8zMDktMSJ9LCB7InNvdXJjZSI6ICIxOS8xNzY2IiwgInRhcmdldCI6ICI1OC83MjExLTIifSwgeyJzb3VyY2UiOiAiNjgvMTQyNSIsICJ0YXJnZXQiOiAicy9TdGV2ZW5Ta2llbmEifSwgeyJzb3VyY2UiOiAiOTIvNDAiLCAidGFyZ2V0IjogIjIyMS82NTE2In0sIHsic291cmNlIjogIjkyLzQwIiwgInRhcmdldCI6ICIzMzkvNzU1OCJ9LCB7InNvdXJjZSI6ICI5Mi80MCIsICJ0YXJnZXQiOiAiNTQvMTIxNiJ9LCB7InNvdXJjZSI6ICI5Mi80MCIsICJ0YXJnZXQiOiAiMjMxLzE5MDQifSwgeyJzb3VyY2UiOiAiOTIvNDAiLCAidGFyZ2V0IjogIjY2LzIxNzMtOSJ9LCB7InNvdXJjZSI6ICI5Mi80MCIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiOTIvNDAiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjkyLzQwIiwgInRhcmdldCI6ICI5My80MDEwLTMxIn0sIHsic291cmNlIjogIjkyLzQwIiwgInRhcmdldCI6ICI2NC8xMDgxMiJ9LCB7InNvdXJjZSI6ICI1OC83MjExLTIiLCAidGFyZ2V0IjogIjIzNC82ODc4In0sIHsic291cmNlIjogIjU4LzcyMTEtMiIsICJ0YXJnZXQiOiAiMzQ4LzY0OTgifSwgeyJzb3VyY2UiOiAiNTgvNzIxMS0yIiwgInRhcmdldCI6ICI3Mi8xMTI0NCJ9LCB7InNvdXJjZSI6ICI1OC83MjExLTIiLCAidGFyZ2V0IjogIjg1LzQ2OTcifSwgeyJzb3VyY2UiOiAiNTgvNzIxMS0yIiwgInRhcmdldCI6ICIxMy8zMTAifSwgeyJzb3VyY2UiOiAiNTgvNzIxMS0yIiwgInRhcmdldCI6ICI5Mi8zMDktMSJ9LCB7InNvdXJjZSI6ICIxMy8zMTAiLCAidGFyZ2V0IjogIjIzNC82ODc4In0sIHsic291cmNlIjogIjEzLzMxMCIsICJ0YXJnZXQiOiAiMzQ4LzY0OTgifSwgeyJzb3VyY2UiOiAiMTMvMzEwIiwgInRhcmdldCI6ICI3Mi8xMTI0NCJ9LCB7InNvdXJjZSI6ICIxMy8zMTAiLCAidGFyZ2V0IjogIjg1LzQ2OTcifSwgeyJzb3VyY2UiOiAiMTMvMzEwIiwgInRhcmdldCI6ICI5Mi8zMDktMSJ9LCB7InNvdXJjZSI6ICI3OC84NzAwIiwgInRhcmdldCI6ICIxODIvMjM1NiJ9LCB7InNvdXJjZSI6ICI3OC84NzAwIiwgInRhcmdldCI6ICIyMjgvNzg5NyJ9LCB7InNvdXJjZSI6ICIyMzQvNjg3OCIsICJ0YXJnZXQiOiAiMzQ4LzY0OTgifSwgeyJzb3VyY2UiOiAiMjM0LzY4NzgiLCAidGFyZ2V0IjogIjcyLzExMjQ0In0sIHsic291cmNlIjogIjIzNC82ODc4IiwgInRhcmdldCI6ICI4NS80Njk3In0sIHsic291cmNlIjogIjIzNC82ODc4IiwgInRhcmdldCI6ICI5Mi8zMDktMSJ9LCB7InNvdXJjZSI6ICIyMTEvMzk2OCIsICJ0YXJnZXQiOiAiMTgyLzIzNTYifSwgeyJzb3VyY2UiOiAiMjExLzM5NjgiLCAidGFyZ2V0IjogIjE2Ni82MjY0In0sIHsic291cmNlIjogIjIxMS8zOTY4IiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICJlL0Fybm9FaWdlbndpbGxpZyIsICJ0YXJnZXQiOiAiMzI0LzI2MTMifSwgeyJzb3VyY2UiOiAiZS9Bcm5vRWlnZW53aWxsaWciLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogImUvQXJub0VpZ2Vud2lsbGlnIiwgInRhcmdldCI6ICIxNjMvMjMzOSJ9LCB7InNvdXJjZSI6ICJlL0Fybm9FaWdlbndpbGxpZyIsICJ0YXJnZXQiOiAiMjIyLzE4ODkifSwgeyJzb3VyY2UiOiAiZS9Bcm5vRWlnZW53aWxsaWciLCAidGFyZ2V0IjogIjI1OC81MDAzIn0sIHsic291cmNlIjogImUvQXJub0VpZ2Vud2lsbGlnIiwgInRhcmdldCI6ICIyNzEvMDc2OSJ9LCB7InNvdXJjZSI6ICJlL0Fybm9FaWdlbndpbGxpZyIsICJ0YXJnZXQiOiAiNDYvNjYxMSJ9LCB7InNvdXJjZSI6ICJlL0Fybm9FaWdlbndpbGxpZyIsICJ0YXJnZXQiOiAiMTM3LzMyNTIifSwgeyJzb3VyY2UiOiAiZS9Bcm5vRWlnZW53aWxsaWciLCAidGFyZ2V0IjogIjI3LzExMjYxIn0sIHsic291cmNlIjogImUvQXJub0VpZ2Vud2lsbGlnIiwgInRhcmdldCI6ICJtL1ZhaGFiU01pcnJva25pIn0sIHsic291cmNlIjogImUvQXJub0VpZ2Vud2lsbGlnIiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICJlL0Fybm9FaWdlbndpbGxpZyIsICJ0YXJnZXQiOiAiMjA1LzM5MDgifSwgeyJzb3VyY2UiOiAiZS9Bcm5vRWlnZW53aWxsaWciLCAidGFyZ2V0IjogIjI0My81NzYwIn0sIHsic291cmNlIjogImUvQXJub0VpZ2Vud2lsbGlnIiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICJlL0Fybm9FaWdlbndpbGxpZyIsICJ0YXJnZXQiOiAiMzI0LzI1NjcifSwgeyJzb3VyY2UiOiAiZS9Bcm5vRWlnZW53aWxsaWciLCAidGFyZ2V0IjogIjE5OC83NDIzIn0sIHsic291cmNlIjogImUvQXJub0VpZ2Vud2lsbGlnIiwgInRhcmdldCI6ICI0Ny8zMTgifSwgeyJzb3VyY2UiOiAiMTMyLzkxMTMiLCAidGFyZ2V0IjogIjIxNS8zNzQyIn0sIHsic291cmNlIjogIjEzMi85MTEzIiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICI1OC83ODAyIiwgInRhcmdldCI6ICJtL1ZhaGFiU01pcnJva25pIn0sIHsic291cmNlIjogIjU4Lzc4MDIiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjU4Lzc4MDIiLCAidGFyZ2V0IjogIjE0OC85NjMyIn0sIHsic291cmNlIjogIjE0Ni83NzQ2IiwgInRhcmdldCI6ICIxNDkvMTI4MyJ9LCB7InNvdXJjZSI6ICIxNDYvNzc0NiIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTQ2Lzc3NDYiLCAidGFyZ2V0IjogIjgwLzcwMyJ9LCB7InNvdXJjZSI6ICIxNDYvNzc0NiIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIxODIvMjM1NiIsICJ0YXJnZXQiOiAiMjcxLzA3NjkifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIxODIvMjM1NiIsICJ0YXJnZXQiOiAiMTQ5LzEyODMifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjE2Ni82MjY0In0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICIyMzIvMTc5NyJ9LCB7InNvdXJjZSI6ICIxODIvMjM1NiIsICJ0YXJnZXQiOiAiMzc4LzQyOTIifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjI2My85OTU5In0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICIxNzYvNDIwNSJ9LCB7InNvdXJjZSI6ICIxODIvMjM1NiIsICJ0YXJnZXQiOiAiMTc1LzEyMzEifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjE3OC8zMjA5In0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICIyMjEvNjUxNiJ9LCB7InNvdXJjZSI6ICIxODIvMjM1NiIsICJ0YXJnZXQiOiAiMzM5Lzc1NTgifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjU0LzEyMTYifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjIzMS8xOTA0In0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICI2Ni8yMTczLTkifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjkzLzQwMTAtMzEifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjY0LzEwODEyIn0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICIzNjkvNzY4OSJ9LCB7InNvdXJjZSI6ICIxODIvMjM1NiIsICJ0YXJnZXQiOiAiMTYzLzMxNzUifSwgeyJzb3VyY2UiOiAiMTgyLzIzNTYiLCAidGFyZ2V0IjogIjIyOC83ODk3In0sIHsic291cmNlIjogIjE4Mi8yMzU2IiwgInRhcmdldCI6ICIxMjcvMzEyOCJ9LCB7InNvdXJjZSI6ICIzMjQvMjYxMyIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMzI0LzI2MTMiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjMyNC8yNjEzIiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICIzMjQvMjYxMyIsICJ0YXJnZXQiOiAiMjU4LzUwMDMifSwgeyJzb3VyY2UiOiAiMzI0LzI2MTMiLCAidGFyZ2V0IjogIjI3MS8wNzY5In0sIHsic291cmNlIjogIjMyNC8yNjEzIiwgInRhcmdldCI6ICI0Ni82NjExIn0sIHsic291cmNlIjogIjMyNC8yNjEzIiwgInRhcmdldCI6ICIxMzcvMzI1MiJ9LCB7InNvdXJjZSI6ICIzMjQvMjYxMyIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiMzI0LzI2MTMiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiMzI0LzI2MTMiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjMyNC8yNjEzIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICIzMjQvMjYxMyIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiMzI0LzI2MTMiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjMyNC8yNjEzIiwgInRhcmdldCI6ICIzMjQvMjU2NyJ9LCB7InNvdXJjZSI6ICIzMjQvMjYxMyIsICJ0YXJnZXQiOiAiMTk4Lzc0MjMifSwgeyJzb3VyY2UiOiAiMzI0LzI2MTMiLCAidGFyZ2V0IjogIjQ3LzMxOCJ9LCB7InNvdXJjZSI6ICIxMzMvMjAyOSIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiMTMzLzIwMjkiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjEzMy8yMDI5IiwgInRhcmdldCI6ICIyNzEvMDc2OSJ9LCB7InNvdXJjZSI6ICIyMTEvODA3MyIsICJ0YXJnZXQiOiAiNDYvNjYxMSJ9LCB7InNvdXJjZSI6ICIyMTEvODA3MyIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIxNDkvNTc5OSIsICJ0YXJnZXQiOiAiNDYvNTQ2NiJ9LCB7InNvdXJjZSI6ICIxNDkvNTc5OSIsICJ0YXJnZXQiOiAiNDYvNjYxMSJ9LCB7InNvdXJjZSI6ICIxNDkvNTc5OSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTQ5LzU3OTkiLCAidGFyZ2V0IjogIjQxLzQ0ODItMiJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICIyODQvOTQwMSJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICIyMzYvNDQ0OSJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICI4Mi85MDU4In0sIHsic291cmNlIjogIjE2LzM0MTEiLCAidGFyZ2V0IjogIjQ1Lzc4MCJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICIyNDYvMjk2MSJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICIyMTMvODA4MCJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICIxMTUvNjQ2Ni0yIn0sIHsic291cmNlIjogIjE2LzM0MTEiLCAidGFyZ2V0IjogIjk5LzQzMyJ9LCB7InNvdXJjZSI6ICIxNi8zNDExIiwgInRhcmdldCI6ICI1MS85Njg3In0sIHsic291cmNlIjogIjE2LzM0MTEiLCAidGFyZ2V0IjogIjIxNS81MDE0In0sIHsic291cmNlIjogIjE2LzM0MTEiLCAidGFyZ2V0IjogIjE5OC8xNDY1In0sIHsic291cmNlIjogIjIyOC83ODk3IiwgInRhcmdldCI6ICIyMTUvNTAxNCJ9LCB7InNvdXJjZSI6ICIyMjgvNzg5NyIsICJ0YXJnZXQiOiAiMjE1LzM3NDIifSwgeyJzb3VyY2UiOiAiMjI4Lzc4OTciLCAidGFyZ2V0IjogIjcyLzExMzM4In0sIHsic291cmNlIjogIjIyOC83ODk3IiwgInRhcmdldCI6ICI0My8zMDExIn0sIHsic291cmNlIjogIjQzLzMwMTEiLCAidGFyZ2V0IjogIjIxNS81MDE0In0sIHsic291cmNlIjogIjQzLzMwMTEiLCAidGFyZ2V0IjogIjIxNS8zNzQyIn0sIHsic291cmNlIjogIjQzLzMwMTEiLCAidGFyZ2V0IjogIjcyLzExMzM4In0sIHsic291cmNlIjogIjEzMC82MzMwIiwgInRhcmdldCI6ICIyOTcvMTgyMSJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIxNDkvMTI4MyJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMjMyLzE3OTcifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjM3OC80MjkyIn0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICJtL1ZhaGFiU01pcnJva25pIn0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIyNjMvOTk1OSJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMTc2LzQyMDUifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIxNzgvMzIwOSJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMjU4LzUwMDMifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjQ2LzY2MTEifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjEzNy8zMjUyIn0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiMjcxLzA3NjkiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjI3MS8wNzY5IiwgInRhcmdldCI6ICIyNzEvMTAyNSJ9LCB7InNvdXJjZSI6ICIyNzEvMDc2OSIsICJ0YXJnZXQiOiAiMjcxLzA2MzcifSwgeyJzb3VyY2UiOiAiMzc4LzQyOTIiLCAidGFyZ2V0IjogIjIzMi8xNzk3In0sIHsic291cmNlIjogIjM3OC80MjkyIiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIzNzgvNDI5MiIsICJ0YXJnZXQiOiAiMTQ5LzEyODMifSwgeyJzb3VyY2UiOiAiMzc4LzQyOTIiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiaC9KaWF3ZWlIYW4iLCAidGFyZ2V0IjogIjY2LzU5MjMtOCJ9LCB7InNvdXJjZSI6ICJoL0ppYXdlaUhhbiIsICJ0YXJnZXQiOiAiMjUwLzk3NTcifSwgeyJzb3VyY2UiOiAiaC9KaWF3ZWlIYW4iLCAidGFyZ2V0IjogIjcxLzY3NjgtMSJ9LCB7InNvdXJjZSI6ICIxOTgvMTQ2NSIsICJ0YXJnZXQiOiAiMjE1LzUwMTQifSwgeyJzb3VyY2UiOiAiMTk4LzE0NjUiLCAidGFyZ2V0IjogIjk5LzQzMyJ9LCB7InNvdXJjZSI6ICIxOTgvMTQ2NSIsICJ0YXJnZXQiOiAiODIvOTA1OCJ9LCB7InNvdXJjZSI6ICI0Ni81NDY2IiwgInRhcmdldCI6ICI0Ni82NjExIn0sIHsic291cmNlIjogIjQ2LzU0NjYiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjQ2LzU0NjYiLCAidGFyZ2V0IjogIjQxLzQ0ODItMiJ9LCB7InNvdXJjZSI6ICIyNDYvMjk2MSIsICJ0YXJnZXQiOiAiNDUvNzgwIn0sIHsic291cmNlIjogIjI0Ni8yOTYxIiwgInRhcmdldCI6ICIyMTMvODA4MCJ9LCB7InNvdXJjZSI6ICIyNDYvMjk2MSIsICJ0YXJnZXQiOiAiMTE1LzY0NjYtMiJ9LCB7InNvdXJjZSI6ICIyNDYvMjk2MSIsICJ0YXJnZXQiOiAiOTkvNDMzIn0sIHsic291cmNlIjogIjI0Ni8yOTYxIiwgInRhcmdldCI6ICI1MS85Njg3In0sIHsic291cmNlIjogIjg1LzQ2OTciLCAidGFyZ2V0IjogIjM0OC82NDk4In0sIHsic291cmNlIjogIjg1LzQ2OTciLCAidGFyZ2V0IjogIjcyLzExMjQ0In0sIHsic291cmNlIjogIjg1LzQ2OTciLCAidGFyZ2V0IjogIjkyLzMwOS0xIn0sIHsic291cmNlIjogIjkyLzQ5OC0xIiwgInRhcmdldCI6ICJzL1N0ZXZlblNraWVuYSJ9LCB7InNvdXJjZSI6ICIxODAvNTQzNiIsICJ0YXJnZXQiOiAiMjA1LzI2NTEifSwgeyJzb3VyY2UiOiAiMTgwLzU0MzYiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjE4MC81NDM2IiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIxODAvNTQzNiIsICJ0YXJnZXQiOiAiNjIvNTg4NCJ9LCB7InNvdXJjZSI6ICI0NS83ODAiLCAidGFyZ2V0IjogIjIxMy84MDgwIn0sIHsic291cmNlIjogIjQ1Lzc4MCIsICJ0YXJnZXQiOiAiMTE1LzY0NjYtMiJ9LCB7InNvdXJjZSI6ICI0NS83ODAiLCAidGFyZ2V0IjogIjk5LzQzMyJ9LCB7InNvdXJjZSI6ICI0NS83ODAiLCAidGFyZ2V0IjogIjUxLzk2ODcifSwgeyJzb3VyY2UiOiAiMjEzLzgwODAiLCAidGFyZ2V0IjogIjExNS82NDY2LTIifSwgeyJzb3VyY2UiOiAiMjEzLzgwODAiLCAidGFyZ2V0IjogIjk5LzQzMyJ9LCB7InNvdXJjZSI6ICIyMTMvODA4MCIsICJ0YXJnZXQiOiAiNTEvOTY4NyJ9LCB7InNvdXJjZSI6ICIyNTAvOTc1NyIsICJ0YXJnZXQiOiAiNjYvNTkyMy04In0sIHsic291cmNlIjogIjI1MC85NzU3IiwgInRhcmdldCI6ICI3MS82NzY4LTEifSwgeyJzb3VyY2UiOiAiNjYvMjE3My05IiwgInRhcmdldCI6ICIyMjEvNjUxNiJ9LCB7InNvdXJjZSI6ICI2Ni8yMTczLTkiLCAidGFyZ2V0IjogIjMzOS83NTU4In0sIHsic291cmNlIjogIjY2LzIxNzMtOSIsICJ0YXJnZXQiOiAiNTQvMTIxNiJ9LCB7InNvdXJjZSI6ICI2Ni8yMTczLTkiLCAidGFyZ2V0IjogIjIzMS8xOTA0In0sIHsic291cmNlIjogIjY2LzIxNzMtOSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiNjYvMjE3My05IiwgInRhcmdldCI6ICI5My80MDEwLTMxIn0sIHsic291cmNlIjogIjY2LzIxNzMtOSIsICJ0YXJnZXQiOiAiNjQvMTA4MTIifSwgeyJzb3VyY2UiOiAiMjE1LzUwMTQiLCAidGFyZ2V0IjogIjIxNS8zNzQyIn0sIHsic291cmNlIjogIjIxNS81MDE0IiwgInRhcmdldCI6ICI3Mi8xMTMzOCJ9LCB7InNvdXJjZSI6ICIyMTUvNTAxNCIsICJ0YXJnZXQiOiAiNjkvODE2NyJ9LCB7InNvdXJjZSI6ICIyMTUvNTAxNCIsICJ0YXJnZXQiOiAiMTI3Lzc0OTcifSwgeyJzb3VyY2UiOiAiMjE1LzUwMTQiLCAidGFyZ2V0IjogIjk5LzQzMyJ9LCB7InNvdXJjZSI6ICIyMTUvNTAxNCIsICJ0YXJnZXQiOiAiODIvOTA1OCJ9LCB7InNvdXJjZSI6ICIyMTUvNTAxNCIsICJ0YXJnZXQiOiAiNzcvMTMxOSJ9LCB7InNvdXJjZSI6ICIxNDkvMTI4MyIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMTQ5LzEyODMiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjE0OS8xMjgzIiwgInRhcmdldCI6ICIyMzIvMTc5NyJ9LCB7InNvdXJjZSI6ICIxNDkvMTI4MyIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIxNDkvMTI4MyIsICJ0YXJnZXQiOiAiMjYzLzk5NTkifSwgeyJzb3VyY2UiOiAiMTQ5LzEyODMiLCAidGFyZ2V0IjogIjE3Ni80MjA1In0sIHsic291cmNlIjogIjE0OS8xMjgzIiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICIxNDkvMTI4MyIsICJ0YXJnZXQiOiAiMTc4LzMyMDkifSwgeyJzb3VyY2UiOiAiMTQ5LzEyODMiLCAidGFyZ2V0IjogIjgwLzcwMyJ9LCB7InNvdXJjZSI6ICI5Mi85OTIyIiwgInRhcmdldCI6ICJzL1N0ZXZlblNraWVuYSJ9LCB7InNvdXJjZSI6ICI5Mi85OTIyIiwgInRhcmdldCI6ICIxODAvNTMzNSJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICIxNjMvMjMzOSJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICIyNTgvNTAwMyJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICIxMzcvMzI1MiJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICIyNy8xMTI2MSJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICJtL1ZhaGFiU01pcnJva25pIn0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjIwNS8zOTA4In0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjI0My81NzYwIn0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjE5OC83NDIzIn0sIHsic291cmNlIjogIjQ2LzY2MTEiLCAidGFyZ2V0IjogIjQ3LzMxOCJ9LCB7InNvdXJjZSI6ICI0Ni82NjExIiwgInRhcmdldCI6ICI0MS80NDgyLTIifSwgeyJzb3VyY2UiOiAiNDYvNjYxMSIsICJ0YXJnZXQiOiAiMTk0LzE1NzAifSwgeyJzb3VyY2UiOiAiNDYvNjYxMSIsICJ0YXJnZXQiOiAiMTgwLzUzMzUifSwgeyJzb3VyY2UiOiAiOTkvNDMzIiwgInRhcmdldCI6ICIxMTUvNjQ2Ni0yIn0sIHsic291cmNlIjogIjk5LzQzMyIsICJ0YXJnZXQiOiAiNTEvOTY4NyJ9LCB7InNvdXJjZSI6ICI5OS80MzMiLCAidGFyZ2V0IjogIjgyLzkwNTgifSwgeyJzb3VyY2UiOiAibC9KdXJlTGVza292ZWMiLCAidGFyZ2V0IjogIjEyNy8zMTI4In0sIHsic291cmNlIjogImwvSnVyZUxlc2tvdmVjIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICJsL0p1cmVMZXNrb3ZlYyIsICJ0YXJnZXQiOiAiMTQyLzMyMDIifSwgeyJzb3VyY2UiOiAibC9KdXJlTGVza292ZWMiLCAidGFyZ2V0IjogIjE2My8zMTc1In0sIHsic291cmNlIjogIjA2LzEwODE0IiwgInRhcmdldCI6ICIyOTcvMTgyMSJ9LCB7InNvdXJjZSI6ICIwNi8xMDgxNCIsICJ0YXJnZXQiOiAiODcvMTAxMDAifSwgeyJzb3VyY2UiOiAiMjMxLzE5MDQiLCAidGFyZ2V0IjogIjIyMS82NTE2In0sIHsic291cmNlIjogIjIzMS8xOTA0IiwgInRhcmdldCI6ICIzMzkvNzU1OCJ9LCB7InNvdXJjZSI6ICIyMzEvMTkwNCIsICJ0YXJnZXQiOiAiNTQvMTIxNiJ9LCB7InNvdXJjZSI6ICIyMzEvMTkwNCIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMjMxLzE5MDQiLCAidGFyZ2V0IjogIjkzLzQwMTAtMzEifSwgeyJzb3VyY2UiOiAiMjMxLzE5MDQiLCAidGFyZ2V0IjogIjY0LzEwODEyIn0sIHsic291cmNlIjogIjI1OC81MDAzIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIyNTgvNTAwMyIsICJ0YXJnZXQiOiAiMTYzLzIzMzkifSwgeyJzb3VyY2UiOiAiMjU4LzUwMDMiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjI1OC81MDAzIiwgInRhcmdldCI6ICIxMzcvMzI1MiJ9LCB7InNvdXJjZSI6ICIyNTgvNTAwMyIsICJ0YXJnZXQiOiAiMjcvMTEyNjEifSwgeyJzb3VyY2UiOiAiMjU4LzUwMDMiLCAidGFyZ2V0IjogIm0vVmFoYWJTTWlycm9rbmkifSwgeyJzb3VyY2UiOiAiMjU4LzUwMDMiLCAidGFyZ2V0IjogIjE3NS8xMjMxIn0sIHsic291cmNlIjogIjI1OC81MDAzIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICIyNTgvNTAwMyIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiMjU4LzUwMDMiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjI1OC81MDAzIiwgInRhcmdldCI6ICIzMjQvMjU2NyJ9LCB7InNvdXJjZSI6ICIyNTgvNTAwMyIsICJ0YXJnZXQiOiAiMTk4Lzc0MjMifSwgeyJzb3VyY2UiOiAiMjU4LzUwMDMiLCAidGFyZ2V0IjogIjQ3LzMxOCJ9LCB7InNvdXJjZSI6ICIzNDgvNjQ5OCIsICJ0YXJnZXQiOiAiNzIvMTEyNDQifSwgeyJzb3VyY2UiOiAiMzQ4LzY0OTgiLCAidGFyZ2V0IjogIjkyLzMwOS0xIn0sIHsic291cmNlIjogIjEzNy8zMjUyIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIxMzcvMzI1MiIsICJ0YXJnZXQiOiAiMTYzLzIzMzkifSwgeyJzb3VyY2UiOiAiMTM3LzMyNTIiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjEzNy8zMjUyIiwgInRhcmdldCI6ICIyNy8xMTI2MSJ9LCB7InNvdXJjZSI6ICIxMzcvMzI1MiIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIxMzcvMzI1MiIsICJ0YXJnZXQiOiAiMTc1LzEyMzEifSwgeyJzb3VyY2UiOiAiMTM3LzMyNTIiLCAidGFyZ2V0IjogIjIwNS8zOTA4In0sIHsic291cmNlIjogIjEzNy8zMjUyIiwgInRhcmdldCI6ICIyNDMvNTc2MCJ9LCB7InNvdXJjZSI6ICIxMzcvMzI1MiIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTM3LzMyNTIiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjEzNy8zMjUyIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIxMzcvMzI1MiIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjkyLzMwOS0xIiwgInRhcmdldCI6ICI3Mi8xMTI0NCJ9LCB7InNvdXJjZSI6ICI5My80MDEwLTMxIiwgInRhcmdldCI6ICIyMjEvNjUxNiJ9LCB7InNvdXJjZSI6ICI5My80MDEwLTMxIiwgInRhcmdldCI6ICIzMzkvNzU1OCJ9LCB7InNvdXJjZSI6ICI5My80MDEwLTMxIiwgInRhcmdldCI6ICI1NC8xMjE2In0sIHsic291cmNlIjogIjkzLzQwMTAtMzEiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjkzLzQwMTAtMzEiLCAidGFyZ2V0IjogIjY0LzEwODEyIn0sIHsic291cmNlIjogIjMzOS83NTU4IiwgInRhcmdldCI6ICIyMjEvNjUxNiJ9LCB7InNvdXJjZSI6ICIzMzkvNzU1OCIsICJ0YXJnZXQiOiAiNTQvMTIxNiJ9LCB7InNvdXJjZSI6ICIzMzkvNzU1OCIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMzM5Lzc1NTgiLCAidGFyZ2V0IjogIjY0LzEwODEyIn0sIHsic291cmNlIjogIjY5LzgxNjciLCAidGFyZ2V0IjogIjEyNy83NDk3In0sIHsic291cmNlIjogIjcyLzExMzM4IiwgInRhcmdldCI6ICIyMTUvMzc0MiJ9LCB7InNvdXJjZSI6ICIyNjMvOTk1OSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMjYzLzk5NTkiLCAidGFyZ2V0IjogIjE3Ni80MjA1In0sIHsic291cmNlIjogIjI2My85OTU5IiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICIyNjMvOTk1OSIsICJ0YXJnZXQiOiAiMTc4LzMyMDkifSwgeyJzb3VyY2UiOiAiMjIxLzY1MTYiLCAidGFyZ2V0IjogIjU0LzEyMTYifSwgeyJzb3VyY2UiOiAiMjIxLzY1MTYiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjIyMS82NTE2IiwgInRhcmdldCI6ICI2NC8xMDgxMiJ9LCB7InNvdXJjZSI6ICIyODQvOTQwMSIsICJ0YXJnZXQiOiAiMjM2LzQ0NDkifSwgeyJzb3VyY2UiOiAiMjg0Lzk0MDEiLCAidGFyZ2V0IjogIjgyLzkwNTgifSwgeyJzb3VyY2UiOiAiMjcvMTEyNjEiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjI3LzExMjYxIiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICIyNy8xMTI2MSIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMjcvMTEyNjEiLCAidGFyZ2V0IjogIjE2My8yMzM5In0sIHsic291cmNlIjogIjI3LzExMjYxIiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICIyNy8xMTI2MSIsICJ0YXJnZXQiOiAibS9WYWhhYlNNaXJyb2tuaSJ9LCB7InNvdXJjZSI6ICIyNy8xMTI2MSIsICJ0YXJnZXQiOiAiMjA1LzM5MDgifSwgeyJzb3VyY2UiOiAiMjcvMTEyNjEiLCAidGFyZ2V0IjogIjI0My81NzYwIn0sIHsic291cmNlIjogIjI3LzExMjYxIiwgInRhcmdldCI6ICIzMjQvMjU2NyJ9LCB7InNvdXJjZSI6ICIyNy8xMTI2MSIsICJ0YXJnZXQiOiAiMTk4Lzc0MjMifSwgeyJzb3VyY2UiOiAiMjcvMTEyNjEiLCAidGFyZ2V0IjogIjQ3LzMxOCJ9LCB7InNvdXJjZSI6ICIyOTcvMTgyMSIsICJ0YXJnZXQiOiAiODcvMTAxMDAifSwgeyJzb3VyY2UiOiAiMTYzLzMxNzUiLCAidGFyZ2V0IjogIjEyNy8zMTI4In0sIHsic291cmNlIjogIjE2My8zMTc1IiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIxNjMvMzE3NSIsICJ0YXJnZXQiOiAiMTQyLzMyMDIifSwgeyJzb3VyY2UiOiAiMTYzLzMxNzUiLCAidGFyZ2V0IjogIjM2OS83Njg5In0sIHsic291cmNlIjogIm0vVmFoYWJTTWlycm9rbmkiLCAidGFyZ2V0IjogIjIzMi8xNzk3In0sIHsic291cmNlIjogIm0vVmFoYWJTTWlycm9rbmkiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIm0vVmFoYWJTTWlycm9rbmkiLCAidGFyZ2V0IjogIjgwLzcwMyJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIxNDgvOTYzMiJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIxNjMvMjMzOSJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIyMjIvMTg4OSJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIxNzUvMTIzMSJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIyNDMvNTc2MCJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIzMjQvMjU2NyJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICJtL1ZhaGFiU01pcnJva25pIiwgInRhcmdldCI6ICI0Ny8zMTgifSwgeyJzb3VyY2UiOiAiMjM2LzQ0NDkiLCAidGFyZ2V0IjogIjgyLzkwNTgifSwgeyJzb3VyY2UiOiAiNTEvOTY4NyIsICJ0YXJnZXQiOiAiMTE1LzY0NjYtMiJ9LCB7InNvdXJjZSI6ICIyNzEvMTAyNSIsICJ0YXJnZXQiOiAiMjcxLzA2MzcifSwgeyJzb3VyY2UiOiAiNTUvNjg5OSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiNTUvNjg5OSIsICJ0YXJnZXQiOiAiMTc1LzEyMzEifSwgeyJzb3VyY2UiOiAiNTUvNjg5OSIsICJ0YXJnZXQiOiAiMTMxLzcxNjkifSwgeyJzb3VyY2UiOiAiMjE1LzM2ODIiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjI2LzI1OTkiLCAidGFyZ2V0IjogInIvQ2hyaXN0b3BoZXJSZSJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTc1LzEyMzEiLCAidGFyZ2V0IjogIjE3Ni80MjA1In0sIHsic291cmNlIjogIjE3NS8xMjMxIiwgInRhcmdldCI6ICIxNzgvMzIwOSJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiMjA1LzI2NTEifSwgeyJzb3VyY2UiOiAiMTc1LzEyMzEiLCAidGFyZ2V0IjogIjQxLzU5NzktMSJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiNjIvNTg4NCJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiMzUyLzQyMzcifSwgeyJzb3VyY2UiOiAiMTc1LzEyMzEiLCAidGFyZ2V0IjogIjQ2LzgzMzMifSwgeyJzb3VyY2UiOiAiMTc1LzEyMzEiLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjE3NS8xMjMxIiwgInRhcmdldCI6ICIyMTUvMzc0MiJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiMTYzLzIzMzkifSwgeyJzb3VyY2UiOiAiMTc1LzEyMzEiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjE3NS8xMjMxIiwgInRhcmdldCI6ICIyMDUvMzkwOCJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiMjQzLzU3NjAifSwgeyJzb3VyY2UiOiAiMTc1LzEyMzEiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjE3NS8xMjMxIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIxNzUvMTIzMSIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjIwNS8zOTA4IiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIyMDUvMzkwOCIsICJ0YXJnZXQiOiAiMTYzLzIzMzkifSwgeyJzb3VyY2UiOiAiMjA1LzM5MDgiLCAidGFyZ2V0IjogIjIyMi8xODg5In0sIHsic291cmNlIjogIjIwNS8zOTA4IiwgInRhcmdldCI6ICIyNDMvNTc2MCJ9LCB7InNvdXJjZSI6ICIyMDUvMzkwOCIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMjA1LzM5MDgiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjIwNS8zOTA4IiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIyMDUvMzkwOCIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjE2My8yMzM5IiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIxNjMvMjMzOSIsICJ0YXJnZXQiOiAiMjIyLzE4ODkifSwgeyJzb3VyY2UiOiAiMTYzLzIzMzkiLCAidGFyZ2V0IjogIjI0My81NzYwIn0sIHsic291cmNlIjogIjE2My8yMzM5IiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIxNjMvMjMzOSIsICJ0YXJnZXQiOiAiMzI0LzI1NjcifSwgeyJzb3VyY2UiOiAiMTYzLzIzMzkiLCAidGFyZ2V0IjogIjE5OC83NDIzIn0sIHsic291cmNlIjogIjE2My8yMzM5IiwgInRhcmdldCI6ICI0Ny8zMTgifSwgeyJzb3VyY2UiOiAiMTI3LzMxMjgiLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjEyNy8zMTI4IiwgInRhcmdldCI6ICIxNDIvMzIwMiJ9LCB7InNvdXJjZSI6ICI3MS82NzY4LTEiLCAidGFyZ2V0IjogIjY2LzU5MjMtOCJ9LCB7InNvdXJjZSI6ICIxOTQvMTU3MCIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTk0LzE1NzAiLCAidGFyZ2V0IjogIjE4MC81MzM1In0sIHsic291cmNlIjogIjgwLzcwMyIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMjE1LzM3NDIiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjE4NC8wMDMzIiwgInRhcmdldCI6ICIxNjkvMTc0MiJ9LCB7InNvdXJjZSI6ICIxODQvMDAzMyIsICJ0YXJnZXQiOiAiNTgvNTQ5MiJ9LCB7InNvdXJjZSI6ICI2Mi81ODg0IiwgInRhcmdldCI6ICIyMDUvMjY1MSJ9LCB7InNvdXJjZSI6ICI2Mi81ODg0IiwgInRhcmdldCI6ICI0MS81OTc5LTEifSwgeyJzb3VyY2UiOiAiNjIvNTg4NCIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMjIyLzE4ODkiLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjIyMi8xODg5IiwgInRhcmdldCI6ICIyNDMvNTc2MCJ9LCB7InNvdXJjZSI6ICIyMjIvMTg4OSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMjIyLzE4ODkiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjIyMi8xODg5IiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIyMjIvMTg4OSIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjIzMi8xNzk3IiwgInRhcmdldCI6ICIyMTcvMTY2OCJ9LCB7InNvdXJjZSI6ICIxNjkvMTc0MiIsICJ0YXJnZXQiOiAiNTgvNTQ5MiJ9LCB7InNvdXJjZSI6ICIxNzgvMzIwOSIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTc4LzMyMDkiLCAidGFyZ2V0IjogIjE3Ni80MjA1In0sIHsic291cmNlIjogIjI0My81NzYwIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIyNDMvNTc2MCIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMjQzLzU3NjAiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjI0My81NzYwIiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIyNDMvNTc2MCIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogInMvU3RldmVuU2tpZW5hIiwgInRhcmdldCI6ICIxODAvNTMzNSJ9LCB7InNvdXJjZSI6ICJzL1N0ZXZlblNraWVuYSIsICJ0YXJnZXQiOiAiODcvNzI5NyJ9LCB7InNvdXJjZSI6ICI4Ny83Mjk3IiwgInRhcmdldCI6ICIxODAvNTMzNSJ9LCB7InNvdXJjZSI6ICI2NC8xMDgxMiIsICJ0YXJnZXQiOiAiNTQvMTIxNiJ9LCB7InNvdXJjZSI6ICI2NC8xMDgxMiIsICJ0YXJnZXQiOiAiMjE3LzE2NjgifSwgeyJzb3VyY2UiOiAiMTgwLzUzMzUiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjQ2LzgzMzMiLCAidGFyZ2V0IjogIjM1Mi80MjM3In0sIHsic291cmNlIjogIjQ2LzgzMzMiLCAidGFyZ2V0IjogIjIxNy8xNjY4In0sIHsic291cmNlIjogIjIxNy8xNjY4IiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIyMTcvMTY2OCIsICJ0YXJnZXQiOiAiMTY2LzYyNjQifSwgeyJzb3VyY2UiOiAiMjE3LzE2NjgiLCAidGFyZ2V0IjogIjE3Ni80MjA1In0sIHsic291cmNlIjogIjIxNy8xNjY4IiwgInRhcmdldCI6ICI1NC8xMjE2In0sIHsic291cmNlIjogIjIxNy8xNjY4IiwgInRhcmdldCI6ICIzNTIvNDIzNyJ9LCB7InNvdXJjZSI6ICIyMTcvMTY2OCIsICJ0YXJnZXQiOiAiMTQ4Lzk2MzIifSwgeyJzb3VyY2UiOiAiMjE3LzE2NjgiLCAidGFyZ2V0IjogIjMyNC8yNTY3In0sIHsic291cmNlIjogIjIxNy8xNjY4IiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIyMTcvMTY2OCIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjIxNy8xNjY4IiwgInRhcmdldCI6ICI0MS80NDgyLTIifSwgeyJzb3VyY2UiOiAiMzI0LzI1NjciLCAidGFyZ2V0IjogIjIzOS82MDcwIn0sIHsic291cmNlIjogIjMyNC8yNTY3IiwgInRhcmdldCI6ICIxOTgvNzQyMyJ9LCB7InNvdXJjZSI6ICIzMjQvMjU2NyIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjE5OC83NDIzIiwgInRhcmdldCI6ICIyMzkvNjA3MCJ9LCB7InNvdXJjZSI6ICIxOTgvNzQyMyIsICJ0YXJnZXQiOiAiNDcvMzE4In0sIHsic291cmNlIjogIjQ3LzMxOCIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiNDEvNTk3OS0xIiwgInRhcmdldCI6ICIyMDUvMjY1MSJ9LCB7InNvdXJjZSI6ICIyMDUvMjY1MSIsICJ0YXJnZXQiOiAiMjM5LzYwNzAifSwgeyJzb3VyY2UiOiAiMjM5LzYwNzAiLCAidGFyZ2V0IjogIjE0Mi8zMjAyIn1dfQ==''')))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2N4mDMcuY1xT"
},
"source": [
"# Test out that you can call Gemini"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "F3KwvNgPYmcJ"
},
"outputs": [],
"source": [
"model = genai.GenerativeModel('gemini-1.5-flash')\n",
"response = model.generate_content(\"How many r's are in the word 'strawberry'?\")\n",
"print(response.text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GefRU-PdfINx"
},
"source": [
"# Ok, let's make some graphs!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DTelg4DpZRMq"
},
"outputs": [],
"source": [
"random_seed = 9876\n",
"\n",
"graphs = generate_graphs(number_of_graphs=500,\n",
" algorithm='er', # Erdos-Reyni random graphs\n",
" directed=False,\n",
" random_seed=random_seed)\n",
"\n",
"from matplotlib import pyplot as plt\n",
"fig, axs = plt.subplots(2, 2)\n",
"i = 0\n",
"for x in [0, 1]:\n",
" for y in [0, 1]:\n",
" nx.draw(graphs[i], ax=axs[x, y])\n",
" axs[x, y].set_title('Graph {}'.format(i))\n",
" i += 1\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4uPk8uewkuBw"
},
"source": [
"Generate zero-shot prompts for the cycle-check task, using the 'adjacency encoding method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xvfHNbb5h9DA"
},
"outputs": [],
"source": [
"task = CycleCheck()\n",
"encoded_examples = task.prepare_examples_dict(\n",
" graphs,\n",
" generator_algorithms = ['er']*len(graphs),\n",
" encoding_method='adjacency')\n",
"\n",
"print('First example: \\n')\n",
"print('Question: ', encoded_examples[0]['question'])\n",
"print('\\n\\nExpected response: ', encoded_examples[0]['answer'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6wN3Da704iMB"
},
"source": [
"Ask Gemini to determine if this graph contains a cycle:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vGMGlmmE4gvH"
},
"outputs": [],
"source": [
"print(model.generate_content(encoded_examples[0]['question']).text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9I2J0p6Xk-4k"
},
"source": [
"Define a simple scoring function for the task, and compute accuracy for the first 10 examples."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "P1GWa-SRlB_M"
},
"outputs": [],
"source": [
"# This is a fairly simple way of checking the answer.\n",
"def judge_cycle_check(example, response_text):\n",
" has_cycle = 'yes' in example['answer'].lower()\n",
" parsed_response = 'yes' in response_text[:10].lower()\n",
" return has_cycle == parsed_response\n",
"\n",
"\n",
"def score_task(model, examples, score_fn):\n",
" num_correct = 0\n",
" for ex in examples:\n",
" response = model.generate_content(ex['question'])\n",
" num_correct += score_fn(ex, response.text)\n",
" return num_correct / len(examples)\n",
"\n",
"score_task(model, [encoded_examples[i] for i in range(10)], judge_cycle_check)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rI9hm8pPfRuZ"
},
"source": [
"We should get a fairly high score (0.9, or 90% in our running), which shows that the model can answer the cycle check task to some level of fidelity."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a9PhAwLcyh6P"
},
"source": [
"## Contest! Design the best Graph Encoding function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zDrqxn7tylHG"
},
"outputs": [],
"source": [
"def my_cool_graph_encoding_function(graph: nx.Graph) -\u003e str:\n",
" # TODO - improve this graph encoding function.\n",
" return 'The graph is cool.'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qAk29lShAkM2"
},
"source": [
"Great, so lets try to score my_cool_graph_encoding_function. Because the prompt has nothing to do with the graph (so far), we don't expect the model to do perform well using it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "etu9L7NzdGWG"
},
"outputs": [],
"source": [
"# use your function to encode and then score\n",
"\n",
"task = CycleCheck()\n",
"cool_encoded_examples = task.prepare_examples_dict(\n",
" graphs,\n",
" generator_algorithms = ['er']*len(graphs),\n",
" encoding_method=my_cool_graph_encoding_function)\n",
"\n",
"score_task(model, [cool_encoded_examples[i] for i in range(10)], judge_cycle_check)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BFLcG4uDfuFy"
},
"source": [
"And the first eval (for us 0.1, or 10%) matches our intuition. A graph encoding should mention something material about the graph!\n",
"\n",
"Let's try to change that now!"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "K7BRk9VDvDCd"
},
"source": [
"## Load DBLP Graph Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "hMzsW1sRpg_N"
},
"outputs": [],
"source": [
"#@title Get Coauthorship Graph\n",
"#@markdown You can use a **saved** coauthorship graph, or get one from the DBLP API.\n",
"\n",
"input_source = \"saved\" # @param [\"saved\",\"dblp\"]\n",
"\n",
"#@markdown If you want to use another author, you'll need the authorid from DBLP. This can be found from your DBLP profile URL.\n",
"\n",
"#@markdown If your profile URL is: https://dblp.org/pid/91/10813.html\n",
"\n",
"#@markdown Then your authorid is:\n",
"\n",
"dblp_author_id = \"91/10813\" # @param {\"type\":\"string\"}\n",
"\n",
"#@markdown Lets also write the corresponding author's name for use later.\n",
"\n",
"author_name = \"Bryan Perozzi\" # @param {\"type\":\"string\"}\n",
"\n",
"if input_source == \"dblp\":\n",
" G = dblp_graph(dblp_author_id, graph_fn=coauthorship_graph)\n",
"else:\n",
" G = saved_coauthorship_graph"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q07pigrcwBW7"
},
"source": [
"### View Graph Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CV_-FeggeDLe"
},
"outputs": [],
"source": [
"# array of node label sizes\n",
"label_size = {x:max(6, 2*math.log2(y+1)) for x,y in G.degree()}\n",
"\n",
"Sigma(G,\n",
" node_label='name',\n",
" raw_node_label_size=label_size,\n",
" start_layout=1,\n",
" node_metrics=[\"louvain\"],\n",
" node_color=\"louvain\",\n",
" default_edge_type=\"line\",\n",
" node_border_color_from=\"node\",\n",
" label_grid_cell_size=50,\n",
" label_font=\"sans-serif\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZIHwHdpEsUad"
},
"source": [
"### Make a simple coauthorship encoder."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8Kc_1cGysFXP"
},
"outputs": [],
"source": [
"def coauthorship_encoder(graph: nx.Graph) -\u003e str:\n",
" text = \"This is the coauthorship information (a graph) of the researcher \" + author_name + \". \"\n",
"\n",
" authorships = []\n",
" for src,dst in G.edges():\n",
" src_name = G.nodes[src][\"name\"]\n",
" dst_name = G.nodes[dst][\"name\"]\n",
" authorships.append(src_name + \" collaborated with \" + dst_name + \".\")\n",
"\n",
" text += '\\n'.join(authorships).strip()\n",
" return text"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6DVmtrldyn1r"
},
"source": [
"## Combining parametric knowledge and graph information"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6lwQKArYuCel"
},
"source": [
"Now that we have an encoder, lets come up with some interesting questions to ask.\n",
"\n",
"**First**, we'll use the LLM's parametric knowledge to try and answer the question, and then we'll with the graph context added.\n",
"\n",
"#### **Example Question: How many colaborators are there (degree)**?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vHXLT7Udt_sC"
},
"outputs": [],
"source": [
"prompt_input = \"How many coauthors does the researcher \" + author_name + \" have?\"\n",
"response = model.generate_content(prompt_input)\n",
"print(\"PROMPT: \" + prompt_input)\n",
"print(\"OUTPUT: \" + response.text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WC-2SurnvIK1"
},
"source": [
"We see that the LLM is hestiant to answer the question. (note: *If it had tried, its quite likely the answer would be wrong*)\n",
"\n",
"Now let's try using our encoder function from above:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "An6qBhg4vMT3"
},
"outputs": [],
"source": [
"prompt_input = coauthorship_encoder(G)+ \"\\nHow many collaborators does the researcher \" + author_name + \" have?\"\n",
"response = model.generate_content(prompt_input)\n",
"# print(\"PROMPT:\" + prompt_input)\n",
"print(\"OUTPUT: \" + response.text)\n",
"print(\"Expected output: \" + str(G.degree(dblp_author_id)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a_3obiIcvNGo"
},
"source": [
"Here we see that the LLM is better able to answer the question, but it still might not be able to do it accurately. 😀\n",
"\n",
"(*If you are interested in improving the LLM's capabilities, check out Part II of this tutorial.*)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "agyudyNWvTAm"
},
"source": [
"We can use the graph to help a model remember knowledge that it already knows (which can benefit factuality, grounding, etc)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ZLAOjx_Wc2VT"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [
"N6cnEIIlXhCj",
"YFxTwdqup_z5",
"-VvPogUCwLUC",
"JUCaM8NDYoMx",
"I88orawwu4BS"
],
"provenance": [
{
"file_id": "1P7nxQJmxy0Jsw8GBDDarUb-vrwPFX-F6",
"timestamp": 1724521885203
},
{
"file_id": "1S9yN9gwUUJmmhPau88lKFNtUYD0G2hnh",
"timestamp": 1724422548058
}
]
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
================================================
FILE: tutorial/KDD-Tutorial-2-Let-Your-Graph-Do-The-Talking.ipynb
================================================
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"gpuType": "V28"
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "TPU"
},
"cells": [
{
"cell_type": "markdown",
"source": [
"This is a noteboook that illustrates how to use Graph Neural Networks to encode structured data for use in Large Language Models. It is Part 2 of a two part tutorial from KDD'24.\n",
"\n",
"**This notebook requires a TPUv2 runtime**\n",
"\n",
"If you find this tutorial useful or want to know more, please consider our publication:\n",
"Let your graph do the talking: Encoding structured data for LLMs\n",
"```\n",
"@article{perozzi2024let,\n",
" title={Let your graph do the talking: Encoding structured data for llms},\n",
" author={Perozzi, Bryan and Fatemi, Bahare and Zelle, Dustin and Tsitsulin, Anton and Kazemi, Mehran and Al-Rfou, Rami and Halcrow, Jonathan},\n",
" journal={arXiv preprint arXiv:2402.05862},\n",
" year={2024}\n",
"}\n",
"```\n",
"\n",
"## Tutorial Part II: GNN Encoding of Graph Information\n",
"This notebook takes the work we did in the first part of the tutorial and extends it to using a Graph Neural Network to directly encode a representation of a graph into a prompt (vs using a text encoding as we did in the previous part).\n",
"\n",
"## Notebook Outline:\n",
"\n",
"Setup (Install Dependencies, download Gemma weights)\n",
"Dataset creation\n",
"Graph-to-Text conversion\n",
"Evaluation\n",
"Exercise: Graph Encoding Challenge\n",
"Exercise: DBLP Dataset\n",
"Setup\n",
"\n",
"## Prework!\n",
"\n",
"Sign-up for Kaggle and consent to the Gemma TOS (this is a requirement to download the Gemma weights used in this notebook).\n",
"https://www.kaggle.com/models/google/gemma/license/consent?returnUrl=%2Fmodels%2Fgoogle%2Fgemma%2FFlax%2F2b-it%2F2"
],
"metadata": {
"id": "EdaaBjBQ3c5g"
}
},
{
"cell_type": "code",
"source": [
"%%capture\n",
"# @title Install Dependencies\n",
"!pip install git+https://github.com/google-deepmind/gemma.git\n",
"!pip install --user kaggle\n",
"!pip install sparse_deferred\n",
"!git clone https://github.com/google-research/talk-like-a-graph.git\n",
"import sys\n",
"sys.path.insert(0, \"/content/talk-like-a-graph\")\n"
],
"metadata": {
"id": "BtYsxQuUQ-K0"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Login to Kaggle\n",
"Follow the link in the login dialog to get an API key if you don't already have one. Also make sure to approve the [Gemma TOS](https://www.kaggle.com/models/google/gemma/license/consent?returnUrl=%2Fmodels%2Fgoogle%2Fgemma%2FFlax%2F2b-it%2F2) as well."
],
"metadata": {
"id": "42csum1-l0bn"
}
},
{
"cell_type": "code",
"source": [
"import kagglehub\n",
"\n",
"kagglehub.login()"
],
"metadata": {
"id": "xmM6CSh7RbIk"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Download Gemma"
],
"metadata": {
"id": "500UNvQOmcMn"
}
},
{
"cell_type": "code",
"source": [
"import os\n",
"VARIANT = '2b-it' # @param ['2b', '2b-it', '7b', '7b-it'] {type:\"string\"}\n",
"weights_dir = kagglehub.model_download(f'google/gemma/Flax/{VARIANT}')\n",
"ckpt_path = os.path.join(weights_dir, VARIANT)\n",
"vocab_path = os.path.join(weights_dir, 'tokenizer.model')"
],
"metadata": {
"id": "W0BzTj2tPROY"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Import dependencies"
],
"metadata": {
"id": "WGFZHBaCRa77"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ycYOnVRpHZuJ",
"cellView": "form"
},
"outputs": [],
"source": [
"# @title\n",
"import os\n",
"from collections.abc import Sequence\n",
"import dataclasses\n",
"from typing import Any, Callable, Mapping\n",
"import sys\n",
"\n",
"\n",
"import chex\n",
"from flax import linen as nn\n",
"import jax\n",
"import jax.numpy as jnp\n",
"import networkx as nx\n",
"import numpy as np\n",
"\n",
"\n",
"from gemma import params as params_lib\n",
"from gemma import transformer as transformer_lib\n",
"from gemma import sampler as sampler_lib\n",
"import sentencepiece as spm\n",
"import sparse_deferred as sd\n",
"from sparse_deferred import jax as sdjnp\n",
"from sparse_deferred.structs import graph_struct\n",
"from sparse_deferred import np as sdnp\n"
]
},
{
"cell_type": "markdown",
"source": [
"## GraphToken library code"
],
"metadata": {
"id": "DzQUN-bCmrjg"
}
},
{
"cell_type": "code",
"source": [
"# @title\n",
"import collections\n",
"from collections.abc import Iterable\n",
"import io\n",
"import json\n",
"from typing import Any, Callable, NamedTuple, Sequence\n",
"\n",
"import numpy as np\n",
"import tqdm\n",
"\n",
"# Code for converting NetworkX graphs to graph tensor\n",
"def laplacian_pos_embedding(graph: nx.Graph, units: int = 4) -\u003e nx.Graph:\n",
" \"\"\"Adds the laplacian positional encoding.\"\"\"\n",
" m = nx.normalized_laplacian_matrix(\n",
" graph, nodelist=sorted(graph.nodes), weight=None\n",
" ).astype(np.float32)\n",
" u, _, _ = np.linalg.svd(m.todense(), compute_uv=True)\n",
" if units \u003e u.shape[1]:\n",
" u = np.pad(u, ((0, 0), (0, units - u.shape[1])))\n",
" nx.set_node_attributes(\n",
" graph, dict(zip(sorted(graph.nodes), u[:, :units])), name='lpe'\n",
" )\n",
" return graph\n",
"\n",
"\n",
"def to_graph_struct(graph: nx.Graph, node_ids: list[int]=None) -\u003e graph_struct.GraphStruct:\n",
" if graph.edges(data=True):\n",
" s, t, w = zip(*[\n",
" (s, t, (d['weight'] if d and 'weight' in d else None))\n",
" for s, t, d in graph.edges(data=True)\n",
" ])\n",
" else:\n",
" s, t, w = (), (), ()\n",
" # tfgnn assumes graphs are directed. Adding the rev edges for an undirected\n",
" # graph.\n",
" if not graph.is_directed():\n",
" s, t, w = s + t, t + s, w + w\n",
"\n",
" graph = laplacian_pos_embedding(graph, units=4)\n",
" return graph_struct.GraphStruct.new(\n",
" nodes={'nodes': {'lpe': np.stack([graph.nodes('lpe')[i] for i in range(graph.number_of_nodes())])}},\n",
" edges={'edges': ((np.array(s, dtype=np.int32), np.array(t, dtype=np.int32)), {})}\n",
" )\n",
"\n",
"\n",
"Tensor = sd.matrix.Tensor\n",
"Features = dict[str, Tensor]\n",
"FeatureSets = dict[str, Features]\n",
"Edge = tuple[tuple[Tensor, ...], Features] # (endpoints, edge features)\n",
"Edges = dict[str, Edge]\n",
"Nodes = FeatureSets\n",
"Schema = dict[str, tuple[str, ...]]\n",
"_Schema = dict[str, tuple[dict[str, int], ...]]\n",
"\n",
"\n",
"\n",
"class FixedSizePadder:\n",
" \"\"\"Adds padding to `GraphStruct` instances for fixed-sized tensors.\n",
"\n",
" Fixed-size tensors can be preferred when running on TPU accelerators.\n",
"\n",
" To use this class, you must first initialize it with statistics of your graphs\n",
" then use it to pad graphs. The statistics can be initialized by invoking\n",
" `calculate_pad_statistics`: this function records the *maximum* observerd size\n",
" of every node and edge set, as well as the standard deviation (std) of sizes.\n",
"\n",
" Once initialized, the function: `pad_graph()` will add padding to the graph.\n",
" Specifically, the node feature (tensors) will be padded with zeros. Similarly,\n",
" edges will be inserted, among newly-added virtual nodes.\n",
"\n",
" Each node (or edge) size will become:\n",
"\n",
" `max observed [per calculate_pad_statistics] + slack*std + 1`\n",
"\n",
" NOTE: there will always be at least one more node or edge, even if the\n",
" statistics show zero std. This is required for making virtual nodes.\n",
"\n",
" All sizes node-set (features) and edge-set (features and adjacency list)\n",
" \"\"\"\n",
"\n",
" def __init__(self, engine: sd.ComputeEngine, slack: float = 1.0):\n",
" # `('edge'|'node', NodeOrEdgeName) -\u003e target size`\n",
" # where `target size` is maximum observed size for node (or edge) set, plus\n",
" # one, plus slack-times-std of observed sizes.\n",
" self.sizes: dict[tuple[str, str], int] = {}\n",
" self.slack = slack\n",
" self._engine = engine\n",
"\n",
" def calculate_pad_statistics(\n",
" self, examples: Iterable[graph_struct.GraphStruct], num_steps: int = 100):\n",
" \"\"\"Measures the max and std of node \u0026 edge sizes of elements of `examples`.\n",
"\n",
" Calling this function is necessary before invoking `pad_graph`.\n",
"\n",
" Args:\n",
" examples: iterable that yields `GraphStruct` examples.\n",
" num_steps: If positive, considers this many samples of `examples`.\n",
" Otherwise, iterates over all `examples`. Warning: this may run\n",
" infinitely on infinite iterators (e.g., `dataset.repeat()`).\n",
" \"\"\"\n",
" sizes: dict[tuple[str, str], list[int]] = collections.defaultdict(list)\n",
" for i, graph in enumerate(examples):\n",
" assert isinstance(graph, graph_struct.GraphStruct)\n",
" if i \u003e 0 and i \u003e= num_steps:\n",
" break\n",
" for node_name, features in graph.nodes.items():\n",
" value_list = sizes[('nodes', node_name)]\n",
" if not features:\n",
" value_list.append(0)\n",
" else:\n",
" value_list.append(list(features.values())[0].shape[0])\n",
"\n",
" for edge_name, edges_tuple in graph.edges.items():\n",
" value_list = sizes[('edges', edge_name)]\n",
" source_nodes = edges_tuple[0][0]\n",
" # if len(value_list) and edge_set.sizes.shape != value_list[-1].shape:\n",
" # continue\n",
" value_list.append(source_nodes.shape[0])\n",
"\n",
" self.sizes = {k: int(1 + max(v) + self.slack * np.std(v))\n",
" for k, v in sizes.items()}\n",
"\n",
" def pad_graph(self, graph: graph_struct.GraphStruct) -\u003e graph_struct.GraphStruct:\n",
" \"\"\"Pads node-sets and edge-sets, with zeros, to max-seen during `calc..`.\n",
"\n",
" This function is useful for running on TPU hardware.\n",
"\n",
" Args:\n",
" graph: contains any number of nodes and edges.\n",
"\n",
" Returns:\n",
" graph with deterministic number of nodes and edges. See class docstring.\n",
" \"\"\"\n",
" if not self.sizes:\n",
" raise ValueError(\n",
" 'No statistics have been initialized. '\n",
" 'Perhaps you forgot to invoke \"calculate_pad_statistics\"?')\n",
" # Edge set name -\u003e (1D vectors containing endpoints**), {\"feature\": Tensor})\n",
" edges: Edges = {}\n",
" # ** tuple should have 2 entries for directed graphs\n",
"\n",
" nodes: Nodes = {}\n",
"\n",
" # For every key in `edges`, store names of node sets that `key` edge\n",
" # connects.\n",
" schema = graph.schema\n",
"\n",
" e = self._engine # for short.\n",
" for node_name, node_features in graph.nodes.items():\n",
" padded_features = {}\n",
" desired_size = self.sizes[('nodes', node_name)]\n",
"\n",
" for feature_name, feature in node_features.items():\n",
" feature = feature[:desired_size] # if `is_oversized`.\n",
" pad = self._engine.maximum(\n",
" desired_size - self._engine.shape(feature)[0], 0)\n",
" zeros = e.zeros(\n",
" tuple([pad] + list(feature.shape[1:])), dtype=feature.dtype)\n",
" padded_feature = e.concat([feature, zeros], axis=0)\n",
" padded_feature = e.reshape(\n",
" padded_feature, [desired_size] + list(padded_feature.shape[1:]))\n",
" padded_features[feature_name] = padded_feature\n",
"\n",
" nodes[node_name] = padded_features\n",
"\n",
" for edge_name, (edge_endpoints, features) in graph.edges.items():\n",
" padded_features = {}\n",
" padded_endpoints = []\n",
" desired_size = self.sizes[('edges', edge_name)]\n",
" current_size = e.shape(edge_endpoints[0])[0]\n",
"\n",
" pad = e.maximum(desired_size - current_size, 0)\n",
" e.assert_greater(pad, -1)\n",
"\n",
" for feature_name, feature in features.items():\n",
" feature = feature[:desired_size] # if `is_oversized`.\n",
" zeros = e.zeros(\n",
" tuple([pad] + list(feature.shape[1:])), dtype=feature.dtype\n",
" )\n",
" padded_feature = e.concat([feature, zeros], axis=0)\n",
" padded_feature = e.reshape(\n",
" padded_feature, [desired_size] + list(padded_feature.shape[1:])\n",
" )\n",
" padded_features[feature_name] = padded_feature\n",
"\n",
" edge_endpoints = [node_ids[:desired_size] for node_ids in edge_endpoints]\n",
" # [[src1_is_valid, src2_is_valid, ...], [tgt1_is_valid, ...]]\n",
" valid = e.cast(\n",
" [\n",
" ids \u003c self.sizes[('nodes', node_name)]\n",
" for ids, node_name in zip(edge_endpoints, schema[edge_name])\n",
" ],\n",
" dtype=bool,\n",
" )\n",
" valid = e.reduce_all(valid, axis=0)\n",
"\n",
" for node_ids, node_name in zip(edge_endpoints, schema[edge_name]):\n",
" # Universe size (e.g., of source or target).\n",
" max_endpoint = self.sizes[('nodes', node_name)] - 1\n",
" node_ids = node_ids[:desired_size]\n",
" node_ids = e.boolean_mask(node_ids, valid)\n",
" pad = desired_size - e.shape(node_ids)[0] # Need only to compute once.\n",
"\n",
" padded_ids = e.concat([\n",
" node_ids,\n",
" e.ones((pad), dtype=node_ids.dtype) * max_endpoint\n",
" ], axis=0)\n",
" padded_ids = e.reshape(padded_ids, [desired_size])\n",
" padded_endpoints.append(padded_ids)\n",
"\n",
" edges[edge_name] = (tuple(padded_endpoints), padded_features)\n",
"\n",
" graph = graph_struct.GraphStruct.new(nodes=nodes, edges=edges, schema=schema)\n",
" return graph\n",
"\n",
"\n",
"\n",
"## gnn.py\n",
"class GIN(nn.Module):\n",
" \"\"\"Graph Isomorphism Network: https://arxiv.org/pdf/1810.00826.pdf.\"\"\"\n",
"\n",
" output_dim: int\n",
" num_hidden_layers: int = 1\n",
" hidden_dim: int = 32\n",
" epsilon: float = 0.1 # See GIN paper (link above)\n",
"\n",
" def setup(self):\n",
" layer_dims = [self.hidden_dim] * self.num_hidden_layers\n",
" self.layers = [\n",
" nn.Dense(dim, use_bias=False, dtype=jnp.bfloat16) for dim in layer_dims\n",
" ]\n",
" self.out_layer = nn.Dense(\n",
" self.output_dim, use_bias=False, dtype=jnp.bfloat16\n",
" )\n",
"\n",
" def __call__(self, graph: graph_struct.GraphStruct) -\u003e jax.Array:\n",
" x = graph.nodes['nodes']['lpe']\n",
" adj = graph.adj(sdjnp.engine, 'edges')\n",
" adj = adj.add_eye(1 + self.epsilon) # self connections with 1+eps weight.\n",
"\n",
" for i, layer in enumerate(self.layers):\n",
" x = layer(adj @ x)\n",
" if i \u003c self.num_hidden_layers:\n",
" x = nn.relu(x)\n",
" x = jnp.concat(x, axis=-1)\n",
" return self.out_layer(x)\n",
"\n",
"\n",
"class GCN(nn.Module):\n",
" \"\"\"Graph convolutional network: https://arxiv.org/pdf/1609.02907.pdf.\"\"\"\n",
"\n",
" output_dim: int\n",
" num_hidden_layers: int = 1\n",
" hidden_dim: int = 32\n",
"\n",
" def setup(self):\n",
" layer_dims = [self.hidden_dim] * self.num_hidden_layers\n",
" self.layers = [nn.Dense(dim, use_bias=False) for dim in layer_dims]\n",
" self.out_layer = nn.Dense(\n",
" self.output_dim, use_bias=False, dtype=jnp.bfloat16\n",
" )\n",
"\n",
" def __call__(self, graph: graph_struct.GraphStruct) -\u003e jax.Array:\n",
" x = graph.nodes['nodes']['lpe']\n",
" adj = graph.adj(sdjnp.engine, 'edges')\n",
" adj_symnorm = (adj + adj.transpose()).add_eye().normalize_symmetric()\n",
"\n",
" for i, layer in enumerate(self.layers):\n",
" x = layer(adj_symnorm @ x)\n",
" if i \u003c self.num_hidden_layers:\n",
" x = nn.relu(x)\n",
" x = jnp.concat(x, axis=-1)\n",
" return self.out_layer(x)\n",
"\n",
"\n",
"## sampler.py\n",
"\n",
"@dataclasses.dataclass\n",
"class SamplerOutput:\n",
"\n",
" # Decoded samples from the model.\n",
" text: list[str]\n",
"\n",
" # Per-step logits used during sampling.\n",
" logits: list[list[float]]\n",
"\n",
" # Tokens corresponding to the generated samples.\n",
" tokens: list[list[int]]\n",
"\n",
" graph_embeddings: list[jnp.ndarray]\n",
"\n",
"\n",
"class GraphTokenSampler:\n",
" \"\"\"Sampler for GraphToken.\"\"\"\n",
"\n",
" def __init__(\n",
" self,\n",
" gnn: nn.Module,\n",
" llm: transformer_lib.Transformer,\n",
" vocab: spm.SentencePieceProcessor,\n",
" params: Mapping[str, Any],\n",
" gnn_token_template: str = r'\u003cunused%d\u003e',\n",
" ):\n",
" \"\"\"Initializes the sampler.\n",
"\n",
" Args:\n",
" gnn: The GNN model.\n",
" llm: The LLM model.\n",
" vocab: The vocab used by the LLM.\n",
" params: The parameters for the GNN and LLM. This should contain the params\n",
" for the gnn under params['gnn'] and the params for the llm under\n",
" params['transformer']\n",
" gnn_token_template: The token used to represent the GNN embedding.\n",
" \"\"\"\n",
"\n",
" self._gnn = gnn\n",
" self._llm = llm\n",
" self._params = params\n",
" self._vocab = vocab\n",
" self._gnn_token_template = gnn_token_template\n",
" self._sampler = sampler_lib.Sampler(\n",
" transformer=self._llm,\n",
" vocab=self._vocab,\n",
" params=self._params['transformer'],\n",
" )\n",
"\n",
" def __call__(\n",
" self,\n",
" input_strings: Sequence[str],\n",
" input_graphs: Sequence[graph_struct.GraphStruct],\n",
" total_generation_steps: int,\n",
" echo: bool = False,\n",
" return_logits: bool = True,\n",
" forbidden_tokens: Sequence[str] | None = None,\n",
" ) -\u003e SamplerOutput:\n",
" \"\"\"Samples from the model.\n",
"\n",
" Args:\n",
" input_strings: The input strings.\n",
" input_graphs: The input graphs.\n",
" total_generation_steps: The number of steps to generate.\n",
" echo: Whether to echo the input.\n",
" return_logits: Whether to return the logits.\n",
" forbidden_tokens: Tokens that are forbidden, in addition to the GNN token.\n",
"\n",
" Returns:\n",
" The sampled output.\n",
" \"\"\"\n",
" assert len(input_graphs) == len(input_strings), (\n",
" len(input_graphs),\n",
" len(input_strings),\n",
" )\n",
" augmented_inputs = []\n",
" full_forbidden_tokens = []\n",
" if forbidden_tokens is not None:\n",
" full_forbidden_tokens += forbidden_tokens\n",
" graph_embeddings = []\n",
" augmented_transformer_params = self._params['transformer']\n",
"\n",
" placeholder_token = PLACEHOLDER_TOKEN\n",
" full_forbidden_tokens.append(placeholder_token)\n",
" placeholder_token_id = self._vocab.EncodeAsIds(placeholder_token)\n",
" assert len(placeholder_token_id) == 1, placeholder_token\n",
" placeholder_token_id = placeholder_token_id[0]\n",
"\n",
" for prompt, graph in zip(input_strings, input_graphs):\n",
" embed = self._gnn.apply(self._params['gnn'], graph)\n",
" assert (\n",
" self._params['transformer']['embedder']['input_embedding'][\n",
" placeholder_token_id\n",
" ].shape\n",
" == embed.shape\n",
" )\n",
" augmented_transformer_params['embedder']['input_embedding'] = (\n",
" augmented_transformer_params['embedder']['input_embedding']\n",
" .at[placeholder_token_id]\n",
" .set(embed)\n",
" )\n",
" graph_embeddings.append(embed)\n",
" augmented_inputs.append(placeholder_token + prompt)\n",
"\n",
" self._sampler.params = augmented_transformer_params\n",
" o = self._sampler(\n",
" input_strings=augmented_inputs,\n",
" total_generation_steps=total_generation_steps,\n",
" echo=echo,\n",
" return_logits=return_logits,\n",
" forbidden_tokens=full_forbidden_tokens,\n",
" )\n",
" return SamplerOutput(\n",
" **dataclasses.asdict(o),\n",
" graph_embeddings=graph_embeddings,\n",
" )\n",
"\n",
"\n",
"\n",
"@chex.dataclass(frozen=True)\n",
"class TrainingInput:\n",
" \"\"\"Batch of training data for a GraphToken model.\"\"\"\n",
"\n",
" # Input tokens given to the model\n",
" input_tokens: np.ndarray # size [B, L]\n",
"\n",
" # A mask that determines which tokens contribute to the target loss\n",
" # calculation.\n",
" target_mask: np.ndarray # size [B, L]\n",
"\n",
" input_graphs: list[graph_struct.GraphStruct] # size [B]\n",
"\n",
" # Ground truth for the input tokens, if representable as an integer.\n",
" # For boolean classification tasks, this is 0/1.\n",
" parsed_ground_truth: np.ndarray | None # size [B]\n",
"\n",
"\n",
"def parse_int(s: str) -\u003e int:\n",
" \"\"\"Parse a string as an integer.\"\"\"\n",
" return int(float(s.strip()))\n",
"\n",
"\n",
"def parse_yes_no(s: str) -\u003e bool:\n",
" \"\"\"Parse a string as a yes/no answer, looking at the first 10 chars.\"\"\"\n",
" return 'yes' in s.lower()[:10]\n",
"\n",
"PLACEHOLDER_TOKEN = '\u003cunused0\u003e'\n",
"\n",
"def graphqa_ds(\n",
" vocab: spm.SentencePieceProcessor,\n",
" encoded_examples: list,\n",
" padder: graph_struct.FixedSizePadder | None = None,\n",
" max_tokens: int = 100,\n",
" gt_parser: Callable[[str], Any] | None = None,\n",
") -\u003e tuple[graph_struct.FixedSizePadder, list[TrainingInput]]:\n",
" \"\"\"Load a GraphQA dataset as a list of TrainingInput.\n",
"\n",
" Args:\n",
" vocab: The vocab to use for tokenization.\n",
" encoded_examples: List of encoded examples generated by GraphQA\n",
" padder: The padder to use for padding the graph. If None, a new padder will\n",
" be created and returned. This is so a padder can be shared across multiple\n",
" datasets / splits.\n",
" max_tokens: The maximum number of tokens to allow in the input. For\n",
" 'task_only' prompting this can be quite small (100 tokens is plenty)\n",
" gt_parser: A function to parse the ground truth from the answer string, used\n",
" to supply the 'parsed_ground_truth' field in the TrainingInput.\n",
"\n",
" Returns:\n",
" The padder used for padding the graphs, and a list of TrainingInput.\n",
" \"\"\"\n",
"\n",
" output = []\n",
" for ex in encoded_examples:\n",
" query = PLACEHOLDER_TOKEN + ex['question'][ex['question'].find('Q:'):]\n",
" answer = ex['answer']\n",
" graph = to_graph_struct(ex['graph'])\n",
" query_tokens = vocab.EncodeAsIds(query)\n",
" answer_tokens = vocab.EncodeAsIds(answer) + [vocab.eos_id()]\n",
" input_tokens = np.array([vocab.bos_id()] + query_tokens + answer_tokens)\n",
" target_mask = np.zeros_like(input_tokens, dtype=jnp.int32)\n",
" # Add one for BOS token\n",
" target_mask[len(query_tokens) + 1 :] = 1\n",
" orig_len = len(query_tokens) + len(answer_tokens) + 1\n",
" input_tokens = np.pad(\n",
" input_tokens,\n",
" [[0, max_tokens - orig_len]],\n",
" constant_values=vocab.pad_id(),\n",
" )\n",
"\n",
" target_mask = np.pad(target_mask, [[0, max_tokens - orig_len]])\n",
"\n",
"\n",
" # The GNN library that we are using requires a global feature. We set\n",
" # a fake value here, but it is unused otherwise.\n",
" #graph = graph.update(nodes={'g': {'foo': np.zeros([1])}})\n",
"\n",
" output.append(\n",
" TrainingInput(\n",
" input_tokens=np.array([input_tokens]),\n",
" target_mask=np.array([target_mask]),\n",
" input_graphs=[graph],\n",
" parsed_ground_truth=np.array(gt_parser(answer)) if gt_parser else None,\n",
" )\n",
" )\n",
" if padder is None:\n",
" padder = FixedSizePadder(sdnp.engine)\n",
" padder.calculate_pad_statistics(\n",
" [e.input_graphs[0] for e in output], len(output)\n",
" )\n",
" for o in output:\n",
" o.input_graphs[0] = padder.pad_graph(o.input_graphs[0])\n",
" return padder, output\n",
"\n",
"\n",
"def decode_questions(\n",
" training_input: TrainingInput, vocab: spm.SentencePieceProcessor\n",
") -\u003e list[str]:\n",
" \"\"\"Decode the question from the input tokens. (ignoring the first 2).\"\"\"\n",
" b, l = training_input.input_tokens.shape\n",
" question_tokens = []\n",
" for i in range(b):\n",
" question_tokens.append([])\n",
" # Skip the first two tokens (BOS and control token).\n",
" for j in range(2, l):\n",
" if training_input.target_mask[i, j] == 1:\n",
" break\n",
" question_tokens[i].append(int(training_input.input_tokens[i, j]))\n",
" return [''.join(vocab.DecodeIds(q)) for q in question_tokens]\n",
"\n",
"\n",
"## training_loop\n",
"import functools\n",
"from typing import Any, MutableMapping\n",
"\n",
"import chex\n",
"from flax import linen as nn\n",
"from gemma import transformer as transformer_lib\n",
"import jax\n",
"import jax.numpy as jnp\n",
"import optax\n",
"import tqdm\n",
"\n",
"Params = MutableMapping[str, Any]\n",
"\n",
"\n",
"def get_attention_mask_and_positions(\n",
" example: jax.Array,\n",
" pad_id: int,\n",
") -\u003e tuple[jax.Array, jax.Array]:\n",
" \"\"\"Builds the position and attention mask vectors from the given tokens.\"\"\"\n",
" pad_mask = example != pad_id\n",
" current_token_position = transformer_lib.build_positions_from_mask(pad_mask)\n",
" attention_mask = transformer_lib.make_causal_attn_mask(pad_mask)\n",
" return current_token_position, attention_mask\n",
"\n",
"\n",
"def forward_and_loss_fn(\n",
" params: Params,\n",
" *,\n",
" gnn: nn.Module,\n",
" llm: transformer_lib.Transformer,\n",
" input_tokens: jax.Array, # Shape [B, L]\n",
" input_graphs: list[graph_struct.GraphStruct], # Shape [B]\n",
" input_mask: jax.Array, # Shape [B, L]\n",
" positions: jax.Array, # Shape [B, L]\n",
" attention_mask: jax.Array, # [B, L, L]\n",
" placeholder_token_id: int,\n",
") -\u003e jax.Array:\n",
" \"\"\"Forward pass and loss function.\n",
"\n",
" Args:\n",
" params: Params for the gnn and transformer. The gnn params are stored in\n",
" params['gnn'] and the llm params are stored in params['transformer'].\n",
" gnn: gnn model to call.\n",
" llm: gemma transformer model to call.\n",
" input_tokens: input tokens sequence, shape [B, L].\n",
" input_graphs: input graphs.\n",
" input_mask: tokens to ignore when computing the loss, shape [B, L].\n",
" positions: relative position of each token, shape [B, L].\n",
" attention_mask: input attention mask, shape [B, L].\n",
" placeholder_token_id: Index in the LLM vocabulary that we are using for passing\n",
" graph embeddings.\n",
"\n",
" Returns:\n",
" Softmax cross-entropy loss for the next-token prediction task.\n",
" \"\"\"\n",
" # Right now we only support batch_size = 1\n",
" chex.assert_axis_dimension(input_tokens, 0, 1)\n",
" chex.assert_equal_shape([input_tokens, input_mask, positions])\n",
" chex.assert_axis_dimension(attention_mask, 0, 1)\n",
" chex.assert_equal(len(input_graphs), 1)\n",
"\n",
" # Get the GNN embedding and update the transformer input embedding for a\n",
" # control token.\n",
" graph_embed = gnn.apply(params['gnn'], input_graphs[0])\n",
" params['transformer']['embedder']['input_embedding'] = (\n",
" params['transformer']['embedder']['input_embedding']\n",
" .at[placeholder_token_id]\n",
" .set(graph_embed)\n",
" )\n",
" # Forward pass on the input data.\n",
" # No attention cache is needed here.\n",
" logits, _ = llm.apply(\n",
" {'params': params['transformer']},\n",
" input_tokens,\n",
" positions,\n",
" None, # Attention cache is None.\n",
" attention_mask,\n",
" )\n",
"\n",
" # Exclude the last step as it does not appear in the targets.\n",
" logits = logits[0, :-1]\n",
"\n",
" # Similarly, the first token cannot be predicted.\n",
" target_tokens = input_tokens[0, 1:]\n",
" target_mask = input_mask[0, 1:]\n",
"\n",
" # Convert the target labels into one-hot encoded vectors.\n",
" one_hot = jax.nn.one_hot(target_tokens, logits.shape[-1])\n",
"\n",
" # Don't update on unwanted tokens.\n",
" one_hot = one_hot * target_mask.astype(one_hot.dtype)[..., jnp.newaxis]\n",
"\n",
" # Normalisation factor.\n",
" norm_factor = 1 / (jnp.sum(target_mask) + 1e-8)\n",
"\n",
" # Return the nll loss.\n",
" return -jnp.sum(jax.nn.log_softmax(logits) * one_hot) * norm_factor\n",
"\n",
"\n",
"@functools.partial(\n",
" jax.jit,\n",
" static_argnames=['gnn', 'llm', 'optimizer', 'pad_id', 'placeholder_token_id'],\n",
")\n",
"def train_step(\n",
" llm: transformer_lib.Transformer,\n",
" gnn: nn.Module,\n",
" params: MutableMapping[str, Any],\n",
" optimizer: optax.GradientTransformation,\n",
" opt_state: optax.OptState,\n",
" pad_id: int,\n",
" example: TrainingInput,\n",
" placeholder_token_id: int,\n",
") -\u003e tuple[jax.Array, Params, optax.OptState]:\n",
" \"\"\"Train step.\n",
"\n",
" Args:\n",
" llm: gemma transformer model.\n",
" gnn: gnn model.\n",
" params: model's input parameters.\n",
" optimizer: optax optimizer to use.\n",
" opt_state: input optimizer's state.\n",
" pad_id: id of the pad token.\n",
" example: input batch.\n",
" placeholder_token_id: Index in the LLM vocabulary that we are using for passing\n",
" graph embeddings.\n",
"\n",
" Returns:\n",
" Training loss, updated parameters, updated optimizer state.\n",
" \"\"\"\n",
"\n",
" # Build the position and attention mask vectors.\n",
" positions, attention_mask = get_attention_mask_and_positions(\n",
" jnp.array(example.input_tokens), pad_id\n",
" )\n",
"\n",
" # Forward and backward passes\n",
" train_loss, grads = jax.value_and_grad(forward_and_loss_fn)(\n",
" params,\n",
" gnn=gnn,\n",
" llm=llm,\n",
" input_tokens=example.input_tokens,\n",
" input_mask=example.target_mask,\n",
" input_graphs=example.input_graphs,\n",
" positions=positions,\n",
" attention_mask=attention_mask,\n",
" placeholder_token_id=placeholder_token_id,\n",
" )\n",
"\n",
" updates, opt_state = optimizer.update(\n",
" grads['gnn'], opt_state, params=params['gnn']\n",
" )\n",
" params['gnn'] = optax.apply_updates(params['gnn'], updates)\n",
"\n",
" return train_loss, params, opt_state\n",
"\n",
"\n",
"@functools.partial(\n",
" jax.jit, static_argnames=['gnn', 'llm', 'pad_id', 'placeholder_token_id']\n",
")\n",
"def validation_step(\n",
" gnn: nn.Module,\n",
" llm: transformer_lib.Transformer,\n",
" params: MutableMapping[str, Any],\n",
" pad_id: int,\n",
" example: TrainingInput,\n",
" placeholder_token_id: int,\n",
") -\u003e jax.Array:\n",
" \"\"\"Validation step.\n",
"\n",
" Args:\n",
" gnn: gnn model.\n",
" llm: gemma transformer model.\n",
" params: model's input parameters. The gnn params are stored in params['gnn']\n",
" and the llm params are stored in params['transformer'].\n",
" pad_id: id of the pad token.\n",
" example: input batch\n",
" placeholder_token_id: Index in the LLM vocabulary that we are using for passing\n",
" graph embeddings.\n",
"\n",
" Returns:\n",
" Validation loss.\n",
" \"\"\"\n",
" jax_input = jax.tree.map(jnp.array, example)\n",
" positions, attention_mask = get_attention_mask_and_positions(\n",
" jax_input.input_tokens, pad_id\n",
" )\n",
" val_loss = forward_and_loss_fn(\n",
" params,\n",
" gnn=gnn,\n",
" llm=llm,\n",
" input_tokens=jax_input.input_tokens,\n",
" input_mask=jax_input.target_mask,\n",
" input_graphs=jax_input.input_graphs,\n",
" positions=positions,\n",
" attention_mask=attention_mask,\n",
" placeholder_token_id=placeholder_token_id,\n",
" )\n",
" return val_loss\n",
"\n",
"\n",
"@chex.dataclass(frozen=True)\n",
"class TrainingConfig:\n",
" learning_rate: float\n",
" num_epochs: int\n",
" eval_every_n: int\n",
" batch_size: int\n",
" max_steps: int | None = None\n",
"\n",
"\n",
"def train_loop(\n",
" llm: transformer_lib.Transformer,\n",
" gnn: nn.Module,\n",
" train_ds: list[TrainingInput],\n",
" validation_ds: list[TrainingInput],\n",
" params: Params,\n",
" training_cfg: TrainingConfig,\n",
" vocab: spm.SentencePieceProcessor,\n",
") -\u003e Params:\n",
" \"\"\"Main training loop for GraphToken.\n",
"\n",
" Args:\n",
" llm: Gemma transformer model.\n",
" gnn: gnn model.\n",
" train_ds: training dataset.\n",
" validation_ds: validation dataset.\n",
" params: Combined params for both the LLM and GNN. The GNN params are stored\n",
" in params['gnn'] and the LLM params are stored in params['transformer'].\n",
" training_cfg: training configuration.\n",
" vocab: sentence piece vocabulary.\n",
"\n",
" Returns:\n",
" Updated model's input parameters.\n",
" \"\"\"\n",
" optimizer = optax.lion(training_cfg.learning_rate)\n",
" opt_state = optimizer.init(params['gnn'])\n",
"\n",
" avg_loss = 0\n",
"\n",
" placeholder_token_id = vocab.EncodeAsIds(PLACEHOLDER_TOKEN)\n",
" assert (\n",
" len(placeholder_token_id) == 1\n",
" ), f'Placeholder token multiple ids: {placeholder_token_id}'\n",
" placeholder_token_id = placeholder_token_id[0]\n",
" # A first round of validation loss\n",
" n_steps_eval = 0\n",
" eval_loss = 0\n",
"\n",
" with tqdm.tqdm(range(training_cfg.num_epochs * len(train_ds))) as pbar:\n",
" averaged_steps = 0\n",
" for n_steps in pbar:\n",
" train_example = train_ds[n_steps % len(train_ds)]\n",
" train_loss, params, opt_state = train_step(\n",
" gnn=gnn,\n",
" llm=llm,\n",
" params=params,\n",
" optimizer=optimizer,\n",
" opt_state=opt_state,\n",
" pad_id=vocab.pad_id(),\n",
" example=train_example,\n",
" placeholder_token_id=placeholder_token_id,\n",
" )\n",
" averaged_steps += 1\n",
" avg_loss += train_loss\n",
" if n_steps and n_steps % training_cfg.eval_every_n == 0:\n",
" val_iterator = validation_ds\n",
" avg_loss /= averaged_steps\n",
" averaged_steps = 0\n",
" pbar.write(\n",
" f'STEP {n_steps} training loss: {avg_loss}'\n",
" )\n",
" avg_loss = 0\n",
" if (\n",
" training_cfg.max_steps is not None\n",
" and n_steps \u003e training_cfg.max_steps\n",
" ):\n",
" break\n",
" if averaged_steps != 0:\n",
" avg_loss /= averaged_steps\n",
" pbar.write(\n",
" f'STEP {n_steps} training loss: {avg_loss}'\n",
" )\n",
" return params\n",
"\n",
"def merge_params(llm_params, gnn_params):\n",
" out = {}\n",
" out.update(llm_params)\n",
" out['gnn'] = gnn_params\n",
" return out"
],
"metadata": {
"id": "RZKA-lU7IsZ8",
"cellView": "form"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Train a GraphToken Model"
],
"metadata": {
"id": "NSuoM8KbnXbF"
}
},
{
"cell_type": "markdown",
"source": [
"Load Gemma Weights"
],
"metadata": {
"id": "oknoqsTMI78J"
}
},
{
"cell_type": "code",
"source": [
"params = params_lib.load_and_format_params(ckpt_path)\n",
"\n",
"# Reshard params over TPU device mesh\n",
"from jax.sharding import PartitionSpec as P\n",
"mesh = jax.sharding.Mesh(np.array(jax.devices()).reshape(4, 2), ('x', 'y'))\n",
"sharding = jax.sharding.NamedSharding(mesh, P('x'))\n",
"def try_to_shard(x):\n",
" try:\n",
" return jax.device_put(x, sharding)\n",
" except:\n",
" return x\n",
"params = jax.tree_map(try_to_shard, params)\n",
"\n",
"\n",
"config_2b = transformer_lib.TransformerConfig.from_params(\n",
" params,\n",
" cache_size=128 # Number of time steps in the transformer's cache\n",
")\n",
"model_2b = transformer_lib.Transformer(config=config_2b)\n",
"\n",
"# Load vocabulary\n",
"vocab = spm.SentencePieceProcessor()\n",
"assert vocab.Load(vocab_path)"
],
"metadata": {
"id": "U1q6PiLSCTTa"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Generate some training data, for the CycleCheck task"
],
"metadata": {
"id": "NUDd2EisJHX_"
}
},
{
"cell_type": "code",
"source": [
"from talk_like_a_graph import graph_generators\n",
"from talk_like_a_graph import graph_tasks\n",
"random_seed = 9876\n",
"\n",
"train_graphs = graph_generators.generate_graphs(number_of_graphs=500,\n",
" algorithm='er', # Erdos-Reyni random graphs\n",
" directed=False,\n",
" random_seed=random_seed)\n",
"test_graphs = graph_generators.generate_graphs(number_of_graphs=10,\n",
" algorithm='er', # Erdos-Reyni random graphs\n",
" directed=False,\n",
" random_seed=random_seed + 12385)\n",
"task = graph_tasks.CycleCheck()\n",
"train_examples = list(task.prepare_examples_dict(\n",
" train_graphs,\n",
" generator_algorithms = ['er']*len(train_graphs),\n",
" encoding_method='adjacency').values())\n",
"test_examples = list(task.prepare_examples_dict(\n",
" test_graphs,\n",
" generator_algorithms = ['er']*len(test_graphs),\n",
" encoding_method='adjacency').values())\n",
"padder, train_ds = graphqa_ds(vocab, train_examples, max_tokens=25)\n",
"_, test_ds = graphqa_ds(vocab, test_examples, max_tokens=25, padder=padder)"
],
"metadata": {
"id": "93oCqfLYnXFr"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Train GraphToken"
],
"metadata": {
"id": "dJTlxG9yUx9y"
}
},
{
"cell_type": "code",
"source": [
"gin = GIN(config_2b.embed_dim, num_hidden_layers=3, hidden_dim=4)\n",
"key = jax.random.PRNGKey(0)\n",
"gnn_params = gin.init(key, train_ds[0].input_graphs[0])\n",
"\n",
"\n",
"train_config = TrainingConfig(\n",
" learning_rate=0.0001, num_epochs=3, eval_every_n=250, batch_size=1\n",
")\n",
"params_learned = train_loop(\n",
" llm=model_2b,\n",
" gnn=gin,\n",
" train_ds=train_ds,\n",
" validation_ds=test_ds,\n",
" params=merge_params(params, gnn_params),\n",
" training_cfg=train_config,\n",
" vocab=vocab,\n",
")"
],
"metadata": {
"id": "Uq3ZF3v7Ivo7"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Sample outputs"
],
"metadata": {
"id": "4eaBMWhPchoJ"
}
},
{
"cell_type": "code",
"source": [
"from IPython.display import Markdown, display\n",
"\n",
"graph_token_sampler = GraphTokenSampler(\n",
" params=params_learned, llm=model_2b, gnn=gin, vocab=vocab\n",
")\n",
"\n",
"\n",
"def get_graph_qa_question(ex):\n",
" with_graph = ex['question']\n",
" q_index = with_graph.find('Q:')\n",
" return with_graph[q_index:]\n",
"\n",
"for i in range(len(test_examples)):\n",
" tokenized_input = test_ds[i]\n",
" ex = test_examples[i]\n",
" prompt = get_graph_qa_question(ex)\n",
" if i == 0:\n",
" display(\n",
" Markdown(\n",
" '**Prompt:** '\n",
" + prompt\n",
" + '\\n\\n'\n",
" )\n",
" )\n",
" llm_output = graph_token_sampler(\n",
" [prompt],\n",
" tokenized_input.input_graphs,\n",
" total_generation_steps=15,\n",
" return_logits=False,\n",
" ).text[0]\n",
" display(Markdown(f'**LLM Output:** \"{llm_output}\"'))\n",
" display(\n",
" Markdown(f\"**Ground Truth:** {ex['answer']}\")\n",
" )\n",
" display(Markdown('-' * 80))\n",
" print()\n",
"\n"
],
"metadata": {
"id": "0p4CxB2Lcilw"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Exercise: Train a model for a different task.\n",
"\n",
"Take the above code and modify it for the NodeCount task.\n",
"How does your model perform?\n",
"\n",
"If your kernel runs out of memory run the following code to clear the TPU memory, then re-run the code block labeled 'Load Gemma weights' and retry.\n",
"```\n",
"for a in jax.live_arrays():\n",
" a.delete()\n",
"```"
],
"metadata": {
"id": "uuW3rA3-qKy9"
}
}
]
}
================================================
FILE: tutorial/README.md
================================================
# Tutorial on Graph Reasoning with LLMs (GReaL)
## **KDD'24 Tutorial**
### Sunday, August 25th, 2024
Centre de Convencions Internacional de Barcelona
P1 122-123
2:00 pm - 5:00 pm CEST
### Overview
Graphs are a powerful tool for representing and analyzing complex relationships
in real-world applications. Large Language Models (LLMs) have demonstrated
impressive capabilities by advancing state-of-the-art on many language-based
benchmarks. Their ability to process and understand natural language open
exciting possibilities in various domains. Despite the remarkable progress in
automated reasoning with natural text, reasoning on graphs with LLMs remains an
understudied problem that has recently gained more attention.
This tutorial builds upon recent advances in expressing reasoning problems
through the lens of tasks on graph data. The first part of the tutorial will
provide an in-depth discussion of techniques for representing graphs as inputs
to LLMs. The second, hands-on, portion will demonstrate these techniques in a
practical setting.
### Schedule
This talk is being given at [KDD 2024](https://kdd2024.kdd.org/) at 2pm in room
P1 122-123 at the Centre de Convencions Internacional de Barcelona.
### Tutorial Material
The slides and notebooks used in the tutorial are available here:
- [Tutorial Slides](https://drive.google.com/file/d/16rHZVyCeGDfY3djVafKHFPhnabOVtd03/)
- [Talk Like a Graph Notebook](https://github.com/google-research/talk-like-a-graph/blob/main/tutorial/KDD-Tutorial-1-Talk-Like-a-Graph.ipynb)
- [GraphToken (Let Your Graph Do the Talking) Notebook](https://github.com/google-research/talk-like-a-graph/blob/main/tutorial/KDD-Tutorial-2-Let-Your-Graph-Do-The-Talking.ipynb)
### Additional Resources
*coming soon*