Showing preview only (373K chars total). Download the full file or copy to clipboard to get everything.
Repository: wannesm/LeuvenMapMatching
Branch: master
Commit: 9ca9f0b73665
Files: 69
Total size: 352.0 KB
Directory structure:
gitextract_6vcq1ios/
├── .gitignore
├── .readthedocs.yaml
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.md
├── docs/
│ ├── Makefile
│ ├── classes/
│ │ ├── map/
│ │ │ ├── BaseMap.rst
│ │ │ ├── InMemMap.rst
│ │ │ └── SqliteMap.rst
│ │ ├── matcher/
│ │ │ ├── BaseMatcher.rst
│ │ │ ├── BaseMatching.rst
│ │ │ ├── DistanceMatcher.rst
│ │ │ └── SimpleMatcher.rst
│ │ ├── overview.rst
│ │ └── util/
│ │ └── Segment.rst
│ ├── conf.py
│ ├── index.rst
│ ├── make.bat
│ ├── requirements.txt
│ └── usage/
│ ├── customdistributions.rst
│ ├── debug.rst
│ ├── incremental.rst
│ ├── installation.rst
│ ├── introduction.rst
│ ├── latitudelongitude.rst
│ ├── openstreetmap.rst
│ └── visualisation.rst
├── leuvenmapmatching/
│ ├── __init__.py
│ ├── map/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── inmem.py
│ │ └── sqlite.py
│ ├── matcher/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── distance.py
│ │ ├── newsonkrumm.py
│ │ └── simple.py
│ ├── util/
│ │ ├── __init__.py
│ │ ├── debug.py
│ │ ├── dist_euclidean.py
│ │ ├── dist_latlon.py
│ │ ├── dist_latlon_nvector.py
│ │ ├── evaluation.py
│ │ ├── gpx.py
│ │ ├── kalman.py
│ │ ├── openstreetmap.py
│ │ ├── projections.py
│ │ └── segment.py
│ └── visualization.py
├── setup.cfg
├── setup.py
└── tests/
├── examples/
│ ├── example_1_simple.py
│ └── example_using_osmnx_and_geopandas.py
├── rsrc/
│ ├── bug2/
│ │ └── readme.md
│ ├── newson_krumm_2009/
│ │ └── readme.md
│ └── path_latlon/
│ ├── readme.md
│ ├── route.gpx
│ └── route2.gpx
├── test_bugs.py
├── test_conversion.py
├── test_examples.py
├── test_newsonkrumm2009.py
├── test_nonemitting.py
├── test_nonemitting_circle.py
├── test_parallelroads.py
├── test_path.py
├── test_path_latlon.py
└── test_path_onlyedges.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.cache
.eggs
.idea
dist
*.egg-info
venv*
README
.ipynb_checkpoints
*.zip
*.gv
*.gv.pdf
*.xml
tests/route.gpx
examples/Leuven\ Stadswandeling*
.git-old
.pytest_cache
docs/_build
build
cache
baselines
./examples
*.pkl
================================================
FILE: .readthedocs.yaml
================================================
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
sphinx:
configuration: docs/conf.py
python:
install:
- requirements: docs/requirements.txt
================================================
FILE: LICENSE
================================================
Leuven.MapMatching
------------------
Copyright 2015-2018 KU Leuven, DTAI Research Group
Copyright 2017-2018 Sirris
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This package/repository contains code from the dtaimapmatching project as well
as some open-source works:
Latitude/longitude spherical geodesy tools
------------------------------------------
Latitude/longitude spherical geodesy tools
(c) Chris Veness 2002-2017, MIT Licence
www.movable-type.co.uk/scripts/latlong.html
www.movable-type.co.uk/scripts/geodesy/docs/module-latlon-spherical.html
Nvector
-------
Gade, K. (2010). A Nonsingular Horizontal Position Representation, The Journal
of Navigation, Volume 63, Issue 03, pp 395-417, July 2010.
(www.navlab.net/Publications/A_Nonsingular_Horizontal_Position_Representation.pdf)
This paper should be cited in publications using this library.
Copyright (c) 2015, Norwegian Defence Research Establishment (FFI)
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above publication
information, copyright notice, this list of conditions and the following
disclaimer.
2. Redistributions in binary form must reproduce the above publication
information, copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS
BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: MANIFEST.in
================================================
include README.md
include LICENSE
================================================
FILE: Makefile
================================================
.PHONY: test3
test3:
@#export PYTHONPATH=.;./venv/bin/py.test --ignore=venv -vv
python3 setup.py test
.PHONY: test2
test2:
python2 setup.py test
.PHONY: test
test: test3 test2
.PHONY: version
version:
@python3 setup.py --version
.PHONY: prepare_dist
prepare_dist:
rm -rf dist/*
python3 setup.py sdist bdist_wheel
.PHONY: prepare_tag
prepare_tag:
@echo "Check whether repo is clean"
git diff-index --quiet HEAD
@echo "Check correct branch"
if [[ "$$(git rev-parse --abbrev-ref HEAD)" != "master" ]]; then echo 'Not master branch'; exit 1; fi
@echo "Add tag"
git tag "v$$(python3 setup.py --version)"
git push --tags
.PHONY: deploy
deploy: prepare_dist prepare_tag
@echo "Check whether repo is clean"
git diff-index --quiet HEAD
@echo "Start uploading"
twine upload --repository leuvenmapmatching dist/*
.PHONY: docs
docs:
export PYTHONPATH=..; cd docs; make html
.PHONY: docsclean
docsclean:
cd docs; make clean
.PHONY: clean
clean: docsclean
================================================
FILE: README.md
================================================
# Leuven.MapMatching
[](https://pypi.org/project/leuvenmapmatching/)
[](https://leuvenmapmatching.readthedocs.io/en/latest/?badge=latest)
Align a trace of GPS measurements to a map or road segments.
The matching is based on a Hidden Markov Model (HMM) with non-emitting
states. The model can deal with missing data and you can plug in custom
transition and emission probability distributions.
Main reference:
> Meert Wannes, Mathias Verbeke, "HMM with Non-Emitting States for Map Matching",
> European Conference on Data Analysis (ECDA), Paderborn, Germany, 2018.
Other references:
> Devos Laurens, Vandebril Raf (supervisor), Meert Wannes (supervisor),
> "Traffic patterns revealed through matrix functions and map matching",
> Master thesis, Faculty of Engineering Science, KU Leuven, 2018
## Installation and usage
$ pip install leuvenmapmatching
More information and examples:
[leuvenmapmatching.readthedocs.io](https://leuvenmapmatching.readthedocs.io)
## Dependencies
Required:
- [numpy](http://www.numpy.org)
- [scipy](https://www.scipy.org)
Optional (only loaded when methods are called to rely on these packages):
- [matplotlib](http://matplotlib.org):
For visualisation
- [smopy](https://github.com/rossant/smopy):
For visualisation
- [nvector](https://github.com/pbrod/Nvector):
For latitude-longitude computations
- [gpxpy](https://github.com/tkrajina/gpxpy):
To import GPX files
- [pykalman](https://pykalman.github.io):
So smooth paths using a Kalman filter
- [pyproj](https://jswhit.github.io/pyproj/):
To project latitude-longitude coordinates to an XY-plane
- [rtree](http://toblerity.org/rtree/):
To quickly search locations
## Contact
Wannes Meert, DTAI, KU Leuven
wannes.meert@cs.kuleuven.be
https://dtai.cs.kuleuven.be
Mathias Verbeke, Sirris
mathias.verbeke@sirris.be
http://www.sirris.be/expertise/data-innovation
Developed with the support of [Elucidata.be](http://www.elucidata.be).
## License
Copyright 2015-2022, KU Leuven - DTAI Research Group, Sirris - Elucidata Group
Apache License, Version 2.0.
================================================
FILE: docs/Makefile
================================================
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = DTAIMap-Matching
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
================================================
FILE: docs/classes/map/BaseMap.rst
================================================
BaseMap
=======
.. autoclass:: leuvenmapmatching.map.base.BaseMap
:members:
================================================
FILE: docs/classes/map/InMemMap.rst
================================================
InMemMap
========
.. autoclass:: leuvenmapmatching.map.inmem.InMemMap
:members:
================================================
FILE: docs/classes/map/SqliteMap.rst
================================================
SqliteMap
=========
.. autoclass:: leuvenmapmatching.map.sqlite.SqliteMap
:members:
================================================
FILE: docs/classes/matcher/BaseMatcher.rst
================================================
BaseMatcher
===========
This a generic base class to be used by matchers. This class itself
does not implement a working matcher. Use a matcher such as
``SimpleMatcher``, ``DistanceMatcher``, ...
.. autoclass:: leuvenmapmatching.matcher.base.BaseMatcher
:members:
================================================
FILE: docs/classes/matcher/BaseMatching.rst
================================================
BaseMatching
============
.. autoclass:: leuvenmapmatching.matcher.base.BaseMatching
:members:
================================================
FILE: docs/classes/matcher/DistanceMatcher.rst
================================================
DistanceMatcher
===============
.. autoclass:: leuvenmapmatching.matcher.distance.DistanceMatcher
:members:
================================================
FILE: docs/classes/matcher/SimpleMatcher.rst
================================================
SimpleMatcher
=============
.. autoclass:: leuvenmapmatching.matcher.simple.SimpleMatcher
:members:
================================================
FILE: docs/classes/overview.rst
================================================
matcher
~~~~~~~
.. toctree::
:caption: Matcher
matcher/BaseMatcher
matcher/SimpleMatcher
matcher/DistanceMatcher
.. toctree::
:caption: Matching
matcher/BaseMatching
map
~~~
.. toctree::
:caption: Map
map/BaseMap
map/InMemMap
map/SqliteMap
util
~~~~
.. toctree::
:caption: Util
util/Segment
================================================
FILE: docs/classes/util/Segment.rst
================================================
Segment
=======
.. autoclass:: leuvenmapmatching.util.segment.Segment
:members:
================================================
FILE: docs/conf.py
================================================
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# Leuven.MapMatching documentation build configuration file, created by
# sphinx-quickstart on Sat Apr 14 23:24:31 2018.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.mathjax',
'sphinx.ext.viewcode']
autoclass_content = 'both'
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'Leuven.MapMatching'
copyright = '2018-2022, Wannes Meert'
author = 'Wannes Meert'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1.1.1'
# The full version, including alpha/beta/rc tags.
release = '1.1.1'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = 'en'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
# html_theme = 'alabaster'
html_theme = "sphinx_rtd_theme"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'LeuvenMapMatchingDoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'LeuvenMapMatching.tex', 'Leuven.MapMatching Documentation',
'Wannes Meert', 'manual'),
]
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'leuvenmapmatching', 'Leuven.MapMatching Documentation',
[author], 1)
]
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'LeuvenMapMatching', 'Leuven.MapMatching Documentation',
author, 'LeuvenMapMatching', 'Map Matching',
'Miscellaneous'),
]
================================================
FILE: docs/index.rst
================================================
.. Leuven.MapMatching documentation master file, created by
sphinx-quickstart on Sat Apr 14 23:24:31 2018.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Leuven.MapMatching's documentation
==================================
Align a trace of coordinates (e.g. GPS measurements) to a map of road segments.
The matching is based on a Hidden Markov Model (HMM) with non-emitting
states. The model can deal with missing data and you can plug in custom
transition and emission probability distributions.
Reference:
Meert Wannes, Mathias Verbeke, "HMM with Non-Emitting States for Map Matching",
European Conference on Data Analysis (ECDA), Paderborn, Germany, 2018.
.. figure:: https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/example1.png?v=2
:alt: example
.. toctree::
:maxdepth: 2
:caption: Contents:
.. toctree::
:caption: Usage
usage/installation
usage/introduction
usage/openstreetmap
usage/visualisation
usage/latitudelongitude
usage/customdistributions
usage/incremental
usage/debug
.. toctree::
:maxdepth: 2
:caption: Classes
classes/overview
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
================================================
FILE: docs/make.bat
================================================
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
set SPHINXPROJ=DTAIMap-Matching
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
================================================
FILE: docs/requirements.txt
================================================
numpy
scipy
sphinx_rtd_theme
================================================
FILE: docs/usage/customdistributions.rst
================================================
Custom probability distributions
================================
You can use your own custom probability distributions for the transition and emission probabilities.
This is achieved by inheriting from the :class:`BaseMatcher` class.
Examples are available in the :class:`SimpleMatching` class and :class:`DistanceMatching` class.
The latter implements a variation based on Newson and Krumm (2009).
Transition probability distribution
-----------------------------------
Overwrite the :meth:`logprob_trans` method.
For example, if you want to use a uniform distribution over the possible road segments:
.. code-block:: python
def logprob_trans(self, prev_m, edge_m, edge_o, is_prev_ne, is_next_ne):
return -math.log(len(self.matcher.map.nodes_nbrto(self.edge_m.last_point())))
Note that ``prev_m.edge_m`` and ``edge_m`` are not necessarily connected. For example if the ``Map`` object
returns a neighbor state that is not connected in the roadmap. This functionality is used to allow switching lanes.
Emission probability distribution
---------------------------------
Overwrite the :meth:`logprob_obs` method for non-emitting nodes.
These methods are given the closest distance as `dist`, the previous :class:`Matching` object
in the lattice, the state as `edge_m`, and the observation as `edge_o`. The latter two are :class:`Segment` objects
that can represent either a segment or a point.
Each segment also has a project point which is the point on the segment that is the closest point.
For example, a simple step function with more tolerance for non-emitting nodes:
.. code-block:: python
def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne):
if is_ne:
if dist < 50:
return -math.log(50)
else:
if dist < 10:
return -math.log(10)
return -np.inf
Note that an emission probability can be given for a non-emitting node. This allows you to rank non-emitting nodes
even when no observations are available. It will then insert pseudo-observations on the line between the previous
and next observations.
To have a pure non-emitting node, the `logprob_obs` method should always return 0 if the
``is_ne`` argument is true.
Custom lattice objects
----------------------
If you need to store additional information in the lattice, inherit from the :class:`Matching` class and
pass your custom object to the :class:`Matcher` object.
.. code-block:: python
from leuvenmapmatching.map.base import BaseMatching
class MyMatching(BaseMatching):
...
matcher = MyMatcher(mapdb, matching=MyMatching)
================================================
FILE: docs/usage/debug.rst
================================================
Debug
=====
Increasing the verbosity level
------------------------------
To inspect the intermediate steps that the algorithm take, you can increase
the verbosity level of the package. For example:
.. code-block:: python
import sys
import logging
import leuvenmapmatching
logger = leuvenmapmatching.logger
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))
Inspect the best matching
-------------------------
The best match is available in ``matcher.lattice_best``. This is a list of
``Matching`` objects. For example after running the first example in the introduction:
.. code-block:: python
>>> matcher.lattice_best
[Matching<A-B-0-0>,
Matching<A-B-1-0>,
Matching<A-B-2-0>,
...
A matching object summarizes its information as a tuple with three values if
the best match is with a vertex: <label-observation-nonemitting>. And a tuple
with four values if the best match is with an edge: <labelstart-labelend-observation-nonemitting>.
In the example above, the first observation (with index 0) is matched to a point on the edge
A-B. If you want to inspect the exact locations, you can query the ``Segment``
objects that express the observation and map: ``matching.edge_o`` and ``matching.edge_m``.
.. code-block:: python
>>> match = matcher.lattice_best[0]
>>> match.edge_m.l1, match.edge_m.l2 # Edge start/end labels
('A', 'B')
>>> match.edge_m.pi # Best point on A-B edge
(1.0, 1.0)
>>> match.edge_m.p1, match.edge_m.p2 # Locations of A and B
((1, 1), (1, 3))
>>> match.edge_o.l1, match.edge_o.l2 # Observation
('O0', None)
>>> match.edge_o.pi # Location of observation O0, because no second location
(0.8, 0.7)
>>> match.edge_o.p1 # Same as pi because no interpolation
(0.8, 0.7)
Inspect the matching lattice
----------------------------
All paths through the lattice are available in ``matcher.lattice``.
The lattice is a dictionary with a ``LatticeColumn`` object for each observation
(in case the full path of observations is matched).
For each observation, you can inspect the ``Matching`` objects with:
.. code-block:: python
>>> matcher.lattice
{0: <leuvenmapmatching.matcher.base.LatticeColumn at 0x12369bf40>,
1: <leuvenmapmatching.matcher.base.LatticeColumn at 0x123639dc0>,
2: <leuvenmapmatching.matcher.base.LatticeColumn at 0x123603f40>,
...
>>> matcher.lattice[0].values_all()
{Matching<A-B-0-0>,
Matching<A-B-0-1>,
Matching<A-C-0-0>,
...
To start backtracking you can, for example, see which matching object
for the last element has the highest probability (thus the best match):
.. code-block:: python
>>> m = max(matcher.lattice[len(path)-1].values_all(), key=lambda m: m.logprob)
>>> m.logprob
-0.6835815469734807
The previous matching objects can be queried with. These are only those
matches that are connected to this matchin the lattice (in this case
nodes in the street graph with an edge to the current node):
.. code-block:: python
>>> m.prev # Best previous match with a connection (multiple if equal probability)
{Matching<E-F-20-0>}
>>> m.prev_other # All previous matches in the lattice with a connection
{Matching<C-E-20-0>,
Matching<D-E-20-0>,
Matching<F-E-20-0>,
Matching<Y-E-20-0>}
================================================
FILE: docs/usage/incremental.rst
================================================
Incremental matching
====================
Example: Incremental matching
-------------------------------
If the observations are collected in a streaming setting. The matching can also be invoked incrementally.
The lattice will be built further every time a new subsequence of the path is given.
.. code-block:: python
from leuvenmapmatching.matcher.distance import DistanceMatcher
from leuvenmapmatching.map.inmemmap import InMemMap
map_con = InMemMap("mymap", graph={
"A": ((1, 1), ["B", "C", "X"]),
"B": ((1, 3), ["A", "C", "D", "K"]),
"C": ((2, 2), ["A", "B", "D", "E", "X", "Y"]),
"D": ((2, 4), ["B", "C", "D", "E", "K", "L"]),
"E": ((3, 3), ["C", "D", "F", "Y"]),
"F": ((3, 5), ["D", "E", "L"]),
"X": ((2, 0), ["A", "C", "Y"]),
"Y": ((3, 1), ["X", "C", "E"]),
"K": ((1, 5), ["B", "D", "L"]),
"L": ((2, 6), ["K", "D", "F"])
}, use_latlon=False)
path = [(0.8, 0.7), (0.9, 0.7), (1.1, 1.0), (1.2, 1.5), (1.2, 1.6), (1.1, 2.0),
(1.1, 2.3), (1.3, 2.9), (1.2, 3.1), (1.5, 3.2), (1.8, 3.5), (2.0, 3.7),
(2.3, 3.5), (2.4, 3.2), (2.6, 3.1), (2.9, 3.1), (3.0, 3.2),
(3.1, 3.8), (3.0, 4.0), (3.1, 4.3), (3.1, 4.6), (3.0, 4.9)]
matcher = DistanceMatcher(map_con, max_dist=2, obs_noise=1, min_prob_norm=0.5)
states, _ = matcher.match(path[:5])
states, _ = matcher.match(path, expand=True)
nodes = matcher.path_pred_onlynodes
print("States\n------")
print(states)
print("Nodes\n------")
print(nodes)
print("")
matcher.print_lattice_stats()
================================================
FILE: docs/usage/installation.rst
================================================
Installation
============
Dependencies
------------
Required:
- `numpy <http://www.numpy.org>`__
- `scipy <https://www.scipy.org>`__
Optional (only loaded when methods are called that rely on these packages):
- `rtree <http://toblerity.org/rtree/>`__
- `nvector <https://github.com/pbrod/Nvector>`__
- `gpxpy <https://github.com/tkrajina/gpxpy>`__
- `pyproj <https://jswhit.github.io/pyproj/>`__
- `pykalman <https://pykalman.github.io>`__
- `matplotlib <http://matplotlib.org>`__
- `smopy <https://github.com/rossant/smopy>`__
Using pip
---------
If you want to install the latest released version using pip:
::
$ pip install leuvenmapmatching
If you want to install the latest non-released version (add ``@develop``) for the
latest development version:
::
$ pip install git+https://github.com/wannesm/leuvenmapmatching
From source
-----------
The library can also be compiled and/or installed directly from source.
* Download the source from https://github.com/wannesm/leuvenmapmatching
* To compile and install in your site-package directory: ``python3 setup.py install``
================================================
FILE: docs/usage/introduction.rst
================================================
Examples
========
Example 1: Simple
-----------------
A first, simple example. Some parameters are given to tune the algorithm.
The ``max_dist`` and ``obs_noise`` are distances that indicate the maximal distance between observation and road
segment and the expected noise in the measurements, respectively.
The ``min_prob_norm`` prunes the lattice in that it drops paths that drop below 0.5 normalized probability.
The probability is normalized to allow for easier reasoning about the probability of a path.
It is computed as the exponential smoothed log probability components instead of the sum as would be the case
for log likelihood.
Because the number of possible paths quickly grows, it is recommended to set the
``max_lattice_width`` argument to speed up the algorithm (available from version 1.0 onwards).
It will only continue the search with this number of possible paths at every step. If no solution is found,
this value can be incremented using the ``increase_max_lattice_width`` method.
.. code-block:: python
from leuvenmapmatching.matcher.distance import DistanceMatcher
from leuvenmapmatching.map.inmem import InMemMap
map_con = InMemMap("mymap", graph={
"A": ((1, 1), ["B", "C", "X"]),
"B": ((1, 3), ["A", "C", "D", "K"]),
"C": ((2, 2), ["A", "B", "D", "E", "X", "Y"]),
"D": ((2, 4), ["B", "C", "F", "E", "K", "L"]),
"E": ((3, 3), ["C", "D", "F", "Y"]),
"F": ((3, 5), ["D", "E", "L"]),
"X": ((2, 0), ["A", "C", "Y"]),
"Y": ((3, 1), ["X", "C", "E"]),
"K": ((1, 5), ["B", "D", "L"]),
"L": ((2, 6), ["K", "D", "F"])
}, use_latlon=False)
path = [(0.8, 0.7), (0.9, 0.7), (1.1, 1.0), (1.2, 1.5), (1.2, 1.6), (1.1, 2.0),
(1.1, 2.3), (1.3, 2.9), (1.2, 3.1), (1.5, 3.2), (1.8, 3.5), (2.0, 3.7),
(2.3, 3.5), (2.4, 3.2), (2.6, 3.1), (2.9, 3.1), (3.0, 3.2),
(3.1, 3.8), (3.0, 4.0), (3.1, 4.3), (3.1, 4.6), (3.0, 4.9)]
matcher = DistanceMatcher(map_con, max_dist=2, obs_noise=1, min_prob_norm=0.5, max_lattice_width=5)
states, _ = matcher.match(path)
nodes = matcher.path_pred_onlynodes
print("States\n------")
print(states)
print("Nodes\n------")
print(nodes)
print("")
matcher.print_lattice_stats()
Example 2: Non-emitting states
------------------------------
In case there are less observations that states (an assumption of HMMs), non-emittings states allow you
to deal with this. States will be inserted that are not associated with any of the given observations if
this improves the probability of the path.
It is possible to also associate a distribtion over the distance between observations and the non-emitting
states (`obs_noise_ne`). This allows the algorithm to prefer nearby road segments. This value should be
larger than `obs_noise` as it is mapped to the line between the previous and next observation, which does
not necessarily run over the relevant segment. Setting this to infinity is the same as using pure
non-emitting states that ignore observations completely.
.. code-block:: python
from leuvenmapmatching.matcher.distance import DistanceMatcher
from leuvenmapmatching.map.inmem import InMemMap
from leuvenmapmatching import visualization as mmviz
path = [(1, 0), (7.5, 0.65), (10.1, 1.9)]
mapdb = InMemMap("mymap", graph={
"A": ((1, 0.00), ["B"]),
"B": ((3, 0.00), ["A", "C"]),
"C": ((4, 0.70), ["B", "D"]),
"D": ((5, 1.00), ["C", "E"]),
"E": ((6, 1.00), ["D", "F"]),
"F": ((7, 0.70), ["E", "G"]),
"G": ((8, 0.00), ["F", "H"]),
"H": ((10, 0.0), ["G", "I"]),
"I": ((10, 2.0), ["H"])
}, use_latlon=False)
matcher = DistanceMatcher(mapdb, max_dist_init=0.2, obs_noise=1, obs_noise_ne=10,
non_emitting_states=True, only_edges=True , max_lattice_width=5)
states, _ = matcher.match(path)
nodes = matcher.path_pred_onlynodes
print("States\n------")
print(states)
print("Nodes\n------")
print(nodes)
print("")
matcher.print_lattice_stats()
mmviz.plot_map(mapdb, matcher=matcher,
show_labels=True, show_matching=True
filename="output.png"))
================================================
FILE: docs/usage/latitudelongitude.rst
================================================
Dealing with Latitude-Longitude
===============================
The toolbox can deal with latitude-longitude coordinates directly.
Map matching, however, requires a lot of repeated computations between points and latitude-longitude
computations will be more expensive than Euclidean distances.
There are three different options how you can handle latitude-longitude coordinates:
Option 1: Use Latitude-Longitude directly
-----------------------------------------
Set the ``use_latlon`` flag in the :class:`Map` to true.
For example to read in an OpenStreetMap file directly to a :class:`InMemMap` object:
.. code-block:: python
from leuvenmapmatching.map.inmem import InMemMap
map_con = InMemMap("myosm", use_latlon=True)
for entity in osmread.parse_file(osm_fn):
if isinstance(entity, osmread.Way) and 'highway' in entity.tags:
for node_a, node_b in zip(entity.nodes, entity.nodes[1:]):
map_con.add_edge(node_a, node_b)
map_con.add_edge(node_b, node_a)
if isinstance(entity, osmread.Node):
map_con.add_node(entity.id, (entity.lat, entity.lon))
map_con.purge()
Option 2: Project Latitude-Longitude to X-Y
-------------------------------------------
Latitude-Longitude coordinates can be transformed two a frame with two orthogonal axis.
.. code-block:: python
from leuvenmapmatching.map.inmem import InMemMap
map_con_latlon = InMemMap("myosm", use_latlon=True)
# Add edges/nodes
map_con_xy = map_con_latlon.to_xy()
route_latlon = []
# Add GPS locations
route_xy = [map_con_xy.latlon2yx(latlon) for latlon in route_latlon]
This can also be done directly using the `pyproj <https://github.com/jswhit/pyproj>`_ toolbox.
For example, using the Lambert Conformal projection to project the route GPS coordinates:
.. code-block:: python
import pyproj
route = [(4.67878,50.864),(4.68054,50.86381),(4.68098,50.86332),(4.68129,50.86303),(4.6817,50.86284),
(4.68277,50.86371),(4.68894,50.86895),(4.69344,50.86987),(4.69354,50.86992),(4.69427,50.87157),
(4.69643,50.87315),(4.69768,50.87552),(4.6997,50.87828)]
lon_0, lat_0 = route[0]
proj = pyproj.Proj(f"+proj=merc +ellps=GRS80 +units=m +lon_0={lon_0} +lat_0={lat_0} +lat_ts={lat_0} +no_defs")
xs, ys = [], []
for lon, lat in route:
x, y = proj(lon, lat)
xs.append(x)
ys.append(y)
Notice that the pyproj package uses the convention to express coordinates as x-y which is
longitude-latitude because it is defined this way in the CRS definitions while the Leuven.MapMatching
toolbox follows the ISO 6709 standard and expresses coordinates as latitude-longitude. If you
want ``pyproj`` to use latitude-longitude you can use set the
`axisswap option <https://proj4.org/operations/conversions/axisswap.html>`_.
If you want to define both the from and to projections:
.. code-block:: python
import pyproj
route = [(4.67878,50.864),(4.68054,50.86381),(4.68098,50.86332),(4.68129,50.86303),(4.6817,50.86284),
(4.68277,50.86371),(4.68894,50.86895),(4.69344,50.86987),(4.69354,50.86992),(4.69427,50.87157),
(4.69643,50.87315),(4.69768,50.87552),(4.6997,50.87828)]
p1 = pyproj.Proj(proj='latlon', datum='WGS84')
p2 = pyproj.Proj(proj='utm', datum='WGS84')
xs, ys = [], []
for lon, lat in route:
x, y = pyproj.transform(lon, lat)
xs.append(x)
ys.append(y)
Option 3: Use Latitude-Longitude as if they are X-Y points
----------------------------------------------------------
A naive solution would be to use latitude-longitude coordinate pairs as if they are X-Y coordinates.
For small distances, far away from the poles and not crossing the dateline, this option might work.
But it is not adviced.
For example, for long distances the error is quite large. In the image beneath, the blue line is the computation
of the intersection using latitude-longitude while the red line is the intersection using Eucludean distances.
.. figure:: https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/latlon_mismatch_1.png?v=1
:alt: Latitude-Longitude mismatch
.. figure:: https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/latlon_mismatch_2.png?v=1
:alt: Latitude-Longitude mismatch detail
================================================
FILE: docs/usage/openstreetmap.rst
================================================
Map from OpenStreetMap
======================
You can download a graph for map-matching from the OpenStreetMap.org service.
Multiple methods exists, we illustrate two.
Using requests, osmread and gpx
-------------------------------
You can perform map matching on a OpenStreetMap database by combing ``leuvenmapmatching``
with the packages ``requests``, ``osmread`` and ``gpx``.
Download a map as XML
~~~~~~~~~~~~~~~~~~~~~
You can use the overpass-api.de service:
.. code-block:: python
from pathlib import Path
import requests
xml_file = Path(".") / "osm.xml"
url = 'http://overpass-api.de/api/map?bbox=4.694933,50.870047,4.709256000000001,50.879628'
r = requests.get(url, stream=True)
with xml_file.open('wb') as ofile:
for chunk in r.iter_content(chunk_size=1024):
if chunk:
ofile.write(chunk)
Create graph using osmread
~~~~~~~~~~~~~~~~~~~~~~~~~~
Once we have a file containing the region we are interested in, we can select the roads we want to use
to create a graph from. In this case we focus on 'ways' with a 'highway' tag. Those represent a variety
of roads. For a more detailed filtering look at the
`possible values of the highway tag <https://wiki.openstreetmap.org/wiki/Key:highway>`_.
.. code-block:: python
from leuvenmapmatching.map.inmem import InMemMap
import osmread
map_con = InMemMap("myosm", use_latlon=True, use_rtree=True, index_edges=True)
for entity in osmread.parse_file(str(xml_file)):
if isinstance(entity, osmread.Way) and 'highway' in entity.tags:
for node_a, node_b in zip(entity.nodes, entity.nodes[1:]):
map_con.add_edge(node_a, node_b)
# Some roads are one-way. We'll add both directions.
map_con.add_edge(node_b, node_a)
if isinstance(entity, osmread.Node):
map_con.add_node(entity.id, (entity.lat, entity.lon))
map_con.purge()
Note that ``InMemMap`` is a simple container for a map. It is recommended to use
your own optimized connecter to your map dataset.
If you want to allow transitions that are not following the exact road segments you can inherit from the ``Map``
class and define a new class with your own transitions.
The transitions are defined using the ``nodes_nbrto`` and ``edges_nbrt`` methods.
Perform map matching on an OpenStreetMap database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can create a list of latitude-longitude coordinates manually. Or read a gpx file.
.. code-block:: python
from leuvenmapmatching.util.gpx import gpx_to_path
track = gpx_to_path("mytrack.gpx")
matcher = DistanceMatcher(map_con,
max_dist=100, max_dist_init=25, # meter
min_prob_norm=0.001,
non_emitting_length_factor=0.75,
obs_noise=50, obs_noise_ne=75, # meter
dist_noise=50, # meter
non_emitting_states=True,
max_lattice_width=5)
states, lastidx = matcher.match(track)
Using osmnx and geopandas
-------------------------
Another great library to interact with OpenStreetMap data is the `osmnx <https://github.com/gboeing/osmnx>`_ package.
The osmnx package can retrieve relevant data automatically, for example when given a name of a region.
This package is build on top of the `geopandas <http://geopandas.org>`_ package.
.. code-block:: python
import osmnx
graph = ox.graph_from_place('Leuven, Belgium', network_type='drive', simplify=False)
graph_proj = ox.project_graph(graph)
# Create GeoDataFrames (gdfs)
# Approach 1
nodes_proj, edges_proj = ox.graph_to_gdfs(graph_proj, nodes=True, edges=True)
for nid, row in nodes_proj[['x', 'y']].iterrows():
map_con.add_node(nid, (row['x'], row['y']))
for eid, _ in edges_proj.iterrows():
map_con.add_edge(eid[0], eid[1])
# Approach 2
nodes, edges = ox.graph_to_gdfs(graph_proj, nodes=True, edges=True)
nodes_proj = nodes.to_crs("EPSG:3395")
edges_proj = edges.to_crs("EPSG:3395")
for nid, row in nodes_proj.iterrows():
map_con.add_node(nid, (row['lat'], row['lon']))
# We can also extract edges also directly from networkx graph
for nid1, nid2, _ in graph.edges:
map_con.add_edge(nid1, nid2)
================================================
FILE: docs/usage/visualisation.rst
================================================
Visualisation
=============
To inspect the results, a plotting function is included.
Simple plotting
---------------
To plot the graph in a matplotlib figure use:
.. code-block:: python
from leuvenmapmatching import visualization as mmviz
mmviz.plot_map(map_con, matcher=matcher,
show_labels=True, show_matching=True, show_graph=True,
filename="my_plot.png")
This will result in the following figure:
.. figure:: https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/plot1.png?v=1
:alt: Plot1
You can also define your own figure by passing a matplotlib axis object:
.. code-block:: python
fig, ax = plt.subplots(1, 1)
mmviz.plot_map(map_con, matcher=matcher,
ax=ax,
show_labels=True, show_matching=True, show_graph=True,
filename="my_plot.png")
Plotting with an OpenStreetMap background
-----------------------------------------
The plotting function also supports a link with the ``smopy`` package.
Set the ``use_osm`` argument to true and pass a map that is defined with
latitude-longitude (thus ``use_latlon=True``).
You can set ``zoom_path`` to true to only see the relevant part and not the
entire map that is available in the map. Alternatively you can also set the
bounding box manually using the ``bb`` argument.
.. code-block:: python
mm_viz.plot_map(map_con, matcher=matcher,
use_osm=True, zoom_path=True,
show_labels=False, show_matching=True, show_graph=False,
filename="my_osm_plot.png")
This will result in the following figure:
.. figure:: https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/plot2.png?v=1
:alt: Plot2
Or when some GPS points are missing in the track, the matching is more
visible as the matched route deviates from the straight line between two
GPS points:
.. figure:: https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/plot3.png?v=1
:alt: Plot3
================================================
FILE: leuvenmapmatching/__init__.py
================================================
# encoding: utf-8
"""
leuvenmapmatching
~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2015-2022 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
import logging
from . import map, matcher, util
# visualization is not loaded by default (avoid loading unnecessary dependencies such as matplotlib).
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
__version__ = '1.1.4'
================================================
FILE: leuvenmapmatching/map/__init__.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.map
~~~~~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
================================================
FILE: leuvenmapmatching/map/base.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.map.base
~~~~~~~~~~~~~~~~~~~~~~~~~~
Base Map class.
To be used in a Matcher object, the following functions need to be defined:
- ``edges_closeto``
- ``nodes_closeto``
- ``nodes_nbrto``
- ``edges_nbrto``
For visualiation purposes the following methods need to be implemented:
- ``bb``
- ``labels``
- ``size``
- ``coordinates``
- ``node_coordinates``
- ``all_edges``
- ``all_nodes``
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
from abc import abstractmethod
import logging
MYPY = False
if MYPY:
from typing import Tuple, Union, List
LabelType = Union[int, str]
LocType = Tuple[float, float]
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
class BaseMap(object):
"""Abstract class for a Map."""
def __init__(self, name, use_latlon=True):
"""Simple database wrapper/stub."""
self.name = name
self._use_latlon = None
self.distance = None
self.distance_point_to_segment = None
self.distance_segment_to_segment = None
self.box_around_point = None
self.use_latlon = use_latlon
@property
def use_latlon(self):
return self._use_latlon
@use_latlon.setter
def use_latlon(self, value):
self._use_latlon = value
if self._use_latlon:
from ..util import dist_latlon as dist_lib
else:
from ..util import dist_euclidean as dist_lib
self.distance = dist_lib.distance
self.distance_point_to_segment = dist_lib.distance_point_to_segment
self.distance_segment_to_segment = dist_lib.distance_segment_to_segment
self.box_around_point = dist_lib.box_around_point
self.lines_parallel = dist_lib.lines_parallel
@abstractmethod
def bb(self):
"""Bounding box.
:return: (lat_min, lon_min, lat_max, lon_max)
"""
@abstractmethod
def labels(self):
"""Labels of all nodes."""
@abstractmethod
def size(self):
"""Number of nodes."""
@abstractmethod
def node_coordinates(self, node_key):
"""Coordinates for given node key."""
@abstractmethod
def edges_closeto(self, loc, max_dist=None, max_elmt=None):
"""Find edges close to a certain location.
:param loc: Latitude, Longitude
:param max_dist: Maximal distance that returned nodes can be from lat-lon
:param max_elmt: Maximal number of elements returned after sorting according to distance.
:return: list[tuple[dist, label, loc]]
"""
return None
@abstractmethod
def nodes_closeto(self, loc, max_dist=None, max_elmt=None):
"""Find nodes close to a certain location.
:param loc: Latitude, Longitude
:param max_dist: Maximal distance that returned nodes can be from lat-lon
:param max_elmt: Maximal number of elements returned after sorting according to distance.
:return: list[tuple[dist, label, loc]]
"""
return None
@abstractmethod
def nodes_nbrto(self, node):
# type: (BaseMap, LabelType) -> List[Tuple[LabelType, LocType]]
"""Return all nodes that are linked to ``node``.
:param node: Node identifier
:return: list[tuple[label, loc]]
"""
return []
def edges_nbrto(self, edge):
# type: (BaseMap, Tuple[LabelType, LabelType]) -> List[Tuple[LabelType, LocType, LabelType, LocType]]
"""Return all edges that are linked to ``edge``.
Defaults to ``nodes_nbrto``.
:param edge: Edge identifier
:return: list[tuple[label1, label2, loc1, loc2]]
"""
results = []
l1, l2 = edge
p2 = self.node_coordinates(l2)
for l3, p3 in self.nodes_nbrto(l2):
results.append((l2, p2, l3, p3))
return results
@abstractmethod
def all_nodes(self, bb=None):
"""All node keys and coordinates.
:return: [(key, (lat, lon))]
"""
@abstractmethod
def all_edges(self, bb=None):
"""All edges.
:return: [(key_a, loc_a, key_b, loc_b)]
"""
================================================
FILE: leuvenmapmatching/map/inmem.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.map.inmem
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Simple in-memory map representation. Not suited for production purposes.
Write your own map class that connects to your map (e.g. a database instance).
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
import logging
import time
from pathlib import Path
import pickle
from functools import partial
try:
import rtree
except ImportError:
rtree = None
try:
import pyproj
except ImportError:
pyproj = None
try:
import tqdm
except ImportError:
tqdm = None
from .base import BaseMap
MYPY = False
if MYPY:
from typing import Optional, Set, Tuple, Dict, Union
LabelType = Union[int, str]
LocType = Tuple[float, float]
EdgeType = Tuple[LabelType, LabelType]
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
class InMemMap(BaseMap):
def __init__(self, name, use_latlon=True, use_rtree=False, index_edges=False,
crs_lonlat=None, crs_xy=None, graph=None, linked_edges=None, dir=None, deserializing=False):
"""In-memory representation of a map.
This is a simple database-like object to perform experiments with map matching.
For production purposes it is recommended to use your own derived
class (e.g. to connect to your database instance).
This class supports:
- Indexing using rtrees to allow for fast searching of points on the map.
When using the rtree index, only integer numbers are allowed as node labels.
- Serializing to write and read from files.
- Projecting points to a different frame (e.g. GPS to Lambert)
:param name: Map name (mandatory)
:param use_latlon: The locations represent latitude-longitude pairs, otherwise y-x coordinates
are assumed.
:param use_rtree: Build an rtree index to quickly search for locations.
:param index_edges: Build an index for the edges in the map instead of the vertices.
:param crs_lonlat: Coordinate reference system for the latitude-longitude coordinates.
:param crs_xy: Coordiante reference system for the y-x coordinates.
:param graph: Initial graph of form Dict[label, Tuple[Tuple[y,x], List[neighbor]]]]
:param dir: Directory where to serialize to. If given, the rtree index structure will be written
to a file immediately.
:param deserializing: Internal variable to indicate that the object is being build from a file.
"""
super(InMemMap, self).__init__(name, use_latlon=use_latlon)
self.dir = None if dir is None else Path(dir)
self.index_edges = index_edges
self.graph = dict() if graph is None else graph
self.rtree = None
self.use_rtree = use_rtree
if self.use_rtree:
self.setup_index(deserializing=deserializing)
self.crs_lonlat = 'EPSG:4326' if crs_lonlat is None else crs_lonlat # GPS
self.crs_xy = 'EPSG:3395' if crs_xy is None else crs_xy # Mercator projection
if pyproj:
# proj_lonlat = pyproj.Proj(self.crs_lonlat, preserve_units=True)
# proj_xy = pyproj.Proj(self.crs_xy, preserve_units=True)
# self.lonlat2xy = partial(pyproj.transform, proj_lonlat, proj_xy)
# self.xy2lonlat = partial(pyproj.transform, proj_xy, proj_lonlat)
tr_lonlat2xy = pyproj.Transformer.from_crs(self.crs_lonlat, self.crs_xy)
self.lonlat2xy = tr_lonlat2xy.transform
tr_xy2lonlat = pyproj.Transformer.from_crs(self.crs_xy, self.crs_lonlat)
self.xy2lonlat = tr_xy2lonlat.transform
else:
def pyproj_notfound(*_args, **_kwargs):
raise Exception("pyproj package not found")
self.lonlat2xy = pyproj_notfound
self.xy2lonlat = pyproj_notfound
self.linked_edges = linked_edges # type: Optional[Dict[EdgeType, Set[Tuple[EdgeType]]]]
self.vertex_label_map = None
def vertex_label_to_int(self, label, create=False):
if type(label) is int:
return label
if self.vertex_label_map is None:
if not create:
return label
self.vertex_label_map = dict()
if label in self.vertex_label_map:
new_label = self.vertex_label_map[label]
else:
new_label = len(self.vertex_label_map)
self.vertex_label_map[label] = new_label
return new_label
def vertices_labels_to_int(self):
graph = dict()
for label, (loc, nbrs) in self.graph.items():
new_label = self.vertex_label_to_int(label, create=True)
new_nbrs = [self.vertex_label_to_int(nbr, create=True) for nbr in nbrs]
graph[new_label] = (loc, new_nbrs)
self.graph = graph
def serialize(self):
"""Create a serializable data structure."""
data = {
"name": self.name,
"graph": self.graph,
"use_latlon": self.use_latlon,
"use_rtree": self.use_rtree,
"index_edges": self.index_edges,
"crs_lonlat": self.crs_lonlat,
"crs_xy": self.crs_xy,
"linked_edges": self.linked_edges
}
if self.dir is not None:
data["dir"] = self.dir
return data
@classmethod
def deserialize(cls, data):
"""Create a new instance from a dictionary."""
nmap = cls(data["name"], dir=data.get("dir", None),
use_latlon=data["use_latlon"], use_rtree=data["use_rtree"],
index_edges=data["index_edges"],
crs_lonlat=data.get("crs_lonlat", None), crs_xy=data.get("crs_xy", None),
graph=data.get("graph", None), linked_edges=data.get("linked_edges", None),
deserializing=True)
return nmap
def dump(self):
"""Serialize map using pickle.
All files will be saved to the `dir` directory using the `name` as filename.
"""
if self.dir is None:
logger.error(f"No directory set where to save (see InMemMap.__init__)")
return
filename = self.dir / (self.name + ".pkl")
with filename.open("wb") as ofile:
pickle.dump(self.serialize(), ofile)
logger.debug(f"Saved map to {filename}")
if self.rtree:
rtree_fn = self.rtree_fn()
if rtree_fn is not None:
self.rtree.close()
self.rtree = rtree.index.Index(str(rtree_fn))
@classmethod
def from_pickle(cls, filename):
"""Deserialize map using pickle to the given filename."""
filename = Path(filename)
with filename.open("rb") as ifile:
data = pickle.load(ifile)
nmap = cls.deserialize(data)
return nmap
def bb(self):
"""Bounding box.
:return: (lat_min, lon_min, lat_max, lon_max) or (y_min, x_min, y_max, x_max)
"""
if self.use_rtree:
lat_min, lon_min, lat_max, lon_max = self.rtree.bounds
else:
glat, glon = zip(*[t[0] for t in self.graph.values()])
lat_min, lat_max = min(glat), max(glat)
lon_min, lon_max = min(glon), max(glon)
return lat_min, lon_min, lat_max, lon_max
def labels(self):
"""All labels."""
return self.graph.keys()
def size(self):
return len(self.graph)
def node_coordinates(self, node_key):
"""Get the coordinates of the given node.
:param node_key: Node label/key
:return: (lat, lon)
"""
return self.graph[node_key][0]
def add_node(self, node, loc):
"""Add new node to the map.
:param node: label
:param loc: (lat, lon) or (y, x)
"""
if node in self.graph:
if self.graph[node][0] is None:
self.graph[node] = (loc, self.graph[node][1])
else:
self.graph[node] = (loc, [])
if self.use_rtree and self.rtree is not None and not self.index_edges:
if type(node) is not int:
raise Exception(f"Rtree index only supports integer keys for vertices")
self.rtree.insert(node, (loc[0], loc[1], loc[0], loc[1]))
def del_node(self, node):
if node not in self.graph:
return
if self.rtree:
data = self.graph[node]
loc = data[0]
self.rtree.delete(node, (loc[0], loc[1], loc[0], loc[1]))
del self.graph[node]
def add_edge(self, node_a, node_b):
"""Add new edge to the map.
:param node_a: Label for the node that is the start of the edge
:param node_b: Label for the node that is the end of the edge
"""
if node_a not in self.graph:
raise ValueError(f"Add {node_a} first as node")
if node_b not in self.graph:
raise ValueError(f"Add {node_b} first as node")
if node_b not in self.graph[node_a][1]:
self.graph[node_a][1].append(node_b)
if self.use_rtree and self.rtree is not None and self.index_edges:
if type(node_a) is not int or type(node_b) is not int:
raise Exception(f"Rtree index only supports integer keys for vertices")
loc_a = self.graph[node_a][0]
loc_b = self.graph[node_b][0]
bb = (min(loc_a[0], loc_b[0]), min(loc_a[1], loc_b[1]), # y_min, x_min
max(loc_a[0], loc_b[0]), max(loc_a[1], loc_b[1])) # y_max, x_max
self.rtree.insert(node_a, bb)
# self.rtree.insert(node_b, bb)
def _items_in_bb(self, bb):
if self.rtree is not None:
node_idxs = self.rtree.intersection(bb)
for key in node_idxs:
yield (key, self.graph[key])
else:
lat_min, lon_min, lat_max, lon_max = bb
for key, value in self.graph.items():
((lat, lon), nbrs) = value
if lat_min <= lat <= lat_max and lon_min <= lon <= lon_max:
yield (key, value)
def all_edges(self, bb=None):
"""Return all edges.
:param bb: Bounding box
:return: (key_a, loc_a, nbr, loc_b)
"""
if bb is None:
keyvals = self.graph.items()
else:
keyvals = self._items_in_bb(bb)
for key_a, (loc_a, nbrs) in keyvals:
if loc_a is not None:
for nbr in nbrs:
try:
loc_b, _ = self.graph[nbr]
if loc_b is not None:
yield (key_a, loc_a, nbr, loc_b)
except KeyError:
# print("Node not found: {}".format(nbr))
pass
def all_nodes(self, bb=None):
"""Return all nodes.
:param bb: Bounding box
:return:
"""
if bb is None:
keyvals = self.graph.items()
else:
keyvals = self._items_in_bb(bb)
for key_a, (loc_a, nbrs) in keyvals:
if loc_a is not None:
yield key_a, loc_a
def purge(self):
cnt_noloc = 0
cnt_noedges = 0
remove = []
for node in self.graph.keys():
if self.graph[node][0] is None:
cnt_noloc += 1
remove.append(node)
# print("No location for node {}".format(node))
elif len(self.graph[node][1]) == 0:
cnt_noedges += 1
remove.append(node)
for node in remove:
self.del_node(node)
logger.debug("Removed {} nodes without location".format(cnt_noloc))
logger.debug("Removed {} nodes without edges".format(cnt_noedges))
def rtree_size(self):
bb = self.rtree.bounds
if bb[0] < bb[2] and bb[1] < bb[3]:
rtree_size = self.rtree.count(bb)
else:
rtree_size = 0
return rtree_size
def rtree_fn(self):
rtree_fn = None
if self.dir is not None:
rtree_fn = self.dir / self.name
return rtree_fn
def setup_index(self, force=False, deserializing=False):
if not self.use_rtree:
return
if self.rtree is not None and not force:
return
if rtree is None:
raise Exception("rtree package not found")
rtree_fn = self.rtree_fn()
args = []
if deserializing and (rtree_fn is None or not rtree_fn.exists()):
deserializing = False
if self.graph and len(self.graph) > 0 and not deserializing:
if self.index_edges:
logger.debug("Generator to index edges")
def generator_function():
for label, data in self.graph.items():
lat_min, lon_min = data[0]
lat_max, lon_max = lat_min, lon_min
for idx in data[1]:
olat, olon = self.graph[idx][0]
lat_min = min(lat_min, olat)
lat_max = max(lat_max, olat)
lon_min = min(lon_min, olon)
lon_max = max(lon_max, olon)
if type(label) is not int:
raise Exception(f"Rtree index only supports integer keys for vertices")
yield (label, (lat_min, lon_min, lat_max, lon_max), None)
else:
logger.debug("Generator to index nodes")
def generator_function():
for label, data in self.graph.items():
lat, lon = data[0]
if type(label) is not int:
raise Exception(f"Rtree index only supports integer keys for vertices")
yield (label, (lat, lon, lat, lon), None)
args.append(generator_function())
t_start = time.time()
if self.dir is not None:
# props = rtree.index.Property()
# if force:
# props.overwrite = True
logger.debug(f"Creating new file-based rtree index ({rtree_fn}) ...")
args.insert(0, str(rtree_fn))
elif deserializing:
raise Exception("Cannot deserialize, no directory given")
else:
logger.debug(f"Creating new in-memory rtree index (args={args}) ...")
self.rtree = rtree.index.Index(*args)
t_delta = time.time() - t_start
logger.debug(f"... done: rtree size = {self.rtree_size()}, time = {t_delta} sec")
def fill_index(self):
if not self.use_rtree or self.rtree is None:
return
for label, data in self.graph.items():
loc = data[0]
self.rtree.insert(label, (loc[1], loc[0], loc[1], loc[0]))
logger.debug(f"After filling rtree, size = {self.rtree_size()}")
def to_xy(self, name=None, use_rtree=None):
"""Create a map that uses a projected XY representation on which Euclidean distances
can be used.
"""
if not self.use_latlon:
return self
if name is None:
name = self.name + "_xy"
if use_rtree is None:
use_rtree = self.use_rtree
ngraph = dict()
for label, row in self.graph.items():
lat, lon = row[0]
x, y = self.lonlat2xy(lon, lat)
ngraph[label] = ((y, x), row[1])
nmap = self.__class__(name, dir=self.dir, graph=ngraph,
use_latlon=False, use_rtree=use_rtree, index_edges=self.index_edges,
crs_xy=self.crs_xy, crs_lonlat=self.crs_lonlat)
return nmap
def latlon2xy(self, lat, lon):
x, y = self.lonlat2xy(lon, lat)
return x, y
def latlon2yx(self, lat, lon):
x, y = self.lonlat2xy(lon, lat)
return y, x
def xy2latlon(self, x, y):
lon, lat = self.xy2lonlat(x, y)
return lat, lon
def yx2latlon(self, y, x):
lon, lat = self.xy2lonlat(x, y)
return lat, lon
def nodes_closeto(self, loc, max_dist=None, max_elmt=None):
"""Return all nodes close to the given location.
:param loc: Location
:param max_dist: Maximal distance from the location
:param max_elmt: Return only the most nearby nodes
"""
t_start = time.time()
lat, lon = loc[:2]
lat_b, lon_l, lat_t, lon_r = self.box_around_point((lat, lon), max_dist)
bb = (lat_b, lon_l, # y_min, x_min
lat_t, lon_r) # y_max, x_max
if self.rtree is not None and max_dist is not None:
logger.debug(f"Search closeby nodes to {loc}, bb={bb}")
nodes = self.rtree.intersection(bb)
else:
logger.warning("Searching closeby nodes with linear search, use an index and set max_dist")
if max_dist is not None:
nodes = (key for key, _ in self._items_in_bb(self.box_around_point((lat, lon), max_dist)))
else:
nodes = self.graph.keys()
t_delta_search = time.time() - t_start
t_start = time.time()
results = []
for label in nodes:
oloc, nbrs = self.graph[label]
dist = self.distance(loc, oloc)
if dist < max_dist:
results.append((dist, label, oloc))
results.sort()
t_delta_dist = time.time() - t_start
logger.debug(f"Found {len(results)} closeby nodes "
f"in {t_delta_search} sec and computed distances in {t_delta_dist} sec")
if max_elmt is not None:
results = results[:max_elmt]
return results
def edges_closeto(self, loc, max_dist=None, max_elmt=None):
"""Return all nodes that are on an edge that is close to the given location.
:param loc: Location
:param max_dist: Maximal distance from the location
:param max_elmt: Return only the most nearby nodes
"""
t_start = time.time()
lat, lon = loc[:2]
if self.rtree is not None and max_dist is not None and self.index_edges:
lat_b, lon_l, lat_t, lon_r = self.box_around_point((lat, lon), max_dist)
bb = (lat_b, lon_l, # y_min, x_min
lat_t, lon_r) # y_max, x_max
logger.debug(f"Search closeby edges to {loc}, bb={bb}")
nodes = self.rtree.intersection(bb)
else:
if self.rtree is not None and max_dist is not None and not self.index_edges:
logger.warning("Index is built for nodes, not for edges, set the index_edges argument to true")
logger.warning("Searching closeby nodes with linear search, use an index and set max_dist")
if max_dist is not None:
bb = self.box_around_point((lat, lon), max_dist)
nodes = (key for key, _ in self._items_in_bb(bb))
else:
nodes = self.graph.keys()
t_delta_search = time.time() - t_start
t_start = time.time()
results = []
for label in nodes:
oloc, nbrs = self.graph[label]
for nbr in nbrs:
if label == nbr:
continue
nbr_data = self.graph[nbr]
dist, pi, ti = self.distance_point_to_segment(loc, oloc, nbr_data[0])
# print(f"label={label}/{oloc}, nbr={nbr}/{nbr_data[0]} -- loc={loc} -> {dist}, {pi}, {ti}")
if dist < max_dist:
results.append((dist, label, oloc, nbr, nbr_data[0], pi, ti))
results.sort()
t_delta_dist = time.time() - t_start
logger.debug(f"Found {len(results)} closeby edges "
f"in {t_delta_search} sec and computed distances in {t_delta_dist} sec")
if max_elmt is not None:
results = results[:max_elmt]
return results
def nodes_nbrto(self, node):
results = []
if node not in self.graph:
return results
loc_node, nbrs = self.graph[node]
for nbr_label in nbrs + [node]:
try:
loc_nbr = self.graph[nbr_label][0]
if loc_nbr is not None:
results.append((nbr_label, loc_nbr))
except KeyError:
pass
return results
def edges_nbrto(self, edge):
results = []
l1, l2 = edge
p1 = self.node_coordinates(l1)
p2 = self.node_coordinates(l2)
# Edges that connect at end of this edge
for l3, p3 in self.nodes_nbrto(l2):
results.append((l2, p2, l3, p3))
# Edges that are in parallel and close
if self.linked_edges:
for (l3, l4) in self.linked_edges.get(edge, []):
p3 = self.node_coordinates(l3)
p4 = self.node_coordinates(l4)
results.append((l3, p3, l4, p4))
return results
def find_duplicates(self, func=None):
"""Find entries with identical locations."""
cnt = 0
for label, data in self.graph.items():
lat, lon = data[0]
idxs = list(self.rtree.nearest((lat, lon, lat, lon), num_results=1))
idxs.remove(label)
if len(idxs) > 0:
# logger.info(f"Found doubles for {label}: {idxs}")
if func:
func(label, idxs)
logger.info(f"Found {cnt} doubles")
def connect_parallelroads(self, dist=0.5, bb=None):
if self.rtree is None or not self.index_edges:
logger.error("Finding parallel roads requires and edge-based index")
return
self.linked_edges = {}
it = self.all_edges(bb=bb)
if tqdm:
it = tqdm.tqdm(list(it))
for key_a, loc_a, key_b, loc_b in it:
bb2 = [min(loc_a[0], loc_b[0]), min(loc_a[1], loc_b[1]),
max(loc_a[0], loc_b[0]), max(loc_a[1], loc_b[1])]
for key_c, loc_c, key_d, loc_d in self.all_edges(bb=bb2):
if key_a == key_c or key_a == key_d or key_b == key_c or key_b == key_d:
continue
# print(f"Test: ({key_a},{key_b}) - ({key_c},{key_d})")
if self.lines_parallel(loc_a, loc_b, loc_c, loc_d, d=dist):
# print(f"Parallel: ({key_a},{key_b}) - ({key_c},{key_d})")
key = (key_a, key_b)
if key in self.linked_edges:
self.linked_edges[key].add((key_c, key_d))
else:
self.linked_edges[key] = {(key_c, key_d)}
logger.debug(f"Linked {len(self.linked_edges)} edges")
def print_stats(self):
print("Graph\n-----")
print("Nodes: {}".format(len(self.graph)))
def __str__(self):
# s = ""
# for label, (loc, nbrs, _) in self.graph.items():
# s += f"{label:<10} - ({loc[0]:10.4f}, {loc[1]:10.4f})\n"
# return s
return f"InMemMap({self.name}, size={self.size()})"
================================================
FILE: leuvenmapmatching/map/sqlite.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.map.sqlite
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Map representation based on a sqlite database. Not optimized for production purposes.
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
import sqlite3
import tempfile
import logging
import time
from pathlib import Path
import pickle
from functools import partial
try:
import pyproj
except ImportError:
pyproj = None
try:
import tqdm
except ImportError:
tqdm = None
from .base import BaseMap
MYPY = False
if MYPY:
from typing import Optional, Set, Tuple, Dict, Union
LabelType = Union[int, str]
LocType = Tuple[float, float]
EdgeType = Tuple[LabelType, LabelType]
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
class SqliteMap(BaseMap):
def __init__(self, name, use_latlon=True,
crs_lonlat=None, crs_xy=None, dir=None, deserializing=False):
"""Store a map as a SQLite instance.
This class supports:
- Indexing using rtrees to allow for fast searching of points on the map.
When using the rtree index, only integer numbers are allowed as node labels.
- Serializing to write and read from files.
- Projecting points to a different frame (e.g. GPS to Lambert)
:param name: Name of database file
:param use_latlon: The locations represent latitude-longitude pairs, otherwise y-x coordinates
are assumed.
:param crs_lonlat: Coordinate reference system for the latitude-longitude coordinates.
:param crs_xy: Coordiante reference system for the y-x coordinates.
:param dir: Directory where to serialize to. If not given, a temporary location will be used.
:param deserializing: Internal variable to indicate that the object is being build from a file.
"""
super(SqliteMap, self).__init__(name, use_latlon=use_latlon)
self.dir = Path(tempfile.gettempdir()) if dir is None else Path(dir)
name = Path(name)
suffix = name.suffix
if suffix == '':
name = name.with_suffix('.sqlite')
self.db_fn = self.dir / name
if deserializing and not self.db_fn.exists():
raise Exception(f"File not found: {self.db_fn}")
logger.debug(f"Opening database: {self.db_fn}")
try:
self.db = sqlite3.connect(str(self.db_fn))
except Exception as exc:
raise Exception(f'Problem with database: {self.db_fn}') from exc
self.crs_lonlat = crs_lonlat
self.crs_xy = crs_xy
self.use_latlon = use_latlon
if deserializing:
self.read_properties()
else:
self.create_db()
if self.crs_lonlat is None:
self.crs_lonlat = 'EPSG:4326' # GPS
if self.crs_xy is None:
self.crs_xy = 'EPSG:3395' # Mercator projection
self.save_properties()
if pyproj:
proj_lonlat = pyproj.Proj(self.crs_lonlat, preserve_units=True)
proj_xy = pyproj.Proj(self.crs_xy, preserve_units=True)
self.lonlat2xy = partial(pyproj.transform, proj_lonlat, proj_xy)
self.xy2lonlat = partial(pyproj.transform, proj_xy, proj_lonlat)
else:
def pyproj_notfound(*_args, **_kwargs):
raise Exception("pyproj package not found")
self.lonlat2xy = pyproj_notfound
self.xy2lonlat = pyproj_notfound
def read_properties(self):
c = self.db.cursor()
for row in c.execute("SELECT key, value FROM properties;"):
key, value = row[0], pickle.loads(row[1])
self.__dict__[key] = value
def save_properties(self):
c = self.db.cursor()
q = "INSERT INTO properties (key, value) VALUES (?, ?)"
v = [('name', pickle.dumps(self.name)),
('use_latlon', pickle.dumps(self.use_latlon)),
('crs_lonlat', pickle.dumps(self.crs_lonlat)),
('crs_xy', pickle.dumps(self.crs_xy))]
c.executemany(q, v)
self.db.commit()
def create_db(self):
logger.debug("Cleaning database file and creating new tables")
c = self.db.cursor()
c.execute("DROP INDEX IF EXISTS edges_from_index")
c.execute("DROP INDEX IF EXISTS close_edges_index")
c.execute("DROP TABLE IF EXISTS nodes_index")
c.execute("DROP TABLE IF EXISTS nodes")
c.execute("DROP TABLE IF EXISTS edges_index")
c.execute("DROP TABLE IF EXISTS edges")
c.execute("DROP TABLE IF EXISTS close_edges")
c.execute("DROP TABLE IF EXISTS properties")
self.db.commit()
# Create tables
q = ("CREATE VIRTUAL TABLE nodes_index USING rtree(\n"
"id, -- Integer primary key\n"
"minX, maxX, -- Minimum and maximum X coordinate\n"
"minY, maxY -- Minimum and maximum Y coordinate\n"
")")
c.execute(q)
q = ("CREATE TABLE nodes(\n"
"id INTEGER PRIMARY KEY,\n"
"x REAL,\n"
"y REAL\n"
")")
c.execute(q)
q = ("CREATE VIRTUAL TABLE edges_index USING rtree(\n"
"id, -- Integer primary key\n"
"minX, maxX, -- Minimum and maximum X coordinate\n"
"minY, maxY -- Minimum and maximum Y coordinate\n"
")")
c.execute(q)
q = ("CREATE TABLE edges(\n"
"id INTEGER PRIMARY KEY,\n"
"path INTEGER,\n" # Not necessarily unique, a pathway id can consist of multiple edges
"pathnum INTEGER,\n"
"id1 INTEGER,\n" # node 1
"id2 INTEGER,\n" # node 2
"speed REAL,\n" # speed m/s
"type INTEGER\n" # extra field
")")
c.execute(q)
q = ("CREATE TABLE close_edges(\n"
"id1 INTEGER,\n" # edge 1
"id2 INTEGER\n" # edge 2
")")
c.execute(q)
q = ("CREATE TABLE properties(\n"
"key TEXT,\n"
"value BLOB\n"
")")
c.execute(q)
q = "CREATE INDEX edges_from_index ON edges(id1)"
c.execute(q)
q = "CREATE INDEX close_edges_index ON close_edges(id1)"
c.execute(q)
self.db.commit()
@classmethod
def from_file(cls, filename):
"""Read from an existing file."""
filename = Path(filename).with_suffix('')
nmap = cls(filename.name, dir=filename.parent, deserializing=True)
return nmap
def bb(self):
"""Bounding box.
:return: (lat_min, lon_min, lat_max, lon_max) or (y_min, x_min, y_max, x_max)
"""
c = self.db.cursor()
c.execute('SELECT min(minX), max(minX), min(maxX), max(maxX) FROM nodes_index;')
lon_min, lon_max, lat_min, lat_max = c.fetchone()
return lat_min, lon_min, lat_max, lon_max
def labels(self):
"""All labels."""
c = self.db.cursor()
c.execute('SELECT id FROM nodes;')
result = [row[0] for row in c.fetchall()]
return result
def size(self):
c = self.db.cursor()
c.execute('SELECT count(*) FROM nodes')
result = c.fetchone()[0]
return result
def node_coordinates(self, node_key):
"""Get the coordinates of the given node.
:param node_key: Node label/key
:return: (lat, lon)
"""
c = self.db.cursor()
c.execute('SELECT y, x FROM nodes WHERE id = ?', (node_key, ))
result = c.fetchone()
if result is None:
raise Exception(f"No coordinates found for node {node_key}")
return result
def add_node(self, node, loc, ignore_doubles=False, no_index=False, no_commit=False):
"""Add new node to the map.
:param node: label
:param loc: (lat, lon) or (y, x)
:param ignore_doubles: When trying to add the same node, ignore it
:param no_commit: Do not commit to database (remember to commit later)
"""
c = self.db.cursor()
lat, lon = loc
# Nodes
q = "INSERT INTO nodes VALUES(?, ?, ?)"
try:
c.execute(q, (node, lon, lat))
except sqlite3.IntegrityError as exc:
if ignore_doubles and "UNIQUE constraint failed: nodes.id" in str(exc):
return
logger.error(f"Problem with adding node {node} {loc}")
raise exc
# Nodes index
if not no_index:
q = "INSERT INTO nodes_index VALUES(?, ?, ?, ?, ?)"
try:
c.execute(q, (node, lon, lon, lat, lat))
except sqlite3.IntegrityError as exc:
logger.error(f"Problem with adding node to index {node} {loc}")
raise exc
if not no_commit:
self.db.commit()
def reindex_nodes(self):
logger.debug("Reindexing nodes ...")
t_start = time.time()
c = self.db.cursor()
c.execute('DELETE FROM nodes_index')
q = ("INSERT INTO nodes_index "
"SELECT id, x, x, y, y FROM nodes")
c.execute(q)
self.db.commit()
c.execute('SELECT count(*) FROM nodes_index')
cnt = c.fetchone()[0]
t_delta = time.time() - t_start
logger.debug(f"... done, #rows = {cnt}, time = {t_delta} sec")
def add_nodes(self, nodes):
"""Add list of nodes to database.
:param nodes: List[Tuple[node_key, Tuple[lat, lon]]]
"""
c = self.db.cursor()
def get_node_index():
for key, (lat, lon) in nodes:
yield key, lon, lon, lat, lat
q = "INSERT INTO nodes_index VALUES(?, ?, ?, ?, ?)"
c.executemany(q, get_node_index())
def get_node_vals():
for key, (lat, lon) in nodes:
yield key, lon, lat
q = "INSERT INTO nodes VALUES(?, ?, ?)"
c.executemany(q, get_node_vals())
self.db.commit()
def del_node(self, node):
raise Exception("TODO")
def add_edge(self, node_a, node_b, loc_a=None, loc_b=None, speed=None, edge_type=None,
path=None, pathnum=None,
no_index=False, no_commit=False):
"""Add new edge to the map.
:param node_a: Label for the node that is the start of the edge
:param node_b: Label for the node that is the end of the edge
:param no_commit: Do not commit to database (remember to commit later)
"""
c = self.db.cursor()
eid = (node_a, node_b).__hash__()
c.execute('INSERT OR IGNORE INTO edges(id, path, pathnum, id1, id2, type, speed) VALUES (?, ?, ?, ?, ?, ?, ?)',
(eid, path, pathnum, node_a, node_b, edge_type, speed))
# c.execute('SELECT last_insert_rowid();')
# eid = c.fetchone()[0]
if not no_index:
if loc_a is None:
c.execute('SELECT y, x FROM nodes WHERE id = ?;', (node_a, ))
loc_a = c.fetchone()
if loc_b is None:
c.execute('SELECT y, x FROM nodes WHERE id = ?;', (node_b, ))
loc_b = c.fetchone()
lat1, lon1 = loc_a
lat2, lon2 = loc_b
if lat1 > lat2:
lat1, lat2 = lat2, lat1
if lon1 > lon2:
lon1, lon2 = lon2, lon1
c.execute('INSERT OR IGNORE INTO edges_index(id, minX, maxX, minY, maxY) VALUES (?, ?, ?, ?, ?)',
(eid, lon1, lon2, lat1, lat2))
if not no_commit:
self.db.commit()
def add_edges(self, edges, no_index=False):
"""Add list of nodes to database.
:param edges: List[Tuple[node_key, node_key]] or
List[Tuple[node_key, node_key, path_key, int]]
"""
c = self.db.cursor()
def get_edge():
for row in edges:
row = list(row) + ([None] * (6 - len(row)))
key_a, key_b, path, pathnum, edge_type, speed = row
eid = (key_a, key_b).__hash__()
yield eid, path, pathnum, key_a, key_b, edge_type, speed
q = "INSERT INTO edges(id, path, pathnum, id1, id2, type, speed) VALUES(?, ?, ?, ?, ?, ?, ?);"
c.executemany(q, get_edge())
self.db.commit()
if not no_index:
self.reindex_edges()
def reindex_edges(self):
logger.debug("Reindexing edges ...")
t_start = time.time()
c = self.db.cursor()
# c2 = self.db.cursor()
c.execute('DELETE FROM edges_index')
q = ('INSERT INTO edges_index '
'SELECT e.id, MIN(n1.x,n2.x), MAX(n1.x,n2.x), '
' MIN(n1.y,n2.y), MAX(n1.y,n2.y) '
'FROM edges e '
'INNER JOIN nodes n1 ON n1.id = e.id1 '
'INNER JOIN nodes n2 ON n2.id = e.id2')
c.execute(q)
# cnt = 0
# for row in c.execute(q):
# # Contained in query
# c2.execute('INSERT INTO edges_index(id, minX, maxX, minY, maxY) VALUES (?, ?, ?, ?, ?)', row)
# cnt += 1
self.db.commit()
c.execute('SELECT count(*) FROM edges_index')
cnt = c.fetchone()[0]
t_delta = time.time() - t_start
logger.debug(f"... done, #rows = {cnt}, time = {t_delta} sec")
def all_edges(self, bb=None):
"""Return all edges.
:param bb: Bounding box
:return: (key_a, loc_a, nbr, loc_b)
"""
c = self.db.cursor()
q = 'SELECT e.id1, e.id2, n1.x AS n1x, n2.x AS n2x, n1.y AS n1y, n2.y AS n2y ' + \
'FROM edges e, edges_index ei ' + \
'LEFT JOIN nodes n1 ON n1.id = e.id1 ' + \
'LEFT JOIN nodes n2 ON n2.id = e.id2 ' + \
'WHERE ei.id == e.id'
if bb:
min_y, min_x, max_y, max_x = bb
# Intersecting with query
q += ' AND ei.maxX >= ? AND ei.minX <= ? AND ei.maxY >= ? AND ei.minY <= ?'
c.execute(q, (min_x, max_x, min_y, max_y))
else:
c.execute(q)
for row in c.fetchall():
key_a, key_b, lon_a, lon_b, lat_a, lat_b = row
yield key_a, (lat_a, lon_a), key_b, (lat_b, lon_b)
def all_nodes(self, bb=None):
"""Return all nodes.
:param bb: Bounding box (minY, minX, maxY, maxX)
:return:
"""
c = self.db.cursor()
q = ('SELECT n.id, n.x, n.y '
'FROM nodes n, nodes_index ni '
'WHERE n.id = ni.id ')
if bb:
minY, minX, maxY, maxX = bb
q += 'AND ni.minX >= ? AND ni.maxX <= ? AND ni.minY >= ? AND ni.maxY <= ?'
c.execute(q, (minX, maxX, minY, maxY))
else:
c.execute(q)
for row in c.fetchall():
key_a, lon_a, lat_a = row
yield key_a, (lat_a, lon_a)
def purge(self):
pass
def to_xy(self, name=None):
"""Create a map that uses a projected XY representation on which Euclidean distances
can be used.
"""
if not self.use_latlon:
return self
if name is None:
name = self.name + "_xy"
logger.debug("Start transformation ...")
t_start = time.time()
nmap = self.__class__(name, dir=self.dir, use_latlon=self.use_latlon,
crs_xy=self.crs_xy, crs_lonlat=self.crs_lonlat)
raise Exception("to implement")
t_delta = time.time() - t_start
logger.debug(f"... done: rtree size = {self.rtree_size()}, time = {t_delta} sec")
return nmap
def latlon2xy(self, lat, lon):
x, y = self.lonlat2xy(lon, lat)
return x, y
def latlon2yx(self, lat, lon):
x, y = self.lonlat2xy(lon, lat)
return y, x
def xy2latlon(self, x, y):
lon, lat = self.xy2lonlat(x, y)
return lat, lon
def yx2latlon(self, y, x):
lon, lat = self.xy2lonlat(x, y)
return lat, lon
def nodes_closeto(self, loc, max_dist=None, max_elmt=None):
"""Return all nodes close to the given location.
:param loc: Location
:param max_dist: Maximal distance from the location
:param max_elmt: Return only the most nearby nodes
"""
t_start = time.time()
lat, lon = loc[:2]
lat_b, lon_l, lat_t, lon_r = self.box_around_point((lat, lon), max_dist)
bb = (lat_b, lon_l, # y_min, x_min
lat_t, lon_r) # y_max, x_max
nodes = self.all_nodes(bb=bb)
t_delta_search = time.time() - t_start
t_start = time.time()
results = []
for key_o, loc_o in nodes:
dist = self.distance(loc, loc_o)
if dist < max_dist:
results.append((dist, key_o, loc_o))
results.sort()
t_delta_dist = time.time() - t_start
logger.debug(f"Found {len(results)} closeby nodes "
f"in {t_delta_search} sec and computed distances in {t_delta_dist} sec")
if max_elmt is not None:
results = results[:max_elmt]
return results
def edges_closeto(self, loc, max_dist=None, max_elmt=None):
"""Return all nodes that are on an edge that is close to the given location.
:param loc: Location
:param max_dist: Maximal distance from the location
:param max_elmt: Return only the most nearby nodes
"""
print(f"edges_closeto({loc})")
t_start = time.time()
lat, lon = loc[:2]
lat_b, lon_l, lat_t, lon_r = self.box_around_point((lat, lon), max_dist)
bb = (lat_b, lon_l, # y_min, x_min
lat_t, lon_r) # y_max, x_max
logger.debug(f"Search in bounding box {bb}")
nodes = self.all_edges(bb=bb)
t_delta_search = time.time() - t_start
t_start = time.time()
results = []
for key_a, loc_a, key_b, loc_b in nodes:
dist, pi, ti = self.distance_point_to_segment(loc, loc_a, loc_b)
if dist < max_dist:
results.append((dist, key_a, loc_a, key_b, loc_b, pi, ti))
results.sort()
t_delta_dist = time.time() - t_start
logger.debug(f"Found {len(results)} closeby edges "
f"in {t_delta_search} sec and computed distances in {t_delta_dist} sec")
if max_elmt is not None:
results = results[:max_elmt]
return results
def nodes_nbrto(self, node):
c = self.db.cursor()
q = ('SELECT e.id2, n2.y, n2.x FROM edges e '
'INNER JOIN nodes n2 ON n2.id = e.id2 '
'WHERE e.id1 = ?')
results = []
for nbr_label, nbr_lat, nbr_lon in c.execute(q, (node, )):
results.append((nbr_label, (nbr_lat, nbr_lon)))
return results
def edges_nbrto(self, edge):
l1, l2 = edge
c = self.db.cursor()
c.execute('SELECT n.y, n.x FROM nodes n WHERE id = ?', (l2, ))
p2 = c.fetchone()
results = []
# Edges that connect at end of this edge
for l3, p3 in self.nodes_nbrto(l2):
results.append((l2, p2, l3, p3))
# Edges that are in parallel and close
edge_id = edge.__hash__()
q = ('SELECT e.id1, e.id2, n1.y, n1.x, n2.y, n2.x FROM close_edges ce '
'INNER JOIN edges e ON e.id = ce.id2 '
'INNER JOIN nodes n1 ON n1.id = e.id1 '
'INNER JOIN nodes n2 ON n2.id = e.id2 '
'WHERE ce.id1 = ?')
for l3, l4, p3lat, p3lon, p4lat, p4lon in c.execute(q, (edge_id,)):
results.append((l3, (p3lat, p3lon), l4, (p4lat, p4lon)))
return results
def find_duplicates(self, func=None):
"""Find entries with identical locations."""
c = self.db.cursor()
logger.debug('Find duplicates ...')
t_start = time.time()
cnt = 0
q = ('select count(*)as qty, group_concat(id) '
'from nodes '
'group by y, x '
'having qty > 1 ')
for ncnt, idxs in c.execute(q):
func(int(idx) for idx in idxs.split(","))
t_delta = time.time() - t_start
logger.info(f"Found {cnt} doubles, time: {t_delta} seconds")
def connect_parallelroads(self, dist=0.5, bb=None):
c = self.db.cursor()
it = self.all_edges(bb=bb)
if tqdm:
it = tqdm.tqdm(list(it))
cnt = 0
for key_a, loc_a, key_b, loc_b in it:
e_id1 = (key_a, key_b).__hash__()
bb2 = [min(loc_a[0], loc_b[0]), min(loc_a[1], loc_b[1]),
max(loc_a[0], loc_b[0]), max(loc_a[1], loc_b[1])]
for key_c, loc_c, key_d, loc_d in self.all_edges(bb=bb2):
e_id2 = (key_c, key_d).__hash__()
if key_a == key_c or key_a == key_d or key_b == key_c or key_b == key_d:
continue
# print(f"Test: ({key_a},{key_b}) - ({key_c},{key_d})")
if self.lines_parallel(loc_a, loc_b, loc_c, loc_d, d=dist):
# print(f"Parallel: ({key_a},{key_b}) - ({key_c},{key_d})")
c.execute('INSERT INTO close_edges(id1, id2) VALUES (?, ?)', (e_id1, e_id2))
c.execute('INSERT INTO close_edges(id1, id2) VALUES (?, ?)', (e_id2, e_id1))
cnt += 1
logger.debug(f"Linked {cnt} edges")
self.db.commit()
def nodes_to_paths(self, nodes, ignore_nopath=True):
c = self.db.cursor()
prev_path = None
paths = []
for begin, end in zip(nodes[:-1], nodes[1:]):
c.execute("SELECT path FROM edges WHERE id1=? AND id2=?", (begin, end))
path = c.fetchone()[0]
if path is None and ignore_nopath:
continue
if path != prev_path:
paths.append(path)
prev_path = path
return paths
def path_dist(self, path):
c = self.db.cursor()
dist = 0
q = ('SELECT n1.y, n1.x, n2.y, n2.x FROM edges e '
'INNER JOIN nodes n1 ON n1.id = e.id1 '
'INNER JOIN nodes n2 ON n2.id = e.id2 '
'WHERE e.pathnum>0 AND e.path=?')
for lat1, lon1, lat2, lon2 in c.execute(q, (path,)):
dist += self.distance((lat1, lon1), (lat2, lon2))
return dist
def print_stats(self):
print("Graph\n-----")
print("Nodes: {}".format(len(self.graph)))
def __str__(self):
# s = ""
# for label, (loc, nbrs, _) in self.graph.items():
# s += f"{label:<10} - ({loc[0]:10.4f}, {loc[1]:10.4f})\n"
# return s
c = self.db.cursor()
c.execute("select sqlite_version()")
row = c.fetchone()
version = row[0]
return f"SqliteMap({self.name}, size={self.size()}, version={version})"
================================================
FILE: leuvenmapmatching/matcher/__init__.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.matcher
~~~~~~~~~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
================================================
FILE: leuvenmapmatching/matcher/base.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.matcher.base
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Base Matcher and Matching classes.
This a generic base class to be used by matchers. This class itself
does not implement a working matcher.
:author: Wannes Meert
:copyright: Copyright 2015-2021 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
from __future__ import print_function
import math
import sys
import logging
import time
from collections import OrderedDict, defaultdict, namedtuple
from itertools import islice
from typing import List, Tuple, Dict, Any, Optional, Set
import numpy as np
from ..util.segment import Segment
from ..util import approx_equal, approx_leq
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
approx_value = 0.0000000001
ema_const = namedtuple('EMAConst', ['prev', 'cur'])(0.7, 0.3)
default_label_width = 25
class BaseMatching(object):
"""Matching object that represents a node in the Viterbi lattice."""
__slots__ = ['matcher', 'edge_m', 'edge_o',
'logprob', 'logprobema', 'logprobe', 'logprobne',
'obs', 'obs_ne', 'dist_obs',
'prev', 'prev_other', 'stop', 'length', 'delayed']
def __init__(self, matcher: 'BaseMatcher', edge_m: Segment, edge_o: Segment,
logprob=-np.inf, logprobema=-np.inf, logprobe=-np.inf, logprobne=-np.inf,
dist_obs: float = 0.0, obs: int = 0, obs_ne: int = 0,
prev: Optional[Set['BaseMatching']] = None, stop: bool = False, length: int = 1,
delayed: int = 0, **_kwargs):
"""
:param matcher: Reference to the Matcher used to generate this matching object.
:param edge_m: Segment in the given graph (thus line between two nodes in the graph).
:param edge_o: Segment in the given observations (thus line in between two observations).
:param logprob: Log probability of this matching.
:param logprobema: Exponential Mean Average of Log probability.
:param logprobe: Emitting
:param logprobne: Non-emitting
:param dist_obs: Distance between map point and observation
:param obs: Reference to path entry index (observation)
:param obs_ne: Number of non-emitting states for this observation
:param prev: Previous best matching objects
:param stop: Stop after this matching (e.g. because probability is too low)
:param length: Lenght of current matching sequence through lattice.
:param delayed: This matching is temporarily stopped if >0 (e.g. to first explore better options).
:param dist_m: Distance over graph
:param dist_o: Distance over observations
:param _kwargs:
"""
self.edge_m: Segment = edge_m
self.edge_o: Segment = edge_o
self.logprob: float = logprob # max probability
self.logprobe: float = logprobe # Emitting
self.logprobne: float = logprobne # Non-emitting
self.logprobema: float = logprobema # exponential moving average log probability # TODO: Not used anymore?
self.obs: int = obs # reference to path entry index (observation)
self.obs_ne: int = obs_ne # number of non-emitting states for this observation
self.dist_obs: float = dist_obs # Distance between map point and observation
self.prev: Set[BaseMatching] = set() if prev is None else prev # Previous best matching objects
self.prev_other: Set[BaseMatching] = set() # Previous matching objects with lower logprob
self.stop: bool = stop
self.length: int = length
self.delayed: int = delayed
self.matcher: BaseMatcher = matcher
@property
def prune_value(self):
"""Pruning the lattice (e.g. to delay) is based on this key."""
return self.logprob
# return self.logprobema
def next(self, edge_m: Segment, edge_o: Segment, obs: int = 0, obs_ne: int = 0):
"""Create a next lattice Matching object with this Matching object as the previous one in the lattice."""
new_stop = False
if edge_m.is_point() and edge_o.is_point():
# node to node
dist = self.matcher.map.distance(edge_m.p1, edge_o.p1)
# proj_m = edge_m.p1
# proj_o = edge_o.pi
elif edge_m.is_point() and not edge_o.is_point():
# node to edge
dist, proj_o, t_o = self.matcher.map.distance_point_to_segment(edge_m.p1, edge_o.p1, edge_o.p2)
# proj_m = edge_m.p1
edge_o.pi = proj_o
edge_o.ti = t_o
elif not edge_m.is_point() and edge_o.is_point():
# edge to node
dist, proj_m, t_m = self.matcher.map.distance_point_to_segment(edge_o.p1, edge_m.p1, edge_m.p2)
if not self.matcher.only_edges and (approx_equal(t_m, 0.0) or approx_equal(t_m, 1.0)):
if __debug__ and logger.isEnabledFor(logging.DEBUG):
logger.debug(f" | Stopped trace: Too close to end, {t_m}")
new_stop = True
else:
return None
edge_m.pi = proj_m
edge_m.ti = t_m
# proj_o = edge_o.pi
elif not edge_m.is_point() and not edge_o.is_point():
# edge to edge
dist, proj_m, proj_o, t_m, t_o = self.matcher.map.distance_segment_to_segment(edge_m.p1, edge_m.p2,
edge_o.p1, edge_o.p2)
edge_m.pi = proj_m
edge_m.ti = t_m
edge_o.pi = proj_o
edge_o.ti = t_o
else:
raise Exception(f"Should not happen")
logprob_trans, props_trans = self.matcher.logprob_trans(self, edge_m, edge_o,
is_prev_ne=(self.obs_ne != 0),
is_next_ne=(obs_ne != 0))
logprob_obs, props_obs = self.matcher.logprob_obs(dist, self, edge_m, edge_o,
is_ne=(obs_ne != 0))
if __debug__ and logprob_trans > 0:
raise Exception(f"logprob_trans = {logprob_trans} > 0")
if __debug__ and logprob_obs > 0:
raise Exception(f"logprob_obs = {logprob_obs} > 0")
new_logprob_delta = logprob_trans + logprob_obs
if obs_ne == 0:
new_logprobe = self.logprob + new_logprob_delta
new_logprobne = 0
new_logprob = new_logprobe
new_length = self.length + 1
else:
# Non-emitting states require normalisation
# "* e^(ne_length_factor_log)" or "- ne_length_factor_log" for every step to a non-emitting
# state to prefer shorter paths
new_logprobe = self.logprobe + self.matcher.ne_length_factor_log
# The obvious choice would be average to compensate for that non-emitting states
# create different path lengths between emitting nodes.
# We use min() as it is a monotonic function, in contrast with an average
new_logprobne = min(self.logprobne, new_logprob_delta)
new_logprob = new_logprobe + new_logprobne
# Alternative approach with an average
# new_logprobne = self.logprobne + new_logprob_delta
# "+ 1" to punish non-emitting states a bit less. Otherwise it would be
# similar to (Pr_tr*Pr_obs)**2, which punishes just one non-emitting state too much.
# new_logprob = new_logprobe + new_logprobne / (obs_ne + 1)
new_length = self.length
new_logprobema = ema_const.cur * new_logprob_delta + ema_const.prev * self.logprobema
new_stop |= self.matcher.do_stop(new_logprob / new_length, dist, logprob_trans, logprob_obs)
if __debug__ and new_logprob > self.logprob:
raise Exception(f"Expecting a monotonic probability, "
f"new_logprob = {new_logprob} > logprob = {self.logprob}")
if not new_stop or (__debug__ and logger.isEnabledFor(logging.DEBUG)):
m_next = self.__class__(self.matcher, edge_m, edge_o,
logprob=new_logprob, logprobne=new_logprobne,
logprobe=new_logprobe, logprobema=new_logprobema,
obs=obs, obs_ne=obs_ne, prev={self}, dist_obs=dist,
stop=new_stop, length=new_length, delayed=self.delayed,
**props_trans, **props_obs)
return m_next
else:
return None
@classmethod
def first(cls, logprob_init, edge_m, edge_o, matcher, dist_obs):
"""Create an initial lattice Matching object."""
logprob_obs, props_obs = matcher.logprob_obs(dist_obs, None, edge_m, edge_o)
logprob = logprob_init + logprob_obs
new_stop = matcher.do_stop(logprob, dist_obs, logprob_init, logprob_obs)
if not new_stop or logger.isEnabledFor(logging.DEBUG):
m_next = cls(matcher, edge_m=edge_m, edge_o=edge_o,
logprob=logprob, logprobema=logprob, logprobe=logprob, logprobne=0,
dist_obs=dist_obs, obs=0, stop=new_stop, **props_obs)
return m_next
else:
return None
def update(self, m_next):
"""Update the current entry if the new matching object for this state is better.
:param m_next: The new matching object representing the same node in the lattice.
:return: True if the current object is replaced, False otherwise
"""
# if self.length != m_next.length:
# slogprob_norm = self.logprob / self.length
# nlogprob_norm = m_next.logprob / m_next.length
# else:
# slogprob_norm = self.logprob
# nlogprob_norm = m_next.logprob
# if (self.stop == m_next.stop and slogprob_norm < nlogprob_norm) or (self.stop and not m_next.stop):
# self._update_inner(m_next)
# return True
# elif abs(slogprob_norm - nlogprob_norm) < approx_value and self.stop == m_next.stop:
# self.prev.update(m_next.prev)
# self.stop = m_next.stop
# return False
assert self.length == m_next.length
if (self.stop and not m_next.stop) \
or (self.stop == m_next.stop and self.logprob < m_next.logprob):
self._update_inner(m_next)
return True
else:
self.prev_other.update(m_next.prev)
return False
def _update_inner(self, m_other: 'BaseMatching'):
self.edge_m = m_other.edge_m
self.edge_o = m_other.edge_o
self.logprob = m_other.logprob
self.logprobe = m_other.logprobe
self.logprobne = m_other.logprobne
self.logprobema = m_other.logprobema
self.dist_obs = m_other.dist_obs
self.obs = m_other.obs
self.obs_ne = m_other.obs_ne
self.prev_other.update(self.prev) # Do we use this?
self.prev = m_other.prev
self.stop = m_other.stop
self.delayed = m_other.delayed
self.length = m_other.length
def is_nonemitting(self):
return self.obs_ne != 0
def is_emitting(self):
return self.obs_ne == 0
def last_emitting_logprob(self):
if self.is_emitting():
return self.logprob
elif self.prev is None or len(self.prev) == 0:
return 0
else:
return next(iter(self.prev)).last_emitting_logprob()
def __str__(self, label_width=None):
stop = ''
if self.stop:
stop = 'x'
else:
stop = f'{self.delayed}'
if label_width is None:
label_width = default_label_width
repr_tmpl = "{:<2} | {:<"+str(label_width)+"} | {:10.5f} | {:10.5f} | {:10.5f} | {:10.5f} | " +\
"{:<3} | {:10.5f} | {:<" + str(label_width) + "} |"
return repr_tmpl.format(stop, self.label, self.logprob, self.logprob / self.length,
self.logprobema, self.logprobne, self.obs,
self.dist_obs, ",".join([str(prev.label) for prev in self.prev]))
def __repr__(self):
return "Matching<"+str(self.label)+">"
@staticmethod
def repr_header(label_width=None, stop=""):
if label_width is None:
label_width = default_label_width
repr_tmpl = "{:<2} | {:<"+str(label_width)+"} | {:<10} | {:<10} | {:<10} | {:<10} | " + \
"{:<3} | {:<10} | {:<"+str(label_width)+"} |"
return repr_tmpl.format(stop, "", "lg(Pr)", "nlg(Pr)", "slg(Pr)", "lg(Pr-ne)", "obs", "d(obs)", "prev")
@staticmethod
def repr_static(fields, label_width=None):
if label_width is None:
label_width = default_label_width
default_fields = ["", "", float('nan'), float('nan'), float('nan'), float('nan'), "", float('nan'), "", ""]
repr_tmpl = "{:<2} | {:<" + str(label_width) + "} | {:10.5f} | {:10.5f} | {:10.5f} | {:10.5f} | " + \
"{:<3} | {:10.5f} | {:<" + str(label_width) + "} |"
if len(fields) < 8:
fields = list(fields) + default_fields[len(fields):]
return repr_tmpl.format(*fields)
@property
def label(self):
if self.edge_m.p2 is None:
return "{}---{}-{}".format(self.edge_m.l1, self.obs, self.obs_ne)
else:
return "{}-{}-{}-{}".format(self.edge_m.l1, self.edge_m.l2, self.obs, self.obs_ne)
@property
def cname(self):
if self.edge_m.l2 is None:
return "{}_{}_{}".format(self.edge_m.l1, self.obs, self.obs_ne)
else:
return "{}_{}_{}_{}".format(self.edge_m.l1, self.edge_m.l2, self.obs, self.obs_ne)
@property
def key(self):
"""Key that indicates the node or edge, observation and non-emitting step.
This is the unique key that is used in the lattice.
"""
if self.edge_m.l2 is None:
return tuple([self.edge_m.l1, self.obs, self.obs_ne])
else:
return tuple([self.edge_m.l1, self.edge_m.l2, self.obs, self.obs_ne])
@property
def shortkey(self):
"""Key that indicates the node or edge. Irrespective of the current observation."""
if self.edge_m.l2 is None:
return self.edge_m.l1
else:
return tuple([self.edge_m.l1, self.edge_m.l2])
@property
def nodes(self):
if self.edge_m.l2 is None:
return [self.edge_m.l1]
else:
return [self.edge_m.l1, self.edge_m.l2]
def __hash__(self):
return self.cname.__hash__()
def __lt__(self, o):
return self.logprob < o.logprob
def __le__(self, o):
return self.logprob <= o.logprob
def __eq__(self, o):
return self.logprob == o.logprob
def __ne__(self, o):
return self.logprob != o.logprob
def __ge__(self, o):
return self.logprob >= o.logprob
def __gt__(self, o):
return self.logprob > o.logprob
class LatticeColumn:
def __init__(self, obs_idx):
# 0 = obs, >0 = non-emitting between this obs and next
self.obs_idx = obs_idx
self.o = [] # type list[dict[label,Matching]]
def __contains__(self, item):
for c in self.o:
if item in c:
return True
return False
def __len__(self):
return len(self.o)
def set_delayed(self, delayed):
"""Update all delayed values."""
for c in self.o:
for m in c.values():
m.delayed = delayed
def dict(self, obs_ne=None):
if obs_ne is None:
raise AttributeError('obs_ne should be value')
while obs_ne >= len(self.o):
self.o.append({})
return self.o[obs_ne]
def values_all(self):
"""All matches for the emitting layer and all non-emitting layers."""
values = set()
for o in self.o:
values.update(o.values())
return values
def values(self, obs_ne=None):
if obs_ne is None:
raise AttributeError('obs_ne should be value')
if len(self.o) <= obs_ne:
return []
return self.o[obs_ne].values()
def upsert(self, matching):
# type: (BaseMatching) -> None
if matching is None:
return None
while matching.obs_ne >= len(self.o):
self.o.append({})
c = self.o[matching.obs_ne]
if matching.key in c:
other_matching = c[matching.key] # type: BaseMatching
other_matching.update(matching)
else:
c[matching.key] = matching
return c[matching.key]
def prune(self, obs_ne, max_lattice_width, expand_upto, prune_thr=None):
"""Prune given column in the lattice to fit in max_lattice_width.
Also ignore all matchings with a probability lower than prune_thr. These are
matchings that are worse than the matchings at the next observation that are
retained after pruning.
:param obs_ne:
:param max_lattice_width:
:param expand_upto: The current expand level
:return:
"""
cur_lattice = [m for m in self.values(obs_ne) if not m.stop]
if __debug__:
logger.debug('Prune lattice[{},{}] from {} to {}, with prune thr {}'
.format(self.obs_idx, obs_ne,
len([m for m in cur_lattice if not m.stop and m.delayed == expand_upto]),
max_lattice_width, prune_thr))
cnt_pruned = 0
if max_lattice_width is not None and len(cur_lattice) > max_lattice_width:
ms = sorted(cur_lattice, key=lambda t: t.prune_value, reverse=True)
cur_width = max_lattice_width
m_last = ms[cur_width - 1]
# Extend current width if next pruned matching has same logprob as last kept matching
# This increases the lattice width but otherwise the algorithm depends on the
# order of edges/nodes and is not deterministic.
while cur_width < len(ms) and ms[cur_width].prune_value == m_last.prune_value:
m_last = ms[cur_width]
cur_width += 1
if prune_thr is not None:
while cur_width > 0 and ms[cur_width - 1].prune_value < prune_thr:
cur_width -= 1
for m in ms[:cur_width]: # type: BaseMatching
if m.delayed > expand_upto:
m.delayed = expand_upto # expand now
for m in ms[cur_width:]:
if m.delayed <= expand_upto:
if __debug__:
cnt_pruned += 1
m.delayed = expand_upto + 1 # expand later
if cur_width > 0:
prune_thr = ms[cur_width - 1].prune_value
if __debug__:
logger.debug(f'Pruned {cnt_pruned} matchings, return {prune_thr}')
return prune_thr
class BaseMatcher:
def __init__(self, map_con, obs_noise=1, max_dist_init=None, max_dist=None, min_prob_norm=None,
non_emitting_states=True, max_lattice_width=None,
only_edges=True, obs_noise_ne=None, matching=BaseMatching,
non_emitting_length_factor=0.75, **kwargs):
"""Initialize a matcher for map matching.
This a generic base class to be used by matchers. This class itself
does not implement a working matcher.
Distances are in meters when using latitude-longitude.
:param map_con: Map object to connect to map database
:param obs_noise: Standard deviation of noise
:param obs_noise_ne: Standard deviation of noise for non-emitting states (is set to obs_noise if not give)
:param max_dist_init: Maximum distance from start location (if not given, uses max_dist)
:param max_dist: Maximum distance from path (this is a hard cut, min_prob_norm should be better)
:param min_prob_norm: Minimum normalized probability of observations (ema)
:param non_emitting_states: Allow non-emitting states. A non-emitting state is a state that is
not associated with an observation. Here we assume it can be associated with a location in between
two observations to allow for pruning. It is advised to set min_prob_norm and/or max_dist to avoid
visiting all possible nodes in the graph.
:param max_lattice_width: Only continue from a limited number of states (thus locations) for a given observation.
This possibly speeds up the matching by a lot.
If there are more possible next states, the states with the best likelihood so far are selected.
The other states are 'delayed'. If the matching is continued later with a larger value using
`increase_max_lattice_width`, the algorithms continuous from these delayed states.
:param only_edges: Do not include nodes as states, only edges. This is the typical setting for HMM methods.
:param matching: Matching type
:param non_emitting_length_factor: Reduce the probability of a sequence of non-emitting states the longer it
is. This can be used to prefer shorter paths. This is separate from the transition probabilities because
transition probabilities are averaged for non-emitting states and thus the length is also averaged out.
To define a custom transition and/or emission probability distribtion, overwrite the following functions:
- :meth:`logprob_trans`
- :meth:`logprob_obs`
"""
self.map = map_con # type: BaseMap
if max_dist:
self.max_dist = max_dist
else:
self.max_dist = np.inf
if max_dist_init:
self.max_dist_init = max_dist_init
else:
self.max_dist_init = self.max_dist
if min_prob_norm:
self.min_logprob_norm = math.log(min_prob_norm)
else:
self.min_logprob_norm = -np.inf
logger.debug(f"Matcher.min_logprob_norm = {self.min_logprob_norm}, Matcher.max_dist = {self.max_dist}")
self.obs_noise = obs_noise
if obs_noise_ne is None:
self.obs_noise_ne = obs_noise
else:
self.obs_noise_ne = obs_noise_ne
self.path = None
self.lattice = None # type: Optional[dict[int,LatticeColumn]]
# Best path through lattice:
self.lattice_best = None # type: Optional[list[BaseMatching]]
self.node_path = None # type: Optional[list[str]]
self.matching = matching
self.non_emitting_states = non_emitting_states # type: bool
self.non_emitting_states_maxnb = 100
self.max_lattice_width = max_lattice_width # type: Optional[int]
self.only_edges = only_edges # type: bool
self.expand_now = 0 # all m.delayed <= expand_upto will be expanded
self.early_stop_idx = None
# Penalties
self.ne_length_factor_log = math.log(non_emitting_length_factor)
def logprob_trans(self, prev_m, edge_m, edge_o,
is_prev_ne=False, is_next_ne=False):
# type: (BaseMatcher, BaseMatching, Segment, Segment, bool, bool) -> Tuple[float, Dict[str, Any]]
"""Transition probability.
Note: In contrast with a regular HMM, this cannot be a probability density function, it needs
to be a proper probability (thus values between 0.0 and 1.0).
:return: probability, properties that are passed to the matching object
"""
return 0, {} # All probabilities are 1 (thus technically not a distribution)
def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne=False):
"""Emission probability.
Note: In contrast with a regular HMM, this cannot be a probability density function, it needs
to be a proper probability (thus values between 0.0 and 1.0).
:return: probability, properties that are passed to the matching object
"""
return 0, {}
def match_gpx(self, gpx_file, unique=True):
"""Map matching from a gpx file"""
from ..util.gpx import gpx_to_path
path = gpx_to_path(gpx_file)
return self.match(path, unique=unique)
def do_stop(self, logprob_norm, dist, logprob_trans, logprob_obs):
if logprob_norm < self.min_logprob_norm:
logger.debug(f" | Stopped trace: norm(log(Pr)) too small: {logprob_norm} < {self.min_logprob_norm}"
f" -- lPr_t = {logprob_trans:.3f}, lPr_o = {logprob_obs:.3f}")
return True
if dist > self.max_dist:
logger.debug(f" | Stopped trace: distance too large: {dist} > {self.max_dist}")
return True
return False
def _insert(self, m_next):
return self.lattice[m_next.obs].upsert(m_next)
def match(self, path, unique=False, tqdm=None, expand=False):
"""Dynamic Programming based (HMM-like) map matcher.
If the matcher fails to match the entire path, the last matched index is returned.
This index can be used to run the matcher again from that observation onwards.
:param path: list[Union[tuple[lat, lon], tuple[lat, lon, time]]
:param unique: Only retain unique nodes in the sequence (avoid repetitions)
:param tqdm: Use a tqdm progress reporter (default is None)
:param expand: Expand the current lattice (delayed matches)
:return: Tuple of (List of BaseMatching, index of last observation that was matched)
"""
if __debug__:
logger.debug("Start matching path of length {}".format(len(path)))
# Initialisation
if expand:
self.expand_now += 1
if self.path != path:
is_path_extended = True
if len(path) > len(self.path):
for pi, spi in zip(path, self.path):
if pi != spi:
is_path_extended = False
break
else:
is_path_extended = False
if is_path_extended:
self.lattice[len(self.path) - 1].set_delayed(self.expand_now)
for obs_idx in range(len(self.path), len(path)):
if obs_idx not in self.lattice:
self.lattice[obs_idx] = LatticeColumn(obs_idx)
self.path = path
else:
raise Exception(f'Cannot expand for a new path, should be the same path (or an extension).')
else:
self.path = path
self.expand_now = 0
nb_start_nodes = self._create_start_nodes(use_edges=self.only_edges)
if nb_start_nodes == 0:
self.lattice_best = []
return [], 0
if __debug__ and logger.isEnabledFor(logging.DEBUG):
self.print_lattice(obs_idx=0, label_width=default_label_width, debug=True)
# Start iterating over observations 1..end
t_start = time.time()
iterator = range(1, len(path))
if tqdm:
iterator = tqdm(iterator)
self.early_stop_idx = None
for obs_idx in iterator:
if __debug__:
logger.debug("--- obs {} --- {} ---".format(obs_idx, self.path[obs_idx]))
# check if early stopping has occured
cnt_lat_size_not_zero = False
for m_tmp in self.lattice[obs_idx - 1].values(0):
if not m_tmp.stop:
cnt_lat_size_not_zero = True
break
# if len(self.lattice[obs_idx - 1]) == 0:
if not cnt_lat_size_not_zero:
if __debug__:
logger.debug("No solutions found anymore")
self.early_stop_idx = obs_idx - 1
logger.info(f'Stopped early at observation {self.early_stop_idx}')
break
# Expand matches
self._match_states(obs_idx)
if self.non_emitting_states:
# Fill in non-emitting states between previous and current observation
self._match_non_emitting_states(obs_idx - 1, expand=expand)
if self.max_lattice_width:
# Prune again if non_emitting_states reactives matches from match_states
self.lattice[obs_idx].prune(0, self.max_lattice_width, self.expand_now)
if __debug__ and logger.isEnabledFor(logging.DEBUG):
self.print_lattice(obs_idx=obs_idx, label_width=default_label_width, debug=True)
logger.debug(f"--- end obs {obs_idx} ---")
t_delta = time.time() - t_start
logger.info("--- end ---")
logger.info("Build lattice in {} seconds".format(t_delta))
# Backtrack to find best path
if not self.early_stop_idx:
one_no_stop = False
for m in self.lattice[len(path) - 1].values_all(): # todo: could be values(0) ?
if not m.stop:
one_no_stop = True
break
if not one_no_stop:
self.early_stop_idx = len(path) - 1
if self.early_stop_idx is not None:
if self.early_stop_idx == 0:
self.lattice_best = []
return [], 0
start_idx = self.early_stop_idx - 1
else:
start_idx = len(self.path) - 1
node_path = self._build_node_path(start_idx, unique)
return node_path, start_idx
def _skip_ne_states(self, prev_m):
# type: (BaseMatcher, BaseMatching) -> bool
return False
def _create_start_nodes(self, use_edges=True):
"""Find those nodes that are close to the first point in the path.
:return: Number of created start points.
"""
# Initialisation on first observation
if self.expand_now > 0:
# No need to search for new points, only activate delayed matches
self.lattice[0].prune(0, self.max_lattice_width, self.expand_now)
return len(self.lattice[0])
t_start = time.time()
self.lattice = dict()
for obs_idx in range(len(self.path)):
self.lattice[obs_idx] = LatticeColumn(obs_idx)
if use_edges:
nodes = self.map.edges_closeto(self.path[0], max_dist=self.max_dist_init)
else:
nodes = self.map.nodes_closeto(self.path[0], max_dist=self.max_dist_init)
if __debug__:
logger.debug("--- obs {} --- {} ---".format(0, self.path[0]))
t_delta = time.time() - t_start
logger.info("Initialized lattice with {} starting points in {} seconds".format(len(nodes), t_delta))
if len(nodes) == 0:
logger.info(f'Stopped early at observation 0'
f', no starting points/edges x found for which '
f'|x - ({self.path[0][0]:.2f},{self.path[0][1]:.2f})| < {self.max_dist_init}')
return 0
if __debug__:
logger.debug(self.matching.repr_header())
logprob_init = 0 # math.log(1.0/len(nodes))
if use_edges:
# Search for nearby edges
for dist_obs, label1, loc1, label2, loc2, pi, ti in nodes:
if label2 == label1:
continue
edge_m = Segment(label1, loc1, label2, loc2, pi, ti)
edge_o = Segment(f"O{0}", self.path[0])
m_next = self.matching.first(logprob_init, edge_m, edge_o, self, dist_obs)
if m_next is not None:
self.lattice[0].upsert(m_next)
if __debug__:
logger.debug(str(m_next))
else:
# Search for nearby nodes
for dist_obs, label, loc in nodes:
edge_m = Segment(label, loc)
edge_o = Segment(f"O{0}", self.path[0])
m_next = self.matching.first(logprob_init, edge_m, edge_o, self, dist_obs)
if m_next is not None:
self.lattice[0].upsert(m_next)
if __debug__:
logger.debug(str(m_next))
if self.max_lattice_width:
self.lattice[0].prune(0, max_lattice_width=self.max_lattice_width, expand_upto=self.expand_now)
# if self.non_emitting_states:
# self._match_non_emitting_states(0, path)
return len(self.lattice[0])
def increase_delayed(self, expand_from=None):
if expand_from is None:
expand_from = self.expand_now + 1
for col in self.lattice.values():
for colo in col.o:
for m in colo.values():
if m.delayed >= expand_from:
m.delayed += 1
def _match_states(self, obs_idx, prev_lattice=None, max_dist=None, inc_delayed=False):
"""Match states
:param obs_idx:
:param prev_lattice: Start from this list instead of the previous
column in the lattice
:param max_dist: Use map.*_closeto instead of map.*_nbrto
:param inc_delayed: Increase delayed property when new state is created
:return: True is new states have been found, False otherwise.
"""
if prev_lattice is None:
prev_lattice = [m for m in self.lattice[obs_idx - 1].values(0) if not m.stop and m.delayed == self.expand_now]
count = 0
for m in prev_lattice: # type: BaseMatching
if m.stop:
assert False # should not happen
continue
count += 1
if m.edge_m.is_point():
# == Move to neighbour from node ==
if max_dist is None:
nbrs = self.map.nodes_nbrto(m.edge_m.l1)
else:
nbrs = self.map.nodes_closeto(m.edge_m.p1, max_dist=max_dist)
# print("Neighbours for {}: {}".format(m, nbrs))
if nbrs is None:
if __debug__:
logger.debug("No neighbours found for node {}".format(m.edge_m.l1))
continue
if __debug__:
logger.debug(" + Move to {} neighbours from node {}".format(len(nbrs), m.edge_m.l1))
logger.debug(m.repr_header())
for nbr_label, nbr_loc in nbrs:
# === Move from node to node (or stay on node) ===
if not self.only_edges:
edge_m = Segment(nbr_label, nbr_loc)
edge_o = Segment(f"O{obs_idx}", self.path[obs_idx])
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if inc_delayed:
m_next.delayed += 1
self._insert(m_next)
if __debug__:
logger.debug(str(m_next))
# === Move from node to edge ===
if m.edge_m.l1 != nbr_label:
edge_m = Segment(m.edge_m.l1, m.edge_m.p1, nbr_label, nbr_loc)
edge_o = Segment(f"O{obs_idx}", self.path[obs_idx])
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if inc_delayed:
m_next.delayed += 1
self._insert(m_next)
if __debug__:
logger.debug(str(m_next))
else:
if __debug__:
logger.debug(self.matching.repr_static(('x', f'{nbr_label}-{nbr_label} < self-loop')))
else:
# == Move to neighbour from edge ==
if __debug__:
logger.debug(" + Move to neighbour from edge {}".format(m.label))
logger.debug(m.repr_header())
# === Stay on edge ===
edge_m = Segment(m.edge_m.l1, m.edge_m.p1, m.edge_m.l2, m.edge_m.p2)
edge_o = Segment(f"O{obs_idx}", self.path[obs_idx])
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if inc_delayed:
m_next.delayed += 1
self._insert(m_next)
if __debug__:
logger.debug(str(m_next))
# === Move from edge to node ===
if not self.only_edges:
edge_m = Segment(m.edge_m.l2, m.edge_m.p2)
edge_o = Segment(f"O{obs_idx}", self.path[obs_idx])
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if inc_delayed:
m_next.delayed += 1
self._insert(m_next)
if __debug__:
logger.debug(str(m_next))
else:
# === Move from edge to next edge ===
if max_dist is None:
nbrs = self.map.edges_nbrto((m.edge_m.l1, m.edge_m.l2)) # type: list
else:
nbrs = [(l1, p1, l2, p2) for _, l1, p1, l2, p2, _, _
in self.map.edges_closeto(m.edge_m.pi, max_dist=max_dist)]
if nbrs is None or len(nbrs) == 0:
if __debug__:
logger.debug(f"No neighbours found for edge {m.edge_m.label}")
continue
for nbr_label1, nbr_loc1, nbr_label2, nbr_loc2 in nbrs:
# same edge is different action, opposite edge should be allowed to return in a one-way street
if m.edge_m.l2 != nbr_label2 and m.edge_m.l1 != nbr_label1:
edge_m = Segment(nbr_label1, nbr_loc1, nbr_label2, nbr_loc2)
edge_o = Segment(f"O{obs_idx}", self.path[obs_idx])
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if inc_delayed:
m_next.delayed += 1
self._insert(m_next)
if __debug__:
mstr = str(m_next)
logger.debug(mstr)
if self.max_lattice_width:
self.lattice[obs_idx].prune(0, self.max_lattice_width, self.expand_now)
if count == 0:
if __debug__:
logger.debug("No active solution found anymore")
return False
return True
def _match_non_emitting_states(self, obs_idx, expand=False):
"""Match sequences of nodes that all refer to the same observation at obs_idx.
Assumptions:
This method assumes that the lattice is filled up for both obs_idx and obs_idx + 1.
:param obs_idx: Index of the first observation used (the second will be obs_idx + 1)
:return: None
"""
obs = self.path[obs_idx]
if obs_idx < len(self.path) - 1:
obs_next = self.path[obs_idx + 1]
else:
obs_next = None
# The current states are the current observation's states
if expand:
cur_lattice = dict((m.key, m) for m in self.lattice[obs_idx].values(0) if not m.stop and m.delayed == self.expand_now)
else:
cur_lattice = dict((m.key, m) for m in self.lattice[obs_idx].values(0) if not (m.stop or m.delayed > 0))
lattice_toinsert = list()
# The current best states are the next observation's states if you would ignore non-emitting states
lattice_best = dict((m.shortkey, m)
for m in self.lattice[obs_idx + 1].values(0) if not m.stop)
lattice_ne = set(m.shortkey
for m in self.lattice[obs_idx + 1].values(0) if not m.stop and self._skip_ne_states(m))
# cur_lattice = set(self.lattice[obs_idx].values())
nb_ne = 0
prune_thr = None
while len(cur_lattice) > 0 and nb_ne < self.non_emitting_states_maxnb:
nb_ne += 1
if __debug__:
logger.debug("--- obs {}:{} --- {} - {} ---".format(obs_idx, nb_ne, obs, obs_next))
cur_lattice = self._match_non_emitting_states_inner(cur_lattice, obs_idx, obs, obs_next, nb_ne,
lattice_best, lattice_ne)
if self.max_lattice_width is not None:
self.lattice[obs_idx].prune(nb_ne, self.max_lattice_width, self.expand_now, prune_thr)
# Link to next observation
self._match_non_emitting_states_end(cur_lattice, obs_idx + 1, obs_next,
lattice_best, expand=expand)
if self.max_lattice_width is not None:
prune_thr = self.lattice[obs_idx + 1].prune(0, self.max_lattice_width, self.expand_now, None)
if self.max_lattice_width is not None:
self.lattice[obs_idx + 1].prune(0, self.max_lattice_width, self.expand_now, None)
# logger.info('Used {} levels of non-emitting states'.format(nb_ne))
# for m in lattice_toinsert:
# self._insert(m)
def _node_in_prev_ne(self, m_next, label):
"""Is the given node already visited in the chain of non-emitting states.
:param m_next:
:param label: Node label
:return: True or False
"""
# for m in itertools.chain(m_next.prev, m_next.prev_other):
for m in m_next.prev: # type: BaseMatching
if m.obs != m_next.obs:
return False
assert(m_next.obs_ne != m.obs_ne)
# print('prev', m.shortkey, 'checking for ', label)
# if label == m.shortkey:
if label in m.nodes:
return True
if m.obs_ne == 0:
return False
if self._node_in_prev_ne(m, label):
return True
return False
@staticmethod
def _insert_tmp(m_next, lattice):
if m_next.key in lattice:
return lattice[m_next.key].update(m_next)
else:
lattice[m_next.key] = m_next
return True
def _match_non_emitting_states_inner(self, cur_lattice, obs_idx, obs, obs_next, nb_ne,
lattice_best, lattice_ne):
# cur_lattice_new = dict()
cur_lattice_new = self.lattice[obs_idx].dict(nb_ne)
for m in cur_lattice.values(): # type: BaseMatching
if m.stop or m.delayed != self.expand_now:
continue
if m.shortkey in lattice_ne:
logger.debug(f"Skip non-emitting states from {m.label}, already visited")
continue
# == Move to neighbour edge from edge ==
if m.edge_m.l2 is not None and self.only_edges:
nbrs = self.map.edges_nbrto((m.edge_m.l1, m.edge_m.l2))
# print("Neighbours for {}: {}".format(m, nbrs))
if nbrs is None or len(nbrs) == 0:
if __debug__:
logger.debug(f"No neighbours found for edge {m.edge_m.label} ({m.label}, non-emitting)")
continue
for nbr_label1, nbr_loc1, nbr_label2, nbr_loc2 in nbrs:
if self._node_in_prev_ne(m, nbr_label2):
if __debug__:
logger.debug(self.matching.repr_static(('x', '{} < node in prev ne'.format(nbr_label2))))
continue
# === Move to next edge ===
if m.edge_m.l2 != nbr_label2 and m.edge_m.l1 != nbr_label2:
edge_m = Segment(nbr_label1, nbr_loc1, nbr_label2, nbr_loc2)
edge_o = Segment(f"O{obs_idx}", obs, f"O{obs_idx+1}", obs_next)
m_next = m.next(edge_m, edge_o, obs=obs_idx, obs_ne=nb_ne)
if m_next is not None:
if m_next.key in cur_lattice_new:
if m_next.shortkey in lattice_best:
if approx_leq(m_next.dist_obs, lattice_best[m_next.shortkey].dist_obs):
cur_lattice_new[m_next.key].update(m_next)
else:
m_next.stop = True
if __debug__ and logger.isEnabledFor(logging.DEBUG):
logger.debug(f" | Stopped trace: distance larger than best for key {m_next.shortkey}: "
f"{m_next.dist_obs} > {lattice_best[m_next.shortkey].dist_obs}")
else:
cur_lattice_new[m_next.key].update(m_next)
else:
if m_next.shortkey in lattice_best:
# if m_next.logprob > lattice_best[m_next.shortkey].logprob:
if approx_leq(m_next.dist_obs, lattice_best[m_next.shortkey].dist_obs):
cur_lattice_new[m_next.key] = m_next
# lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
else:
if __debug__ and logger.isEnabledFor(logging.DEBUG):
logger.debug(f" | Stopped trace: distance larger than best for key {m_next.shortkey}: "
f"{m_next.dist_obs} > {lattice_best[m_next.shortkey].dist_obs}")
m_next.stop = True
else:
cur_lattice_new[m_next.key] = m_next
# lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
# cur_lattice_new.add(m_next)
if __debug__:
logger.debug(str(m_next))
else:
if __debug__:
logger.debug(self.matching.repr_static(('x', f'{nbr_label1}-{nbr_label2} < goes back (ne)')))
# == Move to neighbour node from node==
if m.edge_m.l2 is None and not self.only_edges:
cur_node = m.edge_m.l1
nbrs = self.map.nodes_nbrto(cur_node)
if nbrs is None:
if __debug__:
logger.debug(
f"No neighbours found for node {cur_node} ({m.label}, non-emitting)")
continue
if __debug__:
logger.debug(
f" + Move to {len(nbrs)} neighbours from node {cur_node} ({m.label}, non-emitting)")
logger.debug(m.repr_header())
for nbr_label, nbr_loc in nbrs:
# print(f"self._node_in_prev_ne({m.label}, {nbr_label}) = {self._node_in_prev_ne(m, nbr_label)}")
if self._node_in_prev_ne(m, nbr_label):
if __debug__:
logger.debug(self.matching.repr_static(('x', '{} < node in prev ne'.format(nbr_label))))
continue
# === Move to next node ===
if m.edge_m.l1 != nbr_label:
edge_m = Segment(nbr_label, nbr_loc)
edge_o = Segment(f"O{obs_idx}", obs, f"O{obs_idx+1}", obs_next)
m_next = m.next(edge_m, edge_o, obs=obs_idx, obs_ne=nb_ne)
if m_next is not None:
if m_next.key in cur_lattice_new:
cur_lattice_new[m_next.key].update(m_next)
else:
if m_next.shortkey in lattice_best:
# if m_next.logprob > lattice_best[m_next.shortkey].logprob:
if m_next.dist_obs < lattice_best[m_next.shortkey].dist_obs:
cur_lattice_new[m_next.key] = m_next
lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
elif __debug__ and logger.isEnabledFor(logging.DEBUG):
m_next.stop = True
cur_lattice_new[m_next.key] = m_next
# lattice_toinsert.append(m_next)
else:
cur_lattice_new[m_next.key] = m_next
lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
# cur_lattice_new.add(m_next)
if __debug__:
logger.debug(str(m_next))
else:
if __debug__:
logger.debug(f"x | {m.edge_m.l1}-{nbr_label} < self-loop")
return cur_lattice_new
def _match_non_emitting_states_end(self, cur_lattice, obs_idx, obs_next,
lattice_best, expand=False):
for m in cur_lattice.values(): # type: BaseMatching
if m.stop or m.delayed > self.expand_now:
continue
if m.edge_m.l2 is not None:
# Move to neighbour edge from edge
nbrs = self.map.edges_nbrto((m.edge_m.l1, m.edge_m.l2))
# print("Neighbours for {}: {}".format(m, nbrs))
if nbrs is None or len(nbrs) == 0:
if __debug__:
logger.debug("No neighbours found for edge {} ({})".format(m.edge_m.label, m.label))
continue
if __debug__:
logger.debug(f" + Move to {len(nbrs)} neighbours from edge {m.edge_m.label} "
f"({m.label}, non-emitting->emitting)")
logger.debug(m.repr_header())
for nbr_label1, nbr_loc1, nbr_label2, nbr_loc2 in nbrs:
if self._node_in_prev_ne(m, nbr_label2):
if __debug__:
logger.debug(self.matching.repr_static(('x', '{} < node in prev ne'.format(nbr_label2))))
continue
# Move to next edge
if m.edge_m.l1 != nbr_label2 and m.edge_m.l2 != nbr_label2:
edge_m = Segment(nbr_label1, nbr_loc1, nbr_label2, nbr_loc2)
edge_o = Segment(f"O{obs_idx+1}", obs_next)
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if m_next.shortkey in lattice_best:
# if m_next.dist_obs < lattice_best[m_next.shortkey].dist_obs:
if m_next.logprob > lattice_best[m_next.shortkey].logprob:
lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
self.lattice[obs_idx].upsert(m_next)
elif __debug__ and logger.isEnabledFor(logging.DEBUG):
m_next.stop = True
# lattice_toinsert.append(m_next)
self.lattice[obs_idx].upsert(m_next)
else:
lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
self.lattice[obs_idx].upsert(m_next)
if __debug__:
logger.debug(str(m_next))
else:
if __debug__:
logger.debug(self.matching.repr_static(('x', '{} < going back'.format(nbr_label2))))
else: # m.edge_m.l2 is None:
# Move to neighbour node from node
cur_node = m.edge_m.l1
nbrs = self.map.nodes_nbrto(cur_node)
# print("Neighbours for {}: {}".format(m, nbrs))
if nbrs is None:
if __debug__:
logger.debug("No neighbours found for node {}".format(cur_node, m.label))
continue
if __debug__:
logger.debug(f" + Move to {len(nbrs)} neighbours from node {cur_node} "
f"({m.label}, non-emitting->emitting)")
logger.debug(m.repr_header())
for nbr_label, nbr_loc in nbrs:
if self._node_in_prev_ne(m, nbr_label):
if __debug__:
logger.debug(self.matching.repr_static(('x', '{} < node in prev ne'.format(nbr_label))))
continue
# Move to next node
if m.edge_m.l1 != nbr_label:
# edge_m = Segment(m.edge_m.l1, m.edge_m.p1, nbr_label, nbr_loc)
edge_m = Segment(nbr_label, nbr_loc)
edge_o = Segment(f"O{obs_idx+1}", obs_next)
m_next = m.next(edge_m, edge_o, obs=obs_idx)
if m_next is not None:
if m_next.shortkey in lattice_best:
# if m_next.dist_obs < lattice_best[m_next.shortkey].dist_obs:
if m_next.logprob > lattice_best[m_next.shortkey].logprob:
lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
self.lattice[obs_idx].upsert(m_next)
elif __debug__ and logger.isEnabledFor(logging.DEBUG):
m_next.stop = True
# lattice_toinsert.append(m_next)
self.lattice[obs_idx].upsert(m_next)
else:
lattice_best[m_next.shortkey] = m_next
# lattice_toinsert.append(m_next)
self.lattice[obs_idx].upsert(m_next)
if __debug__:
logger.debug(str(m_next))
else:
if __debug__:
logger.debug(self.matching.repr_static(('x', '{} < self-loop'.format(nbr_label))))
def get_matching(self, identifier=None):
m = None # type: Optional[BaseMatching]
if isinstance(identifier, BaseMatching):
m = identifier
elif identifier is None:
col = self.lattice[len(self.lattice) - 1]
for curm in col.values_all():
if m is None or curm.logprob > m.logprob:
m = curm
elif type(identifier) is int:
# If integer, search for the best matching at this index in the lattice
for cur_m in self.lattice[identifier].values_all(): # type:BaseMatching
if not cur_m.stop and (m is None or cur_m.logprob > m.logprob):
m = cur_m
elif type(identifier) is str:
# If string, try to parse identifier
parts = identifier.split('-')
idx, ne, key = None, None, None
if len(parts) == 4:
nodea, nodeb, idx, ne = [int(part) for part in parts]
key = (nodea, nodeb, idx, ne)
col = self.lattice[idx] # type: LatticeColumn
col_ne = col.o[ne]
m = col_ne[key]
elif len(parts) == 3:
node, idx, ne = [int(part) for part in parts]
key = (node, idx, ne)
col = self.lattice[idx] # type: LatticeColumn
col_ne = col.o[ne]
m = col_ne[key]
elif len(parts) == 1:
m = None
l1 = int(parts[0])
for l in self.lattice.values(): # type: LatticeColumn
for curm in l.values_all():
if (curm.edge_m.l1 == l1 or curm.edge_m.l2 == l1) and \
(m is None or curm.logprob > m.logprob):
m = curm
else:
raise AttributeError(f'Unknown string format for matching. '
'Expects <node>-<idx>-<ne> or <node>-<node>-<idx>-<ne>.')
return m
def get_matching_path(self, start_m):
"""List of Matching objects that end in the given Matching object."""
start_m = self.get_matching(start_m)
return self._build_matching_path(start_m)
def get_node_path(self, start_m, only_nodes=False):
"""List of node/edge names that end in the given Matching object."""
path = self.get_matching_path(start_m)
node_path = [m.shortkey for m in path]
if only_nodes:
node_path = self.node_path_to_only_nodes(node_path)
return node_path
def get_path(self, only_nodes=True, allow_jumps=False, only_closest=True):
"""A list with all the nodes (no edges) the matched path passes through."""
if only_nodes is False:
return self.node_path
if self.node_path is None or len(self.node_path) == 0:
return []
path = self.node_path_to_only_nodes(self.node_path, allow_jumps=allow_jumps)
if only_closest:
m = self.lattice_best[0]
if m.edge_m.ti > 0.5:
path.pop(0)
return path
def node_path_to_only_nodes(self, path, allow_jumps=False):
"""Path of nodes and edges to only nodes.
:param path: List of node names or edges as (node name, node name)
:param allow_jumps: Allow a path over edges that are not connected.
This occurs when matches are added without an edge, for example,
when searching for edges in the distance neighborhood instead in
the graph.
:return: List of node names
"""
nodes = []
prev_state = path[0]
if type(prev_state) is tuple:
nodes.append(prev_state[0])
nodes.append(prev_state[1])
prev_node = prev_state[1]
else:
nodes.append(prev_state)
prev_node = prev_state
for state in path[1:]:
if state == prev_state:
continue
if type(state) is not tuple:
if state != prev_node:
nodes.append(state)
prev_node = state
elif type(state) is tuple:
if state[0] == prev_node:
if state[1] != prev_node:
nodes.append(state[1])
prev_node = state[1]
elif state[1] == prev_node:
if state[0] != prev_node:
nodes.append(state[0])
prev_node = state[0]
elif not allow_jumps:
raise Exception(f"State {state} does not have as previous node {prev_node}")
else:
nodes.append(state[0])
nodes.append(state[1])
prev_node = state[1]
else:
raise Exception(f"Unknown type of state: {state} ({type(state)})")
prev_state = state
return nodes
def _build_matching_path(self, start_m, max_depth=None):
lattice_best = []
node_max = start_m
cur_depth = 0
if __debug__ and logger.isEnabledFor(logging.DEBUG):
logger.debug(self.matching.repr_header(stop=" "))
logger.debug("Start ({}): {}".format(node_max.obs, node_max))
lattice_best.append(node_max)
if node_max.is_emitting():
cur_depth += 1
# for obs_idx in reversed(range(start_idx)):
if max_depth is None:
max_depth = len(self.lattice) + 1
while cur_depth < max_depth and len(node_max.prev) > 0:
node_max_last = node_max
node_max: Optional[BaseMatching] = None
for prev_m in node_max_last.prev:
if prev_m is not None and (node_max is None or prev_m.logprob > node_max.logprob):
node_max = prev_m
if node_max is None:
logger.error("Did not find a matching node for path point at index {}. ".format(node_max_last.obs) +
"Stopped building path.")
break
logger.debug("Max ({}): {}".format(node_max.obs, node_max))
lattice_best.append(node_max)
if node_max.is_emitting():
cur_depth += 1
lattice_best = list(reversed(lattice_best))
return lattice_best
def _build_node_path(self, start_idx, unique=True, max_depth=None, last_is_e=False):
"""Build the path from the lattice.
:param start_idx:
:param unique:
:param max_depth:
:param last_is_e: Last matched lattice node should be an emitting state.
In case the matching stops early, the longest path can be in between two observations
and thus be a nonemitting state (which by definition has a lower probability than the
last emitting state). If this argument is set to true, the longer match is preferred.
:return:
"""
node_max = None
node_max_ne = 0
if last_is_e:
for m in self.lattice[start_idx].values_all(): # type:BaseMatching
if not m.stop and (node_max is None or m.logprob > node_max.logprob):
node_max = m
else:
for m in self.lattice[start_idx].values_all(): # type:BaseMatching
if not m.stop and (node_max is None or m.obs_ne > node_max_ne or m.logprob > node_max.logprob):
node_max_ne = m.obs_ne
node_max = m
if node_max is None:
logger.error("Did not find a matching node for path point at index {}".format(start_idx))
return None
self.lattice_best = self._build_matching_path(node_max, max_depth)
node_path = [m.shortkey for m in self.lattice_best]
if unique:
self.node_path = []
prev_node = None
for node in node_path:
if node != prev_node:
self.node_path.append(node)
prev_node = node
else:
self.node_path = node_path
return self.node_path
def increase_max_lattice_width(self, max_lattice_width, unique=False, tqdm=None):
"""Increase the value for max_lattice_width and continue the matching with all
paths that were ignored so far (up to the new max_lattice_width).
This is useful when the matcher is first run with a small max_lattice_width
to be fast, but when the true path is not obvious and excluded from the first
guesses. When the matcher stops early, this method allows to easily expand the
search space.
:param max_lattice_width: New maximal number of paths to consider
:param unique: See match method
:param tqdm: See match method
"""
self.max_lattice_width = max_lattice_width
return self.match(self.path, unique=unique, tqdm=tqdm, expand=True)
def continue_with_distance(self, from_matches=None, k=2, nb_obs=2, max_dist=None):
"""Continue the matcher but ignore edges and allow jumps
to nearby edged.
:param from_matches: Search in the neigborhood of these matches
:param k: If from_matches is not given, the k best matches are used
in the last nb_obs observations since last early_stop_idx
:praram nb_obs: If from_matches is not given, the k best matches are used
in the last nb_obs observations since last early_stop_idx
:param max_dist: Add edges that are maximally max_dist away from the previous
match. If none, self.max_dist * 3 is used.
"""
if from_matches is None:
from_matches = self.best_last_matches(k=k, nb_obs=nb_obs)
self.increase_delayed()
if max_dist is None:
max_dist = self.max_dist * 3
for obs_idx, cur_matches in from_matches.items():
self._match_states(obs_idx, prev_lattice=cur_matches, max_dist=max_dist, inc_delayed=True)
def path_bb(self):
"""Get boundig box of matched path (if it exists, otherwise return None)."""
path = self.path
plat, plon = islice(zip(*path), 2)
lat_min, lat_max = min(plat), max(plat)
lon_min, lon_max = min(plon), max(plon)
bb = lat_min, lon_min, lat_max, lon_max
return bb
def print_lattice(self, file=None, obs_idx=None, obs_ne=0, label_width=None, debug=False):
if debug:
xprint = logger.debug
else:
if file is None:
file = sys.stdout
xprint = lambda arg: print(arg, file=file)
# print("Lattice:", file=file)
if obs_idx is not None:
idxs = [obs_idx]
else:
idxs = range(len(self.lattice))
for idx in idxs:
if len(self.lattice[idx]) > 0:
if label_width is None:
label_width = 0
for m in self.lattice[idx].values(obs_ne):
label_width = max(label_width, len(str(m.label)))
xprint("--- obs {} ---".format(idx))
xprint(self.matching.repr_header(label_width=label_width))
for m in sorted(self.lattice[idx].values(obs_ne), key=lambda t: str(t.label)):
xprint(m.__str__(label_width=label_width))
def lattice_dot(self, file=None, precision=None, render=False):
"""Write the lattice as a Graphviz DOT file.
:param file: File object to print to. Prints to stdout if None.
:param precision: Precision of (log) probabilities.
:param render: Try to render the generated Graphviz file.
"""
if file is None:
file = sys.stdout
if precision is None:
prfmt = ''
else:
prfmt = f'.{precision}f'
print('digraph lattice {', file=file)
print('\trankdir=LR;', file=file)
# Vertices
for idx_ob in range(len(self.lattice)):
col = self.lattice[idx_ob]
for idx_ne in range(len(col)):
ms = col.values(idx_ne)
if len(ms) == 0:
continue
cnames = [(m.obs_ne, m.cname, m.stop, m.delayed) for m in ms]
cnames.sort()
cur_obs_ne = -1
print('\t{\n\t\trank=same; ', file=file)
for obs_ne, cname, stop, delayed in cnames:
if obs_ne != cur_obs_ne:
if cur_obs_ne != -1:
print('\t};\n\t{\n\t\trank=same; ', file=file)
cur_obs_ne = obs_ne
if stop:
options = 'label="{} x",color=gray,fontcolor=gray'.format(cname)
elif delayed > self.expand_now:
options = 'label="{} d{}",color=gray,fontcolor=gray'.format(cname, delayed)
elif self.expand_now != 0:
options = 'label="{} d{}"'.format(cname, delayed)
else:
options = 'label="{} "'.format(cname)
print('\t\t{} [{}];'.format(cname, options), file=file)
print('\t};', file=file)
# Edges
for idx_ob in range(len(self.lattice)):
col = self.lattice[idx_ob]
for idx_ne in range(len(col)):
ms = col.values(idx_ne)
if len(ms) == 0:
continue
for m in ms:
for mp in m.prev:
if m.stop or m.delayed > self.expand_now:
options = ',color=gray,fontcolor=gray'
else:
options = ''
print(f'\t {mp.cname} -> {m.cname} [label="{m.logprob:{prfmt}}"{options}];', file=file)
for mp in m.prev_other:
if m.stop or m.delayed > self.expand_now:
options = ',color=gray,fontcolor=gray'
else:
options = ''
print(f'\t {mp.cname} -> {m.cname} [color=gray,label="{m.logprob:{prfmt}}"{options}];', file=file)
print('}', file=file)
if render and file is not None:
import subprocess as sp
from pathlib import Path
from io import TextIOWrapper
if isinstance(file, Path):
fn = str(file.canonical())
elif isinstance(file, TextIOWrapper):
file.flush()
fn = file.name
else:
fn = str(file)
cmd = ['dot', '-Tpdf', '-O', fn]
logger.debug(' '.join(cmd))
sp.call(cmd)
def print_lattice_stats(self, file=None, verbose=False):
if file is None:
file = sys.stdout
print("Stats lattice", file=file)
print("-------------", file=file)
stats = OrderedDict()
stats["nbr levels"] = len(self.lattice) if self.lattice else "?"
total_nodes = 0
max_nodes = 0
min_nodes = 9999999
if self.lattice:
sizes = []
for idx in range(len(self.lattice)):
level = self.lattice[idx].values(0)
# stats["#nodes[{}]".format(idx)] = len(level)
sizes.append(len(level))
total_nodes += len(level)
if len(level) < min_nodes:
min_nodes = len(level)
if len(level) > max_nodes:
max_nodes = len(level)
stats["nbr lattice"] = total_nodes
if verbose:
stats["nbr lattice[level]"] = ", ".join([str(s) for s in sizes])
stats["avg lattice[level]"] = total_nodes/len(self.lattice)
stats["min lattice[level]"] = min_nodes
stats["max lattice[level]"] = max_nodes
if self.lattice_best and len(self.lattice_best) > 0:
stats["avg obs distance"] = np.mean([m.dist_obs for m in self.lattice_best])
stats["last logprob"] = self.lattice_best[-1].logprob
stats["last length"] = self.lattice_best[-1].length
stats["last norm logprob"] = self.lattice_best[-1].logprob / self.lattice_best[-1].length
if verbose:
stats["best logprob"] = ", ".join(["{:.3f}".format(m.logprob) for m in self.lattice_best])
stats["best norm logprob"] = \
", ".join(["{:.3f}".format(m.logprob/m.length) for i, m in enumerate(self.lattice_best)])
stats["best norm prob"] = \
", ".join(["{:.3f}".format(math.exp(m.logprob/m.length)) for i, m in enumerate(self.lattice_best)])
for key, val in stats.items():
print("{:<24} : {}".format(key, val), file=file)
def node_counts(self):
if self.lattice is None:
return None
counts = defaultdict(lambda: 0)
for level in self.lattice.values():
for m in level.values_all():
counts[m.label] += 1
return counts
def inspect_early_stopping(self):
"""Analyze the lattice and try to find most plausible reason why the
matching stopped early and print to stdout."""
if self.early_stop_idx is None:
print("No early stopping.")
return
col = self.lattice[self.early_stop_idx - 1]
print("The last matched nodes or edges were:")
first_row = True
ignore = set()
for ne_i in range(len(col.o) - 1, -1, -1):
for v in col.o[ne_i].values():
if v.key not in ignore:
if first_row:
print(v.repr_header())
first_row = False
print(v)
ignore.update(r.key for r in v.prev)
def best_last_matches(self, k=1, nb_obs=3):
"""Return the k best last matches.
:param k: Number of best matches to keep for an observation
:param nb_obs: How many last matched observations to consider
"""
import heapq
if self.early_stop_idx is None:
col_idx = len(self.lattice) - 1
else:
col_idx = self.early_stop_idx - 1
hh = []
obs_cnt = 0
while col_idx >= 0 and obs_cnt < nb_obs:
h = []
col = self.lattice[col_idx]
col_oneselected = False
for ne_i in range(len(col.o) - 1, -1, -1):
for v in col.o[ne_i].values():
if v.stop:
continue
if len(h) < k:
heapq.heappush(h, (v.logprob, v))
col_oneselected = True
elif v.logprob > h[0][0]:
heapq.heappop(h)
heapq.heappush(h, (v.logprob, v))
col_oneselected = True
hh.extend(h)
if col_oneselected is False:
print(f'break in {col_idx=}')
break
col_idx -= 1
obs_cnt += 1
result = defaultdict(list)
for m in hh:
m = m[1]
result[m.obs + 1].append(m)
# return [m[1] for m in hh]
return result
def copy_lastinterface(self, nb_interfaces=1):
"""Copy the current matcher and keep the last interface as the start point.
This method allows you to perform incremental matching without keeping the entire
lattice in memory.
You need to run :meth:`match_incremental` on this object to continue from the existing
(partial) lattice. Otherwise, if you use :meth:`match`, it will be overwritten.
Open question, if there is no need to keep track of older lattices, it will probably
be more efficient to clear the older parts of the interface instead of copying the newer
parts.
:param nb_interfaces: Nb of interfaces (columns in lattice) to keep. Default is 1, the last one.
:return: new Matcher object
"""
matcher = self.__class__(self.map, obs_noise=self.obs_noise, max_dist_init=self.max_dist_init,
max_dist=self.max_dist, min_prob_norm=self.min_logprob_norm,
non_emitting_states=self.non_emitting_states,
max_lattice_width=self.max_lattice_width, only_edges=self.only_edges,
obs_noise_ne=self.obs_noise_ne, matching=self.matching,
avoid_goingback=self.avoid_goingback,
non_emitting_length_factor=math.exp(self.ne_length_factor_log))
matcher.lattice = []
matcher.path = []
for int_i in range(len(self.lattice) - nb_interfaces, len(self.lattice)):
matcher.lattice.append(self.lattice[int_i])
matcher.path.append(self.path[int_i])
return matcher
@property
def path_pred(self):
"""The matched path, both nodes and/or edges (depending on your settings)."""
return self.node_path
@property
def path_pred_onlynodes(self):
"""A list with all the nodes (no edges) the matched path passes through."""
return self.get_path(only_nodes=True, allow_jumps=False)
@property
def path_pred_onlynodes_withjumps(self):
"""A list with all the nodes (no edges) the matched path passes through."""
return self.get_path(only_nodes=True, allow_jumps=True)
def path_pred_distance(self):
"""Total distance of the matched path."""
if self.lattice_best is None:
return None
if len(self.lattice_best) == 1:
return 0
dist = 0
m_prev = self.lattice_best[0]
for idx, m in enumerate(self.lattice_best[1:]):
if m_prev.edge_m.label != m.edge_m.label and m_prev.edge_m.l2 == m.edge_m.l1:
# Go over the connection between two edges to compute the distance
cdist = self.map.distance(m_prev.edge_m.pi, m_prev.edge_m.p2)
cdist += self.map.distance(m_prev.edge_m.p2, m.edge_m.pi)
else:
cdist = self.map.distance(m_prev.edge_m.pi, m.edge_m.pi)
dist += cdist
m_prev = m
return dist
def path_distance(self):
"""Total distance of the observations."""
if self.lattice_best is None:
return None
if len(self.lattice_best) == 1:
return 0
dist = 0
m_prev = self.lattice_best[0]
for m in self.lattice_best[1:]:
dist += self.map.distance(m_prev.edge_o.pi, m.edge_o.pi)
m_prev = m
return dist
def path_all_distances(self):
"""Return a list of all distances between observed trace and map.
One entry for each point in the map and point in the trace that are mapped to each other.
In case non-emitting nodes are used, extra entries can be present where a point in the trace
or a point in the map is mapped to a segment.
"""
path = self.lattice_best
dists = [m.dist_obs for m in path]
return dists
================================================
FILE: leuvenmapmatching/matcher/distance.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.matcher.distance
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
import logging
import math
from .base import BaseMatching, BaseMatcher
from ..util.segment import Segment
from ..util.debug import printd
MYPY = False
if MYPY:
from typing import Tuple, Any, Dict
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
class DistanceMatching(BaseMatching):
__slots__ = ['d_s', 'd_o', 'lpe', 'lpt'] # Additional fields
def __init__(self, *args, d_s=0.0, d_o=0.0, lpe=0.0, lpt=0.0, **kwargs):
"""
:param args: Arguments for BaseMatching
:param d_s: Distance between two (interpolated) states
:param d_o: Distance between two (interpolated) observations
:param lpe: Log probability of emission
:param lpt: Log probablity of transition
:param kwargs: Arguments for BaseMatching
"""
super().__init__(*args, **kwargs)
self.d_o: float = d_o
self.d_s: float = d_s
self.lpe: float = lpe
self.lpt: float = lpt
def _update_inner(self, m_other):
# type: (DistanceMatching, DistanceMatching) -> None
super()._update_inner(m_other)
self.d_s = m_other.d_s
self.d_o = m_other.d_o
self.lpe = m_other.lpe
self.lpt = m_other.lpt
@staticmethod
def repr_header(label_width=None, stop=""):
res = BaseMatching.repr_header(label_width)
res += f" {'dt(o)':<6} | {'dt(s)':<6} |"
if logger.isEnabledFor(logging.DEBUG):
res += f" {'lg(Pr-t)':<9} | {'lg(Pr-e)':<9} |"
return res
def __str__(self, label_width=None):
res = super().__str__(label_width)
res += f" {self.d_o:>6.2f} | {self.d_s:>6.2f} |"
if logger.isEnabledFor(logging.DEBUG):
res += f" {self.lpt:>9.2f} | {self.lpe:>9.2f} |"
return res
def __repr__(self):
return self.label
class DistanceMatcher(BaseMatcher):
"""
Map Matching that takes into account the distance between matched locations on the map compared to
the distance between the observations (that are matched to these locations). It thus prefers matched
paths that have a similar distance than the observations.
Inspired on the method presented in:
P. Newson and J. Krumm. Hidden markov map matching through noise and sparseness.
In Proceedings of the 17th ACM SIGSPATIAL international conference on advances
in geographic information systems, pages 336–343. ACM, 2009.
The options available in :class:``BaseMatcher`` are inherited. Additionally, this class
offers:
- Transition probability is lower if the distance between observations and states is different
- Transition probability is lower if the the next match is going back on an edge or to a previous edge
- Transition probability is lower if two neighboring states represent not connected edges
- Skip non-emitting states if distance between states and observations is close to each other
"""
def __init__(self, *args, **kwargs):
"""Create a new object.
:param map_con: Map object to connect to map database
:param obs_noise: Standard deviation of noise
:param obs_noise_ne: Standard deviation of noise for non-emitting states (is set to obs_noise if not given)
:param max_dist_init: Maximum distance from start location (if not given, uses max_dist)
:param max_dist: Maximum distance from path (this is a hard cut, min_prob_norm should be better)
:param min_prob_norm: Minimum normalized probability of observations (ema)
:param non_emitting_states: Allow non-emitting states. A non-emitting state is a state that is
not associated with an observation. Here we assume it can be associated with a location in between
two observations to allow for pruning. It is advised to set min_prob_norm and/or max_dist to avoid
visiting all possible nodes in the graph.
:param non_emitting_length_factor: Reduce the probability of a sequence of non-emitting states the longer it
is. This can be used to prefer shorter paths. This is separate from the transition probabilities because
transition probabilities are averaged for non-emitting states and thus the length is also averaged out.
:param max_lattice_width: Restrict the lattice (or possible candidate states per observation) to this value.
If there are more possible next states, the states with the best likelihood so far are selected.
:param dist_noise: Standard deviation of difference between distance between states and distance
between observatoins. If not given, set to obs_noise
:param dist_noise_ne: If not given, set to dist_noise
:param restrained_ne: Avoid non-emitting states if the distance between states and between
observations is close to each other.
:param avoid_goingback: If true, the probability is lowered for a transition that returns back to a
previous edges or returns to a position on an edge.
:param args: Arguments for BaseMatcher
:param kwargs: Arguments for BaseMatcher
"""
if not kwargs.get("only_edges", True):
logger.warning("The MatcherDistance method only works on edges as states. Nodes have been disabled.")
kwargs["only_edges"] = True
if "matching" not in kwargs:
kwargs["matching"] = DistanceMatching
super().__init__(*args, **kwargs)
self.use_original = kwargs.get('use_original', False)
# if not use_original, the following value for beta gives a prob of 0.5 at dist=x_half:
# beta = np.sqrt(np.power(x_half, 2) / (np.log(2)*2))
self.dist_noise = kwargs.get('dist_noise', self.obs_noise)
self.dist_noise_ne = kwargs.get('dist_noise_ne', self.dist_noise)
self.beta = 2 * self.dist_noise ** 2
self.beta_ne = 2 * self.dist_noise_ne ** 2
self.sigma = 2 * self.obs_noise ** 2
self.sigma_ne = 2 * self.obs_noise_ne ** 2
self.restrained_ne = kwargs.get('restrained_ne', True)
self.restrained_ne_thr = 1.25 # Threshold
self.exact_dt_s = True # Newson and Krumm is 'True'
self.avoid_goingback = kwargs.get('avoid_goingback', True)
self.gobackonedge_factor_log = math.log(0.5)
self.gobacktoedge_factor_log = math.log(0.5)
self.first_farend_penalty = math.log(0.75) # should be > gobacktoedge_factor_log
self.notconnectededges_factor_log = math.log(0.5)
def logprob_trans(self, prev_m, edge_m, edge_o,
is_prev_ne=False, is_next_ne=False):
# type: (DistanceMatcher, DistanceMatching, Segment, Segment, bool, bool) -> Tuple[float, Dict[str, Any]]
"""Transition probability.
The probability is defined with a formula from the exponential family.
:math:`P(dt) = exp(-d_t^2 / (2 * dist_{noise}^2))`
with :math:`d_t = |d_s - d_o|,
d_s = |loc_{prev\_state} - loc_{cur\_state}|,
d_o = |loc_{prev\_obs} - loc_{cur\_obs}|`
This function is more tolerant for low values. The intuition is that values under a certain
distance should all be close to probability 1.0.
Note: We should also smooth the distance between observations to handle outliers better.
:param prev_m: Previous matching / state
:param edge_m: Edge between matchings / states
:param edge_o: Edge between observations
:param is_prev_ne: Is previous state non-emitting
:param is_next_ne: Is the next state non-emitting
:param dist_o: First output of distance_progress
:param dist_m: Second output of distance_progress
:return:
"""
d_z = self.map.distance(prev_m.edge_o.pi, edge_o.pi)
is_same_edge = False
if (prev_m.edge_m.l1 == edge_m.l1 and prev_m.edge_m.l2 == edge_m.l2) or \
(prev_m.edge_m.l1 == edge_m.l2 and prev_m.edge_m.l2 == edge_m.l1):
is_same_edge = True
if ((not self.exact_dt_s) or
is_same_edge or # On same edge
prev_m.edge_m.l2 != edge_m.l1): # Edges are not connected
d_x = self.map.distance(prev_m.edge_m.pi, edge_m.pi)
else:
# Take into account the curvature
d_x = self.map.distance(prev_m.edge_m.pi, prev_m.edge_m.p2) + self.map.distance(prev_m.edge_m.p2, edge_m.pi)
if is_next_ne:
# For non-emitting states, the distances are added
# Otherwise it can map to a sequence of short segments and stay at the same
# observation because the difference is then always small.
d_z += prev_m.d_o
d_x += prev_m.d_s
d_t = abs(d_z - d_x)
# p_dt = 1 / beta * math.exp(-d_t / beta)
if is_prev_ne or is_next_ne:
beta = self.beta_ne
else:
beta = self.beta
logprob = -d_t ** 2 / beta
# Penalties
if prev_m.edge_m.label == edge_m.label:
# Staying in same state
if self.avoid_goingback and edge_m.key == prev_m.edge_m.key and edge_m.ti < prev_m.edge_m.ti:
# Going back on edge (direction is from p1 to p2 of the segment)
logprob += self.gobackonedge_factor_log # Prefer not going back
elif (prev_m.edge_m.l1, prev_m.edge_m.l2) == (edge_m.l2, edge_m.l1):
if self.avoid_goingback:
logprob += self.gobackonedge_factor_log
else:
# Moving states
if prev_m.edge_m.l2 != edge_m.l1:
# We are moving between states that represent edges that are not connected through a node
logprob += self.notconnectededges_factor_log
elif self.avoid_goingback:
# Goin back on state
going_back = False
for m in prev_m.prev:
if edge_m.label == m.edge_m.label:
going_back = True
break
if going_back:
logprob += self.gobacktoedge_factor_log # prefer not going back
props = {
'd_o': d_z,
'd_s': d_x,
'lpt': logprob
}
return logprob, props
def logprob_obs(self, dist, prev_m=None, new_edge_m=None, new_edge_o=None, is_ne=False):
# type: (DistanceMatcher, float, DistanceMatching, Segment, Segment, bool) -> Tuple[float, Dict[str, Any]]
"""Emission probability for emitting states.
Exponential family:
:math:`P(dt) = exp(-d_o^2 / (2 * obs_{noise}^2))`
with :math:`d_o = |loc_{state} - loc_{obs}|`
"""
if is_ne:
sigma = self.sigma_ne
else:
sigma = self.sigma
result = -dist ** 2 / sigma
props = {
'lpe': result
}
return result, props
def _skip_ne_states(self, next_ne_m):
# type: (DistanceMatcher, DistanceMatching) -> bool
# Skip searching for non-emitting states when the distances between nodes
# on the map are similar to the distances between the observation
if not self.restrained_ne:
return False
if next_ne_m.d_s > 0:
factor = (next_ne_m.d_o + next_ne_m.dist_obs) / next_ne_m.d_s
else:
factor = 0
if factor < self.restrained_ne_thr:
logger.debug(f"Skip non-emitting states to {next_ne_m.label}: {factor} < {self.restrained_ne_thr} "
"(observations close enough to each other)")
return True
return False
================================================
FILE: leuvenmapmatching/matcher/newsonkrumm.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.matcher.newsonkrumm
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Methods similar to Newson Krumm 2009 for comparison purposes.
P. Newson and J. Krumm. Hidden markov map matching through noise and sparseness.
In Proceedings of the 17th ACM SIGSPATIAL international conference on advances
in geographic information systems, pages 336–343. ACM, 2009.
:author: Wannes Meert
:copyright: Copyright 2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
from scipy.stats import norm
import math
import logging
from .base import BaseMatching, BaseMatcher
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
class NewsonKrummMatching(BaseMatching):
__slots__ = ['d_s', 'd_o', 'lpe', 'lpt'] # Additional fields
def __init__(self, *args, d_s=1.0, d_o=1.0, lpe=0.0, lpt=0.0, **kwargs):
"""
:param args: Arguments for BaseMatching
:param d_s: Distance between two (interpolated) states
:param d_o: Distance between two (interpolated) observations
:param lpe: Log probability of emission
:param lpt: Log probablity of transition
:param kwargs: Arguments for BaseMatching
"""
super().__init__(*args, **kwargs)
self.d_o: float = d_o
self.d_s: float = d_s
self.lpe: float = lpe
self.lpt: float = lpt
def _update_inner(self, m_other):
# type: (NewsonKrummMatching, NewsonKrummMatching) -> None
super()._update_inner(m_other)
self.d_s = m_other.d_s
self.d_o = m_other.d_o
self.lpe = m_other.lpe
self.lpt = m_other.lpt
@staticmethod
def repr_header(label_width=None, stop=""):
res = BaseMatching.repr_header(label_width)
res += f" {'dt(o)':<6} | {'dt(s)':<6} |"
if logger.isEnabledFor(logging.DEBUG):
res += f" {'lg(Pr-t)':<9} | {'lg(Pr-e)':<9} |"
return res
def __str__(self, label_width=None):
res = super().__str__(label_width)
res += f" {self.d_o:>6.2f} | {self.d_s:>6.2f} |"
if logger.isEnabledFor(logging.DEBUG):
res += f" {self.lpt:>9.2f} | {self.lpe:>9.2f} |"
return res
class NewsonKrummMatcher(BaseMatcher):
"""
Take distance between observations vs states into account. Based on the
method presented in:
P. Newson and J. Krumm. Hidden markov map matching through noise and sparseness.
In Proceedings of the 17th ACM SIGSPATIAL international conference on advances
in geographic information systems, pages 336–343. ACM, 2009.
Two important differences:
* Newson and Krumm use shortest path to handle situations where the distances between
observations are larger than distances between nodes in the graph. The LeuvenMapMatching
toolbox uses non-emitting states to handle this. We thus do not implement the shortest
path algorithm in this class.
* Transition and emission probability are transformed from densities to probababilities by
taking the 1 - CDF instead of the PDF.
Newson and Krumm defaults:
- max_dist = 200 m
- obs_noise = 4.07 m
- beta = 1/6
- only_edges = True
"""
def __init__(self, *args, **kwargs):
"""
:param beta: Default is 1/6
:param beta_ne: Default is beta
:param args: Arguments for BaseMatcher
:param kwargs: Arguments for BaseMatcher
"""
if not kwargs.get("only_edges", True):
logger.warning("The MatcherDistance method only works on edges as states. Nodes have been disabled.")
kwargs["only_edges"] = True
if "matching" not in kwargs:
kwargs["matching"] = NewsonKrummMatching
super().__init__(*args, **kwargs)
# if not use_original, the following value for beta gives a prob of 0.5 at dist=x_half:
# beta = np.sqrt(np.power(x_half, 2) / (np.log(2)*2))
self.beta = kwargs.get('beta', 1/6)
self.beta_ne = kwargs.get('beta_ne', self.beta)
self.obs_noise_dist = norm(scale=self.obs_noise)
self.obs_noise_dist_ne = norm(scale=self.obs_noise_ne)
self.ne_thr = 1.25
def logprob_trans(self, prev_m: NewsonKrummMatching, edge_m, edge_o,
is_prev_ne=False, is_next_ne=False):
"""Transition probability.
Main difference with Newson and Krumm: we know all points are connected thus do not compute the
shortest path but the distance between two points.
Original PDF:
p(dt) = 1 / beta * e^(-dt / beta)
with beta = 1/6
Transformed to probability:
P(dt) = p(d > dt) = e^(-dt / beta)
:param prev_m:
:param edge_m:
:param edge_o:
:param is_prev_ne:
:param is_next_ne:
:return:
"""
d_z = self.map.distance(prev_m.edge_o.pi, edge_o.pi)
if prev_m.edge_m.label == edge_m.label:
d_x = self.map.distance(prev_m.edge_m.pi, edge_m.pi)
else:
d_x = self.map.distance(prev_m.edge_m.pi, prev_m.edge_m.p2) + self.map.distance(prev_m.edge_m.p2, edge_m.pi)
d_t = abs(d_z - d_x)
# p_dt = 1 / beta * math.exp(-d_t / beta)
if is_prev_ne or is_next_ne:
beta = self.beta_ne
else:
beta = self.beta
# icp_dt = math.exp(-d_t / beta)
# try:
# licp_dt = math.log(icp_dt)
# except ValueError:
# licp_dt = float('-inf')
licp_dt = -d_t / beta
props = {
'd_o': d_z,
'd_s': d_x,
'lpt': licp_dt
}
return licp_dt, props
def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne=False):
"""Emission probability for emitting states.
Original pdf:
p(d) = N(0, sigma)
with sigma = 4.07m
Transformed to probability:
P(d) = 2 * (1 - p(d > D)) = 2 * (1 - cdf)
"""
if is_ne:
result = 2 * (1 - self.obs_noise_dist_ne.cdf(dist))
else:
result = 2 * (1 - self.obs_noise_dist.cdf(dist))
try:
result = math.log(result)
except ValueError:
result = -float("inf")
props = {
'lpe': result
}
return result, props
================================================
FILE: leuvenmapmatching/matcher/simple.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.matcher.simple
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2015-2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
import math
from scipy.stats import halfnorm, norm
from .base import BaseMatcher, BaseMatching
from ..util.segment import Segment
class SimpleMatching(BaseMatching):
pass
class SimpleMatcher(BaseMatcher):
def __init__(self, *args, **kwargs):
"""A simple matcher that prefers paths where each matched location is as close as possible to the
observed position.
:param avoid_goingback: Change the transition probability to be lower for the direction the path is coming
from.
:param kwargs: Arguments passed to :class:`BaseMatcher`.
"""
if "matching" not in kwargs:
kwargs['matching'] = SimpleMatching
super().__init__(*args, **kwargs)
self.obs_noise_dist = halfnorm(scale=self.obs_noise)
self.obs_noise_dist_ne = halfnorm(scale=self.obs_noise_ne)
# normalize to max 1 to simulate a prob instead of density
self.obs_noise_logint = math.log(self.obs_noise * math.sqrt(2 * math.pi) / 2)
self.obs_noise_logint_ne = math.log(self.obs_noise_ne * math.sqrt(2 * math.pi) / 2)
# Transition probability is divided (in logprob_trans) by this factor if we move back on the
# current edge.
self.avoid_goingback = kwargs.get('avoid_goingback', True)
self.gobackonedge_factor_log = math.log(0.99)
# Transition probability is divided (in logprob_trans) by this factor if the next state is
# also the previous state, thus if we go back
self.gobacktoedge_factor_log = math.log(0.5)
# Transition probability is divided (in logprob_trans) by this factor if a transition is made
# This is to try to stay on the same node unless there is a good reason
self.transition_factor = math.log(0.9)
def logprob_trans(self, prev_m: BaseMatching, edge_m: Segment, edge_o: Segment,
is_prev_ne=False, is_next_ne=False):
"""Transition probability.
Note: In contrast with a regular HMM, this is not a probability density function, it needs
to be a proper probability (thus values between 0.0 and 1.0).
"""
logprob = 0
if prev_m.edge_m.label == edge_m.label:
# Staying in same state
if self.avoid_goingback and edge_m.key == prev_m.edge_m.key and edge_m.ti < prev_m.edge_m.ti:
# Going back on edge
logprob += self.gobackonedge_factor_log # prefer not going back
else:
# Moving states
logprob += self.transition_factor
if self.avoid_goingback:
# Goin back on state
going_back = False
for m in prev_m.prev:
if edge_m.label == m.edge_m.label:
going_back = True
break
if going_back:
logprob += self.gobacktoedge_factor_log # prefer not going back
return logprob, {} # All probabilities are 1 (thus technically not a distribution)
def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne=False):
"""Emission probability.
Note: In contrast with a regular HMM, this is not a probability density function, it needs
to be a proper probability (thus values between 0.0 and 1.0).
"""
if is_ne:
result = self.obs_noise_dist_ne.logpdf(dist) + self.obs_noise_logint_ne
else:
result = self.obs_noise_dist.logpdf(dist) + self.obs_noise_logint
# print("logprob_obs: {} -> {:.5f} = {:.5f}".format(dist, result, math.exp(result)))
return result, {}
================================================
FILE: leuvenmapmatching/util/__init__.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.util
~~~~~~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2015-2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
# No automatic loading to avoid dependency on packages such as nvector and gpxpy if not used.
def approx_equal(a, b, rtol=0.0, atol=1e-08):
return abs(a - b) <= (atol + rtol * abs(b))
def approx_leq(a, b, rtol=0.0, atol=1e-08):
return (a - b) <= (atol + rtol * abs(b))
================================================
FILE: leuvenmapmatching/util/debug.py
================================================
import logging
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
def printd(*args, **kwargs):
"""Print to debug output."""
logger.debug(*args, **kwargs)
================================================
FILE: leuvenmapmatching/util/dist_euclidean.py
================================================
# encoding: utf-8
"""
leuvenmapmatching.util.dist_euclidean
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:author: Wannes Meert
:copyright: Copyright 2015-2018 DTAI, KU Leuven and Sirris.
:license: Apache License, Version 2.0, see LICENSE for details.
"""
import logging
import math
import numpy as np
logger = logging.getLogger("be.kuleuven.cs.dtai.mapmatching")
def distance(p1, p2):
result = math.sqrt((p1[0] - p2[0]) ** 2 + (p1[1] - p2[1]) ** 2)
# print("distance({}, {}) -> {}".format(p1, p2, result))
return result
def distance_point_to_segment(p, s1, s2, delta=0.0):
p_int, ti = project(s1, s2, p, delta=delta)
return distance(p_int, p), p_int, ti
# l1a = np.array(s1)
# l2a = np.array(s2)
# pa = np.array(p)
# return np.linalg.norm(np.cross(l2a - l1a, l1a - pa)) / np.linalg.norm(l2a - l1a)
def distance_segment_to_segment(f1, f2, t1, t2):
"""Distance between segments..
:param f1: From
:param f2:
:param t1: To
:param t2:
:return: (distance, proj on f, proj on t, rel pos on f, rel pos on t)
"""
x1, y1 = f1
x2, y2 = f2
x3, y3 = t1
x4, y4 = t2
n = ((y4 - y3) * (x2 - x1) - (x4 - x3) * (y2 - y1))
if np.allclose([n], [0], rtol=0):
# parallel
is_parallel = True
n = 0.0001 # TODO: simulates a point far away
else:
is_parallel = False
u_f = ((x4 - x3) * (y1 - y3) - (y4 - y3) * (x1 - x3)) / n
u_t = ((x2 - x1) * (y1 - y3) - (y2 - y1) * (x1 - x3)) / n
xi = x1 + u_f * (x2 - x1)
yi = y1 + u_f * (y2 - y1)
changed_f = False
changed_t = False
if u_t > 1:
u_t = 1
changed_t = True
elif u_t < 0:
u_t = 0
changed_t = True
if u_f > 1:
u_f = 1
changed_f = True
elif u_f < 0:
u_f = 0
changed_f = True
if not changed_t and not changed_f:
return 0, (xi, yi), (xi, yi), u_f, u_t
xf = x1 + u_f
gitextract_6vcq1ios/
├── .gitignore
├── .readthedocs.yaml
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.md
├── docs/
│ ├── Makefile
│ ├── classes/
│ │ ├── map/
│ │ │ ├── BaseMap.rst
│ │ │ ├── InMemMap.rst
│ │ │ └── SqliteMap.rst
│ │ ├── matcher/
│ │ │ ├── BaseMatcher.rst
│ │ │ ├── BaseMatching.rst
│ │ │ ├── DistanceMatcher.rst
│ │ │ └── SimpleMatcher.rst
│ │ ├── overview.rst
│ │ └── util/
│ │ └── Segment.rst
│ ├── conf.py
│ ├── index.rst
│ ├── make.bat
│ ├── requirements.txt
│ └── usage/
│ ├── customdistributions.rst
│ ├── debug.rst
│ ├── incremental.rst
│ ├── installation.rst
│ ├── introduction.rst
│ ├── latitudelongitude.rst
│ ├── openstreetmap.rst
│ └── visualisation.rst
├── leuvenmapmatching/
│ ├── __init__.py
│ ├── map/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── inmem.py
│ │ └── sqlite.py
│ ├── matcher/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── distance.py
│ │ ├── newsonkrumm.py
│ │ └── simple.py
│ ├── util/
│ │ ├── __init__.py
│ │ ├── debug.py
│ │ ├── dist_euclidean.py
│ │ ├── dist_latlon.py
│ │ ├── dist_latlon_nvector.py
│ │ ├── evaluation.py
│ │ ├── gpx.py
│ │ ├── kalman.py
│ │ ├── openstreetmap.py
│ │ ├── projections.py
│ │ └── segment.py
│ └── visualization.py
├── setup.cfg
├── setup.py
└── tests/
├── examples/
│ ├── example_1_simple.py
│ └── example_using_osmnx_and_geopandas.py
├── rsrc/
│ ├── bug2/
│ │ └── readme.md
│ ├── newson_krumm_2009/
│ │ └── readme.md
│ └── path_latlon/
│ ├── readme.md
│ ├── route.gpx
│ └── route2.gpx
├── test_bugs.py
├── test_conversion.py
├── test_examples.py
├── test_newsonkrumm2009.py
├── test_nonemitting.py
├── test_nonemitting_circle.py
├── test_parallelroads.py
├── test_path.py
├── test_path_latlon.py
└── test_path_onlyedges.py
SYMBOL INDEX (320 symbols across 30 files)
FILE: leuvenmapmatching/map/base.py
class BaseMap (line 42) | class BaseMap(object):
method __init__ (line 45) | def __init__(self, name, use_latlon=True):
method use_latlon (line 56) | def use_latlon(self):
method use_latlon (line 60) | def use_latlon(self, value):
method bb (line 73) | def bb(self):
method labels (line 80) | def labels(self):
method size (line 84) | def size(self):
method node_coordinates (line 88) | def node_coordinates(self, node_key):
method edges_closeto (line 92) | def edges_closeto(self, loc, max_dist=None, max_elmt=None):
method nodes_closeto (line 103) | def nodes_closeto(self, loc, max_dist=None, max_elmt=None):
method nodes_nbrto (line 114) | def nodes_nbrto(self, node):
method edges_nbrto (line 123) | def edges_nbrto(self, edge):
method all_nodes (line 140) | def all_nodes(self, bb=None):
method all_edges (line 147) | def all_edges(self, bb=None):
FILE: leuvenmapmatching/map/inmem.py
class InMemMap (line 46) | class InMemMap(BaseMap):
method __init__ (line 47) | def __init__(self, name, use_latlon=True, use_rtree=False, index_edges...
method vertex_label_to_int (line 103) | def vertex_label_to_int(self, label, create=False):
method vertices_labels_to_int (line 117) | def vertices_labels_to_int(self):
method serialize (line 125) | def serialize(self):
method deserialize (line 142) | def deserialize(cls, data):
method dump (line 152) | def dump(self):
method from_pickle (line 171) | def from_pickle(cls, filename):
method bb (line 179) | def bb(self):
method labels (line 192) | def labels(self):
method size (line 196) | def size(self):
method node_coordinates (line 199) | def node_coordinates(self, node_key):
method add_node (line 207) | def add_node(self, node, loc):
method del_node (line 223) | def del_node(self, node):
method add_edge (line 232) | def add_edge(self, node_a, node_b):
method _items_in_bb (line 254) | def _items_in_bb(self, bb):
method all_edges (line 266) | def all_edges(self, bb=None):
method all_nodes (line 287) | def all_nodes(self, bb=None):
method purge (line 301) | def purge(self):
method rtree_size (line 318) | def rtree_size(self):
method rtree_fn (line 326) | def rtree_fn(self):
method setup_index (line 332) | def setup_index(self, force=False, deserializing=False):
method fill_index (line 388) | def fill_index(self):
method to_xy (line 397) | def to_xy(self, name=None, use_rtree=None):
method latlon2xy (line 419) | def latlon2xy(self, lat, lon):
method latlon2yx (line 423) | def latlon2yx(self, lat, lon):
method xy2latlon (line 427) | def xy2latlon(self, x, y):
method yx2latlon (line 431) | def yx2latlon(self, y, x):
method nodes_closeto (line 435) | def nodes_closeto(self, loc, max_dist=None, max_elmt=None):
method edges_closeto (line 472) | def edges_closeto(self, loc, max_dist=None, max_elmt=None):
method nodes_nbrto (line 517) | def nodes_nbrto(self, node):
method edges_nbrto (line 531) | def edges_nbrto(self, edge):
method find_duplicates (line 547) | def find_duplicates(self, func=None):
method connect_parallelroads (line 560) | def connect_parallelroads(self, dist=0.5, bb=None):
method print_stats (line 584) | def print_stats(self):
method __str__ (line 588) | def __str__(self):
FILE: leuvenmapmatching/map/sqlite.py
class SqliteMap (line 43) | class SqliteMap(BaseMap):
method __init__ (line 44) | def __init__(self, name, use_latlon=True,
method read_properties (line 104) | def read_properties(self):
method save_properties (line 110) | def save_properties(self):
method create_db (line 120) | def create_db(self):
method from_file (line 179) | def from_file(cls, filename):
method bb (line 185) | def bb(self):
method labels (line 195) | def labels(self):
method size (line 202) | def size(self):
method node_coordinates (line 208) | def node_coordinates(self, node_key):
method add_node (line 221) | def add_node(self, node, loc, ignore_doubles=False, no_index=False, no...
method reindex_nodes (line 251) | def reindex_nodes(self):
method add_nodes (line 265) | def add_nodes(self, nodes):
method del_node (line 287) | def del_node(self, node):
method add_edge (line 290) | def add_edge(self, node_a, node_b, loc_a=None, loc_b=None, speed=None,...
method add_edges (line 324) | def add_edges(self, edges, no_index=False):
method reindex_edges (line 346) | def reindex_edges(self):
method all_edges (line 370) | def all_edges(self, bb=None):
method all_nodes (line 394) | def all_nodes(self, bb=None):
method purge (line 415) | def purge(self):
method to_xy (line 418) | def to_xy(self, name=None):
method latlon2xy (line 439) | def latlon2xy(self, lat, lon):
method latlon2yx (line 443) | def latlon2yx(self, lat, lon):
method xy2latlon (line 447) | def xy2latlon(self, x, y):
method yx2latlon (line 451) | def yx2latlon(self, y, x):
method nodes_closeto (line 455) | def nodes_closeto(self, loc, max_dist=None, max_elmt=None):
method edges_closeto (line 483) | def edges_closeto(self, loc, max_dist=None, max_elmt=None):
method nodes_nbrto (line 513) | def nodes_nbrto(self, node):
method edges_nbrto (line 523) | def edges_nbrto(self, edge):
method find_duplicates (line 543) | def find_duplicates(self, func=None):
method connect_parallelroads (line 558) | def connect_parallelroads(self, dist=0.5, bb=None):
method nodes_to_paths (line 581) | def nodes_to_paths(self, nodes, ignore_nopath=True):
method path_dist (line 595) | def path_dist(self, path):
method print_stats (line 606) | def print_stats(self):
method __str__ (line 610) | def __str__(self):
FILE: leuvenmapmatching/matcher/base.py
class BaseMatching (line 37) | class BaseMatching(object):
method __init__ (line 44) | def __init__(self, matcher: 'BaseMatcher', edge_m: Segment, edge_o: Se...
method prune_value (line 86) | def prune_value(self):
method next (line 91) | def next(self, edge_m: Segment, edge_o: Segment, obs: int = 0, obs_ne:...
method first (line 176) | def first(cls, logprob_init, edge_m, edge_o, matcher, dist_obs):
method update (line 189) | def update(self, m_next):
method _update_inner (line 217) | def _update_inner(self, m_other: 'BaseMatching'):
method is_nonemitting (line 233) | def is_nonemitting(self):
method is_emitting (line 236) | def is_emitting(self):
method last_emitting_logprob (line 239) | def last_emitting_logprob(self):
method __str__ (line 247) | def __str__(self, label_width=None):
method __repr__ (line 261) | def __repr__(self):
method repr_header (line 265) | def repr_header(label_width=None, stop=""):
method repr_static (line 273) | def repr_static(fields, label_width=None):
method label (line 284) | def label(self):
method cname (line 291) | def cname(self):
method key (line 298) | def key(self):
method shortkey (line 308) | def shortkey(self):
method nodes (line 316) | def nodes(self):
method __hash__ (line 322) | def __hash__(self):
method __lt__ (line 325) | def __lt__(self, o):
method __le__ (line 328) | def __le__(self, o):
method __eq__ (line 331) | def __eq__(self, o):
method __ne__ (line 334) | def __ne__(self, o):
method __ge__ (line 337) | def __ge__(self, o):
method __gt__ (line 340) | def __gt__(self, o):
class LatticeColumn (line 344) | class LatticeColumn:
method __init__ (line 346) | def __init__(self, obs_idx):
method __contains__ (line 351) | def __contains__(self, item):
method __len__ (line 357) | def __len__(self):
method set_delayed (line 360) | def set_delayed(self, delayed):
method dict (line 366) | def dict(self, obs_ne=None):
method values_all (line 373) | def values_all(self):
method values (line 380) | def values(self, obs_ne=None):
method upsert (line 387) | def upsert(self, matching):
method prune (line 401) | def prune(self, obs_ne, max_lattice_width, expand_upto, prune_thr=None):
class BaseMatcher (line 447) | class BaseMatcher:
method __init__ (line 449) | def __init__(self, map_con, obs_noise=1, max_dist_init=None, max_dist=...
method logprob_trans (line 523) | def logprob_trans(self, prev_m, edge_m, edge_o,
method logprob_obs (line 535) | def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne=False):
method match_gpx (line 545) | def match_gpx(self, gpx_file, unique=True):
method do_stop (line 551) | def do_stop(self, logprob_norm, dist, logprob_trans, logprob_obs):
method _insert (line 561) | def _insert(self, m_next):
method match (line 564) | def match(self, path, unique=False, tqdm=None, expand=False):
method _skip_ne_states (line 667) | def _skip_ne_states(self, prev_m):
method _create_start_nodes (line 671) | def _create_start_nodes(self, use_edges=True):
method increase_delayed (line 731) | def increase_delayed(self, expand_from=None):
method _match_states (line 740) | def _match_states(self, obs_idx, prev_lattice=None, max_dist=None, inc...
method _match_non_emitting_states (line 861) | def _match_non_emitting_states(self, obs_idx, expand=False):
method _node_in_prev_ne (line 908) | def _node_in_prev_ne(self, m_next, label):
method _insert_tmp (line 931) | def _insert_tmp(m_next, lattice):
method _match_non_emitting_states_inner (line 938) | def _match_non_emitting_states_inner(self, cur_lattice, obs_idx, obs, ...
method _match_non_emitting_states_end (line 1051) | def _match_non_emitting_states_end(self, cur_lattice, obs_idx, obs_next,
method get_matching (line 1143) | def get_matching(self, identifier=None):
method get_matching_path (line 1187) | def get_matching_path(self, start_m):
method get_node_path (line 1192) | def get_node_path(self, start_m, only_nodes=False):
method get_path (line 1200) | def get_path(self, only_nodes=True, allow_jumps=False, only_closest=Tr...
method node_path_to_only_nodes (line 1213) | def node_path_to_only_nodes(self, path, allow_jumps=False):
method _build_matching_path (line 1259) | def _build_matching_path(self, start_m, max_depth=None):
method _build_node_path (line 1289) | def _build_node_path(self, start_idx, unique=True, max_depth=None, las...
method increase_max_lattice_width (line 1330) | def increase_max_lattice_width(self, max_lattice_width, unique=False, ...
method continue_with_distance (line 1346) | def continue_with_distance(self, from_matches=None, k=2, nb_obs=2, max...
method path_bb (line 1366) | def path_bb(self):
method print_lattice (line 1375) | def print_lattice(self, file=None, obs_idx=None, obs_ne=0, label_width...
method lattice_dot (line 1398) | def lattice_dot(self, file=None, precision=None, render=False):
method print_lattice_stats (line 1475) | def print_lattice_stats(self, file=None, verbose=False):
method node_counts (line 1516) | def node_counts(self):
method inspect_early_stopping (line 1525) | def inspect_early_stopping(self):
method best_last_matches (line 1544) | def best_last_matches(self, k=1, nb_obs=3):
method copy_lastinterface (line 1585) | def copy_lastinterface(self, nb_interfaces=1):
method path_pred (line 1616) | def path_pred(self):
method path_pred_onlynodes (line 1621) | def path_pred_onlynodes(self):
method path_pred_onlynodes_withjumps (line 1626) | def path_pred_onlynodes_withjumps(self):
method path_pred_distance (line 1630) | def path_pred_distance(self):
method path_distance (line 1649) | def path_distance(self):
method path_all_distances (line 1662) | def path_all_distances(self):
FILE: leuvenmapmatching/matcher/distance.py
class DistanceMatching (line 25) | class DistanceMatching(BaseMatching):
method __init__ (line 28) | def __init__(self, *args, d_s=0.0, d_o=0.0, lpe=0.0, lpt=0.0, **kwargs):
method _update_inner (line 44) | def _update_inner(self, m_other):
method repr_header (line 53) | def repr_header(label_width=None, stop=""):
method __str__ (line 60) | def __str__(self, label_width=None):
method __repr__ (line 67) | def __repr__(self):
class DistanceMatcher (line 71) | class DistanceMatcher(BaseMatcher):
method __init__ (line 92) | def __init__(self, *args, **kwargs):
method logprob_trans (line 152) | def logprob_trans(self, prev_m, edge_m, edge_o,
method logprob_obs (line 237) | def logprob_obs(self, dist, prev_m=None, new_edge_m=None, new_edge_o=N...
method _skip_ne_states (line 257) | def _skip_ne_states(self, next_ne_m):
FILE: leuvenmapmatching/matcher/newsonkrumm.py
class NewsonKrummMatching (line 27) | class NewsonKrummMatching(BaseMatching):
method __init__ (line 30) | def __init__(self, *args, d_s=1.0, d_o=1.0, lpe=0.0, lpt=0.0, **kwargs):
method _update_inner (line 46) | def _update_inner(self, m_other):
method repr_header (line 55) | def repr_header(label_width=None, stop=""):
method __str__ (line 62) | def __str__(self, label_width=None):
class NewsonKrummMatcher (line 70) | class NewsonKrummMatcher(BaseMatcher):
method __init__ (line 97) | def __init__(self, *args, **kwargs):
method logprob_trans (line 122) | def logprob_trans(self, prev_m: NewsonKrummMatching, edge_m, edge_o,
method logprob_obs (line 167) | def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne=False):
FILE: leuvenmapmatching/matcher/simple.py
class SimpleMatching (line 17) | class SimpleMatching(BaseMatching):
class SimpleMatcher (line 21) | class SimpleMatcher(BaseMatcher):
method __init__ (line 23) | def __init__(self, *args, **kwargs):
method logprob_trans (line 52) | def logprob_trans(self, prev_m: BaseMatching, edge_m: Segment, edge_o:...
method logprob_obs (line 79) | def logprob_obs(self, dist, prev_m, new_edge_m, new_edge_o, is_ne=False):
FILE: leuvenmapmatching/util/__init__.py
function approx_equal (line 14) | def approx_equal(a, b, rtol=0.0, atol=1e-08):
function approx_leq (line 18) | def approx_leq(a, b, rtol=0.0, atol=1e-08):
FILE: leuvenmapmatching/util/debug.py
function printd (line 7) | def printd(*args, **kwargs):
FILE: leuvenmapmatching/util/dist_euclidean.py
function distance (line 18) | def distance(p1, p2):
function distance_point_to_segment (line 24) | def distance_point_to_segment(p, s1, s2, delta=0.0):
function distance_segment_to_segment (line 33) | def distance_segment_to_segment(f1, f2, t1, t2):
function project (line 97) | def project(s1, s2, p, delta=0.0):
function interpolate_path (line 114) | def interpolate_path(path, dd):
function box_around_point (line 137) | def box_around_point(p, dist):
function lines_parallel (line 144) | def lines_parallel(la, lb, lc, ld, d=None):
FILE: leuvenmapmatching/util/dist_latlon.py
function distance (line 25) | def distance(p1, p2):
function distance_point_to_segment (line 40) | def distance_point_to_segment(p, s1, s2, delta=0.0, constrain=True):
function distance_segment_to_segment (line 98) | def distance_segment_to_segment(f1, f2, t1, t2):
function project (line 143) | def project(s1, s2, p, delta=0.0):
function box_around_point (line 148) | def box_around_point(p, dist):
function interpolate_path (line 160) | def interpolate_path(path, dd):
function bearing_radians (line 186) | def bearing_radians(lat1, lon1, lat2, lon2):
function distance_haversine_radians (line 194) | def distance_haversine_radians(lat1, lon1, lat2, lon2, radius=earth_radi...
function destination_radians (line 204) | def destination_radians(lat1, lon1, bearing, dist):
function lines_parallel (line 211) | def lines_parallel(f1, f2, t1, t2, d=None):
FILE: leuvenmapmatching/util/dist_latlon_nvector.py
function distance (line 22) | def distance(p1, p2):
function distance_gp (line 36) | def distance_gp(p1, p2):
function distance_point_to_segment (line 41) | def distance_point_to_segment(p, s1, s2, delta=0.0):
function distance_segment_to_segment (line 60) | def distance_segment_to_segment(f1, f2, t1, t2):
function project (line 92) | def project(s1, s2, p, delta=0.0):
function _project_nvector (line 100) | def _project_nvector(s1, s2, p, delta=0.0):
function _cross_track_point (line 111) | def _cross_track_point(path, point):
function interpolate_path (line 136) | def interpolate_path(path, dd):
FILE: leuvenmapmatching/util/evaluation.py
function route_mismatch_factor (line 25) | def route_mismatch_factor(map_con, path_pred, path_grnd, window=None, di...
FILE: leuvenmapmatching/util/gpx.py
function gpx_to_path (line 21) | def gpx_to_path(gpx_file):
function path_to_gpx (line 37) | def path_to_gpx(path, filename=None):
FILE: leuvenmapmatching/util/kalman.py
function smooth_path (line 19) | def smooth_path(path, dt=1, obs_noise=1e-4, loc_var=1e-4, vel_var=1e-6, ...
FILE: leuvenmapmatching/util/openstreetmap.py
function locations_to_map (line 19) | def locations_to_map(locations, map_con, filename=None):
function bb_to_map (line 27) | def bb_to_map(bb, map_con, filename=None):
function file_to_map (line 54) | def file_to_map(filename, map_con):
function download_map_xml (line 70) | def download_map_xml(fn, bbox, force=False, verbose=False):
function create_map_from_xml (line 101) | def create_map_from_xml(fn, include_footways=False, include_parking=False,
FILE: leuvenmapmatching/util/projections.py
function latlon2equirectangular (line 18) | def latlon2equirectangular(lat, lon, phi_er=0, lambda_er=0):
function equirectangular2latlon (line 32) | def equirectangular2latlon(y, x, phi_er=0, lambda_er=0):
function latlon2grs80 (line 44) | def latlon2grs80(coordinates, lon_0=0.0, lat_ts=0.0, y_0=0, x_0=0.0, zon...
FILE: leuvenmapmatching/util/segment.py
class Segment (line 16) | class Segment(object):
method __init__ (line 20) | def __init__(self, l1, p1, l2=None, p2=None, pi=None, ti=None):
method label (line 39) | def label(self):
method rlabel (line 45) | def rlabel(self):
method key (line 51) | def key(self):
method pi (line 57) | def pi(self):
method pi (line 63) | def pi(self, value):
method ti (line 70) | def ti(self):
method ti (line 76) | def ti(self, value):
method is_point (line 79) | def is_point(self):
method last_point (line 82) | def last_point(self):
method loc_to_str (line 87) | def loc_to_str(self):
method __str__ (line 94) | def __str__(self):
method __repr__ (line 101) | def __repr__(self):
FILE: leuvenmapmatching/visualization.py
function plot_map (line 31) | def plot_map(map_con, path=None, nodes=None, counts=None, ax=None, use_o...
function plot_lattice (line 316) | def plot_lattice(ax, to_pixels, matcher):
function plot_obs_noise_dist (line 342) | def plot_obs_noise_dist(obs_fn, obs_noise, min_dist=0, max_dist=10):
FILE: tests/examples/example_using_osmnx_and_geopandas.py
function run (line 15) | def run():
FILE: tests/test_bugs.py
function test_bug1 (line 32) | def test_bug1():
function test_bug2 (line 71) | def test_bug2():
FILE: tests/test_conversion.py
function test_path_to_gpx (line 20) | def test_path_to_gpx():
function test_grs80 (line 31) | def test_grs80():
function test_distance1 (line 43) | def test_distance1():
function test_distance2 (line 50) | def test_distance2():
function test_bearing1 (line 59) | def test_bearing1():
function test_destination1 (line 68) | def test_destination1():
function test_distance_segment_to_segment1 (line 78) | def test_distance_segment_to_segment1():
function test_distance_segment_to_segment2 (line 93) | def test_distance_segment_to_segment2():
function test_distance_segment_to_segment3 (line 108) | def test_distance_segment_to_segment3():
function test_distance_segment_to_segment4 (line 123) | def test_distance_segment_to_segment4():
function test_distance_point_to_segment1 (line 138) | def test_distance_point_to_segment1():
function plot_distance_point_to_segment_latlon (line 171) | def plot_distance_point_to_segment_latlon(f, t1, t2, pt, fn):
function plot_distance_segment_to_segment_latlon (line 192) | def plot_distance_segment_to_segment_latlon(f1, f2, t1, t2, pf, pt, fn):
function plot_distance_segment_to_segment_euc (line 215) | def plot_distance_segment_to_segment_euc(f1, f2, t1, t2, pf, pt, fn):
FILE: tests/test_examples.py
function test_examples (line 27) | def test_examples():
function importrun_file (line 33) | def importrun_file(fn, cmp_with_previous=False):
function execute_file (line 41) | def execute_file(fn, cmp_with_previous=False):
FILE: tests/test_newsonkrumm2009.py
function read_gps (line 60) | def read_gps(route_fn):
function read_paths (line 76) | def read_paths(paths_fn):
function parse_linestring (line 90) | def parse_linestring(line):
function read_map (line 100) | def read_map(map_fn):
function correct_map (line 163) | def correct_map(mmap):
function load_data (line 176) | def load_data():
function test_route_slice1 (line 238) | def test_route_slice1():
function test_bug1 (line 258) | def test_bug1():
function test_route (line 299) | def test_route():
function test_bug2 (line 386) | def test_bug2():
FILE: tests/test_nonemitting.py
function setup_map (line 34) | def setup_map():
function visualize_map (line 53) | def visualize_map(pathnb=1):
function test_path1 (line 64) | def test_path1():
function test_path1_inc (line 85) | def test_path1_inc():
function test_path1_dist (line 123) | def test_path1_dist():
function test_path2 (line 146) | def test_path2():
function test_path2_dist (line 167) | def test_path2_dist():
function test_path2_incremental (line 186) | def test_path2_incremental():
function test_path_duplicate (line 207) | def test_path_duplicate():
function test_path3_many_obs (line 244) | def test_path3_many_obs():
function test_path3_few_obs_en (line 274) | def test_path3_few_obs_en():
function test_path3_few_obs_e (line 301) | def test_path3_few_obs_e():
function test_path3_dist (line 328) | def test_path3_dist():
FILE: tests/test_nonemitting_circle.py
function setup_map (line 30) | def setup_map(disconnect=True):
function visualize_map (line 65) | def visualize_map():
function visualize_path (line 84) | def visualize_path(matcher, mapdb, name="test"):
function test_path1 (line 97) | def test_path1():
function test_path1_dist (line 108) | def test_path1_dist():
function test_path2 (line 119) | def test_path2():
function test_path2_dist (line 138) | def test_path2_dist():
FILE: tests/test_parallelroads.py
function create_map1 (line 25) | def create_map1():
function create_path1 (line 52) | def create_path1():
function test_parallel (line 56) | def test_parallel():
function test_bb1 (line 61) | def test_bb1():
function test_merge1 (line 97) | def test_merge1():
function test_path1 (line 108) | def test_path1():
FILE: tests/test_path.py
function test_path1 (line 26) | def test_path1():
function test_path1_dist (line 57) | def test_path1_dist():
function test_path2 (line 85) | def test_path2():
function test_path2_inc (line 121) | def test_path2_inc():
function test_path2_dist (line 176) | def test_path2_dist():
function test_path_outlier (line 208) | def test_path_outlier():
function test_path_outlier2 (line 246) | def test_path_outlier2():
function test_path_outlier_dist (line 280) | def test_path_outlier_dist():
function test_path3 (line 314) | def test_path3():
function test_path3_dist (line 339) | def test_path3_dist():
function test_path4_dist_inc (line 360) | def test_path4_dist_inc():
function test_path4_dist_inc_missing (line 403) | def test_path4_dist_inc_missing():
FILE: tests/test_path_latlon.py
function prepare_files (line 37) | def prepare_files(verbose=False, force=False, download_from_osm=False):
function test_path1 (line 59) | def test_path1(use_rtree=False):
function test_path1_serialization (line 85) | def test_path1_serialization(use_rtree=False):
function test_path1_full (line 118) | def test_path1_full():
function test_path2_proj (line 137) | def test_path2_proj():
function test_path2 (line 166) | def test_path2():
function test_path3 (line 193) | def test_path3():
FILE: tests/test_path_onlyedges.py
function test_path1 (line 24) | def test_path1():
function test_path3 (line 55) | def test_path3():
Condensed preview — 69 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (378K chars).
[
{
"path": ".gitignore",
"chars": 216,
"preview": ".cache\n.eggs\n.idea\ndist\n*.egg-info\nvenv*\nREADME\n.ipynb_checkpoints\n*.zip\n*.gv\n*.gv.pdf\n*.xml\ntests/route.gpx\nexamples/Le"
},
{
"path": ".readthedocs.yaml",
"chars": 167,
"preview": "version: 2\n\nbuild:\n os: ubuntu-22.04\n tools:\n python: \"3.11\"\n\nsphinx:\n configuration: docs/conf.py\n\npython:\n ins"
},
{
"path": "LICENSE",
"chars": 2719,
"preview": "Leuven.MapMatching\n------------------\n\nCopyright 2015-2018 KU Leuven, DTAI Research Group\nCopyright 2017-2018 Sirris\n\nLi"
},
{
"path": "MANIFEST.in",
"chars": 33,
"preview": "include README.md\ninclude LICENSE"
},
{
"path": "Makefile",
"chars": 972,
"preview": "\n.PHONY: test3\ntest3:\n\t@#export PYTHONPATH=.;./venv/bin/py.test --ignore=venv -vv\n\tpython3 setup.py test\n\n.PHONY: test2\n"
},
{
"path": "README.md",
"chars": 2350,
"preview": "# Leuven.MapMatching\n\n[](https://pypi.org/project/le"
},
{
"path": "docs/Makefile",
"chars": 613,
"preview": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHI"
},
{
"path": "docs/classes/map/BaseMap.rst",
"chars": 81,
"preview": "BaseMap\n=======\n\n\n.. autoclass:: leuvenmapmatching.map.base.BaseMap\n :members:\n"
},
{
"path": "docs/classes/map/InMemMap.rst",
"chars": 85,
"preview": "InMemMap\n========\n\n\n.. autoclass:: leuvenmapmatching.map.inmem.InMemMap\n :members:\n"
},
{
"path": "docs/classes/map/SqliteMap.rst",
"chars": 89,
"preview": "SqliteMap\n=========\n\n\n.. autoclass:: leuvenmapmatching.map.sqlite.SqliteMap\n :members:\n"
},
{
"path": "docs/classes/matcher/BaseMatcher.rst",
"chars": 269,
"preview": "BaseMatcher\n===========\n\nThis a generic base class to be used by matchers. This class itself\ndoes not implement a workin"
},
{
"path": "docs/classes/matcher/BaseMatching.rst",
"chars": 100,
"preview": "BaseMatching\n============\n\n\n.. autoclass:: leuvenmapmatching.matcher.base.BaseMatching\n :members:\n"
},
{
"path": "docs/classes/matcher/DistanceMatcher.rst",
"chars": 113,
"preview": "DistanceMatcher\n===============\n\n\n.. autoclass:: leuvenmapmatching.matcher.distance.DistanceMatcher\n :members:\n"
},
{
"path": "docs/classes/matcher/SimpleMatcher.rst",
"chars": 105,
"preview": "SimpleMatcher\n=============\n\n\n.. autoclass:: leuvenmapmatching.matcher.simple.SimpleMatcher\n :members:\n"
},
{
"path": "docs/classes/overview.rst",
"chars": 343,
"preview": "\n\nmatcher\n~~~~~~~\n\n.. toctree::\n :caption: Matcher\n\n matcher/BaseMatcher\n matcher/SimpleMatcher\n matcher/Distanc"
},
{
"path": "docs/classes/util/Segment.rst",
"chars": 85,
"preview": "Segment\n=======\n\n\n.. autoclass:: leuvenmapmatching.util.segment.Segment\n :members:\n"
},
{
"path": "docs/conf.py",
"chars": 4940,
"preview": "#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n#\n# Leuven.MapMatching documentation build configuration file, created by"
},
{
"path": "docs/index.rst",
"chars": 1292,
"preview": ".. Leuven.MapMatching documentation master file, created by\n sphinx-quickstart on Sat Apr 14 23:24:31 2018.\n You can"
},
{
"path": "docs/make.bat",
"chars": 820,
"preview": "@ECHO OFF\r\n\r\npushd %~dp0\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sp"
},
{
"path": "docs/requirements.txt",
"chars": 28,
"preview": "numpy\nscipy\nsphinx_rtd_theme"
},
{
"path": "docs/usage/customdistributions.rst",
"chars": 2624,
"preview": "Custom probability distributions\n================================\n\nYou can use your own custom probability distributions"
},
{
"path": "docs/usage/debug.rst",
"chars": 3358,
"preview": "Debug\n=====\n\nIncreasing the verbosity level\n------------------------------\n\nTo inspect the intermediate steps that the a"
},
{
"path": "docs/usage/incremental.rst",
"chars": 1617,
"preview": "Incremental matching\n====================\n\nExample: Incremental matching\n-------------------------------\n\nIf the observa"
},
{
"path": "docs/usage/installation.rst",
"chars": 1110,
"preview": "Installation\n============\n\nDependencies\n------------\n\nRequired:\n\n- `numpy <http://www.numpy.org>`__\n- `scipy <https://"
},
{
"path": "docs/usage/introduction.rst",
"chars": 4256,
"preview": "Examples\n========\n\nExample 1: Simple\n-----------------\n\nA first, simple example. Some parameters are given to tune the a"
},
{
"path": "docs/usage/latitudelongitude.rst",
"chars": 4297,
"preview": "Dealing with Latitude-Longitude\n===============================\n\nThe toolbox can deal with latitude-longitude coordinate"
},
{
"path": "docs/usage/openstreetmap.rst",
"chars": 4408,
"preview": "Map from OpenStreetMap\n======================\n\nYou can download a graph for map-matching from the OpenStreetMap.org serv"
},
{
"path": "docs/usage/visualisation.rst",
"chars": 2009,
"preview": "Visualisation\n=============\n\nTo inspect the results, a plotting function is included.\n\nSimple plotting\n---------------\n\n"
},
{
"path": "leuvenmapmatching/__init__.py",
"chars": 446,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching\n~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyright 2015-2022 DTAI, K"
},
{
"path": "leuvenmapmatching/map/__init__.py",
"chars": 212,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.map\n~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyright 2018 DTAI"
},
{
"path": "leuvenmapmatching/map/base.py",
"chars": 4229,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.map.base\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBase Map class.\n\nTo be used in a Matcher ob"
},
{
"path": "leuvenmapmatching/map/inmem.py",
"chars": 23304,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.map.inmem\n~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSimple in-memory map representation. Not "
},
{
"path": "leuvenmapmatching/map/sqlite.py",
"chars": 22986,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.map.sqlite\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nMap representation based on a sqlite da"
},
{
"path": "leuvenmapmatching/matcher/__init__.py",
"chars": 220,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.matcher\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyright 2"
},
{
"path": "leuvenmapmatching/matcher/base.py",
"chars": 79237,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.matcher.base\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBase Matcher and Matching classes.\n"
},
{
"path": "leuvenmapmatching/matcher/distance.py",
"chars": 11911,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.matcher.distance\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copy"
},
{
"path": "leuvenmapmatching/matcher/newsonkrumm.py",
"chars": 6372,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.matcher.newsonkrumm\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nMethods similar to Ne"
},
{
"path": "leuvenmapmatching/matcher/simple.py",
"chars": 3900,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.matcher.simple\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyrigh"
},
{
"path": "leuvenmapmatching/util/__init__.py",
"chars": 501,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util\n~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyright 2015-20"
},
{
"path": "leuvenmapmatching/util/debug.py",
"chars": 177,
"preview": "import logging\n\n\nlogger = logging.getLogger(\"be.kuleuven.cs.dtai.mapmatching\")\n\n\ndef printd(*args, **kwargs):\n \"\"\"Pri"
},
{
"path": "leuvenmapmatching/util/dist_euclidean.py",
"chars": 4541,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.dist_euclidean\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert"
},
{
"path": "leuvenmapmatching/util/dist_latlon.py",
"chars": 8186,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.dist_latlon\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBased on:\nhttps://www.movab"
},
{
"path": "leuvenmapmatching/util/dist_latlon_nvector.py",
"chars": 5544,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.dist_latlon_nvector\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wa"
},
{
"path": "leuvenmapmatching/util/evaluation.py",
"chars": 2613,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.evaluation\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nMethods to help set up and ev"
},
{
"path": "leuvenmapmatching/util/gpx.py",
"chars": 1506,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.gpx\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSome additional functions to interact with "
},
{
"path": "leuvenmapmatching/util/kalman.py",
"chars": 2901,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.kalman\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Cop"
},
{
"path": "leuvenmapmatching/util/openstreetmap.py",
"chars": 4944,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.openstreetmap\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:"
},
{
"path": "leuvenmapmatching/util/projections.py",
"chars": 2395,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.projections\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copy"
},
{
"path": "leuvenmapmatching/util/segment.py",
"chars": 2780,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.util.segment\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: C"
},
{
"path": "leuvenmapmatching/visualization.py",
"chars": 14449,
"preview": "# encoding: utf-8\n\"\"\"\nleuvenmapmatching.visualization\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright:"
},
{
"path": "setup.cfg",
"chars": 1158,
"preview": "[metadata]\nname = leuvenmapmatching\nversion = attr: leuvenmapmatching.__version__\nauthor = Wannes Meert\ndescription = Ma"
},
{
"path": "setup.py",
"chars": 1171,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\nsetup.py\n~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyright 2017-2021 DT"
},
{
"path": "tests/examples/example_1_simple.py",
"chars": 1150,
"preview": "from leuvenmapmatching.matcher.distance import DistanceMatcher\nfrom leuvenmapmatching.map.inmem import InMemMap\n\nmap_con"
},
{
"path": "tests/examples/example_using_osmnx_and_geopandas.py",
"chars": 2737,
"preview": "import os\nimport sys\nimport logging\nfrom pathlib import Path\n\nthis_path = Path(os.path.realpath(__file__)).parent.parent"
},
{
"path": "tests/rsrc/bug2/readme.md",
"chars": 174,
"preview": "Test data\n=========\n\nDownload from https://people.cs.kuleuven.be/wannes.meert/leuvenmapmatching/leuvenmapmatching_testda"
},
{
"path": "tests/rsrc/newson_krumm_2009/readme.md",
"chars": 239,
"preview": "Newson Krum testdata\n====================\n\nFiles will be downloaded from https://www.microsoft.com/en-us/research/public"
},
{
"path": "tests/rsrc/path_latlon/readme.md",
"chars": 238,
"preview": "Test data for path_latlon\n=========================\n\nDownload from https://people.cs.kuleuven.be/wannes.meert/leuvenmapm"
},
{
"path": "tests/rsrc/path_latlon/route.gpx",
"chars": 5797,
"preview": "<?xml version=\"1.0\"?>\r\n<gpx\r\n version=\"1.0\"\r\n creator=\"VB Net GPS: vermeiren-willy@pandora.be\"\r\n xmlns:xsi=\"http://w"
},
{
"path": "tests/rsrc/path_latlon/route2.gpx",
"chars": 1224,
"preview": "<?xml version=\"1.0\"?>\r\n<gpx\r\n version=\"1.0\"\r\n creator=\"Wannes\"\r\n xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance"
},
{
"path": "tests/test_bugs.py",
"chars": 5260,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_bugs\n~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyrigh"
},
{
"path": "tests/test_conversion.py",
"chars": 9680,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\nimport sys\nimport logging\nfrom datetime import datetime\nimport pytest\nimport ma"
},
{
"path": "tests/test_examples.py",
"chars": 2295,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_examples\n~~~~~~~~~~~~~~~~~~~\n\nRun standalone python files that a"
},
{
"path": "tests/test_newsonkrumm2009.py",
"chars": 20697,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_path_newsonkrumm2009\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBased on t"
},
{
"path": "tests/test_nonemitting.py",
"chars": 15618,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_nonemitting\n~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyr"
},
{
"path": "tests/test_nonemitting_circle.py",
"chars": 5476,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_nonemitting_circle\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wanne"
},
{
"path": "tests/test_parallelroads.py",
"chars": 4062,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_parallelroads\n~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:c"
},
{
"path": "tests/test_path.py",
"chars": 21121,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_path\n~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyright: Copyrigh"
},
{
"path": "tests/test_path_latlon.py",
"chars": 16578,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_path_latlon\n~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n:copyr"
},
{
"path": "tests/test_path_onlyedges.py",
"chars": 3307,
"preview": "#!/usr/bin/env python3\n# encoding: utf-8\n\"\"\"\ntests.test_path_onlyedges\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n:author: Wannes Meert\n"
}
]
About this extraction
This page contains the full source code of the wannesm/LeuvenMapMatching GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 69 files (352.0 KB), approximately 100.9k tokens, and a symbol index with 320 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.