Repository: RobertLucian/cortex-license-plate-reader-client
Branch: master
Commit: bccaabd0d4a9
Files: 14
Total size: 58.4 KB
Directory structure:
gitextract_0wtqovzi/
├── README.md
├── app.py
├── broadcast.py
├── config.json
├── deps/
│ ├── requirements_base.txt
│ ├── requirements_dpkg_rpi.txt
│ └── requirements_rpi.txt
├── drawings/
│ └── License Plate Identifier Client.drawio
├── gps.py
├── utils/
│ ├── bbox.py
│ ├── colors.py
│ ├── image.py
│ └── queue.py
└── workers.py
================================================
FILE CONTENTS
================================================
================================================
FILE: README.md
================================================
# Client for License Plate Identification in Real-time on AWS w/ Cortex [](https://robertlucian.mit-license.org)
**READ THIS: This is a client for 3 (YOLOv3, CRAFT text detector, CRNN text recognizer) [cortex](https://github.com/cortexlabs/cortex)-deployed ML models. This client only works in conjunction with [these cortex APIs](https://github.com/cortexlabs/cortex/tree/master/examples/tensorflow/license-plate-reader).**
*- The above GIF was taken from [this video](https://www.youtube.com/watch?v=gsYEZtecXlA) of whose predictions were computed on the fly with cortex/AWS -*
## Description
This app which uses the deployed cortex APIs as a PaaS captures the frames from a video camera, sends them for inferencing to the cortex APIs, recombines them after the responses are received and then the detections/recognitions are overlayed on the output stream. This is done on the car's dashcam (composed of the Raspberry Pi + GSM module) in real-time. Access to the internet is provided through the GSM module's 4G connection.
The app must be configured to use the API endpoints as shown when calling `cortex get yolov3` and `cortex get crnn`. Checkout how the APIs are defined in [their repository](https://github.com/cortexlabs/cortex/tree/master/examples/tensorflow/license-plate-reader).
The app also saves a `csv` file containing the dates and GPS coordinates of each identified license plate.
### Latency
The observable latency between capturing the frame and broadcasting the predictions in the browser (with all the inference stuff going on) takes about *0.5-1.0 seconds* depending on:
* How many replicas are assigned for each API.
* Internet connection bandwidth and latency.
* Broadcast buffer size. To get a smoother stream, use a higher buffer size (*10-30*) or if you want the stream to be displayed as quickly as possible but with possible dropped frames, go with lower values (*<10*).
To learn more about how the actual device was constructed, check out [this](https://towardsdatascience.com/i-built-a-diy-license-plate-reader-with-a-raspberry-pi-and-machine-learning-7e428d3c7401) article.
## Target Machine
Target machine **1**: Raspberry Pi.
Target machine **2**: Any x86 machine.
---
The app's primary target machine is the *Raspberry Pi* (3/3B+/4) - the Raspberry Pi is a small embedded computer system that got adopted by many hobbyists and institutions all around the world as the de-facto choice for hardware/software experiments.
Unfortunately for the Raspberry Pi, the $35 pocket-sized computer, doesn't have enough oomph (far from it) to do any inference: not only it doesn't have enough memory to load a model, but should it have enough RAM, it would still take dozens of minutes just to get a single inference. Let alone run inferences at 30 FPS, with a number of inferences for each frame.
The app is built to be used with a *Pi Camera* alongside the *Raspberry Pi*. The details on how to build such a system are found [here](#creating-your-own-device).
Since many developers don't have a Raspberry Pi laying around or are just purely interested in seeing results right away, the app can also be configured to take in a video file and treat it exactly as if it was a camera.
## Dependencies
For either target machine, the minimum version of Python has to be `3.6.x`.
#### For the Raspberry Pi
```bash
pip3 install --user -r deps/requirements_rpi.txt
sudo apt-get update
# dependencies for opencv and pandas package
sudo apt-get install --no-install-recommends $(cat deps/requirements_dpkg_rpi.txt)
```
#### For x86 Machine
```bash
pip3 install -r deps/requirements_base.txt
```
## Configuring
The configuration file can be in this form
```jsonc
{
"video_source": {
// "file" for reading from file or "camera" for pi camera
"type": "file",
// video file to read from; applicable just for "file" type
"input": "airport_ride_480p.mp4",
// scaling for the video file; applicable just for "file" type
"scale_video": 1.0,
// how many frames to skip on the video file; applicable just for "file" type
"frames_to_skip": 0,
// framerate
"framerate": 30
// camera sensor mode; applicable just for "camera" type
// "sensor_mode": 5
// where to save camera's output; applicable just for "camera" type
// "output_file": "recording.h264"
// resolution of the input video; applicable just for the "camera" type
// "resolution": [480, 270]
},
"broadcaster": {
// when broadcasting, a buffer is required to provide framerate fluidity; measured in frames
"target_buffer_size": 10,
// how much the size of the buffer can variate +-
"max_buffer_size_variation": 5,
// the maximum variation of the fps when extracting frames from the queue
"max_fps_variation": 15,
// target fps - must match the camera/file's framerate
"target_fps": 30,
// address to bind the web server to
"serve_address": ["0.0.0.0", 8000]
},
"inferencing_worker": {
// YOLOv3's input size of the image in pixels (must consider the existing model)
"yolov3_input_size_px": 416,
// when drawing the bounding boxes, use a higher res image to draw boxes more precisely
// (this way text is more readable)
"bounding_boxes_upscale_px": 640,
// object detection accuracy threshold in percentages with range (0, 1)
"yolov3_obj_thresh": 0.8,
// the jpeg quality of the image for the CRAFT/CRNN models in percentanges;
// these models receive the cropped images of each detected license plate
"crnn_quality": 98,
// broadcast quality - aim for a lower value since this stream doesn't influence the predictions; measured in percentages
"broadcast_quality": 90,
// connection timeout for both API endpoints measured in seconds
"timeout": 1.20,
// YOLOv3 API endpoint
"api_endpoint_yolov3": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/yolov3",
// CRNN API endpoint
// Can be set to "" value to turn off the recognition inference
// By turning it off, the latency is reduced and the output video appears smoother
"api_endpoint_crnn": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/crnn"
},
"inferencing_pool": {
// number of workers to do inferencing (YOLOv3 + CRAFT + CRNN)
// depending on the source's framerate, a balance must be achieved
"workers": 24,
// pick the nth frame from the input stream
// if the input stream runs at 30 fps, then setting this to 2 would act
// as if the input stream runs at 30/2=15 fps
// ideally, you have a high fps camera (90-180) and you only pick every 3rd-6th frame
"pick_every_nth_frame": 1
},
"flusher": {
// if there are more than this given number of frames in the input stream's buffer, flush them
// it's useful if the inference workers (due to a number of reasons) can't keep up with the input flow
// also keeps the current broadcasted stream up-to-date with the reality
"frame_count_threshold": 5
},
"gps": {
// set to false when using a video file to read the stream from
// set to true when you have a GPS connected to your system
"use_gps": false,
// port to write to to activate the GPS (built for EC25-E modules)
"write_port": "/dev/ttyUSB2",
// port to read from in NMEA standard
"read_port": "/dev/ttyUSB1",
// baudrate as measured in bits/s
"baudrate": 115200
},
"general": {
// to which IP the requests module is bound to
// useful if you only want to route the traffic through a specific interface
"bind_ip": "0.0.0.0",
// where to save the csv data containing the date, the predicted license plate number and GPS data
// can be an empty string, in which case, nothing is stored
"saved_data": "saved_data.csv"
}
}
```
Be aware that for having a functional application, the minimum amount of things that have to be adjusted in the config file are:
1. The input file in case you are using a video to feed the application with. You can download the following `mp4` video file to use as input. Download it by running `wget -O airport_ride_480p.mp4 "https://www.dropbox.com/s/q9j57y5k95wg2zt/airport_ride_480p.mp4?dl=0"`
1. Both API endpoints from your cortex APIs. Use `cortex get your-api-name-here` command to get those.
## Running It
Make sure both APIs are already running in the cluster. Launching it the first time might raise some timeout exceptions, but let it run for a few moments. If there's enough compute capacity, you'll start getting `200` response codes.
Run it like
```bash
python app.py -c config.json
```
Once it's running, you can head off to its browser page to see the live broadcast with its predictions overlayed on top.
To save the broascasted MJPEG stream, you can run the following command (check the broadcast serve address in the `config.json` file)
```bash
PORT=8000
FRAMERATE=30
ffmpeg -i http://localhost:PORT/stream.mjpg -an -vcodec libx264 -r FRAMERATE saved_video.h264
```
To terminate the app, press `CTRL-C` and wait a bit.
## Creating Your Own Device
To create your own Raspberry Pi-powered device to record and display the predictions in real time in your car, you're gonna need the following things:
1. A Raspberry Pi - preferably a 4, because that one has more oomph.
1. A Pi Camera - doesn't matter which version of it.
1. A good buck converter to step-down from 12V down to 5V - aim for 4-5 amps. You can use a SBEC/UBEC/BEC regulators - they are easy to find and cheap.
1. A power outlet for the car's cigarette port to get the 12V DC.
1. 4G/GPS shield to host a GSM module - in this project, an EC25-E module has been used. You will also need antennas.
1. A 3D-printed support to hold the electronics and be able to hold it against the rear mirror or dashboard. Must be built to accomodate to your own car.
Without convoluting this README too much:
* Here are the [STLs/SLDPRTs/Renders](https://www.dropbox.com/sh/fw16vy1okrp606y/AAAwkoWXODmoaOP4yR-z4T8Va?dl=0) to the car's 3D printed support.
* Here's an [article](https://towardsdatascience.com/i-built-a-diy-license-plate-reader-with-a-raspberry-pi-and-machine-learning-7e428d3c7401) that talks about this in full detail.
================================================
FILE: app.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
import signal, os, time, json, queue, socket, click, cv2, pandas as pd
import multiprocessing as mp
import threading as td
import logging
logger = logging.getLogger()
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_format = logging.Formatter(
"%(asctime)s - %(name)s - %(threadName)s - %(levelname)s - %(message)s"
)
stream_handler.setFormatter(stream_format)
logger.addHandler(stream_handler)
logger.setLevel(logging.DEBUG)
disable_loggers = ["urllib3.connectionpool"]
for name, logger in logging.root.manager.loggerDict.items():
if name in disable_loggers:
logger.disabled = True
from gps import ReadGPSData
from workers import BroadcastReassembled, InferenceWorker, Flusher, session
from utils.image import resize_image, image_to_jpeg_bytes
from utils.queue import MPQueue
from requests_toolbelt.adapters.source import SourceAddressAdapter
class GracefullKiller:
"""
For killing the app gracefully.
"""
kill_now = False
def __init__(self):
signal.signal(signal.SIGINT, self.exit_gracefully)
signal.signal(signal.SIGTERM, self.exit_gracefully)
def exit_gracefully(self, signum, frame):
self.kill_now = True
class WorkerPool(mp.Process):
"""
Pool of threads running in a different process.
"""
def __init__(self, name, worker, pool_size, *args, **kwargs):
"""
name - Name of the process.
worker - Derived class of thread to execute.
pool_size - Number of workers to have.
"""
super(WorkerPool, self).__init__(name=name)
self.event_stopper = mp.Event()
self.Worker = worker
self.pool_size = pool_size
self.args = args
self.kwargs = kwargs
def run(self):
logger.info("spawning workers on separate process")
pool = [
self.Worker(
self.event_stopper,
*self.args,
**self.kwargs,
name="{}-Worker-{}".format(self.name, i),
)
for i in range(self.pool_size)
]
[worker.start() for worker in pool]
while not self.event_stopper.is_set():
time.sleep(0.001)
logger.info("stoppping workers on separate process")
[worker.join() for worker in pool]
def stop(self):
self.event_stopper.set()
class DistributeFramesAndInfer:
"""
Custom output class primarly built for the PiCamera class.
Has 3 process-safe queues: in_queue for the incoming frames from the source,
bc_queue for the frames with the predicted overlays heading off to the broadcaster,
predicts_queue for the predictions to be written off to the disk.
"""
def __init__(self, pool_cfg, worker_cfg):
"""
pool_cfg - Configuration dictionary for the pool manager.
worker_cfg - Configuration dictionary for the pool workers.
"""
self.frame_num = 0
self.in_queue = MPQueue()
self.bc_queue = MPQueue()
self.predicts_queue = MPQueue()
for key, value in pool_cfg.items():
setattr(self, key, value)
self.pool = WorkerPool(
"InferencePool",
InferenceWorker,
self.workers,
self.in_queue,
self.bc_queue,
self.predicts_queue,
worker_cfg,
)
self.pool.start()
def write(self, buf):
"""
Mandatory custom output method for the PiCamera class.
buf - Frame as a bytes object.
"""
if buf.startswith(b"\xff\xd8"):
# start of new frame; close the old one (if any) and
if self.frame_num % self.pick_every_nth_frame == 0:
self.in_queue.put({"frame_num": self.frame_num, "jpeg": buf})
self.frame_num += 1
def stop(self):
"""
Stop all workers and the process altogether.
"""
self.pool.stop()
self.pool.join()
qs = [self.in_queue, self.bc_queue]
[q.cancel_join_thread() for q in qs]
def get_queues(self):
"""
Retrieve all queues.
"""
return self.in_queue, self.bc_queue, self.predicts_queue
@click.command(
help=(
"Identify license plates from a given video source"
" while outsourcing the predictions using REST API endpoints."
)
)
@click.option("--config", "-c", required=True, type=str)
def main(config):
killer = GracefullKiller()
# open config file
try:
file = open(config)
cfg = json.load(file)
file.close()
except Exception as error:
logger.critical(str(error), exc_info=1)
return
# give meaningful names to each sub config
source_cfg = cfg["video_source"]
broadcast_cfg = cfg["broadcaster"]
pool_cfg = cfg["inferencing_pool"]
worker_cfg = cfg["inferencing_worker"]
flusher_cfg = cfg["flusher"]
gps_cfg = cfg["gps"]
gen_cfg = cfg["general"]
# bind requests module to use a given network interface
try:
socket.inet_aton(gen_cfg["bind_ip"])
session.mount("http://", SourceAddressAdapter(gen_cfg["bind_ip"]))
logger.info("binding requests module to {} IP".format(gen_cfg["bind_ip"]))
except OSError as e:
logger.error("bind IP is invalid, resorting to default interface", exc_info=True)
# start polling the GPS
if gps_cfg["use_gps"]:
wport = gps_cfg["write_port"]
rport = gps_cfg["read_port"]
br = gps_cfg["baudrate"]
gps = ReadGPSData(wport, rport, br)
gps.start()
else:
gps = None
# workers on a separate process to run inference on the data
logger.info("initializing pool w/ " + str(pool_cfg["workers"]) + " workers")
output = DistributeFramesAndInfer(pool_cfg, worker_cfg)
frames_queue, bc_queue, predicts_queue = output.get_queues()
logger.info("initialized worker pool")
# a single worker in a separate process to reassemble the data
reassembler = BroadcastReassembled(bc_queue, broadcast_cfg, name="BroadcastReassembled")
reassembler.start()
# a single thread to flush the producing queue
# when there are too many frames in the pipe
flusher = Flusher(frames_queue, threshold=flusher_cfg["frame_count_threshold"], name="Flusher")
flusher.start()
# data aggregator to write things to disk
def results_writer():
if len(gen_cfg["saved_data"]) > 0:
df = pd.DataFrame(columns=["Date", "License Plate", "Coordinates"])
while not killer.kill_now:
time.sleep(0.01)
try:
data = predicts_queue.get_nowait()
except queue.Empty:
continue
predicts = data["predicts"]
date = data["date"]
for lp in predicts:
if len(lp) > 0:
lp = " ".join(lp)
entry = {"Date": date, "License Plate": lp, "Coordinates": ""}
if gps:
entry["Coordinates"] = "{}, {}".format(
gps.latitude, gps.longitude
).upper()
df = df.append(entry, ignore_index=True)
logger.info("dumping results to csv file {}".format(gen_cfg["saved_data"]))
if os.path.isfile(gen_cfg["saved_data"]):
header = False
else:
header = True
with open(gen_cfg["saved_data"], "a") as f:
df.to_csv(f, header=header)
# data aggregator thread
results_thread = td.Thread(target=results_writer)
results_thread.start()
if source_cfg["type"] == "camera":
# import module
import picamera
# start the pi camera
with picamera.PiCamera() as camera:
# configure the camera
camera.sensor_mode = source_cfg["sensor_mode"]
camera.resolution = source_cfg["resolution"]
camera.framerate = source_cfg["framerate"]
logger.info(
"picamera initialized w/ mode={} resolution={} framerate={}".format(
camera.sensor_mode, camera.resolution, camera.framerate
)
)
# start recording both to disk and to the queue
camera.start_recording(
output=source_cfg["output_file"], format="h264", splitter_port=0, bitrate=10000000,
)
camera.start_recording(
output=output, format="mjpeg", splitter_port=1, bitrate=10000000, quality=95,
)
logger.info("started recording to file and to queue")
# wait until SIGINT is detected
while not killer.kill_now:
camera.wait_recording(timeout=0.5, splitter_port=0)
camera.wait_recording(timeout=0.5, splitter_port=1)
logger.info(
"frames qsize: {}, broadcast qsize: {}, predicts qsize: {}".format(
frames_queue.qsize(), bc_queue.qsize(), predicts_queue.qsize()
)
)
# stop recording
logger.info("gracefully exiting")
camera.stop_recording(splitter_port=0)
camera.stop_recording(splitter_port=1)
output.stop()
elif source_cfg["type"] == "file":
# open video file
video_reader = cv2.VideoCapture(source_cfg["input"])
video_reader.set(cv2.CAP_PROP_POS_FRAMES, source_cfg["frames_to_skip"])
# get # of frames and determine target width
nb_frames = int(video_reader.get(cv2.CAP_PROP_FRAME_COUNT))
frame_h = int(video_reader.get(cv2.CAP_PROP_FRAME_HEIGHT))
frame_w = int(video_reader.get(cv2.CAP_PROP_FRAME_WIDTH))
target_h = int(frame_h * source_cfg["scale_video"])
target_w = int(frame_w * source_cfg["scale_video"])
period = 1.0 / source_cfg["framerate"]
logger.info(
"file-based video stream initialized w/ resolution={} framerate={} and {} skipped frames".format(
(target_w, target_h), source_cfg["framerate"], source_cfg["frames_to_skip"],
)
)
# serve each frame to the workers iteratively
last_log = time.time()
for i in range(nb_frames):
start = time.time()
try:
# write frame to queue
_, frame = video_reader.read()
if target_w != frame_w:
frame = resize_image(frame, target_w)
jpeg = image_to_jpeg_bytes(frame)
output.write(jpeg)
except Exception as error:
logger.error("unexpected error occurred", exc_info=True)
break
end = time.time()
spent = end - start
left = period - spent
if left > 0:
# maintain framerate
time.sleep(period)
# check if SIGINT has been sent
if killer.kill_now:
break
# do logs every second
current = time.time()
if current - last_log >= 1.0:
logger.info(
"frames qsize: {}, broadcast qsize: {}, predicts qsize: {}".format(
frames_queue.qsize(), bc_queue.qsize(), predicts_queue.qsize()
)
)
last_log = current
logger.info("gracefully exiting")
video_reader.release()
output.stop()
if gps_cfg["use_gps"]:
gps.stop()
reassembler.stop()
flusher.stop()
if __name__ == "__main__":
main()
================================================
FILE: broadcast.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
import io
import logging
import socketserver
from threading import Condition
from http import server
PAGE = """\
<html>
<head>
<title>License Plate Predictions</title>
</head>
<body>
<h1>Live License Plate Predictions</h1>
<img src="stream.mjpg" width="1280" height="720" />
</body>
</html>
"""
logger = logging.getLogger(__name__)
class StreamingOutput(object):
def __init__(self):
self.frame = None
self.buffer = io.BytesIO()
self.condition = Condition()
def write(self, buf):
if buf.startswith(b"\xff\xd8"):
# New frame, copy the existing buffer's content and notify all
# clients it's available
self.buffer.truncate()
with self.condition:
self.frame = self.buffer.getvalue()
self.condition.notify_all()
self.buffer.seek(0)
return self.buffer.write(buf)
class StreamingHandler(server.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/":
self.send_response(301)
self.send_header("Location", "/index.html")
self.end_headers()
elif self.path == "/index.html":
content = PAGE.encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/html")
self.send_header("Content-Length", len(content))
self.end_headers()
self.wfile.write(content)
elif self.path == "/stream.mjpg":
self.send_response(200)
self.send_header("Age", 0)
self.send_header("Cache-Control", "no-cache, private")
self.send_header("Pragma", "no-cache")
self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=FRAME")
self.end_headers()
try:
while True:
with output.condition:
output.condition.wait()
frame = output.frame
self.wfile.write(b"--FRAME\r\n")
self.send_header("Content-Type", "image/jpeg")
self.send_header("Content-Length", len(frame))
self.end_headers()
self.wfile.write(frame)
self.wfile.write(b"\r\n")
except Exception as e:
logging.warning("Removed streaming client %s: %s", self.client_address, str(e))
else:
self.send_error(404)
self.end_headers()
def set_output(output):
self.output = output
class StreamingServer(socketserver.ThreadingMixIn, server.HTTPServer):
allow_reuse_address = True
daemon_threads = True
output = StreamingOutput()
================================================
FILE: config.json
================================================
{
"video_source": {
"type": "file",
"input": "airport_ride_480p.mp4",
"scale_video": 1.0,
"frames_to_skip": 0,
"framerate": 30
},
"broadcaster": {
"target_buffer_size": 10,
"max_buffer_size_variation": 5,
"max_fps_variation": 15,
"target_fps": 30,
"serve_address": ["0.0.0.0", 8000]
},
"inferencing_worker": {
"yolov3_input_size_px": 416,
"bounding_boxes_upscale_px": 640,
"yolov3_obj_thresh": 0.8,
"crnn_quality": 98,
"broadcast_quality": 90,
"timeout": 1.20,
"api_endpoint_yolov3": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/yolov3",
"api_endpoint_crnn": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/crnn"
},
"inferencing_pool": {
"workers": 24,
"pick_every_nth_frame": 1
},
"flusher": {
"frame_count_threshold": 5
},
"gps": {
"use_gps": false,
"write_port": "/dev/ttyUSB2",
"read_port": "/dev/ttyUSB1",
"baudrate": 115200
},
"general": {
"bind_ip": "0.0.0.0",
"saved_data": "saved_data.csv"
}
}
================================================
FILE: deps/requirements_base.txt
================================================
certifi==2019.11.28
idna==2.8
numpy==1.18.1
opencv-contrib-python==4.1.0.25
requests==2.22.0
requests-toolbelt==0.9.1
urllib3==1.25.8
pynmea2==1.15.0
Click==7.0
================================================
FILE: deps/requirements_dpkg_rpi.txt
================================================
build-essential
cmake
unzip
pkg-config
libjpeg-dev
libtiff5-dev
libjasper-dev
libpng-dev
libavcodec-dev
libavformat-dev
libswscale-dev
libv4l-dev
libxvidcore-dev
libx264-dev
libfontconfig1-dev
libcairo2-dev
libgdk-pixbuf2.0-dev
libpango1.0-dev
libgtk2.0-dev
libgtk-3-dev
libatlas-base-dev
gfortran
libhdf5-dev
libhdf5-serial-dev
libhdf5-103
libqtgui4
libqtwebkit4
libqt4-test
python3-pyqt5
python3-dev
python3-pandas
================================================
FILE: deps/requirements_rpi.txt
================================================
certifi==2019.11.28
idna==2.8
numpy==1.18.1
opencv-contrib-python==4.1.0.25
picamera==1.13
requests==2.22.0
requests-toolbelt==0.9.1
urllib3==1.25.8
pynmea2==1.15.0
Click==7.0
================================================
FILE: drawings/License Plate Identifier Client.drawio
================================================
<mxfile host="www.draw.io" modified="2020-02-13T20:55:49.174Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.2 Safari/605.1.15" etag="NyBexjbPMC27BK2-6498" version="12.7.0" type="device"><diagram name="Page-1" id="c7558073-3199-34d8-9f00-42111426c3f3">7Vxbc9o4GP01zLQP7UiWr48Bkm6n3Vm2dNtmX3YUrIC3xqK2aML++pVsGdtCECf4EkrSmWJdLGSdT+e7yQzQaHn/Lsarxe/UJ+HAAP79AI0HhgE92+UfomYja1zLyWrmceDLuqJiGvxHZCWQtevAJ0mlI6M0ZMGqWjmjUURmrFKH45jeVbvd0rD6rSs8JzsV0xkOd2u/Bj5bZLUGQl7R8BsJ5gv51QgBOfMbPPs+j+k6kl84MNBt+pc1L3E+mOyfLLBP70pV6HKARjGlLLta3o9IKFY3X7fsvqs9rduJxyRidW74+HcQLK4/vHHG3/768H7849M/DLyBpoTrJw7XJH+QdLpsk69R+pBEDAMGaHi3CBiZrvBMtN5xseB1C7YMeQnySx8ni7RvXphgxkgcpTUG4I89TFhMv5MRDWmcfgFfUvHHW25pxKSMQFuWS/3Msfgn6oMw/INPIWAbOasQ35BwQpOABVR814yvCuG3DX+SmAUc749KB0bFxHEYzLXdL2TDDWWMLnmDXCbeTO73IgC3uPIdQ+iSsHjDu+Q3WLnsyN1imrJ8V4ge8mTdoiR1Ti5EWIr7fDt4gTi/kKA/SgCsFwHoSgBMpOBv7eLvmd3ib6CG8T8Mbj0wa8hQE/sRwCoe2/1ZwsNwNXggqzU8QA08iM91mCzSmC3onEY4vCxqh1XEij4fqZD6dFn/JYxt5F7Da0arKO7Dii9tvPkmx00L16Lw1sqL4/ty4zhHNHsGMfHDYPHnpOt4Rg4skWdLOwHHc8IOdXT18MckxCz4WZ1JC0hafQD3VID2Ad4LcF6/wNm9AHcfsG+l6xJsvFSgJgqbMoQnDjYEoE+083lW+NYOmVylihjYP9Y0b3iTpEBe8A7QXt0XjfxqLj7fR7eELxtfJANMYjojSZIPzCeajZ31PKBv4cP6VmjPEpa3ACOAdXp4OHa0elh6LY2oVNepalTb2NWo0NBoVLs1hQrdPnZzwuWeXQgHVRiXIU6SYJZXXwVhwdb+bideWerSxV516+5VaB+5V9Nb+SPjTanDigYRS0ojT0RFyVAzFLGyLMXXfegG27EO32ChI2/wUOUGfpE9ZSG52+U6gqzcDsjqK42/c1eId22Fr3Z4yRg79kFeUhjOBtiDTlMugKtI1jMgLO9XNz9Oixy9EyNHG50pOXpdkqNxhuRoa+IjHZPjNpp+Ek71SREdzNE9GaZzwXky3RapbqguOkOqc5zeqQ5ADcgq9xXMEdGIqKHz/QHdq/SvQm0Fm12XaW8PteX2YmEjXpdNRK292AtZ7QsmloC1NLjmdY+jtF2CgErOz0GKwGSPKm87RDXQUrjJVUbKlmJnpOZYx2iGdVK6UYgnEjyzXt6khENvBT4p/dSOmfG9z6oS/2Bqbxn4fmYfED5DfJMOJeRUKiI+rjUcWELNCpMgkXnJA9pYTV82wky2IkBQw0y2RoLbyxkiuLP8nRphb02zaod5nvkQXYnShMQBXwIhDLvmmeTPsm0G6ttmdZLWjdKfU9dWA/1mwZBGaXUqK4/SaucgEd6xxvuRAnFSadFzEAiI0JES8SR3zlFMbu8Bb07tv9WM7fpaqJf8kd68zlv05vV5yKrTK3uZ/aizvoLzv6wUHXvC5EmM51qPYzy1//bAWsvRJd2Z4CaiS1MhRoP0OMQ8xstuwkoXFgAm2A0f+S4ADjoQbmrAeTMV383tP6oE60SVzvp8ofSYavlWjdDIzr631UiPety09UhPW+cMJgFvGuElifFAyP0V//9L4BPKP7nyEGHnaYaPAT4vYoL9bkgCAPvy4uoAGSjkAW8wJA0dQzZVtHVHEHQBnhZJoh8f7QlWTidmRd0TAD171vk8G9+1V1xZix3555rwgWtuyGSBV+KSLzkOQxJmKh8NV6UgXKWtFJ179P4dAs96TO6IoJkLQDP7165z5lH3VkeL+7eBQ62uThJGNF0oA1xM3nNDF7z6dDn9nBXFdov8zAR+3Q1rm8gYW0591B3bBi5uBnVoKbBvwxPll0eQBnYEW8Pd0Bl3j8PdsPR5YfH6DWFnn41B6pGYPJxZ3uxAh3p7oPf6ugJ4a/QSoWpb6efvxT3sBBhmr1rf6Dk+6Zw7/P0GJJHZL/z10d+Xkyuc+V8uIGmYdaXo6BTMkSRituQ6cFHD/gwn7MV92GNSeGqM0NO4D7qXkNtzH3Jn5flzyilzQ3fJiiO5oa10wCQmfjATCZMXatB7G/D5cQPS/T5Bo5oiPQL4Ckd+KuJ8w5IldwbjdmIKT/m1ksqLtAdoY5+IbF/RbUJ7eFUJgQD1nmIyUS/q4zSOfD0lyN2k2kHPPEcODTXpDatZ8raO+exquXeT6aBIQ3XNMpWU1BNYptEUla2yTP+JbLSb7RgHyXdeM14vV5kSGU2/vH6BUG4jRVN4mnh114ri+DQF0toSozAg0fMNVtd3NY5LTLlWBXFL9+YA7DRWbR5/sGAf4nTtC9XwdXrusEPLU3DX/Myg0VCOgheLn7DMlHHxS6Ho8n8=</diagram></mxfile>
================================================
FILE: gps.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
import serial, pynmea2, time, threading as td
import logging
logger = logging.getLogger(__name__)
class ReadGPSData(td.Thread):
"""
Class to read the data off of the EC25-E's GPS module.
Can be easily adapted to work with any other GPS module.
"""
def __init__(self, write_port, read_port, baudrate, name="GPS"):
"""
write_port - The serial port to use for activating the GPS.
read_port - The serial port from which to read the GPS data.
baudrate - Transport rate over the serial port.
name - Name of the thread.
"""
super(ReadGPSData, self).__init__(name=name)
self.write_port = write_port
self.read_port = read_port
self.baudrate = baudrate
self.event = td.Event()
self.lock = td.Lock()
def run(self):
logger.info("configuring GPS on port {}".format(self.write_port))
self.serw = serial.Serial(
self.write_port, baudrate=self.baudrate, timeout=1, rtscts=True, dsrdtr=True
)
self.serw.write("AT+QGPS=1\r".encode("utf-8"))
self.serw.close()
time.sleep(0.5)
self.serr = serial.Serial(
self.read_port, baudrate=self.baudrate, timeout=1, rtscts=True, dsrdtr=True
)
logger.info("configured GPS to read from port {}".format(self.read_port))
while not self.event.is_set():
data = self.serr.readline()
self.lock.acquire()
try:
self.__msg = pynmea2.parse(data.decode("utf-8"))
except:
pass
finally:
self.lock.release()
logger.info(self.__msg)
time.sleep(1)
logger.info("stopped GPS thread")
@property
def parsed(self):
"""
Get the whole parsed data.
"""
self.lock.acquire()
try:
data = self.__msg
except:
data = None
finally:
self.lock.release()
return data
@property
def latitude(self):
"""
Returns latitude expressed as a float.
"""
self.lock.acquire()
try:
latitude = self.__msg.latitude
except:
latitude = 0.0
finally:
self.lock.release()
return latitude
@property
def longitude(self):
"""
Returns longitude expressed as a float.
"""
self.lock.acquire()
try:
longitude = self.__msg.longitude
except:
longitude = 0.0
finally:
self.lock.release()
return longitude
def stop(self):
"""
Stop the thread.
"""
self.event.set()
================================================
FILE: utils/bbox.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
import numpy as np
import cv2
from .colors import get_color
class BoundBox:
def __init__(self, xmin, ymin, xmax, ymax, c=None, classes=None):
self.xmin = xmin
self.ymin = ymin
self.xmax = xmax
self.ymax = ymax
self.c = c
self.classes = classes
self.label = -1
self.score = -1
def get_label(self):
if self.label == -1:
self.label = np.argmax(self.classes)
return self.label
def get_score(self):
if self.score == -1:
self.score = self.classes[self.get_label()]
return self.score
def draw_boxes(image, boxes, overlay_text, labels, obj_thresh, quiet=True):
for box, overlay in zip(boxes, overlay_text):
label_str = ""
label = -1
for i in range(len(labels)):
if box.classes[i] > obj_thresh:
if label_str != "":
label_str += ", "
label_str += labels[i] + " " + str(round(box.get_score() * 100, 2)) + "%"
label = i
if not quiet:
print(label_str)
if label >= 0:
if len(overlay) > 0:
text = label_str + ": [" + " ".join(overlay) + "]"
else:
text = label_str
text = text.upper()
text_size = cv2.getTextSize(text, cv2.FONT_HERSHEY_SIMPLEX, 1.1e-3 * image.shape[0], 5)
width, height = text_size[0][0], text_size[0][1]
region = np.array(
[
[box.xmin - 3, box.ymin],
[box.xmin - 3, box.ymin - height - 26],
[box.xmin + width + 13, box.ymin - height - 26],
[box.xmin + width + 13, box.ymin],
],
dtype="int32",
)
# cv2.rectangle(img=image, pt1=(box.xmin,box.ymin), pt2=(box.xmax,box.ymax), color=get_color(label), thickness=5)
rec = (box.xmin, box.ymin, box.xmax - box.xmin, box.ymax - box.ymin)
rec = tuple(int(i) for i in rec)
cv2.rectangle(img=image, rec=rec, color=get_color(label), thickness=3)
cv2.fillPoly(img=image, pts=[region], color=get_color(label))
cv2.putText(
img=image,
text=text,
org=(box.xmin + 13, box.ymin - 13),
fontFace=cv2.FONT_HERSHEY_SIMPLEX,
fontScale=1e-3 * image.shape[0],
color=(0, 0, 0),
thickness=1,
)
return image
================================================
FILE: utils/colors.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
def get_color(label):
""" Return a color from a set of predefined colors. Contains 80 colors in total.
code originally from https://github.com/fizyr/keras-retinanet/
Args
label: The label to get the color for.
Returns
A list of three values representing a RGB color.
"""
if label < len(colors):
return colors[label]
else:
print("Label {} has no color, returning default.".format(label))
return (0, 255, 0)
colors = [
[31, 0, 255],
[0, 159, 255],
[255, 95, 0],
[255, 19, 0],
[255, 0, 0],
[255, 38, 0],
[0, 255, 25],
[255, 0, 133],
[255, 172, 0],
[108, 0, 255],
[0, 82, 255],
[0, 255, 6],
[255, 0, 152],
[223, 0, 255],
[12, 0, 255],
[0, 255, 178],
[108, 255, 0],
[184, 0, 255],
[255, 0, 76],
[146, 255, 0],
[51, 0, 255],
[0, 197, 255],
[255, 248, 0],
[255, 0, 19],
[255, 0, 38],
[89, 255, 0],
[127, 255, 0],
[255, 153, 0],
[0, 255, 255],
[0, 255, 216],
[0, 255, 121],
[255, 0, 248],
[70, 0, 255],
[0, 255, 159],
[0, 216, 255],
[0, 6, 255],
[0, 63, 255],
[31, 255, 0],
[255, 57, 0],
[255, 0, 210],
[0, 255, 102],
[242, 255, 0],
[255, 191, 0],
[0, 255, 63],
[255, 0, 95],
[146, 0, 255],
[184, 255, 0],
[255, 114, 0],
[0, 255, 235],
[255, 229, 0],
[0, 178, 255],
[255, 0, 114],
[255, 0, 57],
[0, 140, 255],
[0, 121, 255],
[12, 255, 0],
[255, 210, 0],
[0, 255, 44],
[165, 255, 0],
[0, 25, 255],
[0, 255, 140],
[0, 101, 255],
[0, 255, 82],
[223, 255, 0],
[242, 0, 255],
[89, 0, 255],
[165, 0, 255],
[70, 255, 0],
[255, 0, 172],
[255, 76, 0],
[203, 255, 0],
[204, 0, 255],
[255, 0, 229],
[255, 133, 0],
[127, 0, 255],
[0, 235, 255],
[0, 255, 197],
[255, 0, 191],
[0, 44, 255],
[50, 255, 0],
]
================================================
FILE: utils/image.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
import cv2
import numpy as np
def resize_image(image, desired_width):
current_width = image.shape[1]
scale_percent = desired_width / current_width
width = int(image.shape[1] * scale_percent)
height = int(image.shape[0] * scale_percent)
resized = cv2.resize(image, (width, height), interpolation=cv2.INTER_AREA)
return resized
def compress_image(image, grayscale=True, desired_width=416, top_crop_percent=0.45):
if grayscale:
image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
image = resize_image(image, desired_width)
height = image.shape[0]
if top_crop_percent:
image[: int(height * top_crop_percent)] = 128
return image
def image_from_bytes(byte_im):
nparr = np.frombuffer(byte_im, np.uint8)
img_np = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
return img_np
def image_to_jpeg_nparray(image, quality=[int(cv2.IMWRITE_JPEG_QUALITY), 95]):
is_success, im_buf_arr = cv2.imencode(".jpg", image, quality)
return im_buf_arr
def image_to_jpeg_bytes(image, quality=[int(cv2.IMWRITE_JPEG_QUALITY), 95]):
buf = image_to_jpeg_nparray(image, quality)
byte_im = buf.tobytes()
return byte_im
================================================
FILE: utils/queue.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
from multiprocessing.queues import Queue as mp_queue
import multiprocessing as mp
class SharedCounter(object):
""" A synchronized shared counter.
The locking done by multiprocessing.Value ensures that only a single
process or thread may read or write the in-memory ctypes object. However,
in order to do n += 1, Python performs a read followed by a write, so a
second process may read the old value before the new one is written by the
first process. The solution is to use a multiprocessing.Lock to guarantee
the atomicity of the modifications to Value.
This class comes almost entirely from Eli Bendersky's blog:
http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/
"""
def __init__(self, n=0):
self.count = mp.Value("i", n)
def increment(self, n=1):
""" Increment the counter by n (default = 1) """
with self.count.get_lock():
self.count.value += n
def reset(self):
""" Reset the counter to 0 """
with self.count.get_lock():
self.count.value = 0
@property
def value(self):
""" Return the value of the counter """
return self.count.value
class MPQueue(mp_queue):
""" A portable implementation of multiprocessing.Queue.
Because of multithreading / multiprocessing semantics, Queue.qsize() may
raise the NotImplementedError exception on Unix platforms like Mac OS X
where sem_getvalue() is not implemented. This subclass addresses this
problem by using a synchronized shared counter (initialized to zero) and
increasing / decreasing its value every time the put() and get() methods
are called, respectively. This not only prevents NotImplementedError from
being raised, but also allows us to implement a reliable version of both
qsize() and empty().
"""
def __init__(self, *args, **kwargs):
self.ctx = mp.get_context()
super(MPQueue, self).__init__(*args, **kwargs, ctx=self.ctx)
self.size = SharedCounter(0)
def put(self, *args, **kwargs):
self.size.increment(1)
super(MPQueue, self).put(*args, **kwargs)
def get(self, *args, **kwargs):
self.size.increment(-1)
if self.size.value < 0:
self.size.reset()
return super(MPQueue, self).get(*args, **kwargs)
def qsize(self):
""" Reliable implementation of multiprocessing.Queue.qsize() """
return self.size.value
def empty(self):
""" Reliable implementation of multiprocessing.Queue.empty() """
return not self.qsize()
================================================
FILE: workers.py
================================================
# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`
from utils.image import (
resize_image,
compress_image,
image_from_bytes,
image_to_jpeg_nparray,
image_to_jpeg_bytes,
)
from utils.bbox import BoundBox, draw_boxes
from statistics import mean
import time, base64, pickle, json, cv2, logging, requests, queue, broadcast, copy, statistics
import numpy as np
import threading as td
import multiprocessing as mp
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
session = requests.Session()
class WorkerTemplateThread(td.Thread):
def __init__(self, event_stopper, name=None, runnable=None):
td.Thread.__init__(self, name=name)
self.event_stopper = event_stopper
self.runnable = runnable
def run(self):
if self.runnable:
logger.debug("worker started")
while not self.event_stopper.is_set():
self.runnable()
time.sleep(0.030)
logger.debug("worker stopped")
def stop(self):
self.event_stopper.set()
class WorkerTemplateProcess(mp.Process):
def __init__(self, event_stopper, name=None, runnable=None):
mp.Process.__init__(self, name=name)
self.event_stopper = event_stopper
self.runnable = runnable
def run(self):
if self.runnable:
logger.debug("worker started")
while not self.event_stopper.is_set():
self.runnable()
time.sleep(0.030)
logger.debug("worker stopped")
def stop(self):
self.event_stopper.set()
class BroadcastReassembled(WorkerTemplateProcess):
"""
Separate process to broadcast the stream with the overlayed predictions on top of it.
"""
def __init__(self, in_queue, cfg, name=None):
"""
in_queue - Queue from which to extract the frames with the overlayed predictions on top of it.
cfg - The dictionary config for the broadcaster.
name - Name of the process.
"""
super(BroadcastReassembled, self).__init__(event_stopper=mp.Event(), name=name)
self.in_queue = in_queue
self.yolo3_rtt = None
self.crnn_rtt = None
self.detections = 0
self.current_detections = 0
self.recognitions = 0
self.current_recognitions = 0
self.buffer = []
self.oldest_broadcasted_frame = 0
for key, value in cfg.items():
setattr(self, key, value)
def run(self):
# start streaming server
def lambda_func():
server = broadcast.StreamingServer(
tuple(self.serve_address), broadcast.StreamingHandler
)
server.serve_forever()
td.Thread(target=lambda_func, args=(), daemon=True).start()
logger.info("listening for stream clients on {}".format(self.serve_address))
# start polling for new processed frames from the queue and broadcast
logger.info("worker started")
counter = 0
while not self.event_stopper.is_set():
if counter == self.target_fps:
logger.debug("buffer queue size: {}".format(len(self.buffer)))
counter = 0
self.reassemble()
time.sleep(0.001)
counter += 1
logger.info("worker stopped")
def reassemble(self):
"""
Main method to run in the loop.
"""
start = time.time()
self.pull_and_push()
self.purge_stale_frames()
frame, delay = self.pick_new_frame()
# delay loop to stabilize the video fps
end = time.time()
elapsed_time = end - start
elapsed_time += 0.001 # count in the millisecond in self.run
if delay - elapsed_time > 0.0:
time.sleep(delay - elapsed_time)
if frame:
# pull and push again in case
# write buffer (assume it takes an insignificant time to execute)
self.pull_and_push()
broadcast.output.write(frame)
def pull_and_push(self):
"""
Get new frame and push it in the broadcaster's little buffer for stabilization.
"""
try:
data = self.in_queue.get_nowait()
except queue.Empty:
# logger.warning("no data available for worker")
return
# extract data
boxes = data["boxes"]
frame_num = data["frame_num"]
yolo3_rtt = data["avg_yolo3_rtt"]
crnn_rtt = data["avg_crnn_rtt"]
byte_im = data["image"]
# run statistics
self.statistics(yolo3_rtt, crnn_rtt, len(boxes), 0)
# push frames to buffer and pick new frame
self.buffer.append({"image": byte_im, "frame_num": frame_num})
def purge_stale_frames(self):
"""
Remove any frames older than the latest broadcasted frame.
"""
new_buffer = []
for frame in self.buffer:
if frame["frame_num"] > self.oldest_broadcasted_frame:
new_buffer.append(frame)
self.buffer = new_buffer
def pick_new_frame(self):
"""
Get the oldest frame from the buffer that isn't older than the last broadcasted frame.
"""
current_desired_fps = self.target_fps - self.max_fps_variation
delay = 1 / current_desired_fps
if len(self.buffer) == 0:
return None, delay
newlist = sorted(self.buffer, key=lambda k: k["frame_num"])
idx_to_del = 0
for idx, frame in enumerate(newlist):
if frame["frame_num"] < self.oldest_broadcasted_frame:
idx_to_del = idx + 1
newlist = newlist[idx_to_del:]
if len(newlist) == 0:
return None, delay
self.buffer = newlist[::-1]
element = self.buffer.pop()
frame = element["image"]
self.oldest_broadcasted_frame = element["frame_num"]
size = len(self.buffer)
variation = size - self.target_buffer_size
var_perc = variation / self.max_buffer_size_variation
current_desired_fps = self.target_fps + var_perc * self.max_fps_variation
if current_desired_fps < 0:
current_desired_fps = self.target_fps - self.max_fps_variation
try:
delay = 1 / current_desired_fps
except ZeroDivisionError:
current_desired_fps = self.target_fps - self.max_fps_variation
delay = 1 / current_desired_fps
return frame, delay
def statistics(self, yolo3_rtt, crnn_rtt, detections, recognitions):
"""
A bunch of RTT and detection/recognition statistics. Not used.
"""
if not self.yolo3_rtt:
self.yolo3_rtt = yolo3_rtt
else:
self.yolo3_rtt = self.yolo3_rtt * 0.98 + yolo3_rtt * 0.02
if not self.crnn_rtt:
self.crnn_rtt = crnn_rtt
else:
self.crnn_rtt = self.crnn_rtt * 0.98 + crnn_rtt * 0.02
self.detections += detections
self.current_detections = detections
self.recognitions += recognitions
self.current_recognitions = recognitions
class InferenceWorker(WorkerTemplateThread):
"""
Worker that receives frames from a queue, sends requests to 2 cortex APIs for inference reasons,
and retrieves the results and puts them in their appropriate queues.
"""
def __init__(self, event_stopper, in_queue, bc_queue, predicts_queue, cfg, name=None):
"""
event_stopper - Event to stop the worker.
in_queue - Queue that holds the unprocessed frames.
bc_queue - Queue to push into the frames with the overlayed predictions.
predicts_queue - Queue to push into the detected license plates that will eventually get written to the disk.
cfg - Dictionary config for the worker.
name - Name of the worker thread.
"""
super(InferenceWorker, self).__init__(event_stopper=event_stopper, name=name)
self.in_queue = in_queue
self.bc_queue = bc_queue
self.predicts_queue = predicts_queue
self.rtt_yolo3_ms = None
self.rtt_crnn_ms = 0
self.runnable = self.cloud_infer
for key, value in cfg.items():
setattr(self, key, value)
def cloud_infer(self):
"""
Main method that runs in the loop.
"""
try:
data = self.in_queue.get_nowait()
except queue.Empty:
# logger.warning("no data available for worker")
return
#############################
# extract frame
frame_num = data["frame_num"]
img = data["jpeg"]
# preprocess/compress the image
image = image_from_bytes(img)
reduced = compress_image(image)
byte_im = image_to_jpeg_bytes(reduced)
# encode image
img_enc = base64.b64encode(byte_im).decode("utf-8")
img_dump = json.dumps({"img": img_enc})
# make inference request
resp = self.yolov3_api_request(img_dump)
if not resp:
return
#############################
# parse response
r_dict = resp.json()
boxes_raw = r_dict["boxes"]
boxes = []
for b in boxes_raw:
box = BoundBox(*b)
boxes.append(box)
# purge bounding boxes with a low confidence score
aux = []
for b in boxes:
label = -1
for i in range(len(b.classes)):
if b.classes[i] > self.yolov3_obj_thresh:
label = i
if label >= 0:
aux.append(b)
boxes = aux
del aux
# also scale the boxes for later uses
camera_source_width = image.shape[1]
boxes640 = self.scale_bbox(boxes, self.yolov3_input_size_px, self.bounding_boxes_upscale_px)
boxes_source = self.scale_bbox(boxes, self.yolov3_input_size_px, camera_source_width)
#############################
# recognize the license plates in case
# any bounding boxes have been detected
dec_words = []
if len(boxes) > 0 and len(self.api_endpoint_crnn) > 0:
# create set of images of the detected license plates
lps = []
try:
for b in boxes_source:
lp = image[b.ymin : b.ymax, b.xmin : b.xmax]
jpeg = image_to_jpeg_nparray(
lp, [int(cv2.IMWRITE_JPEG_QUALITY), self.crnn_quality]
)
lps.append(jpeg)
except:
logger.warning("encountered error while converting to jpeg")
pass
lps = pickle.dumps(lps, protocol=0)
lps_enc = base64.b64encode(lps).decode("utf-8")
lps_dump = json.dumps({"imgs": lps_enc})
# make request to rcnn API
dec_lps = self.rcnn_api_request(lps_dump)
dec_lps = self.reorder_recognized_words(dec_lps)
for dec_lp in dec_lps:
dec_words.append([word[0] for word in dec_lp])
if len(dec_words) > 0:
logger.info("Detected the following words: {}".format(dec_words))
else:
dec_words = [[] for i in range(len(boxes))]
#############################
# draw detections
upscaled = resize_image(image, self.bounding_boxes_upscale_px)
draw_image = draw_boxes(
upscaled,
boxes640,
overlay_text=dec_words,
labels=["LP"],
obj_thresh=self.yolov3_obj_thresh,
)
draw_byte_im = image_to_jpeg_bytes(
draw_image, [int(cv2.IMWRITE_JPEG_QUALITY), self.broadcast_quality]
)
#############################
# push data for further processing in the queue
output = {
"boxes": boxes,
"frame_num": frame_num,
"avg_yolo3_rtt": self.rtt_yolo3_ms,
"avg_crnn_rtt": self.rtt_crnn_ms,
"image": draw_byte_im,
}
self.bc_queue.put(output)
# push predictions to write to disk
if len(dec_words) > 0:
timestamp = time.time()
literal_time = time.ctime(timestamp)
predicts = {"predicts": dec_words, "date": literal_time}
self.predicts_queue.put(predicts)
logger.info(
"Frame Count: {} - Avg YOLO3 RTT: {}ms - Avg CRNN RTT: {}ms - Detected: {}".format(
frame_num, int(self.rtt_yolo3_ms), int(self.rtt_crnn_ms), len(boxes)
)
)
def scale_bbox(self, boxes, old_width, new_width):
"""
Scale a bounding box.
"""
boxes = copy.deepcopy(boxes)
scale_percent = new_width / old_width
for b in boxes:
b.xmin = int(b.xmin * scale_percent)
b.ymin = int(b.ymin * scale_percent)
b.xmax = int(b.xmax * scale_percent)
b.ymax = int(b.ymax * scale_percent)
return boxes
def yolov3_api_request(self, img_dump):
"""
Make a request to the YOLOv3 API.
"""
# make inference request
try:
start = time.time()
resp = None
resp = session.post(
self.api_endpoint_yolov3,
data=img_dump,
headers={"content-type": "application/json"},
timeout=self.timeout,
)
except requests.exceptions.Timeout as e:
logger.warning("timeout on yolov3 inference request")
time.sleep(0.10)
return None
except Exception as e:
time.sleep(0.10)
logger.warning("timing/connection error on yolov3", exc_info=True)
return None
finally:
end = time.time()
if not resp:
pass
elif resp.status_code != 200:
logger.warning("received {} status code from yolov3 api".format(resp.status_code))
return None
# calculate average rtt (use complementary filter)
current = int((end - start) * 1000)
if not self.rtt_yolo3_ms:
self.rtt_yolo3_ms = current
else:
self.rtt_yolo3_ms = self.rtt_yolo3_ms * 0.98 + current * 0.02
return resp
def rcnn_api_request(self, lps_dump, timeout=1.200):
"""
Make a request to the CRNN API.
"""
# make request to rcnn API
try:
start = time.time()
resp = None
resp = session.post(
self.api_endpoint_crnn,
data=lps_dump,
headers={"content-type": "application/json"},
timeout=self.timeout,
)
except requests.exceptions.Timeout as e:
logger.warning("timeout on crnn inference request")
except:
logger.warning("timing/connection error on crnn", exc_info=True)
finally:
end = time.time()
dec_lps = []
if not resp:
pass
elif resp.status_code != 200:
logger.warning("received {} status code from crnn api".format(resp.status_code))
else:
r_dict = resp.json()
dec_lps = r_dict["license-plates"]
# calculate average rtt (use complementary filter)
current = int((end - start) * 1000)
self.rtt_crnn_ms = self.rtt_crnn_ms * 0.98 + current * 0.02
return dec_lps
def reorder_recognized_words(self, detected_images):
"""
Reorder the detected words in each image based on the average horizontal position of each word.
Sorting them in ascending order.
"""
reordered_images = []
for detected_image in detected_images:
# computing the mean average position for each word
mean_horizontal_positions = []
for words in detected_image:
box = words[1]
y_positions = [point[0] for point in box]
mean_y_position = mean(y_positions)
mean_horizontal_positions.append(mean_y_position)
indexes = np.argsort(mean_horizontal_positions)
# and reordering them
reordered = []
for index, words in zip(indexes, detected_image):
reordered.append(detected_image[index])
reordered_images.append(reordered)
return reordered_images
class Flusher(WorkerTemplateThread):
"""
Thread which removes the elements of a queue when its size crosses a threshold.
Used when there are too many frames are pilling up in the queue.
"""
def __init__(self, queue, threshold, name=None):
"""
queue - Queue to remove the elements from when the threshold is triggered.
threshold - Number of elements.
name - Name of the thread.
"""
super(Flusher, self).__init__(event_stopper=td.Event(), name=name)
self.queue = queue
self.threshold = threshold
self.runnable = self.flush_pipe
def flush_pipe(self):
"""
Main method to run in the loop.
"""
current = self.queue.qsize()
if current > self.threshold:
try:
for i in range(current):
self.queue.get_nowait()
logger.warning("flushed {} elements from the frames queue".format(current))
except queue.Empty:
logger.debug("flushed too many elements from the queue")
time.sleep(0.5)
gitextract_0wtqovzi/ ├── README.md ├── app.py ├── broadcast.py ├── config.json ├── deps/ │ ├── requirements_base.txt │ ├── requirements_dpkg_rpi.txt │ └── requirements_rpi.txt ├── drawings/ │ └── License Plate Identifier Client.drawio ├── gps.py ├── utils/ │ ├── bbox.py │ ├── colors.py │ ├── image.py │ └── queue.py └── workers.py
SYMBOL INDEX (75 symbols across 8 files)
FILE: app.py
class GracefullKiller (line 31) | class GracefullKiller:
method __init__ (line 38) | def __init__(self):
method exit_gracefully (line 42) | def exit_gracefully(self, signum, frame):
class WorkerPool (line 46) | class WorkerPool(mp.Process):
method __init__ (line 51) | def __init__(self, name, worker, pool_size, *args, **kwargs):
method run (line 64) | def run(self):
method stop (line 81) | def stop(self):
class DistributeFramesAndInfer (line 85) | class DistributeFramesAndInfer:
method __init__ (line 93) | def __init__(self, pool_cfg, worker_cfg):
method write (line 115) | def write(self, buf):
method stop (line 126) | def stop(self):
method get_queues (line 135) | def get_queues(self):
function main (line 149) | def main(config):
FILE: broadcast.py
class StreamingOutput (line 24) | class StreamingOutput(object):
method __init__ (line 25) | def __init__(self):
method write (line 30) | def write(self, buf):
class StreamingHandler (line 42) | class StreamingHandler(server.BaseHTTPRequestHandler):
method do_GET (line 43) | def do_GET(self):
method set_output (line 79) | def set_output(output):
class StreamingServer (line 83) | class StreamingServer(socketserver.ThreadingMixIn, server.HTTPServer):
FILE: gps.py
class ReadGPSData (line 9) | class ReadGPSData(td.Thread):
method __init__ (line 16) | def __init__(self, write_port, read_port, baudrate, name="GPS"):
method run (line 30) | def run(self):
method parsed (line 59) | def parsed(self):
method latitude (line 73) | def latitude(self):
method longitude (line 87) | def longitude(self):
method stop (line 100) | def stop(self):
FILE: utils/bbox.py
class BoundBox (line 8) | class BoundBox:
method __init__ (line 9) | def __init__(self, xmin, ymin, xmax, ymax, c=None, classes=None):
method get_label (line 21) | def get_label(self):
method get_score (line 27) | def get_score(self):
function draw_boxes (line 34) | def draw_boxes(image, boxes, overlay_text, labels, obj_thresh, quiet=True):
FILE: utils/colors.py
function get_color (line 4) | def get_color(label):
FILE: utils/image.py
function resize_image (line 7) | def resize_image(image, desired_width):
function compress_image (line 16) | def compress_image(image, grayscale=True, desired_width=416, top_crop_pe...
function image_from_bytes (line 27) | def image_from_bytes(byte_im):
function image_to_jpeg_nparray (line 33) | def image_to_jpeg_nparray(image, quality=[int(cv2.IMWRITE_JPEG_QUALITY),...
function image_to_jpeg_bytes (line 38) | def image_to_jpeg_bytes(image, quality=[int(cv2.IMWRITE_JPEG_QUALITY), 9...
FILE: utils/queue.py
class SharedCounter (line 7) | class SharedCounter(object):
method __init__ (line 22) | def __init__(self, n=0):
method increment (line 25) | def increment(self, n=1):
method reset (line 30) | def reset(self):
method value (line 36) | def value(self):
class MPQueue (line 41) | class MPQueue(mp_queue):
method __init__ (line 55) | def __init__(self, *args, **kwargs):
method put (line 60) | def put(self, *args, **kwargs):
method get (line 64) | def get(self, *args, **kwargs):
method qsize (line 70) | def qsize(self):
method empty (line 74) | def empty(self):
FILE: workers.py
class WorkerTemplateThread (line 24) | class WorkerTemplateThread(td.Thread):
method __init__ (line 25) | def __init__(self, event_stopper, name=None, runnable=None):
method run (line 30) | def run(self):
method stop (line 38) | def stop(self):
class WorkerTemplateProcess (line 42) | class WorkerTemplateProcess(mp.Process):
method __init__ (line 43) | def __init__(self, event_stopper, name=None, runnable=None):
method run (line 48) | def run(self):
method stop (line 56) | def stop(self):
class BroadcastReassembled (line 60) | class BroadcastReassembled(WorkerTemplateProcess):
method __init__ (line 65) | def __init__(self, in_queue, cfg, name=None):
method run (line 85) | def run(self):
method reassemble (line 108) | def reassemble(self):
method pull_and_push (line 130) | def pull_and_push(self):
method purge_stale_frames (line 153) | def purge_stale_frames(self):
method pick_new_frame (line 163) | def pick_new_frame(self):
method statistics (line 201) | def statistics(self, yolo3_rtt, crnn_rtt, detections, recognitions):
class InferenceWorker (line 220) | class InferenceWorker(WorkerTemplateThread):
method __init__ (line 226) | def __init__(self, event_stopper, in_queue, bc_queue, predicts_queue, ...
method cloud_infer (line 246) | def cloud_infer(self):
method scale_bbox (line 375) | def scale_bbox(self, boxes, old_width, new_width):
method yolov3_api_request (line 388) | def yolov3_api_request(self, img_dump):
method rcnn_api_request (line 427) | def rcnn_api_request(self, lps_dump, timeout=1.200):
method reorder_recognized_words (line 462) | def reorder_recognized_words(self, detected_images):
class Flusher (line 489) | class Flusher(WorkerTemplateThread):
method __init__ (line 495) | def __init__(self, queue, threshold, name=None):
method flush_pipe (line 506) | def flush_pipe(self):
Condensed preview — 14 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (63K chars).
[
{
"path": "README.md",
"chars": 10750,
"preview": "# Client for License Plate Identification in Real-time on AWS w/ Cortex [![License: MIT](https://img.shields.io/badge/Li"
},
{
"path": "app.py",
"chars": 11910,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "broadcast.py",
"chars": 2840,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "config.json",
"chars": 1248,
"preview": "{\n \"video_source\": {\n \"type\": \"file\",\n \"input\": \"airport_ride_480p.mp4\",\n \"scale_video\": 1.0,\n "
},
{
"path": "deps/requirements_base.txt",
"chars": 161,
"preview": "certifi==2019.11.28\nidna==2.8\nnumpy==1.18.1\nopencv-contrib-python==4.1.0.25\nrequests==2.22.0\nrequests-toolbelt==0.9.1\nur"
},
{
"path": "deps/requirements_dpkg_rpi.txt",
"chars": 417,
"preview": "build-essential\ncmake\nunzip\npkg-config\nlibjpeg-dev\nlibtiff5-dev\nlibjasper-dev\nlibpng-dev\nlibavcodec-dev\nlibavformat-dev\n"
},
{
"path": "deps/requirements_rpi.txt",
"chars": 176,
"preview": "certifi==2019.11.28\nidna==2.8\nnumpy==1.18.1\nopencv-contrib-python==4.1.0.25\npicamera==1.13\nrequests==2.22.0\nrequests-too"
},
{
"path": "drawings/License Plate Identifier Client.drawio",
"chars": 2844,
"preview": "<mxfile host=\"www.draw.io\" modified=\"2020-02-13T20:55:49.174Z\" agent=\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) Appl"
},
{
"path": "gps.py",
"chars": 2873,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "utils/bbox.py",
"chars": 2684,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "utils/colors.py",
"chars": 2096,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "utils/image.py",
"chars": 1295,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "utils/queue.py",
"chars": 2742,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
},
{
"path": "workers.py",
"chars": 17730,
"preview": "# WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version`\n\n"
}
]
About this extraction
This page contains the full source code of the RobertLucian/cortex-license-plate-reader-client GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 14 files (58.4 KB), approximately 15.8k tokens, and a symbol index with 75 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.