Repository: RobertLucian/cortex-license-plate-reader-client Branch: master Commit: bccaabd0d4a9 Files: 14 Total size: 58.4 KB Directory structure: gitextract_0wtqovzi/ ├── README.md ├── app.py ├── broadcast.py ├── config.json ├── deps/ │ ├── requirements_base.txt │ ├── requirements_dpkg_rpi.txt │ └── requirements_rpi.txt ├── drawings/ │ └── License Plate Identifier Client.drawio ├── gps.py ├── utils/ │ ├── bbox.py │ ├── colors.py │ ├── image.py │ └── queue.py └── workers.py ================================================ FILE CONTENTS ================================================ ================================================ FILE: README.md ================================================ # Client for License Plate Identification in Real-time on AWS w/ Cortex [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://robertlucian.mit-license.org) **READ THIS: This is a client for 3 (YOLOv3, CRAFT text detector, CRNN text recognizer) [cortex](https://github.com/cortexlabs/cortex)-deployed ML models. This client only works in conjunction with [these cortex APIs](https://github.com/cortexlabs/cortex/tree/master/examples/tensorflow/license-plate-reader).** ![Imgur](https://i.imgur.com/jgkJB59.gif) *- The above GIF was taken from [this video](https://www.youtube.com/watch?v=gsYEZtecXlA) of whose predictions were computed on the fly with cortex/AWS -* ## Description This app which uses the deployed cortex APIs as a PaaS captures the frames from a video camera, sends them for inferencing to the cortex APIs, recombines them after the responses are received and then the detections/recognitions are overlayed on the output stream. This is done on the car's dashcam (composed of the Raspberry Pi + GSM module) in real-time. Access to the internet is provided through the GSM module's 4G connection. The app must be configured to use the API endpoints as shown when calling `cortex get yolov3` and `cortex get crnn`. Checkout how the APIs are defined in [their repository](https://github.com/cortexlabs/cortex/tree/master/examples/tensorflow/license-plate-reader). The app also saves a `csv` file containing the dates and GPS coordinates of each identified license plate. ### Latency The observable latency between capturing the frame and broadcasting the predictions in the browser (with all the inference stuff going on) takes about *0.5-1.0 seconds* depending on: * How many replicas are assigned for each API. * Internet connection bandwidth and latency. * Broadcast buffer size. To get a smoother stream, use a higher buffer size (*10-30*) or if you want the stream to be displayed as quickly as possible but with possible dropped frames, go with lower values (*<10*). To learn more about how the actual device was constructed, check out [this](https://towardsdatascience.com/i-built-a-diy-license-plate-reader-with-a-raspberry-pi-and-machine-learning-7e428d3c7401) article. ## Target Machine Target machine **1**: Raspberry Pi. Target machine **2**: Any x86 machine. --- The app's primary target machine is the *Raspberry Pi* (3/3B+/4) - the Raspberry Pi is a small embedded computer system that got adopted by many hobbyists and institutions all around the world as the de-facto choice for hardware/software experiments. Unfortunately for the Raspberry Pi, the $35 pocket-sized computer, doesn't have enough oomph (far from it) to do any inference: not only it doesn't have enough memory to load a model, but should it have enough RAM, it would still take dozens of minutes just to get a single inference. Let alone run inferences at 30 FPS, with a number of inferences for each frame. The app is built to be used with a *Pi Camera* alongside the *Raspberry Pi*. The details on how to build such a system are found [here](#creating-your-own-device). Since many developers don't have a Raspberry Pi laying around or are just purely interested in seeing results right away, the app can also be configured to take in a video file and treat it exactly as if it was a camera. ## Dependencies For either target machine, the minimum version of Python has to be `3.6.x`. #### For the Raspberry Pi ```bash pip3 install --user -r deps/requirements_rpi.txt sudo apt-get update # dependencies for opencv and pandas package sudo apt-get install --no-install-recommends $(cat deps/requirements_dpkg_rpi.txt) ``` #### For x86 Machine ```bash pip3 install -r deps/requirements_base.txt ``` ## Configuring The configuration file can be in this form ```jsonc { "video_source": { // "file" for reading from file or "camera" for pi camera "type": "file", // video file to read from; applicable just for "file" type "input": "airport_ride_480p.mp4", // scaling for the video file; applicable just for "file" type "scale_video": 1.0, // how many frames to skip on the video file; applicable just for "file" type "frames_to_skip": 0, // framerate "framerate": 30 // camera sensor mode; applicable just for "camera" type // "sensor_mode": 5 // where to save camera's output; applicable just for "camera" type // "output_file": "recording.h264" // resolution of the input video; applicable just for the "camera" type // "resolution": [480, 270] }, "broadcaster": { // when broadcasting, a buffer is required to provide framerate fluidity; measured in frames "target_buffer_size": 10, // how much the size of the buffer can variate +- "max_buffer_size_variation": 5, // the maximum variation of the fps when extracting frames from the queue "max_fps_variation": 15, // target fps - must match the camera/file's framerate "target_fps": 30, // address to bind the web server to "serve_address": ["0.0.0.0", 8000] }, "inferencing_worker": { // YOLOv3's input size of the image in pixels (must consider the existing model) "yolov3_input_size_px": 416, // when drawing the bounding boxes, use a higher res image to draw boxes more precisely // (this way text is more readable) "bounding_boxes_upscale_px": 640, // object detection accuracy threshold in percentages with range (0, 1) "yolov3_obj_thresh": 0.8, // the jpeg quality of the image for the CRAFT/CRNN models in percentanges; // these models receive the cropped images of each detected license plate "crnn_quality": 98, // broadcast quality - aim for a lower value since this stream doesn't influence the predictions; measured in percentages "broadcast_quality": 90, // connection timeout for both API endpoints measured in seconds "timeout": 1.20, // YOLOv3 API endpoint "api_endpoint_yolov3": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/yolov3", // CRNN API endpoint // Can be set to "" value to turn off the recognition inference // By turning it off, the latency is reduced and the output video appears smoother "api_endpoint_crnn": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/crnn" }, "inferencing_pool": { // number of workers to do inferencing (YOLOv3 + CRAFT + CRNN) // depending on the source's framerate, a balance must be achieved "workers": 24, // pick the nth frame from the input stream // if the input stream runs at 30 fps, then setting this to 2 would act // as if the input stream runs at 30/2=15 fps // ideally, you have a high fps camera (90-180) and you only pick every 3rd-6th frame "pick_every_nth_frame": 1 }, "flusher": { // if there are more than this given number of frames in the input stream's buffer, flush them // it's useful if the inference workers (due to a number of reasons) can't keep up with the input flow // also keeps the current broadcasted stream up-to-date with the reality "frame_count_threshold": 5 }, "gps": { // set to false when using a video file to read the stream from // set to true when you have a GPS connected to your system "use_gps": false, // port to write to to activate the GPS (built for EC25-E modules) "write_port": "/dev/ttyUSB2", // port to read from in NMEA standard "read_port": "/dev/ttyUSB1", // baudrate as measured in bits/s "baudrate": 115200 }, "general": { // to which IP the requests module is bound to // useful if you only want to route the traffic through a specific interface "bind_ip": "0.0.0.0", // where to save the csv data containing the date, the predicted license plate number and GPS data // can be an empty string, in which case, nothing is stored "saved_data": "saved_data.csv" } } ``` Be aware that for having a functional application, the minimum amount of things that have to be adjusted in the config file are: 1. The input file in case you are using a video to feed the application with. You can download the following `mp4` video file to use as input. Download it by running `wget -O airport_ride_480p.mp4 "https://www.dropbox.com/s/q9j57y5k95wg2zt/airport_ride_480p.mp4?dl=0"` 1. Both API endpoints from your cortex APIs. Use `cortex get your-api-name-here` command to get those. ## Running It Make sure both APIs are already running in the cluster. Launching it the first time might raise some timeout exceptions, but let it run for a few moments. If there's enough compute capacity, you'll start getting `200` response codes. Run it like ```bash python app.py -c config.json ``` Once it's running, you can head off to its browser page to see the live broadcast with its predictions overlayed on top. To save the broascasted MJPEG stream, you can run the following command (check the broadcast serve address in the `config.json` file) ```bash PORT=8000 FRAMERATE=30 ffmpeg -i http://localhost:PORT/stream.mjpg -an -vcodec libx264 -r FRAMERATE saved_video.h264 ``` To terminate the app, press `CTRL-C` and wait a bit. ## Creating Your Own Device ![Imgur](https://i.imgur.com/MvDAXWU.jpg) To create your own Raspberry Pi-powered device to record and display the predictions in real time in your car, you're gonna need the following things: 1. A Raspberry Pi - preferably a 4, because that one has more oomph. 1. A Pi Camera - doesn't matter which version of it. 1. A good buck converter to step-down from 12V down to 5V - aim for 4-5 amps. You can use a SBEC/UBEC/BEC regulators - they are easy to find and cheap. 1. A power outlet for the car's cigarette port to get the 12V DC. 1. 4G/GPS shield to host a GSM module - in this project, an EC25-E module has been used. You will also need antennas. 1. A 3D-printed support to hold the electronics and be able to hold it against the rear mirror or dashboard. Must be built to accomodate to your own car. Without convoluting this README too much: * Here are the [STLs/SLDPRTs/Renders](https://www.dropbox.com/sh/fw16vy1okrp606y/AAAwkoWXODmoaOP4yR-z4T8Va?dl=0) to the car's 3D printed support. * Here's an [article](https://towardsdatascience.com/i-built-a-diy-license-plate-reader-with-a-raspberry-pi-and-machine-learning-7e428d3c7401) that talks about this in full detail. ================================================ FILE: app.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` import signal, os, time, json, queue, socket, click, cv2, pandas as pd import multiprocessing as mp import threading as td import logging logger = logging.getLogger() stream_handler = logging.StreamHandler() stream_handler.setLevel(logging.INFO) stream_format = logging.Formatter( "%(asctime)s - %(name)s - %(threadName)s - %(levelname)s - %(message)s" ) stream_handler.setFormatter(stream_format) logger.addHandler(stream_handler) logger.setLevel(logging.DEBUG) disable_loggers = ["urllib3.connectionpool"] for name, logger in logging.root.manager.loggerDict.items(): if name in disable_loggers: logger.disabled = True from gps import ReadGPSData from workers import BroadcastReassembled, InferenceWorker, Flusher, session from utils.image import resize_image, image_to_jpeg_bytes from utils.queue import MPQueue from requests_toolbelt.adapters.source import SourceAddressAdapter class GracefullKiller: """ For killing the app gracefully. """ kill_now = False def __init__(self): signal.signal(signal.SIGINT, self.exit_gracefully) signal.signal(signal.SIGTERM, self.exit_gracefully) def exit_gracefully(self, signum, frame): self.kill_now = True class WorkerPool(mp.Process): """ Pool of threads running in a different process. """ def __init__(self, name, worker, pool_size, *args, **kwargs): """ name - Name of the process. worker - Derived class of thread to execute. pool_size - Number of workers to have. """ super(WorkerPool, self).__init__(name=name) self.event_stopper = mp.Event() self.Worker = worker self.pool_size = pool_size self.args = args self.kwargs = kwargs def run(self): logger.info("spawning workers on separate process") pool = [ self.Worker( self.event_stopper, *self.args, **self.kwargs, name="{}-Worker-{}".format(self.name, i), ) for i in range(self.pool_size) ] [worker.start() for worker in pool] while not self.event_stopper.is_set(): time.sleep(0.001) logger.info("stoppping workers on separate process") [worker.join() for worker in pool] def stop(self): self.event_stopper.set() class DistributeFramesAndInfer: """ Custom output class primarly built for the PiCamera class. Has 3 process-safe queues: in_queue for the incoming frames from the source, bc_queue for the frames with the predicted overlays heading off to the broadcaster, predicts_queue for the predictions to be written off to the disk. """ def __init__(self, pool_cfg, worker_cfg): """ pool_cfg - Configuration dictionary for the pool manager. worker_cfg - Configuration dictionary for the pool workers. """ self.frame_num = 0 self.in_queue = MPQueue() self.bc_queue = MPQueue() self.predicts_queue = MPQueue() for key, value in pool_cfg.items(): setattr(self, key, value) self.pool = WorkerPool( "InferencePool", InferenceWorker, self.workers, self.in_queue, self.bc_queue, self.predicts_queue, worker_cfg, ) self.pool.start() def write(self, buf): """ Mandatory custom output method for the PiCamera class. buf - Frame as a bytes object. """ if buf.startswith(b"\xff\xd8"): # start of new frame; close the old one (if any) and if self.frame_num % self.pick_every_nth_frame == 0: self.in_queue.put({"frame_num": self.frame_num, "jpeg": buf}) self.frame_num += 1 def stop(self): """ Stop all workers and the process altogether. """ self.pool.stop() self.pool.join() qs = [self.in_queue, self.bc_queue] [q.cancel_join_thread() for q in qs] def get_queues(self): """ Retrieve all queues. """ return self.in_queue, self.bc_queue, self.predicts_queue @click.command( help=( "Identify license plates from a given video source" " while outsourcing the predictions using REST API endpoints." ) ) @click.option("--config", "-c", required=True, type=str) def main(config): killer = GracefullKiller() # open config file try: file = open(config) cfg = json.load(file) file.close() except Exception as error: logger.critical(str(error), exc_info=1) return # give meaningful names to each sub config source_cfg = cfg["video_source"] broadcast_cfg = cfg["broadcaster"] pool_cfg = cfg["inferencing_pool"] worker_cfg = cfg["inferencing_worker"] flusher_cfg = cfg["flusher"] gps_cfg = cfg["gps"] gen_cfg = cfg["general"] # bind requests module to use a given network interface try: socket.inet_aton(gen_cfg["bind_ip"]) session.mount("http://", SourceAddressAdapter(gen_cfg["bind_ip"])) logger.info("binding requests module to {} IP".format(gen_cfg["bind_ip"])) except OSError as e: logger.error("bind IP is invalid, resorting to default interface", exc_info=True) # start polling the GPS if gps_cfg["use_gps"]: wport = gps_cfg["write_port"] rport = gps_cfg["read_port"] br = gps_cfg["baudrate"] gps = ReadGPSData(wport, rport, br) gps.start() else: gps = None # workers on a separate process to run inference on the data logger.info("initializing pool w/ " + str(pool_cfg["workers"]) + " workers") output = DistributeFramesAndInfer(pool_cfg, worker_cfg) frames_queue, bc_queue, predicts_queue = output.get_queues() logger.info("initialized worker pool") # a single worker in a separate process to reassemble the data reassembler = BroadcastReassembled(bc_queue, broadcast_cfg, name="BroadcastReassembled") reassembler.start() # a single thread to flush the producing queue # when there are too many frames in the pipe flusher = Flusher(frames_queue, threshold=flusher_cfg["frame_count_threshold"], name="Flusher") flusher.start() # data aggregator to write things to disk def results_writer(): if len(gen_cfg["saved_data"]) > 0: df = pd.DataFrame(columns=["Date", "License Plate", "Coordinates"]) while not killer.kill_now: time.sleep(0.01) try: data = predicts_queue.get_nowait() except queue.Empty: continue predicts = data["predicts"] date = data["date"] for lp in predicts: if len(lp) > 0: lp = " ".join(lp) entry = {"Date": date, "License Plate": lp, "Coordinates": ""} if gps: entry["Coordinates"] = "{}, {}".format( gps.latitude, gps.longitude ).upper() df = df.append(entry, ignore_index=True) logger.info("dumping results to csv file {}".format(gen_cfg["saved_data"])) if os.path.isfile(gen_cfg["saved_data"]): header = False else: header = True with open(gen_cfg["saved_data"], "a") as f: df.to_csv(f, header=header) # data aggregator thread results_thread = td.Thread(target=results_writer) results_thread.start() if source_cfg["type"] == "camera": # import module import picamera # start the pi camera with picamera.PiCamera() as camera: # configure the camera camera.sensor_mode = source_cfg["sensor_mode"] camera.resolution = source_cfg["resolution"] camera.framerate = source_cfg["framerate"] logger.info( "picamera initialized w/ mode={} resolution={} framerate={}".format( camera.sensor_mode, camera.resolution, camera.framerate ) ) # start recording both to disk and to the queue camera.start_recording( output=source_cfg["output_file"], format="h264", splitter_port=0, bitrate=10000000, ) camera.start_recording( output=output, format="mjpeg", splitter_port=1, bitrate=10000000, quality=95, ) logger.info("started recording to file and to queue") # wait until SIGINT is detected while not killer.kill_now: camera.wait_recording(timeout=0.5, splitter_port=0) camera.wait_recording(timeout=0.5, splitter_port=1) logger.info( "frames qsize: {}, broadcast qsize: {}, predicts qsize: {}".format( frames_queue.qsize(), bc_queue.qsize(), predicts_queue.qsize() ) ) # stop recording logger.info("gracefully exiting") camera.stop_recording(splitter_port=0) camera.stop_recording(splitter_port=1) output.stop() elif source_cfg["type"] == "file": # open video file video_reader = cv2.VideoCapture(source_cfg["input"]) video_reader.set(cv2.CAP_PROP_POS_FRAMES, source_cfg["frames_to_skip"]) # get # of frames and determine target width nb_frames = int(video_reader.get(cv2.CAP_PROP_FRAME_COUNT)) frame_h = int(video_reader.get(cv2.CAP_PROP_FRAME_HEIGHT)) frame_w = int(video_reader.get(cv2.CAP_PROP_FRAME_WIDTH)) target_h = int(frame_h * source_cfg["scale_video"]) target_w = int(frame_w * source_cfg["scale_video"]) period = 1.0 / source_cfg["framerate"] logger.info( "file-based video stream initialized w/ resolution={} framerate={} and {} skipped frames".format( (target_w, target_h), source_cfg["framerate"], source_cfg["frames_to_skip"], ) ) # serve each frame to the workers iteratively last_log = time.time() for i in range(nb_frames): start = time.time() try: # write frame to queue _, frame = video_reader.read() if target_w != frame_w: frame = resize_image(frame, target_w) jpeg = image_to_jpeg_bytes(frame) output.write(jpeg) except Exception as error: logger.error("unexpected error occurred", exc_info=True) break end = time.time() spent = end - start left = period - spent if left > 0: # maintain framerate time.sleep(period) # check if SIGINT has been sent if killer.kill_now: break # do logs every second current = time.time() if current - last_log >= 1.0: logger.info( "frames qsize: {}, broadcast qsize: {}, predicts qsize: {}".format( frames_queue.qsize(), bc_queue.qsize(), predicts_queue.qsize() ) ) last_log = current logger.info("gracefully exiting") video_reader.release() output.stop() if gps_cfg["use_gps"]: gps.stop() reassembler.stop() flusher.stop() if __name__ == "__main__": main() ================================================ FILE: broadcast.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` import io import logging import socketserver from threading import Condition from http import server PAGE = """\ License Plate Predictions

Live License Plate Predictions

""" logger = logging.getLogger(__name__) class StreamingOutput(object): def __init__(self): self.frame = None self.buffer = io.BytesIO() self.condition = Condition() def write(self, buf): if buf.startswith(b"\xff\xd8"): # New frame, copy the existing buffer's content and notify all # clients it's available self.buffer.truncate() with self.condition: self.frame = self.buffer.getvalue() self.condition.notify_all() self.buffer.seek(0) return self.buffer.write(buf) class StreamingHandler(server.BaseHTTPRequestHandler): def do_GET(self): if self.path == "/": self.send_response(301) self.send_header("Location", "/index.html") self.end_headers() elif self.path == "/index.html": content = PAGE.encode("utf-8") self.send_response(200) self.send_header("Content-Type", "text/html") self.send_header("Content-Length", len(content)) self.end_headers() self.wfile.write(content) elif self.path == "/stream.mjpg": self.send_response(200) self.send_header("Age", 0) self.send_header("Cache-Control", "no-cache, private") self.send_header("Pragma", "no-cache") self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=FRAME") self.end_headers() try: while True: with output.condition: output.condition.wait() frame = output.frame self.wfile.write(b"--FRAME\r\n") self.send_header("Content-Type", "image/jpeg") self.send_header("Content-Length", len(frame)) self.end_headers() self.wfile.write(frame) self.wfile.write(b"\r\n") except Exception as e: logging.warning("Removed streaming client %s: %s", self.client_address, str(e)) else: self.send_error(404) self.end_headers() def set_output(output): self.output = output class StreamingServer(socketserver.ThreadingMixIn, server.HTTPServer): allow_reuse_address = True daemon_threads = True output = StreamingOutput() ================================================ FILE: config.json ================================================ { "video_source": { "type": "file", "input": "airport_ride_480p.mp4", "scale_video": 1.0, "frames_to_skip": 0, "framerate": 30 }, "broadcaster": { "target_buffer_size": 10, "max_buffer_size_variation": 5, "max_fps_variation": 15, "target_fps": 30, "serve_address": ["0.0.0.0", 8000] }, "inferencing_worker": { "yolov3_input_size_px": 416, "bounding_boxes_upscale_px": 640, "yolov3_obj_thresh": 0.8, "crnn_quality": 98, "broadcast_quality": 90, "timeout": 1.20, "api_endpoint_yolov3": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/yolov3", "api_endpoint_crnn": "http://a23893c574c0511ea9f430a8bed50c69-1100298247.eu-central-1.elb.amazonaws.com/crnn" }, "inferencing_pool": { "workers": 24, "pick_every_nth_frame": 1 }, "flusher": { "frame_count_threshold": 5 }, "gps": { "use_gps": false, "write_port": "/dev/ttyUSB2", "read_port": "/dev/ttyUSB1", "baudrate": 115200 }, "general": { "bind_ip": "0.0.0.0", "saved_data": "saved_data.csv" } } ================================================ FILE: deps/requirements_base.txt ================================================ certifi==2019.11.28 idna==2.8 numpy==1.18.1 opencv-contrib-python==4.1.0.25 requests==2.22.0 requests-toolbelt==0.9.1 urllib3==1.25.8 pynmea2==1.15.0 Click==7.0 ================================================ FILE: deps/requirements_dpkg_rpi.txt ================================================ build-essential cmake unzip pkg-config libjpeg-dev libtiff5-dev libjasper-dev libpng-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libfontconfig1-dev libcairo2-dev libgdk-pixbuf2.0-dev libpango1.0-dev libgtk2.0-dev libgtk-3-dev libatlas-base-dev gfortran libhdf5-dev libhdf5-serial-dev libhdf5-103 libqtgui4 libqtwebkit4 libqt4-test python3-pyqt5 python3-dev python3-pandas ================================================ FILE: deps/requirements_rpi.txt ================================================ certifi==2019.11.28 idna==2.8 numpy==1.18.1 opencv-contrib-python==4.1.0.25 picamera==1.13 requests==2.22.0 requests-toolbelt==0.9.1 urllib3==1.25.8 pynmea2==1.15.0 Click==7.0 ================================================ FILE: drawings/License Plate Identifier Client.drawio ================================================ 7Vxbc9o4GP01zLQP7UiWr48Bkm6n3Vm2dNtmX3YUrIC3xqK2aML++pVsGdtCECf4EkrSmWJdLGSdT+e7yQzQaHn/Lsarxe/UJ+HAAP79AI0HhgE92+UfomYja1zLyWrmceDLuqJiGvxHZCWQtevAJ0mlI6M0ZMGqWjmjUURmrFKH45jeVbvd0rD6rSs8JzsV0xkOd2u/Bj5bZLUGQl7R8BsJ5gv51QgBOfMbPPs+j+k6kl84MNBt+pc1L3E+mOyfLLBP70pV6HKARjGlLLta3o9IKFY3X7fsvqs9rduJxyRidW74+HcQLK4/vHHG3/768H7849M/DLyBpoTrJw7XJH+QdLpsk69R+pBEDAMGaHi3CBiZrvBMtN5xseB1C7YMeQnySx8ni7RvXphgxkgcpTUG4I89TFhMv5MRDWmcfgFfUvHHW25pxKSMQFuWS/3Msfgn6oMw/INPIWAbOasQ35BwQpOABVR814yvCuG3DX+SmAUc749KB0bFxHEYzLXdL2TDDWWMLnmDXCbeTO73IgC3uPIdQ+iSsHjDu+Q3WLnsyN1imrJ8V4ge8mTdoiR1Ti5EWIr7fDt4gTi/kKA/SgCsFwHoSgBMpOBv7eLvmd3ib6CG8T8Mbj0wa8hQE/sRwCoe2/1ZwsNwNXggqzU8QA08iM91mCzSmC3onEY4vCxqh1XEij4fqZD6dFn/JYxt5F7Da0arKO7Dii9tvPkmx00L16Lw1sqL4/ty4zhHNHsGMfHDYPHnpOt4Rg4skWdLOwHHc8IOdXT18MckxCz4WZ1JC0hafQD3VID2Ad4LcF6/wNm9AHcfsG+l6xJsvFSgJgqbMoQnDjYEoE+083lW+NYOmVylihjYP9Y0b3iTpEBe8A7QXt0XjfxqLj7fR7eELxtfJANMYjojSZIPzCeajZ31PKBv4cP6VmjPEpa3ACOAdXp4OHa0elh6LY2oVNepalTb2NWo0NBoVLs1hQrdPnZzwuWeXQgHVRiXIU6SYJZXXwVhwdb+bideWerSxV516+5VaB+5V9Nb+SPjTanDigYRS0ojT0RFyVAzFLGyLMXXfegG27EO32ChI2/wUOUGfpE9ZSG52+U6gqzcDsjqK42/c1eId22Fr3Z4yRg79kFeUhjOBtiDTlMugKtI1jMgLO9XNz9Oixy9EyNHG50pOXpdkqNxhuRoa+IjHZPjNpp+Ek71SREdzNE9GaZzwXky3RapbqguOkOqc5zeqQ5ADcgq9xXMEdGIqKHz/QHdq/SvQm0Fm12XaW8PteX2YmEjXpdNRK292AtZ7QsmloC1NLjmdY+jtF2CgErOz0GKwGSPKm87RDXQUrjJVUbKlmJnpOZYx2iGdVK6UYgnEjyzXt6khENvBT4p/dSOmfG9z6oS/2Bqbxn4fmYfED5DfJMOJeRUKiI+rjUcWELNCpMgkXnJA9pYTV82wky2IkBQw0y2RoLbyxkiuLP8nRphb02zaod5nvkQXYnShMQBXwIhDLvmmeTPsm0G6ttmdZLWjdKfU9dWA/1mwZBGaXUqK4/SaucgEd6xxvuRAnFSadFzEAiI0JES8SR3zlFMbu8Bb07tv9WM7fpaqJf8kd68zlv05vV5yKrTK3uZ/aizvoLzv6wUHXvC5EmM51qPYzy1//bAWsvRJd2Z4CaiS1MhRoP0OMQ8xstuwkoXFgAm2A0f+S4ADjoQbmrAeTMV383tP6oE60SVzvp8ofSYavlWjdDIzr631UiPety09UhPW+cMJgFvGuElifFAyP0V//9L4BPKP7nyEGHnaYaPAT4vYoL9bkgCAPvy4uoAGSjkAW8wJA0dQzZVtHVHEHQBnhZJoh8f7QlWTidmRd0TAD171vk8G9+1V1xZix3555rwgWtuyGSBV+KSLzkOQxJmKh8NV6UgXKWtFJ179P4dAs96TO6IoJkLQDP7165z5lH3VkeL+7eBQ62uThJGNF0oA1xM3nNDF7z6dDn9nBXFdov8zAR+3Q1rm8gYW0591B3bBi5uBnVoKbBvwxPll0eQBnYEW8Pd0Bl3j8PdsPR5YfH6DWFnn41B6pGYPJxZ3uxAh3p7oPf6ugJ4a/QSoWpb6efvxT3sBBhmr1rf6Dk+6Zw7/P0GJJHZL/z10d+Xkyuc+V8uIGmYdaXo6BTMkSRituQ6cFHD/gwn7MV92GNSeGqM0NO4D7qXkNtzH3Jn5flzyilzQ3fJiiO5oa10wCQmfjATCZMXatB7G/D5cQPS/T5Bo5oiPQL4Ckd+KuJ8w5IldwbjdmIKT/m1ksqLtAdoY5+IbF/RbUJ7eFUJgQD1nmIyUS/q4zSOfD0lyN2k2kHPPEcODTXpDatZ8raO+exquXeT6aBIQ3XNMpWU1BNYptEUla2yTP+JbLSb7RgHyXdeM14vV5kSGU2/vH6BUG4jRVN4mnh114ri+DQF0toSozAg0fMNVtd3NY5LTLlWBXFL9+YA7DRWbR5/sGAf4nTtC9XwdXrusEPLU3DX/Myg0VCOgheLn7DMlHHxS6Ho8n8= ================================================ FILE: gps.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` import serial, pynmea2, time, threading as td import logging logger = logging.getLogger(__name__) class ReadGPSData(td.Thread): """ Class to read the data off of the EC25-E's GPS module. Can be easily adapted to work with any other GPS module. """ def __init__(self, write_port, read_port, baudrate, name="GPS"): """ write_port - The serial port to use for activating the GPS. read_port - The serial port from which to read the GPS data. baudrate - Transport rate over the serial port. name - Name of the thread. """ super(ReadGPSData, self).__init__(name=name) self.write_port = write_port self.read_port = read_port self.baudrate = baudrate self.event = td.Event() self.lock = td.Lock() def run(self): logger.info("configuring GPS on port {}".format(self.write_port)) self.serw = serial.Serial( self.write_port, baudrate=self.baudrate, timeout=1, rtscts=True, dsrdtr=True ) self.serw.write("AT+QGPS=1\r".encode("utf-8")) self.serw.close() time.sleep(0.5) self.serr = serial.Serial( self.read_port, baudrate=self.baudrate, timeout=1, rtscts=True, dsrdtr=True ) logger.info("configured GPS to read from port {}".format(self.read_port)) while not self.event.is_set(): data = self.serr.readline() self.lock.acquire() try: self.__msg = pynmea2.parse(data.decode("utf-8")) except: pass finally: self.lock.release() logger.info(self.__msg) time.sleep(1) logger.info("stopped GPS thread") @property def parsed(self): """ Get the whole parsed data. """ self.lock.acquire() try: data = self.__msg except: data = None finally: self.lock.release() return data @property def latitude(self): """ Returns latitude expressed as a float. """ self.lock.acquire() try: latitude = self.__msg.latitude except: latitude = 0.0 finally: self.lock.release() return latitude @property def longitude(self): """ Returns longitude expressed as a float. """ self.lock.acquire() try: longitude = self.__msg.longitude except: longitude = 0.0 finally: self.lock.release() return longitude def stop(self): """ Stop the thread. """ self.event.set() ================================================ FILE: utils/bbox.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` import numpy as np import cv2 from .colors import get_color class BoundBox: def __init__(self, xmin, ymin, xmax, ymax, c=None, classes=None): self.xmin = xmin self.ymin = ymin self.xmax = xmax self.ymax = ymax self.c = c self.classes = classes self.label = -1 self.score = -1 def get_label(self): if self.label == -1: self.label = np.argmax(self.classes) return self.label def get_score(self): if self.score == -1: self.score = self.classes[self.get_label()] return self.score def draw_boxes(image, boxes, overlay_text, labels, obj_thresh, quiet=True): for box, overlay in zip(boxes, overlay_text): label_str = "" label = -1 for i in range(len(labels)): if box.classes[i] > obj_thresh: if label_str != "": label_str += ", " label_str += labels[i] + " " + str(round(box.get_score() * 100, 2)) + "%" label = i if not quiet: print(label_str) if label >= 0: if len(overlay) > 0: text = label_str + ": [" + " ".join(overlay) + "]" else: text = label_str text = text.upper() text_size = cv2.getTextSize(text, cv2.FONT_HERSHEY_SIMPLEX, 1.1e-3 * image.shape[0], 5) width, height = text_size[0][0], text_size[0][1] region = np.array( [ [box.xmin - 3, box.ymin], [box.xmin - 3, box.ymin - height - 26], [box.xmin + width + 13, box.ymin - height - 26], [box.xmin + width + 13, box.ymin], ], dtype="int32", ) # cv2.rectangle(img=image, pt1=(box.xmin,box.ymin), pt2=(box.xmax,box.ymax), color=get_color(label), thickness=5) rec = (box.xmin, box.ymin, box.xmax - box.xmin, box.ymax - box.ymin) rec = tuple(int(i) for i in rec) cv2.rectangle(img=image, rec=rec, color=get_color(label), thickness=3) cv2.fillPoly(img=image, pts=[region], color=get_color(label)) cv2.putText( img=image, text=text, org=(box.xmin + 13, box.ymin - 13), fontFace=cv2.FONT_HERSHEY_SIMPLEX, fontScale=1e-3 * image.shape[0], color=(0, 0, 0), thickness=1, ) return image ================================================ FILE: utils/colors.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` def get_color(label): """ Return a color from a set of predefined colors. Contains 80 colors in total. code originally from https://github.com/fizyr/keras-retinanet/ Args label: The label to get the color for. Returns A list of three values representing a RGB color. """ if label < len(colors): return colors[label] else: print("Label {} has no color, returning default.".format(label)) return (0, 255, 0) colors = [ [31, 0, 255], [0, 159, 255], [255, 95, 0], [255, 19, 0], [255, 0, 0], [255, 38, 0], [0, 255, 25], [255, 0, 133], [255, 172, 0], [108, 0, 255], [0, 82, 255], [0, 255, 6], [255, 0, 152], [223, 0, 255], [12, 0, 255], [0, 255, 178], [108, 255, 0], [184, 0, 255], [255, 0, 76], [146, 255, 0], [51, 0, 255], [0, 197, 255], [255, 248, 0], [255, 0, 19], [255, 0, 38], [89, 255, 0], [127, 255, 0], [255, 153, 0], [0, 255, 255], [0, 255, 216], [0, 255, 121], [255, 0, 248], [70, 0, 255], [0, 255, 159], [0, 216, 255], [0, 6, 255], [0, 63, 255], [31, 255, 0], [255, 57, 0], [255, 0, 210], [0, 255, 102], [242, 255, 0], [255, 191, 0], [0, 255, 63], [255, 0, 95], [146, 0, 255], [184, 255, 0], [255, 114, 0], [0, 255, 235], [255, 229, 0], [0, 178, 255], [255, 0, 114], [255, 0, 57], [0, 140, 255], [0, 121, 255], [12, 255, 0], [255, 210, 0], [0, 255, 44], [165, 255, 0], [0, 25, 255], [0, 255, 140], [0, 101, 255], [0, 255, 82], [223, 255, 0], [242, 0, 255], [89, 0, 255], [165, 0, 255], [70, 255, 0], [255, 0, 172], [255, 76, 0], [203, 255, 0], [204, 0, 255], [255, 0, 229], [255, 133, 0], [127, 0, 255], [0, 235, 255], [0, 255, 197], [255, 0, 191], [0, 44, 255], [50, 255, 0], ] ================================================ FILE: utils/image.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` import cv2 import numpy as np def resize_image(image, desired_width): current_width = image.shape[1] scale_percent = desired_width / current_width width = int(image.shape[1] * scale_percent) height = int(image.shape[0] * scale_percent) resized = cv2.resize(image, (width, height), interpolation=cv2.INTER_AREA) return resized def compress_image(image, grayscale=True, desired_width=416, top_crop_percent=0.45): if grayscale: image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) image = resize_image(image, desired_width) height = image.shape[0] if top_crop_percent: image[: int(height * top_crop_percent)] = 128 return image def image_from_bytes(byte_im): nparr = np.frombuffer(byte_im, np.uint8) img_np = cv2.imdecode(nparr, cv2.IMREAD_COLOR) return img_np def image_to_jpeg_nparray(image, quality=[int(cv2.IMWRITE_JPEG_QUALITY), 95]): is_success, im_buf_arr = cv2.imencode(".jpg", image, quality) return im_buf_arr def image_to_jpeg_bytes(image, quality=[int(cv2.IMWRITE_JPEG_QUALITY), 95]): buf = image_to_jpeg_nparray(image, quality) byte_im = buf.tobytes() return byte_im ================================================ FILE: utils/queue.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` from multiprocessing.queues import Queue as mp_queue import multiprocessing as mp class SharedCounter(object): """ A synchronized shared counter. The locking done by multiprocessing.Value ensures that only a single process or thread may read or write the in-memory ctypes object. However, in order to do n += 1, Python performs a read followed by a write, so a second process may read the old value before the new one is written by the first process. The solution is to use a multiprocessing.Lock to guarantee the atomicity of the modifications to Value. This class comes almost entirely from Eli Bendersky's blog: http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/ """ def __init__(self, n=0): self.count = mp.Value("i", n) def increment(self, n=1): """ Increment the counter by n (default = 1) """ with self.count.get_lock(): self.count.value += n def reset(self): """ Reset the counter to 0 """ with self.count.get_lock(): self.count.value = 0 @property def value(self): """ Return the value of the counter """ return self.count.value class MPQueue(mp_queue): """ A portable implementation of multiprocessing.Queue. Because of multithreading / multiprocessing semantics, Queue.qsize() may raise the NotImplementedError exception on Unix platforms like Mac OS X where sem_getvalue() is not implemented. This subclass addresses this problem by using a synchronized shared counter (initialized to zero) and increasing / decreasing its value every time the put() and get() methods are called, respectively. This not only prevents NotImplementedError from being raised, but also allows us to implement a reliable version of both qsize() and empty(). """ def __init__(self, *args, **kwargs): self.ctx = mp.get_context() super(MPQueue, self).__init__(*args, **kwargs, ctx=self.ctx) self.size = SharedCounter(0) def put(self, *args, **kwargs): self.size.increment(1) super(MPQueue, self).put(*args, **kwargs) def get(self, *args, **kwargs): self.size.increment(-1) if self.size.value < 0: self.size.reset() return super(MPQueue, self).get(*args, **kwargs) def qsize(self): """ Reliable implementation of multiprocessing.Queue.qsize() """ return self.size.value def empty(self): """ Reliable implementation of multiprocessing.Queue.empty() """ return not self.qsize() ================================================ FILE: workers.py ================================================ # WARNING: you are on the master branch, please refer to the examples on the branch that matches your `cortex version` from utils.image import ( resize_image, compress_image, image_from_bytes, image_to_jpeg_nparray, image_to_jpeg_bytes, ) from utils.bbox import BoundBox, draw_boxes from statistics import mean import time, base64, pickle, json, cv2, logging, requests, queue, broadcast, copy, statistics import numpy as np import threading as td import multiprocessing as mp logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) session = requests.Session() class WorkerTemplateThread(td.Thread): def __init__(self, event_stopper, name=None, runnable=None): td.Thread.__init__(self, name=name) self.event_stopper = event_stopper self.runnable = runnable def run(self): if self.runnable: logger.debug("worker started") while not self.event_stopper.is_set(): self.runnable() time.sleep(0.030) logger.debug("worker stopped") def stop(self): self.event_stopper.set() class WorkerTemplateProcess(mp.Process): def __init__(self, event_stopper, name=None, runnable=None): mp.Process.__init__(self, name=name) self.event_stopper = event_stopper self.runnable = runnable def run(self): if self.runnable: logger.debug("worker started") while not self.event_stopper.is_set(): self.runnable() time.sleep(0.030) logger.debug("worker stopped") def stop(self): self.event_stopper.set() class BroadcastReassembled(WorkerTemplateProcess): """ Separate process to broadcast the stream with the overlayed predictions on top of it. """ def __init__(self, in_queue, cfg, name=None): """ in_queue - Queue from which to extract the frames with the overlayed predictions on top of it. cfg - The dictionary config for the broadcaster. name - Name of the process. """ super(BroadcastReassembled, self).__init__(event_stopper=mp.Event(), name=name) self.in_queue = in_queue self.yolo3_rtt = None self.crnn_rtt = None self.detections = 0 self.current_detections = 0 self.recognitions = 0 self.current_recognitions = 0 self.buffer = [] self.oldest_broadcasted_frame = 0 for key, value in cfg.items(): setattr(self, key, value) def run(self): # start streaming server def lambda_func(): server = broadcast.StreamingServer( tuple(self.serve_address), broadcast.StreamingHandler ) server.serve_forever() td.Thread(target=lambda_func, args=(), daemon=True).start() logger.info("listening for stream clients on {}".format(self.serve_address)) # start polling for new processed frames from the queue and broadcast logger.info("worker started") counter = 0 while not self.event_stopper.is_set(): if counter == self.target_fps: logger.debug("buffer queue size: {}".format(len(self.buffer))) counter = 0 self.reassemble() time.sleep(0.001) counter += 1 logger.info("worker stopped") def reassemble(self): """ Main method to run in the loop. """ start = time.time() self.pull_and_push() self.purge_stale_frames() frame, delay = self.pick_new_frame() # delay loop to stabilize the video fps end = time.time() elapsed_time = end - start elapsed_time += 0.001 # count in the millisecond in self.run if delay - elapsed_time > 0.0: time.sleep(delay - elapsed_time) if frame: # pull and push again in case # write buffer (assume it takes an insignificant time to execute) self.pull_and_push() broadcast.output.write(frame) def pull_and_push(self): """ Get new frame and push it in the broadcaster's little buffer for stabilization. """ try: data = self.in_queue.get_nowait() except queue.Empty: # logger.warning("no data available for worker") return # extract data boxes = data["boxes"] frame_num = data["frame_num"] yolo3_rtt = data["avg_yolo3_rtt"] crnn_rtt = data["avg_crnn_rtt"] byte_im = data["image"] # run statistics self.statistics(yolo3_rtt, crnn_rtt, len(boxes), 0) # push frames to buffer and pick new frame self.buffer.append({"image": byte_im, "frame_num": frame_num}) def purge_stale_frames(self): """ Remove any frames older than the latest broadcasted frame. """ new_buffer = [] for frame in self.buffer: if frame["frame_num"] > self.oldest_broadcasted_frame: new_buffer.append(frame) self.buffer = new_buffer def pick_new_frame(self): """ Get the oldest frame from the buffer that isn't older than the last broadcasted frame. """ current_desired_fps = self.target_fps - self.max_fps_variation delay = 1 / current_desired_fps if len(self.buffer) == 0: return None, delay newlist = sorted(self.buffer, key=lambda k: k["frame_num"]) idx_to_del = 0 for idx, frame in enumerate(newlist): if frame["frame_num"] < self.oldest_broadcasted_frame: idx_to_del = idx + 1 newlist = newlist[idx_to_del:] if len(newlist) == 0: return None, delay self.buffer = newlist[::-1] element = self.buffer.pop() frame = element["image"] self.oldest_broadcasted_frame = element["frame_num"] size = len(self.buffer) variation = size - self.target_buffer_size var_perc = variation / self.max_buffer_size_variation current_desired_fps = self.target_fps + var_perc * self.max_fps_variation if current_desired_fps < 0: current_desired_fps = self.target_fps - self.max_fps_variation try: delay = 1 / current_desired_fps except ZeroDivisionError: current_desired_fps = self.target_fps - self.max_fps_variation delay = 1 / current_desired_fps return frame, delay def statistics(self, yolo3_rtt, crnn_rtt, detections, recognitions): """ A bunch of RTT and detection/recognition statistics. Not used. """ if not self.yolo3_rtt: self.yolo3_rtt = yolo3_rtt else: self.yolo3_rtt = self.yolo3_rtt * 0.98 + yolo3_rtt * 0.02 if not self.crnn_rtt: self.crnn_rtt = crnn_rtt else: self.crnn_rtt = self.crnn_rtt * 0.98 + crnn_rtt * 0.02 self.detections += detections self.current_detections = detections self.recognitions += recognitions self.current_recognitions = recognitions class InferenceWorker(WorkerTemplateThread): """ Worker that receives frames from a queue, sends requests to 2 cortex APIs for inference reasons, and retrieves the results and puts them in their appropriate queues. """ def __init__(self, event_stopper, in_queue, bc_queue, predicts_queue, cfg, name=None): """ event_stopper - Event to stop the worker. in_queue - Queue that holds the unprocessed frames. bc_queue - Queue to push into the frames with the overlayed predictions. predicts_queue - Queue to push into the detected license plates that will eventually get written to the disk. cfg - Dictionary config for the worker. name - Name of the worker thread. """ super(InferenceWorker, self).__init__(event_stopper=event_stopper, name=name) self.in_queue = in_queue self.bc_queue = bc_queue self.predicts_queue = predicts_queue self.rtt_yolo3_ms = None self.rtt_crnn_ms = 0 self.runnable = self.cloud_infer for key, value in cfg.items(): setattr(self, key, value) def cloud_infer(self): """ Main method that runs in the loop. """ try: data = self.in_queue.get_nowait() except queue.Empty: # logger.warning("no data available for worker") return ############################# # extract frame frame_num = data["frame_num"] img = data["jpeg"] # preprocess/compress the image image = image_from_bytes(img) reduced = compress_image(image) byte_im = image_to_jpeg_bytes(reduced) # encode image img_enc = base64.b64encode(byte_im).decode("utf-8") img_dump = json.dumps({"img": img_enc}) # make inference request resp = self.yolov3_api_request(img_dump) if not resp: return ############################# # parse response r_dict = resp.json() boxes_raw = r_dict["boxes"] boxes = [] for b in boxes_raw: box = BoundBox(*b) boxes.append(box) # purge bounding boxes with a low confidence score aux = [] for b in boxes: label = -1 for i in range(len(b.classes)): if b.classes[i] > self.yolov3_obj_thresh: label = i if label >= 0: aux.append(b) boxes = aux del aux # also scale the boxes for later uses camera_source_width = image.shape[1] boxes640 = self.scale_bbox(boxes, self.yolov3_input_size_px, self.bounding_boxes_upscale_px) boxes_source = self.scale_bbox(boxes, self.yolov3_input_size_px, camera_source_width) ############################# # recognize the license plates in case # any bounding boxes have been detected dec_words = [] if len(boxes) > 0 and len(self.api_endpoint_crnn) > 0: # create set of images of the detected license plates lps = [] try: for b in boxes_source: lp = image[b.ymin : b.ymax, b.xmin : b.xmax] jpeg = image_to_jpeg_nparray( lp, [int(cv2.IMWRITE_JPEG_QUALITY), self.crnn_quality] ) lps.append(jpeg) except: logger.warning("encountered error while converting to jpeg") pass lps = pickle.dumps(lps, protocol=0) lps_enc = base64.b64encode(lps).decode("utf-8") lps_dump = json.dumps({"imgs": lps_enc}) # make request to rcnn API dec_lps = self.rcnn_api_request(lps_dump) dec_lps = self.reorder_recognized_words(dec_lps) for dec_lp in dec_lps: dec_words.append([word[0] for word in dec_lp]) if len(dec_words) > 0: logger.info("Detected the following words: {}".format(dec_words)) else: dec_words = [[] for i in range(len(boxes))] ############################# # draw detections upscaled = resize_image(image, self.bounding_boxes_upscale_px) draw_image = draw_boxes( upscaled, boxes640, overlay_text=dec_words, labels=["LP"], obj_thresh=self.yolov3_obj_thresh, ) draw_byte_im = image_to_jpeg_bytes( draw_image, [int(cv2.IMWRITE_JPEG_QUALITY), self.broadcast_quality] ) ############################# # push data for further processing in the queue output = { "boxes": boxes, "frame_num": frame_num, "avg_yolo3_rtt": self.rtt_yolo3_ms, "avg_crnn_rtt": self.rtt_crnn_ms, "image": draw_byte_im, } self.bc_queue.put(output) # push predictions to write to disk if len(dec_words) > 0: timestamp = time.time() literal_time = time.ctime(timestamp) predicts = {"predicts": dec_words, "date": literal_time} self.predicts_queue.put(predicts) logger.info( "Frame Count: {} - Avg YOLO3 RTT: {}ms - Avg CRNN RTT: {}ms - Detected: {}".format( frame_num, int(self.rtt_yolo3_ms), int(self.rtt_crnn_ms), len(boxes) ) ) def scale_bbox(self, boxes, old_width, new_width): """ Scale a bounding box. """ boxes = copy.deepcopy(boxes) scale_percent = new_width / old_width for b in boxes: b.xmin = int(b.xmin * scale_percent) b.ymin = int(b.ymin * scale_percent) b.xmax = int(b.xmax * scale_percent) b.ymax = int(b.ymax * scale_percent) return boxes def yolov3_api_request(self, img_dump): """ Make a request to the YOLOv3 API. """ # make inference request try: start = time.time() resp = None resp = session.post( self.api_endpoint_yolov3, data=img_dump, headers={"content-type": "application/json"}, timeout=self.timeout, ) except requests.exceptions.Timeout as e: logger.warning("timeout on yolov3 inference request") time.sleep(0.10) return None except Exception as e: time.sleep(0.10) logger.warning("timing/connection error on yolov3", exc_info=True) return None finally: end = time.time() if not resp: pass elif resp.status_code != 200: logger.warning("received {} status code from yolov3 api".format(resp.status_code)) return None # calculate average rtt (use complementary filter) current = int((end - start) * 1000) if not self.rtt_yolo3_ms: self.rtt_yolo3_ms = current else: self.rtt_yolo3_ms = self.rtt_yolo3_ms * 0.98 + current * 0.02 return resp def rcnn_api_request(self, lps_dump, timeout=1.200): """ Make a request to the CRNN API. """ # make request to rcnn API try: start = time.time() resp = None resp = session.post( self.api_endpoint_crnn, data=lps_dump, headers={"content-type": "application/json"}, timeout=self.timeout, ) except requests.exceptions.Timeout as e: logger.warning("timeout on crnn inference request") except: logger.warning("timing/connection error on crnn", exc_info=True) finally: end = time.time() dec_lps = [] if not resp: pass elif resp.status_code != 200: logger.warning("received {} status code from crnn api".format(resp.status_code)) else: r_dict = resp.json() dec_lps = r_dict["license-plates"] # calculate average rtt (use complementary filter) current = int((end - start) * 1000) self.rtt_crnn_ms = self.rtt_crnn_ms * 0.98 + current * 0.02 return dec_lps def reorder_recognized_words(self, detected_images): """ Reorder the detected words in each image based on the average horizontal position of each word. Sorting them in ascending order. """ reordered_images = [] for detected_image in detected_images: # computing the mean average position for each word mean_horizontal_positions = [] for words in detected_image: box = words[1] y_positions = [point[0] for point in box] mean_y_position = mean(y_positions) mean_horizontal_positions.append(mean_y_position) indexes = np.argsort(mean_horizontal_positions) # and reordering them reordered = [] for index, words in zip(indexes, detected_image): reordered.append(detected_image[index]) reordered_images.append(reordered) return reordered_images class Flusher(WorkerTemplateThread): """ Thread which removes the elements of a queue when its size crosses a threshold. Used when there are too many frames are pilling up in the queue. """ def __init__(self, queue, threshold, name=None): """ queue - Queue to remove the elements from when the threshold is triggered. threshold - Number of elements. name - Name of the thread. """ super(Flusher, self).__init__(event_stopper=td.Event(), name=name) self.queue = queue self.threshold = threshold self.runnable = self.flush_pipe def flush_pipe(self): """ Main method to run in the loop. """ current = self.queue.qsize() if current > self.threshold: try: for i in range(current): self.queue.get_nowait() logger.warning("flushed {} elements from the frames queue".format(current)) except queue.Empty: logger.debug("flushed too many elements from the queue") time.sleep(0.5)