In Chapter 4: Digit Recognition, we'll add a few new techniques to our image processing toolset by attempting to build a digit recognition pipeline from start to finish. Throughout the exercise, we will get to practice the image preprocessing tricks we've picked up from previous chapters:
- Image manipulations such as resizing, cropping, rotation, color conversion
- Blurring and sharpening operations
- Thresholding and Edge Detection
- Contour approximation
New method and strategies that you'll be learning include:
- Drawing operations (rectangles, text) on our image
- Region of interest and bounding rectangles
- Morphological transformations
- The Seven-Segment Display
To be clear, specialised deep learning libraries that have sprung out in recent years are a lot more robust in their approach. By utilizing machine learning principles (cost function, gradient descent etc), these specialised libraries can handle highly complex object recognition and OCR (optical character recognition) tasks at the cost of brute computing power.
The overarching motivation of this free course however, was to make clear to beginners what constitutes artificial intelligence, and to illustrate the principle benefits of machine learning. I try to achieve that by demonstrating -- over multiple chapters of this course -- how computer visions were traditionally, or rather "classically", performed prior to the emergence of deep learning.
By learning the classical approaches to computer vision, the student (you) can compare the effort it takes to hand-tuning parameters and this adds a new dimension of appreciation towards self-learning methods that we'll discuss in the near future.
Do a quick google search on "digit recognition" or "digit classification" and it's hard to find an introductory deep learning course that doesn't use the famous MNIST (Modified National Institute of Standards and Technology) database. This is a handwritten digit database that has long become the de facto in pretty much any machine learning tutorials:
But I'd argue, that for a budding computer vision developer, your learning objectives are better served by taking a different approach.
By choosing real life images, you are confronted with a few more key challenges that are not present from using a well-curated database such as MNIST. These challenges present new opportunities to learn about key concepts such as region of interest, and morphological operations, that you will come to rely upon greatly in the future.
First, take a look at 4 real-life pictures of security tokens issued by banks and institutional agencies (left-to-right: Bank Central Asia, DBS, OCBC Bank, OneKey for Singapore Government e-services):
Notice how noisy these images are, as each image is shot with a different background, different lighting conditions, each token is of a different size and shape, and the different colors in each security token etc.
Your task, as a computer vision developer, is to develop a pipeline that, in each phase, take you closer to the goal. Roughly speaking, given the above task, we would formulate a pipeline that looks like the following:
- Preprocessing, noise reduction
- Contour approximation
- Find region of interest (ROI), that is the area of the LCD display in each of these pictures
- Extract ROI for further preprocessing, discarding the rest of the image
- Isolate each digit from the ROI
- Iteratively classify each digit in the image
- Combine the per-digit classification to a final string ("output")
In practice, step (1) and (2) above is the "application" of the methods you've learned in previous chapters of this series. As we'll soon observe, we will use a combination of blurring operations and edge detection to draw our contours. Among the contours, one of them would be the LCD display containing the digits to be classified. That is our Region of Interest.
The GIF above demonstrates the code in roi_01.py but essentially it shows the selectROI method in action. You'll commonly combined the selectROI method with a either a slicing operation to crop your region of interest, or a drawing operation to call attention to the specific region of the image.
x,y,w,h = cv2.selectROI("Region of interest", img)
cropped = img[y:y+h, x:x+w]
cv2.rectangle(img_color, (x,y), (x+w,y+h), (255,0,0), 2)
In most cases, it simply wouldn't be realistic to render an image before manually specifying our region of interest. We'll need this operation to be as close to automatic as possible. But how exactly? That depends greatly on the specific problem set.
In some cases, the obvious choice of strategy would be simply shape recognition, say by counting the number of vertices from each contour. The following code is an example implementation of that:
peri = cv2.arcLength(cnt, True)
cnt_appro = cv2.approxPolyDP(cnt, 0.03 * peri, True)
if len(cnt_approx) == 3:
est_shape = 'triangle'
...
elif len(cnt_approx) == 5:
est_shape = 'pentagon'
...
In other cases, you may employ a strategy that try to match contour based on Hu moments (which we'll study in details in future chapters).
Other methods may involve a saliency map, or a visual attention map, for ROI extraction. These methods create a new representation of the original image where each pixel's unique quality are amplified or emphasized. One example implementation on Wikipedia demonstrates how straightforward this concept really is:
$$SALS(I_K) = \sum^{N}_{i=1}|I_k-I_i|$$
As you add new tools and strategies to your computer vision toolbox, you will pick up new approaches to ROI extraction. It is an interesting field of research that has been gaining a lot in popularity with the emergence of deep learning.
As for the images of bank security tokens, can you think of an approach that may be a good fit? Our region of interest is the LCD screen at the top of the button pad on each device, and they all seem to be rather consistent in shape and size. Give it some thought and read on to find out.
Arc Length and Area Size
I've hinted at the shape and size being a factor, so maybe that would be a good starting point. The good news is the OpenCV made this incredibly easy through the contourArea() and arcLength() function.
The following snippet of code, lifted from contourarea_01.py, finds all contours and sort them by area size in descending order before storing the first 10 in cnts:
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:9]
We can also obtain the contour area and parameter iteratively in a for-loop, like the following:
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for i in range(len(cnts)):
area = cv2.contourArea(cnts[i])
peri = cv2.arcLength(cnts[i], closed=True)
print(f'Area:{area}, Perimeter:{peri}')
In effect, we're looping through each contour that the findContours() operation found, and computing two values each time, area and peri.
Note that the contour perimeter is also known as the arc length. The second argument closed specify whether the shape is a closed contour (True) or just a curve (closed=False).
Execute contourarea_01.py and observe how each contour is displayed, from the one with the largest area to the one with the least, for a total of 10 contours. As you run the script on different pictures of bank security tokens, you see that it does a reliable job at finding the contours, sorting them, and returning our LCD display screen as the first in the list. This makes sense, because visually it is apparent that the LCD display occupy the largest area among other closed shapes in our picture.
-
Use assets/dbs.jpg instead of assets/ocbc.jpg in contourarea_01.py. Were you able to extract the region of interest (LCD Display) successfully without any changes to the script?
-
Could we have successfully extract our region of interest have we used arcLength in our strategy?
-
Supposed we only wanted to extract the region of interest and not the rest, which line of code would you change? Reflect the change in the code and execute it to confirm that you have performed this exercise correctly.
-
Supposed we wanted the contours sorted according to their respective area, from the smallest to the largest, which line of code would you change? Reflect the change in the code and execute it to confirm that you have performed this exercise correctly.
While working through the exercises above, you may find it helpful to also draw the text describing the area size and perimeter next to each contour. I've shown you how this can be done in contourarea_02.py but the essential addition we make to the earlier code is the two calls to putText():
PURPLE = (75, 0, 130)
THICKNESS = 1
FONT = cv2.FONT_HERSHEY_SIMPLEX
cv2.putText(img_color, "Area:" + str(area), (x, y - 15), FONT, 0.4, PURPLE,THICKNESS)
cv2.putText(img_color, "Perimeter:" + str(peri), (x, y - 5), FONT, 0.4,PURPLE, THICKNESS)
With these foundations, we are now ready to write a simple utility script that:
- Find our region of interest
- Crop ROI into a new image
- Save it into an folder named
/inter (intermediary) for the actual digit recognition later
Much of what you need to do has already been presented so far, but the core pieces are, lifted from roi_02.py the following few lines of code:
img = cv2.imread(...)
blurred = cv2.GaussianBlur(img, (7, 7), 0)
edged = cv2.Canny(blurred, 130, 150, 255)
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:1]
x, y, w, h = cv2.boundingRect(cnts[0])
roi = img[y : y + h, x : x + w]
cv2.imwrite("roi.png", roi)
The roi_02.py utility script uses the argparse library so user can specify a file path with a flag -p (or --path) like such:
python roi_02.py -p assets/ocbc.jpg
python roi_02.py --path assets/ocbc.jpg
If the user do not specify a file path using the -p flag, the default value would be assets/ocbc.jpg. If you wish to change this, edit roi_02.py and specify a different value for the default parameter.
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--path", default="assets/ocbc.jpg")
You should run this exercise using dbs.jpg, ocbc2.jpg, or onekey.jpg at least once. Execute the script and check the inter folder to confirm that the ROI has been saved. When you're done, you are ready to move on to the next phase of the digit recognition pipeline.
Once the region of interest is obtained, we now have an image that may still contain noises. This is especially the case when our ROI is obtained by means of thresholding methods, since you can expect some "non-features" (noises) to also be included in the resulting image.
To account for these imperfections, we will now perform a series of operations on our image. We'll learn what they are formally, but let's begin by seeing what is it that they offer to our image processing pipeline. I've included a picture with some random noise, as follow:
The digit "0417" is clearly discernible to the human eye despite the presence of noise. However, consider the perspective of a global thresholding operation; These pixel values are "noise" to us but a computer has no such notion of which pixel values are meaningful and what others are not. A thresold value such as the global mean will take all values into account indiscriminately. A contour finding operation will, instead of 4, return thousands of tiny round segments (they may be tiny, but they are completely valid contours).
An image processing pipeline that fail to account for these may result in sub-optimal performance or, very often, completely undesired results.
Enter two of the most fundamental morphological transformations: erosion and dilation.
Erosion "erodes away the boundaries of foreground object" by sliding a kernel through the image and set a pixel to 1 only if all the pixels under the kernel is 1.
This in effect discard pixels near the boundary and any floating pixels that are not part of a larger blob (which is what the human eye is interested in). Because pixels are eroded, your foreground object will shrink in size.
The opposite of erosion, Dilation sets a pixel to 1 if at least one pixel under the kernel is 1, essentially "growing" the foreground object.
Because of how these operations work, there are a couple of things to note:
- Morphological transformations are usually performed on binary images. Recall that pixel values in binary images are either a full white (i.e 1) or black (i.e 0).
- As per convention, we want to keep our foregound in white and background in black
- Because erosion results in a shrinking foreground and dilation results in a growing foreground, these two operations are also commonly used in combinations, i.e erosion followed by dilation, or vice versa
As we read our image in grayscale mode (flags=0), we obtain a white blackground and a mostly-black foreground. This is illustrated in the subplot titled "Original" above. We begin our preprocessing steps by first binarizing the image (step 1), followed by inverting the colors (step 2) to get a white-on-black image.
An erosion operation is then performed (step 3). This works by creating our kernel (either through numpy or through opencv's structuring element) and sliding that kernel across our image to remove white noises in our image.
The side-effect is that our foreground object has now shrunk in size as it's boundaries are eroded away. We grow it back by applying a dilation (step 4) and finally show the output as illustrated in the bottom-right pane of the image above.
roi = cv2.imread("assets/0417s.png", flags=0)
_, thresh = cv2.threshold(roi, 170, 255, cv2.THRESH_BINARY)
inv = cv2.bitwise_not(thresh)
kernel = np.ones((5,5), np.uint8)
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
eroded = cv2.erode(inv, kernel, iterations=1)
dilated = cv2.dilate(eroded, kernel, iterations=1)
cv2.imshow("Transformed", dilated)
cv2.waitKey(0)
OpenCV provides the three shapes for our kernel:
- Rectangular box:
MORPH_RECT
- Cross:
MORPH_CROSS
- Ellipse:
MORPH_ELLIPSE
They are fed as the first argument into cv2.getStructuringElement(), with the second being the kernel size (ksize) itself. The third argument is the anchor point, which defaults to the center.
Opening and Closing
Another name for Erosion, followed by Dilation is the Opening. It is useful in removing noise in our image. The reverse of Opening is Closing, where we first perform Dilation followed by Erosion, particularly suited for closing small holes inside foreground objects.
OpenCV includes the more generic morphologyEx method for all other morphological operations beyond Erosion and Dilation. The function takes an image as the first argument, an operation as the second operation and finally the kernel. Compare how your code will differ between cv2.erode and cv2.dilate, and their respective equivalence in cv2.morphologyEx():
import cv2
import numpy as np
img = cv2.imread('image.png',0)
kernel = np.ones((5,5),np.uint8)
erosion = cv2.erode(img,kernel,iterations = 1)
dilation = cv2.dilate(img,kernel,iterations = 1)
opening = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel)
closing = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
In the homework directory, you'll find 0417h.png. Your job is to apply what you've learned in this lesson to clean up the image. Your output should have these qualities:
- As free of noise as possible (remove the lines, and the red splatted dots across the image)
- If you run
findContours() on the output, you should have exactly 4 contours
- Foreground object in white, background in black
You are free to pick your strategy, but a reference solution would look like the following:
The seven-segment display (known also as "seven-segment indicator") is a form of electronic display device for displaying decimal numerals widely used in digital clocks, electronic meters, calculators and banking security tokens.
This is relevant because it is the character representation of our digits in each of these security tokens. If we can isolate each digit from each other, we can iteratively predict the "class" of each digit (0 to 9). Specifically, we are going to perform a classification task based on the state of each segment.
To ease our understanding, let's refer to each segment using the letters A to G:
We can then create a lookup table that match the collective states to the corresponding class:
| Class |
a |
b |
c |
d |
e |
f |
g |
| 0 |
1 |
1 |
1 |
1 |
1 |
1 |
0 |
| 1 |
0 |
1 |
1 |
0 |
0 |
0 |
0 |
| 2 |
1 |
1 |
0 |
1 |
1 |
0 |
1 |
| 3 |
1 |
1 |
1 |
1 |
0 |
0 |
1 |
| 4 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
| 5 |
1 |
0 |
1 |
1 |
0 |
1 |
1 |
| 6 |
1 |
0 |
1 |
1 |
1 |
1 |
1 |
| 7 |
1 |
1 |
1 |
0 |
0 |
1 |
0 |
| 8 |
1 |
1 |
1 |
1 |
1 |
1 |
1 |
| 9 |
1 |
1 |
1 |
1 |
0 |
1 |
1 |
How would we represent such a lookup table in our Python code and how would we use it? The obvious answer to the first question is a dictionary. Notice that DIGITSDICT is just a representation of the "binary state" of each segment. The digit "8" for example correspond to all seven segments being activated, or "on" (state of 1).
DIGITSDICT = {
(1,1,1,1,1,1,0):0,
(0,1,1,0,0,0,0):1,
(1,1,0,1,1,0,1):2,
(1,1,1,1,0,0,1):3,
(0,1,1,0,0,1,1):4,
(1,0,1,1,0,1,1):5,
(1,0,1,1,1,1,1):6,
(1,1,1,0,0,1,0):7,
(1,1,1,1,1,1,1):8,
(1,1,1,1,0,1,1):9
}
Then, for each digit, we would look at the pixel values in each of the seven segments, and if the majority of pixels are white, we would classify that segment as being in an activated state (1), otherwise in a state of 0. As we iterate over the 7 segments, we now have an array of length 7, each element a binary value(0 or 1).
We would then find the corresponding value in our dictionary using that array. Your code would resemble the following:
sevensegs = [
((x0, y0), (x1, y1)),
((x2, y2), (x3, y3)),
...
]
on = [0] * 7
for (i, ((p1x, p1y), (p2x, p2y))) in enumerate(sevensegs):
region = roi[p1y:p2y, p1x:p2x]
if np.sum(region == 255) > region.size *0.5:
on[i] = 1
digit = DIGITSDICT[tuple(on)]
There are multiple ways to write a for-loop but it's important that you are aware of the order in which your for-loop your executing. Referring to our seven-segment illustration below,the first iteration is only concerned with the state of 'A' while the second interation handles the state of 'B', and so on.
Using enumerate, we obtain an additional counter (i) to our iterable (sevensegs); This is convenient for the purpose of setting states. At the first iteration, the first element is our list is conditionally set to 1 if more than half of the pixels in segment 'A' are white. A more detailed example of python's enumeration is in utils/enumerate.py.
If you are paying close attention to the digit '0' in our LCD display, you will notice that the absence of the 'G' segment causes a pretty visible and significant gap. When you test your digit recognition script without special consideration to this attribute, you will find it consistently failing to account for the numbers "0","1" and "7". In fact, you may not even be able to isolate the aforementioned numbers altogether using the findContour operation, because they were treated as two disjointed pieces instead of a whole piece.
A reasonable strategy to handle this is the Dilation or Closing (Dilation followed by Erosion) operation that you've learned earlier.
Similarly, your ROI may necessitate other pre-processing and the specific tactical solution vary greatly depending on the problem set at hand.
As I inspect the bounding box we retrieved around the LCD screen, the observation that these bouding boxes often have their digits centered around the bottom half of the display led me to insert an additional step prior to the morphological transformation in the final code solution. The step uses numpy subsetting to trim away the top 20% as well as 20% on each side of the image:
roi = cv2.imread("roi.png", flags=0)
RATIO = roi.shape[0] * 0.2
trimmed = roi[
int(RATIO) :,
int(RATIO) : roi.shape[1] - int(RATIO)]
That said, whenever possible, you want to be cautious of not hand-tuning your problem in a way that is overly specific to the images you have at hand lest risking the solution only working on those specific images and not others, a phenomenon fondly termed as "overfitting" in the machine learning community.
I've re-executed the solution code against some sample image sets, once with the "trimming" in-place and then without the trimming, before settling on the decision. As you will see later, the trimming improves our accuracy and is a relatively safe strategy given how every LCD screen regardless of the issuer (bank) has the same asymmetry with more "blank space" at the top half compared to the bottom half.
Furthermore, in many cases of digit recognition / digit classification you will want to predict the class for each digit in an ordered fashion. Supposed the LCD screen contains the digits "40710382", our algorithm should correctly isolate these digits, classify them iteratively, but do so from the leftmost digit to the rightmost. Failing to account for this may result in your algorithm correctly classifying each digit, but produce an unreasonable output such as "1740238".
There are a few strategies you can employ here. We've seen in contourarea_01.py and contourarea_02.py how contour has attributes that can be retrieved using the contourArea() and arcLength() functions. Inspect the following snippet and it should help jog your memory:
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:9]
for i, cnt in enumerate(cnts):
cv2.drawContours(img_color, cnts, i, BCOLOR, THICKNESS)
area = cv2.contourArea(cnt)
peri = cv2.arcLength(cnt, closed=True)
print(f"Area:{area}; Perimeter: {peri}")
Indeed, we're using countour area as a good indicator to search for our region of interest. When we take this idea a little further, we can further place a constraint on our search criteria. In the following code, we draw a bounding rectangle and for an extra layer of precaution, only takes any bounding boxes that are taller than 20 pixels (step 1).
Calling boundingRect() on a contour returns 4 values, respectively the x and y coordinate along with the width and height of the contour.
We then use another property of the contour, its top-left coordinate to determine the logical order of our digits. Specifically, we use the first returned value (cv2.boundingRect(cnt)[0]) since that's the x value for the top-left coordinate of each region. By sorting against this value, our digits are stored in the Python list in an ordered fashion, determined by their respective coordinate value.
digits_cnts = []
cnts, _ = cv2.findContours(eroded, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for cnt in cnts:
(x, y, w, h) = cv2.boundingRect(cnt)
if h > 20:
digits_cnts += [cnt]
sorted_digits = sorted(digits_cnts, key=lambda cnt: cv2.boundingRect(cnt)[0])
When we put these together, we now have a complete pipeline:
The full solution code is in digit_01.py but the essential parts are as follow:
import cv2
import numpy as np
DIGITSDICT = {
(1, 1, 1, 1, 1, 1, 0): 0,
(0, 1, 1, 0, 0, 0, 0): 1,
(1, 1, 0, 1, 1, 0, 1): 2,
(1, 1, 1, 1, 0, 0, 1): 3,
(0, 1, 1, 0, 0, 1, 1): 4,
(1, 0, 1, 1, 0, 1, 1): 5,
(1, 0, 1, 1, 1, 1, 1): 6,
(1, 1, 1, 0, 0, 1, 0): 7,
(1, 1, 1, 1, 1, 1, 1): 8,
(1, 1, 1, 1, 0, 1, 1): 9,
}
roi = cv2.imread("inter/ocbc-roi.png", flags=0)
RATIO = roi.shape[0] * 0.2
roi = cv2.bilateralFilter(roi, 5, 30, 60)
trimmed = roi[int(RATIO) :, int(RATIO) : roi.shape[1] - int(RATIO)]
edged = cv2.adaptiveThreshold(
trimmed, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 5, 5
)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 5))
dilated = cv2.dilate(edged, kernel, iterations=1)
eroded = cv2.erode(dilated, kernel, iterations=1)
cnts, _ = cv2.findContours(eroded, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
digits_cnts = []
for cnt in cnts:
(x, y, w, h) = cv2.boundingRect(cnt)
if h > 20:
digits_cnts += [cnt]
sorted_digits = sorted(digits_cnts, key=lambda cnt: cv2.boundingRect(cnt)[0])
digits = []
for cnt in sorted_digits:
(x, y, w, h) = cv2.boundingRect(cnt)
roi = eroded[y : y + h, x : x + w]
qW, qH = int(w * 0.25), int(h * 0.15)
fractionH, halfH, fractionW = int(h * 0.05), int(h * 0.5), int(w * 0.25)
sevensegs = [
((0, 0), (w, qH)),
((w - qW, 0), (w, halfH)),
((w - qW, halfH), (w, h)),
((0, h - qH), (w, h)),
((0, halfH), (qW, h)),
((0, 0), (qW, halfH)),
(
(0 + fractionW, halfH - fractionH),
(w - fractionW, halfH + fractionH),
),
]
on = [0] * 7
for (i, ((p1x, p1y), (p2x, p2y))) in enumerate(sevensegs):
region = roi[p1y:p2y, p1x:p2x]
print(
f"{i}: Sum of 1: {np.sum(region == 255)}, Sum of 0: {np.sum(region == 0)}, Shape: {region.shape}, Size: {region.size}"
)
if np.sum(region == 255) > region.size * 0.5:
on[i] = 1
print(f"State of ON: {on}")
digit = DIGITSDICT[tuple(on)]
print(f"Digit is: {digit}")
digits += [digit]
cv2.rectangle(canvas, (x, y), (x + w, y + h), CYAN, 1)
cv2.putText(canvas, str(digit), (x - 5, y + 6), FONT, 0.3, (0, 0, 0), 1)
cv2.imshow("Digit", canvas)
cv2.waitKey(0)
print(f"Digits on the token are: {digits}")
- Step 1: Initialize the lookup dictionary
- Step 2: Read our ROI image using OpenCV
- Step 3: Noise reduction and trim away asymmetrical white space in our ROI
- Step 4: Binarize our image using adaptive thresholding
- Step 5: Morphological transformation to remove noise and fill the small holes in our digit
- Step 6: Find contours in our image with a height greater than 20px
- Step 7: Sort the contours in-place, using the x value of their coordinates (hence, left to right)
- Step 8
- Step 8a: Create rectangle bounding box on each digit, and some convenience units that we later use to slice the seven segments. Notice that these convenience units are not hard-coded values, but are proportional to the Height (
h) of our rectangular box
- Step 8b: Slice the seven segments; The first segment ("A") is from point (0,0) to (w,
int(h * 0.15)); This segment is w in width and 15% the height of the full digit contour, starting from position (0, 0)
- Step 8c: Initialize the state to
0 for each of the 7 segments, then conditionally set regions with more white than black pixels to 1
- Step 8d: Once all 7 states have been set, perform lookup against the digit dictionary created in step 1; Append the value to the
digits list created at the beginning of step 8
- Step 9: Draw rectangle and add predicted text for each bounding box. Finally, use a print statement to print the
digits list.
================================================
FILE: digitrecognition/digitrec.md
================================================
# Background
In Chapter 4: Digit Recognition, we'll add a few new techniques to our image processing toolset by attempting to build a digit recognition pipeline from start to finish. Throughout the exercise, we will get to practice the image preprocessing tricks we've picked up from previous chapters:
- Image manipulations such as resizing, cropping, rotation, color conversion
- Blurring and sharpening operations
- Thresholding and Edge Detection
- Contour approximation
New method and strategies that you'll be learning include:
- Drawing operations (rectangles, text) on our image
- Region of interest and bounding rectangles
- Morphological transformations
- The Seven-Segment Display
## What about Deep Learning?
To be clear, specialised deep learning libraries that have sprung out in recent years are a lot more robust in their approach. By utilizing machine learning principles (cost function, gradient descent etc), these specialised libraries can handle highly complex object recognition and OCR (optical character recognition) tasks at the cost of brute computing power.
The overarching motivation of this free course however, was to make clear to beginners what constitutes artificial intelligence, and to illustrate the principle benefits of machine learning. I try to achieve that by demonstrating -- over multiple chapters of this course -- how computer visions were traditionally, or rather "classically", performed prior to the emergence of deep learning.
By learning the classical approaches to computer vision, the student (you) can compare the effort it takes to hand-tuning parameters and this adds a new dimension of appreciation towards self-learning methods that we'll discuss in the near future.
## Region of Interest
Do a quick google search on "digit recognition" or "digit classification" and it's hard to find an introductory deep learning course that **doesn't use** the famous MNIST (Modified National Institute of Standards and Technology)[^1] database. This is a handwritten digit database that has long become the _de facto_ in pretty much any machine learning tutorials:
But I'd argue, that for a budding computer vision developer, your learning objectives are better served by taking a different approach.
By choosing real life images, you are confronted with a few more key challenges that are not present from using a well-curated database such as MNIST. These challenges present new opportunities to learn about key concepts such as **region of interest**, and **morphological operations**, that you will come to rely upon greatly in the future.
First, take a look at 4 real-life pictures of security tokens issued by banks and institutional agencies (left-to-right: Bank Central Asia, DBS, OCBC Bank, OneKey for Singapore Government e-services):
Notice how noisy these images are, as each image is shot with a different background, different lighting conditions, each token is of a different size and shape, and the different colors in each security token etc.
Your task, as a computer vision developer, is to develop a pipeline that, in each phase, take you closer to the goal. Roughly speaking, given the above task, we would formulate a pipeline that looks like the following:
1. Preprocessing, noise reduction
2. Contour approximation
3. Find region of interest (ROI), that is the area of the LCD display in each of these pictures
4. Extract ROI for further preprocessing, discarding the rest of the image
5. Isolate each digit from the ROI
6. Iteratively classify each digit in the image
7. Combine the per-digit classification to a final string ("output")
In practice, step (1) and (2) above is the "application" of the methods you've learned in previous chapters of this series. As we'll soon observe, we will use a combination of blurring operations and edge detection to draw our contours. Among the contours, one of them would be the LCD display containing the digits to be classified. That is our **Region of Interest**.
### Selecting Region of Interest
The GIF above demonstrates the code in `roi_01.py` but essentially it shows the `selectROI` method in action. You'll commonly combined the `selectROI` method with a either a slicing operation to crop your region of interest, or a drawing operation to call attention to the specific region of the image.
```py
x,y,w,h = cv2.selectROI("Region of interest", img)
cropped = img[y:y+h, x:x+w]
# draw rectangle
cv2.rectangle(img_color, (x,y), (x+w,y+h), (255,0,0), 2)
```
In most cases, it simply wouldn't be realistic to render an image before manually specifying our region of interest. We'll need this operation to be as close to automatic as possible. But how exactly? That depends greatly on the specific problem set.
In some cases, the obvious choice of strategy would be simply shape recognition, say by counting the number of vertices from each contour. The following code is an example implementation of that:
```py
# cnt = contour
peri = cv2.arcLength(cnt, True)
# contour approximation
cnt_appro = cv2.approxPolyDP(cnt, 0.03 * peri, True)
if len(cnt_approx) == 3:
est_shape = 'triangle'
...
elif len(cnt_approx) == 5:
est_shape = 'pentagon'
...
```
In other cases, you may employ a strategy that try to match contour based on Hu moments (which we'll study in details in future chapters).
Other methods may involve a saliency map, or a visual attention map, for ROI extraction. These methods create a new representation of the original image where each pixel's **unique quality** are amplified or emphasized. One example implementation on Wikipedia[^2] demonstrates how straightforward this concept really is:
$$SALS(I_K) = \sum^{N}_{i=1}|I_k-I_i|$$
As you add new tools and strategies to your computer vision toolbox, you will pick up new approaches to ROI extraction. It is an interesting field of research that has been gaining a lot in popularity with the emergence of deep learning.
As for the images of bank security tokens, can you think of an approach that may be a good fit? Our region of interest is the LCD screen at the top of the button pad on each device, and they all seem to be rather consistent in shape and size. Give it some thought and read on to find out.
### Arc Length and Area Size
I've hinted at the shape and size being a factor, so maybe that would be a good starting point. The good news is the OpenCV made this incredibly easy through the `contourArea()` and `arcLength()` function.
The following snippet of code, lifted from `contourarea_01.py`, finds all contours and sort them by area size in descending order before storing the first 10 in `cnts`:
```py
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# sort contours by contourArea, and get the first 10
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:9]
```
We can also obtain the contour area and parameter iteratively in a for-loop, like the following:
```py
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for i in range(len(cnts)):
area = cv2.contourArea(cnts[i])
peri = cv2.arcLength(cnts[i], closed=True)
print(f'Area:{area}, Perimeter:{peri}')
```
In effect, we're looping through each contour that the `findContours()` operation found, and computing two values each time, `area` and `peri`.
Note that the contour perimeter is also known as the arc length. The second argument `closed` specify whether the shape is a closed contour (`True`) or just a curve (`closed=False`).
Execute `contourarea_01.py` and observe how each contour is displayed, from the one with the largest area to the one with the least, for a total of 10 contours. As you run the script on different pictures of bank security tokens, you see that it does a reliable job at finding the contours, sorting them, and returning our LCD display screen as the first in the list. This makes sense, because visually it is apparent that the LCD display occupy the largest area among other closed shapes in our picture.
#### Dive Deeper: ROI
1. Use `assets/dbs.jpg` instead of `assets/ocbc.jpg` in `contourarea_01.py`. Were you able to extract the region of interest (LCD Display) successfully without any changes to the script?
2. Could we have successfully extract our region of interest have we used `arcLength` in our strategy?
3. Supposed we only wanted to extract the region of interest and not the rest, which line of code would you change? Reflect the change in the code and execute it to confirm that you have performed this exercise correctly.
4. Supposed we wanted the contours sorted according to their respective area, from the smallest to the largest, which line of code would you change? Reflect the change in the code and execute it to confirm that you have performed this exercise correctly.
While working through the exercises above, you may find it helpful to also draw the text describing the area size and perimeter next to each contour. I've shown you how this can be done in `contourarea_02.py` but the essential addition we make to the earlier code is the two calls to `putText()`:
```py
PURPLE = (75, 0, 130)
THICKNESS = 1
FONT = cv2.FONT_HERSHEY_SIMPLEX
cv2.putText(img_color, "Area:" + str(area), (x, y - 15), FONT, 0.4, PURPLE,THICKNESS)
cv2.putText(img_color, "Perimeter:" + str(peri), (x, y - 5), FONT, 0.4,PURPLE, THICKNESS)
```
### ROI extraction
With these foundations, we are now ready to write a simple utility script that:
1. Find our region of interest
2. Crop ROI into a new image
3. Save it into an folder named `/inter` (intermediary) for the actual digit recognition later
Much of what you need to do has already been presented so far, but the core pieces are, lifted from `roi_02.py` the following few lines of code:
```py
img = cv2.imread(...)
blurred = cv2.GaussianBlur(img, (7, 7), 0)
edged = cv2.Canny(blurred, 130, 150, 255)
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:1]
x, y, w, h = cv2.boundingRect(cnts[0])
roi = img[y : y + h, x : x + w]
cv2.imwrite("roi.png", roi)
```
The `roi_02.py` utility script uses the `argparse` library so user can specify a file path with a flag `-p` (or `--path`) like such:
```bash
python roi_02.py -p assets/ocbc.jpg
# equivalent:
python roi_02.py --path assets/ocbc.jpg
```
If the user do not specify a file path using the `-p` flag, the default value would be `assets/ocbc.jpg`. If you wish to change this, edit `roi_02.py` and specify a different value for the `default` parameter.
```py
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--path", default="assets/ocbc.jpg")
```
You should run this exercise using `dbs.jpg`, `ocbc2.jpg`, or `onekey.jpg` at least once. Execute the script and check the `inter` folder to confirm that the ROI has been saved. When you're done, you are ready to move on to the next phase of the digit recognition pipeline.
## Morphological Transformations
Once the region of interest is obtained, we now have an image that may still contain noises. This is especially the case when our ROI is obtained by means of thresholding methods, since you can expect some "non-features" (noises) to also be included in the resulting image.
To account for these imperfections, we will now perform a series of operations on our image. We'll learn what they are formally, but let's begin by seeing what is it that they _offer_ to our image processing pipeline. I've included a picture with some random noise, as follow:
The digit "0417" is clearly discernible to the human eye despite the presence of noise. However, consider the perspective of a global thresholding operation; These pixel values are "noise" to us but a computer has no such notion of which pixel values are meaningful and what others are not. A thresold value such as the global mean will take all values into account indiscriminately. A contour finding operation will, instead of 4, return thousands of tiny round segments (they may be tiny, but they are completely valid contours).
An image processing pipeline that fail to account for these may result in sub-optimal performance or, very often, completely undesired results.
Enter two of the most fundamental morphological transformations: **erosion** and **dilation**.
### Erosion
Erosion "erodes away the boundaries of foreground object"[^3] by sliding a kernel through the image and set a pixel to 1 **only if all the pixels under the kernel is 1**.
This in effect discard pixels near the boundary and any floating pixels that are not part of a larger blob (which is what the human eye is interested in). Because pixels are eroded, your foreground object will shrink in size.
### Dilation
The opposite of erosion, Dilation sets a pixel to 1 if **at least one pixel under the kernel is 1**, essentially "growing" the foreground object.
Because of how these operations work, there are a couple of things to note:
1. Morphological transformations are usually performed on binary images. Recall that pixel values in binary images are either a full white (i.e 1) or black (i.e 0).
2. As per convention, we want to keep our foregound in white and background in black
3. Because erosion results in a shrinking foreground and dilation results in a growing foreground, these two operations are also commonly used in combinations, i.e erosion followed by dilation, or vice versa
The full code solution is in `morphological_02.py`.
As we read our image in grayscale mode (`flags=0`), we obtain a white blackground and a mostly-black foreground. This is illustrated in the subplot titled "Original" above. We begin our preprocessing steps by first binarizing the image (step 1), followed by inverting the colors (step 2) to get a white-on-black image.
An erosion operation is then performed (step 3). This works by creating our kernel (either through `numpy` or through `opencv`'s structuring element) and sliding that kernel across our image to remove white noises in our image.
The side-effect is that our foreground object has now shrunk in size as it's boundaries are eroded away. We grow it back by applying a dilation (step 4) and finally show the output as illustrated in the bottom-right pane of the image above.
```py
# read as grayscale
roi = cv2.imread("assets/0417s.png", flags=0)
# step 1:
_, thresh = cv2.threshold(roi, 170, 255, cv2.THRESH_BINARY)
# step 2:
inv = cv2.bitwise_not(thresh)
# step 3 (option 1):
kernel = np.ones((5,5), np.uint8)
# step 3 (option 2):
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
eroded = cv2.erode(inv, kernel, iterations=1)
# step 4:
dilated = cv2.dilate(eroded, kernel, iterations=1)
cv2.imshow("Transformed", dilated)
cv2.waitKey(0)
```
OpenCV provides the three shapes for our kernel:
- Rectangular box: `MORPH_RECT`
- Cross: `MORPH_CROSS`
- Ellipse: `MORPH_ELLIPSE`
They are fed as the first argument into `cv2.getStructuringElement()`, with the second being the kernel size (`ksize`) itself. The third argument is the _anchor point_, which defaults to the center.
### Opening and Closing
Another name for **Erosion, followed by Dilation** is the Opening. It is useful in removing noise in our image. The reverse of Opening is Closing, where we first **perform Dilation followed by Erosion**, particularly suited for closing small holes inside foreground objects.
OpenCV includes the more generic `morphologyEx` method for all other morphological operations beyond Erosion and Dilation. The function takes an image as the first argument, an operation as the second operation and finally the kernel. Compare how your code will differ between `cv2.erode` and `cv2.dilate`, and their respective equivalence in `cv2.morphologyEx()`:
```py
import cv2
import numpy as np
img = cv2.imread('image.png',0)
kernel = np.ones((5,5),np.uint8)
erosion = cv2.erode(img,kernel,iterations = 1)
# Equivalent:
# cv2.morphologyEx(img, cv2.MORPH_ERODE, kernel,iterations=1)
dilation = cv2.dilate(img,kernel,iterations = 1)
# Equivalent:
# cv2.morphologyEx(img, cv2.MORPH_DILATE, kernel,iterations=1)
opening = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel)
closing = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
```
### Learn-by-building: Morphological Transformation
In the `homework` directory, you'll find `0417h.png`. Your job is to apply what you've learned in this lesson to clean up the image. Your output should have these qualities:
1. As free of noise as possible (remove the lines, and the red splatted dots across the image)
2. If you run `findContours()` on the output, you should have exactly 4 contours
3. Foreground object in white, background in black
You are free to pick your strategy, but a reference solution would look like the following:
## Seven-segment display
The seven-segment display (known also as "seven-segment indicator") is a form of electronic display device for displaying decimal numerals[^4] widely used in digital clocks, electronic meters, calculators and banking security tokens.
This is relevant because it is the character representation of our digits in each of these security tokens. If we can isolate each digit from each other, we can iteratively predict the "class" of each digit (0 to 9). Specifically, we are going to perform a classification task based on the state of each segment.
To ease our understanding, let's refer to each segment using the letters A to G:
We can then create a lookup table that match the collective states to the corresponding class:
| Class | a | b | c | d | e | f | g |
|------- |--- |--- |--- |--- |--- |--- |--- |
| 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
| 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| 2 | 1 | 1 | 0 | 1 | 1 | 0 | 1 |
| 3 | 1 | 1 | 1 | 1 | 0 | 0 | 1 |
| 4 | 0 | 1 | 1 | 0 | 0 | 1 | 1 |
| 5 | 1 | 0 | 1 | 1 | 0 | 1 | 1 |
| 6 | 1 | 0 | 1 | 1 | 1 | 1 | 1 |
| 7 | 1 | 1 | 1 | 0 | 0 | 1 | 0 |
| 8 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| 9 | 1 | 1 | 1 | 1 | 0 | 1 | 1 |
How would we represent such a lookup table in our Python code and how would we use it? The obvious answer to the first question is a dictionary. Notice that `DIGITSDICT` is just a representation of the "binary state" of each segment. The digit "8" for example correspond to all seven segments being activated, or "on" (state of `1`).
```py
DIGITSDICT = {
(1,1,1,1,1,1,0):0,
(0,1,1,0,0,0,0):1,
(1,1,0,1,1,0,1):2,
(1,1,1,1,0,0,1):3,
(0,1,1,0,0,1,1):4,
(1,0,1,1,0,1,1):5,
(1,0,1,1,1,1,1):6,
(1,1,1,0,0,1,0):7,
(1,1,1,1,1,1,1):8,
(1,1,1,1,0,1,1):9
}
```
Then, for each digit, we would look at the pixel values in each of the seven segments, and if the majority of pixels are white, we would classify that segment as being in an activated state (`1`), otherwise in a state of `0`. As we iterate over the 7 segments, we now have an array of length 7, each element a binary value(`0` or `1`).
We would then find the corresponding value in our dictionary using that array. Your code would resemble the following:
```py
# define the rectangle areas corresponding each segment
sevensegs = [
((x0, y0), (x1, y1)),
((x2, y2), (x3, y3)),
... # 7 of them
]
# initialize the state to OFF
on = [0] * 7
# set each segment to ON / OFF based on majority
for (i, ((p1x, p1y), (p2x, p2y))) in enumerate(sevensegs):
# numpy slicing to extract only one region
region = roi[p1y:p2y, p1x:p2x]
# if majority pixels are white, set state to ON
if np.sum(region == 255) > region.size *0.5:
on[i] = 1
# lookup on dictionary
digit = DIGITSDICT[tuple(on)] # digit is one of 0-9
```
There are multiple ways to write a for-loop but it's important that you are aware of the order in which your for-loop your executing. Referring to our seven-segment illustration below,the first iteration is only concerned with the state of 'A' while the second interation handles the state of 'B', and so on.
Using `enumerate`, we obtain an additional counter (`i`) to our iterable (`sevensegs`); This is convenient for the purpose of setting states. At the first iteration, the first element is our list is conditionally set to 1 if more than half of the pixels in segment 'A' are white. A more detailed example of python's enumeration is in `utils/enumerate.py`.
### Practical Strategies
If you are paying close attention to the digit '0' in our LCD display, you will notice that the absence of the 'G' segment causes a pretty visible and significant gap. When you test your digit recognition script without special consideration to this attribute, you will find it consistently failing to account for the numbers "0","1" and "7". In fact, you may not even be able to isolate the aforementioned numbers altogether using the `findContour` operation, because they were treated as two disjointed pieces instead of a whole piece.
A reasonable strategy to handle this is the Dilation or Closing (Dilation followed by Erosion) operation that you've learned earlier.
Similarly, your ROI may necessitate other pre-processing and the specific tactical solution vary greatly depending on the problem set at hand.
As I inspect the bounding box we retrieved around the LCD screen, the observation that these bouding boxes often have their digits centered around the bottom half of the display led me to insert an additional step prior to the morphological transformation in the final code solution. The step uses numpy subsetting to trim away the top 20% as well as 20% on each side of the image:
```py
roi = cv2.imread("roi.png", flags=0)
RATIO = roi.shape[0] * 0.2
trimmed = roi[
int(RATIO) :,
int(RATIO) : roi.shape[1] - int(RATIO)]
```
That said, whenever possible, you want to be cautious of not hand-tuning your problem in a way that is overly specific to the images you have at hand lest risking the solution **only** working on those specific images and not others, a phenomenon fondly termed as "overfitting" in the machine learning community.
I've re-executed the solution code against some sample image sets, once with the "trimming" in-place and then without the trimming, before settling on the decision. As you will see later, the trimming improves our accuracy and is a relatively safe strategy given how every LCD screen regardless of the issuer (bank) has the same asymmetry with more "blank space" at the top half compared to the bottom half.
#### Contour Properties
Furthermore, in many cases of digit recognition / digit classification you will want to predict the class for each digit in an ordered fashion. Supposed the LCD screen contains the digits "40710382", our algorithm should correctly isolate these digits, classify them iteratively, but do so from the leftmost digit to the rightmost. Failing to account for this may result in your algorithm correctly classifying each digit, but produce an unreasonable output such as "1740238".
There are a few strategies you can employ here. We've seen in `contourarea_01.py` and `contourarea_02.py` how contour has attributes that can be retrieved using the `contourArea()` and `arcLength()` functions. Inspect the following snippet and it should help jog your memory:
```py
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:9]
for i, cnt in enumerate(cnts):
cv2.drawContours(img_color, cnts, i, BCOLOR, THICKNESS)
area = cv2.contourArea(cnt)
peri = cv2.arcLength(cnt, closed=True)
print(f"Area:{area}; Perimeter: {peri}")
```
Indeed, we're using countour area as a good indicator to search for our region of interest. When we take this idea a little further, we can further place a constraint on our search criteria. In the following code, we draw a bounding rectangle and for an extra layer of precaution, only takes any bounding boxes that are taller than 20 pixels (step 1).
Calling `boundingRect()` on a contour returns 4 values, respectively the x and y coordinate along with the width and height of the contour.
We then use another property of the contour, its top-left coordinate to determine the logical order of our digits. Specifically, we use the first returned value (`cv2.boundingRect(cnt)[0]`) since that's the x value for the top-left coordinate of each region. By sorting against this value, our digits are stored in the Python list in an ordered fashion, determined by their respective coordinate value.
```py
digits_cnts = []
cnts, _ = cv2.findContours(eroded, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
for cnt in cnts:
(x, y, w, h) = cv2.boundingRect(cnt)
# step 1
if h > 20:
digits_cnts += [cnt]
# step 2
sorted_digits = sorted(digits_cnts, key=lambda cnt: cv2.boundingRect(cnt)[0])
```
When we put these together, we now have a complete pipeline:
The full solution code is in `digit_01.py` but the essential parts are as follow:
```py
import cv2
import numpy as np
# step 1:
DIGITSDICT = {
(1, 1, 1, 1, 1, 1, 0): 0,
(0, 1, 1, 0, 0, 0, 0): 1,
(1, 1, 0, 1, 1, 0, 1): 2,
(1, 1, 1, 1, 0, 0, 1): 3,
(0, 1, 1, 0, 0, 1, 1): 4,
(1, 0, 1, 1, 0, 1, 1): 5,
(1, 0, 1, 1, 1, 1, 1): 6,
(1, 1, 1, 0, 0, 1, 0): 7,
(1, 1, 1, 1, 1, 1, 1): 8,
(1, 1, 1, 1, 0, 1, 1): 9,
}
# step 2
roi = cv2.imread("inter/ocbc-roi.png", flags=0)
# step 3
RATIO = roi.shape[0] * 0.2
roi = cv2.bilateralFilter(roi, 5, 30, 60)
trimmed = roi[int(RATIO) :, int(RATIO) : roi.shape[1] - int(RATIO)]
# step 4
edged = cv2.adaptiveThreshold(
trimmed, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 5, 5
)
# step 5
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 5))
dilated = cv2.dilate(edged, kernel, iterations=1)
eroded = cv2.erode(dilated, kernel, iterations=1)
# step 6
cnts, _ = cv2.findContours(eroded, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
digits_cnts = []
for cnt in cnts:
(x, y, w, h) = cv2.boundingRect(cnt)
if h > 20:
digits_cnts += [cnt]
# step 7
sorted_digits = sorted(digits_cnts, key=lambda cnt: cv2.boundingRect(cnt)[0])
# step 8
digits = []
for cnt in sorted_digits:
# step 8a
(x, y, w, h) = cv2.boundingRect(cnt)
roi = eroded[y : y + h, x : x + w]
qW, qH = int(w * 0.25), int(h * 0.15)
fractionH, halfH, fractionW = int(h * 0.05), int(h * 0.5), int(w * 0.25)
# step 8b
sevensegs = [
((0, 0), (w, qH)), # a (top bar)
((w - qW, 0), (w, halfH)), # b (upper right)
((w - qW, halfH), (w, h)), # c (lower right)
((0, h - qH), (w, h)), # d (lower bar)
((0, halfH), (qW, h)), # e (lower left)
((0, 0), (qW, halfH)), # f (upper left)
# ((0, halfH - fractionH), (w, halfH + fractionH)) # center
(
(0 + fractionW, halfH - fractionH),
(w - fractionW, halfH + fractionH),
), # center
]
# step 8c
on = [0] * 7
for (i, ((p1x, p1y), (p2x, p2y))) in enumerate(sevensegs):
region = roi[p1y:p2y, p1x:p2x]
print(
f"{i}: Sum of 1: {np.sum(region == 255)}, Sum of 0: {np.sum(region == 0)}, Shape: {region.shape}, Size: {region.size}"
)
if np.sum(region == 255) > region.size * 0.5:
on[i] = 1
print(f"State of ON: {on}")
# step 8d
digit = DIGITSDICT[tuple(on)]
print(f"Digit is: {digit}")
digits += [digit]
# step 9
cv2.rectangle(canvas, (x, y), (x + w, y + h), CYAN, 1)
cv2.putText(canvas, str(digit), (x - 5, y + 6), FONT, 0.3, (0, 0, 0), 1)
cv2.imshow("Digit", canvas)
cv2.waitKey(0)
print(f"Digits on the token are: {digits}")
```
- Step 1: Initialize the lookup dictionary
- Step 2: Read our ROI image using OpenCV
- Step 3: Noise reduction and trim away asymmetrical white space in our ROI
- Step 4: Binarize our image using adaptive thresholding
- Step 5: Morphological transformation to remove noise and fill the small holes in our digit
- Step 6: Find contours in our image with a height greater than 20px
- Step 7: Sort the contours in-place, using the x value of their coordinates (hence, left to right)
- Step 8
- Step 8a: Create rectangle bounding box on each digit, and some convenience units that we later use to slice the seven segments. Notice that these convenience units are not hard-coded values, but are proportional to the Height (`h`) of our rectangular box
- Step 8b: Slice the seven segments; The first segment ("A") is from point (0,0) to (w, `int(h * 0.15)`); This segment is `w` in width and 15% the height of the full digit contour, starting from position (0, 0)
- Step 8c: Initialize the state to `0` for each of the 7 segments, then conditionally set regions with more white than black pixels to `1`
- Step 8d: Once all 7 states have been set, perform lookup against the digit dictionary created in step 1; Append the value to the `digits` list created at the beginning of step 8
- Step 9: Draw rectangle and add predicted text for each bounding box. Finally, use a print statement to print the `digits` list.
# References
[^1]: LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278–2324
[^2]: Saliency map, Wikipedia
[^3]: Morphological Transformations, OpenCV Documentation
[^4]: Seven-segment display, Wikipedia
[^5]: Seven-segment display character representations, Wikipedia
================================================
FILE: digitrecognition/morphological_01.py
================================================
import cv2
import matplotlib.pyplot as plt
roi = cv2.imread("inter/ocbc-roi.png", flags=0)
blurred = cv2.bilateralFilter(roi, 5, 30, 60)
edged = cv2.adaptiveThreshold(
blurred, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 5, 5
)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 5))
dilated = cv2.dilate(edged, kernel, iterations=1)
plt.subplot(2, 2, 1), plt.imshow(roi, cmap="gray")
plt.title("Original"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 2), plt.imshow(blurred, cmap="gray")
plt.title("Blurred"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 3), plt.imshow(edged, cmap="gray")
plt.title("Edged"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 4), plt.imshow(dilated, cmap="gray")
plt.title("Dilated"), plt.xticks([]), plt.yticks([])
plt.show()
================================================
FILE: digitrecognition/morphological_02.py
================================================
import cv2
import matplotlib.pyplot as plt
roi = cv2.imread("assets/0417s.png", flags=0)
cv2.imshow("Original", roi)
cv2.waitKey(0)
_, thresh = cv2.threshold(roi, 170, 255, cv2.THRESH_BINARY)
# thresh = cv2.adaptiveThreshold(dilated, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, 5, 5)
cv2.imshow("Threshold", thresh)
cv2.waitKey(0)
inv = cv2.bitwise_not(thresh)
cv2.imshow("Inverted", inv)
cv2.waitKey(0)
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (4, 4))
eroded = cv2.erode(inv, kernel, iterations=1)
cv2.imshow("Eroded", eroded)
cv2.waitKey(0)
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (6, 6))
dilated = cv2.dilate(eroded, kernel, iterations=1)
cv2.imshow("Dilated", dilated)
cv2.waitKey(0)
plt.subplot(2, 2, 1), plt.imshow(roi, cmap="gray")
plt.title("Original"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 2), plt.imshow(thresh, cmap="gray")
plt.title("Thresholded"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 3), plt.imshow(inv, cmap="gray")
plt.title("Inverted"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 4), plt.imshow(dilated, cmap="gray")
plt.title("Transformed"), plt.xticks([]), plt.yticks([])
plt.show()
================================================
FILE: digitrecognition/roi_01.py
================================================
import cv2
BCOLOR = (75, 0, 130)
THICKNESS = 4
img_color = cv2.imread("assets/ocbc.jpg")
img_color = cv2.resize(img_color, None, None, fx=0.5, fy=0.5)
img = cv2.cvtColor(img_color, cv2.COLOR_BGR2GRAY)
x,y,w,h = cv2.selectROI("Region of interest", img)
print(x,y,w,h)
cropped = img[y:y+h, x:x+w]
cv2.imshow("Cropped", cropped)
cv2.waitKey(0)
cv2.rectangle(img_color, (x,y), (x+w,y+h), (255,0,0), 2)
cv2.imshow("Original Image", img_color)
cv2.waitKey(0)
================================================
FILE: digitrecognition/roi_02.py
================================================
import cv2
import argparse
import re
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--path", default="assets/ocbc.jpg")
args = vars(parser.parse_args())
# test: dbs.jpg | ocbc.jpg
img_color = cv2.imread(args["path"])
img_color = cv2.resize(img_color, None, None, fx=0.5, fy=0.5)
img = cv2.cvtColor(img_color, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(img, (7, 7), 0)
blurred = cv2.bilateralFilter(blurred, 5, sigmaColor=50, sigmaSpace=50)
edged = cv2.Canny(blurred, 130, 150, 255)
cv2.imshow("Outline of device", edged)
cv2.waitKey(0)
cnts, _ = cv2.findContours(edged, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# sort contours by area, and get the largest
cnts = sorted(cnts, key=cv2.contourArea, reverse=True)[:1]
cv2.drawContours(img_color, cnts, 0, (75, 0, 130), 4)
cv2.imshow("Target Contour", img_color)
cv2.waitKey(0)
x, y, w, h = cv2.boundingRect(cnts[0])
roi = img[y : y + h, x : x + w]
cv2.imshow("ROI", roi)
img_name = re.search("(?<=\/)(.*)(?=\.jpg)", args["path"]).group(1)
cv2.imwrite(f"inter/{img_name}-roi.png", roi)
cv2.waitKey(0)
================================================
FILE: digitrecognition/utils/enumerate.py
================================================
digits = ['a', 'b', 'c', 'd']
contracts = {
# salesperson: contract value, duration
'adam':(500, 2),
'brian':(300, 1.5),
'canny':(1000, 4)
}
# for i in range(len(digits)):
# print(i, digits[i])
# better written as:
for i, d in enumerate(digits):
print(i, d)
print('---')
print(dict(enumerate(digits)))
for i, c in enumerate(contracts):
print(i, c)
for i, v in enumerate(contracts.values()):
print(i, v)
d = {i+1:(k,f'${v1} for {v2} years') for i, (k,(v1, v2)) in enumerate(contracts.items())}
print(d)
================================================
FILE: edgedetect/adaptivethresholding_01.py
================================================
import cv2
import matplotlib.pyplot as plt
import numpy as np
img = cv2.imread("assets/sudoku.jpg", flags=0)
_, img_threshold = cv2.threshold(img, 50, 255, cv2.THRESH_BINARY)
img = cv2.medianBlur(img, 5)
mean_adaptive = cv2.adaptiveThreshold(
img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 11, 2
)
gaussian_adaptive = cv2.adaptiveThreshold(
img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2
)
plt.subplot(2, 2, 1), plt.imshow(img, cmap="gray")
plt.title("Original"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 2), plt.imshow(img_threshold, cmap="gray")
plt.title("Binary Threshold (global:50)"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 3), plt.imshow(mean_adaptive, cmap="gray")
plt.title("Mean Adaptive"), plt.xticks([]), plt.yticks([])
plt.subplot(2, 2, 4), plt.imshow(gaussian_adaptive, cmap="gray")
plt.title("Gaussian Adaptive"), plt.xticks([]), plt.yticks([])
plt.show()
================================================
FILE: edgedetect/canny_01.py
================================================
import numpy as np
import cv2
import matplotlib.pyplot as plt
img = cv2.imread("assets/castello.png", flags=0)
img = cv2.medianBlur(img, 9)
img = cv2.GaussianBlur(img, (9, 9), 0)
def sobel(img, k):
gradient_x = cv2.Sobel(img, cv2.CV_64F, 1, 0)
gradient_y = cv2.Sobel(img, cv2.CV_64F, 0, 1)
gradient_x = cv2.convertScaleAbs(gradient_x)
gradient_y = cv2.convertScaleAbs(gradient_y)
return cv2.addWeighted(gradient_x, 0.5, gradient_y, 0.5, 0)
sobel = sobel(img, 3)
canny = cv2.Canny(img, 50, 180)
plt.subplot(1, 2, 1)
plt.imshow(sobel, cmap="gray")
plt.title("Sobel Edge Detector"), plt.xticks([]), plt.yticks([])
plt.subplot(1, 2, 2)
plt.imshow(canny, cmap="gray")
plt.title("Canny Edge Detector"), plt.xticks([]), plt.yticks([])
plt.show()
================================================
FILE: edgedetect/contour_01.py
================================================
import cv2
import numpy as np
image = cv2.imread("assets/pens.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
cv2.imshow("Grayscale", image)
cv2.waitKey(0)
image = cv2.GaussianBlur(image, (3, 3), 0)
cv2.imshow("After Smoothing", image)
cv2.waitKey(0)
def sobel(image):
# run with col.png for best effect
# cv2.Sobel last 2 argument -> order of derivatives in x and y direction respectively
sobelX = cv2.Sobel(image, cv2.CV_64F, 1, 0) # find vertical edges
sobelY = cv2.Sobel(image, cv2.CV_64F, 0, 1) # find horizontal edges along y-axis
gradient_x = np.uint8(np.absolute(sobelX))
gradient_y = np.uint8(np.absolute(sobelY))
sobelCombined = cv2.bitwise_or(gradient_x, gradient_y)
cv2.imshow("Sobel Combined", sobelCombined)
cv2.waitKey(0)
return sobelCombined
def counting_penguins(sobel, image):
sobeled = sobel(image)
_, edged = cv2.threshold(sobeled, 20, 255, cv2.THRESH_BINARY)
cv2.imshow("(Edged)", edged)
cv2.waitKey(0)
cnts, _ = cv2.findContours(
# does this need to be changed?
edged,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE,
)
canvas = np.ones(image.shape)
cv2.drawContours(canvas, cnts, -1, (0, 255, 255), 1)
cv2.imshow("Contour", canvas)
cv2.waitKey(0)
print(f"Found {len(cnts)} penguins")
if __name__ == "__main__":
counting_penguins(sobel, image)
================================================
FILE: edgedetect/contourapprox.py
================================================
import cv2
import numpy as np
image = cv2.imread("homework/equal.png", flags=0)
cv2.imshow("Original", image)
cv2.waitKey(0)
def edge(image):
_, edged = cv2.threshold(image, 220, 255, cv2.THRESH_BINARY_INV)
cv2.imshow("(Edged)", edged)
cv2.waitKey(0)
cnts, _ = cv2.findContours(
# does this need to be changed?
edged,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE,
)
print(f"Cnts Simple Shape (1): {cnts[0].shape}")
print(f"Cnts Simple Shape (2): {cnts[0].shape}")
cnts2, _ = cv2.findContours(
# does this need to be changed?
edged,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_NONE,
)
print(f"Cnts NoApprox Shape:{cnts2[0].shape}")
print(cnts)
canvas = np.ones(image.shape)
cv2.drawContours(canvas, cnts, -1, (0, 255, 255), 1)
cv2.imshow("Contour", canvas)
cv2.waitKey(0)
print(f"Found {len(cnts)} shapes!")
if __name__ == "__main__":
edge(image)
================================================
FILE: edgedetect/edgedetect.html
================================================
An edge can be defined as boundary between regions in an image. Edge detection techniques we'll learn in this course builds upon what we've learned from our lessons in kernel convolution. It is the process of using kernels to reduce the information in our data and preserving only the necessary structural properties in our image.
Gradient points in the direction of the most rapid increase in intensity. When we apply a gradient based edge detection method, we are searching for the maximum and minimum in the first derivative of the image.
When we apply our convolution onto the image, we are finding for regions in the image where there's a sharp change in intensity or color. Arguably the most common edge detection method using this approach is the Sobel Operator.
The Sobel operator applies a filtering operation to produce an image output where the edge is emphasized. It convolves our original image using two 3x3 kernels to capture approximations of the derivatives in both the horizontal and vertical directions.
The x-direction and y-direction kernels would be:
$$G_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix} G_y = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 0 & 0 \\ -1 & -2 & -1 \end{bmatrix}$$
Each kernel is applied separately to obtain the gradient component in each orientation, $G_x$ and $G_y$. Expressed in formula, the gradient magnitude is:
$$|G| = \sqrt{G^2_x + G^2_y}$$
Where the slope $\theta$ of the gradient is calculated as follow:
$$\theta(x,y)=tan^{-1}(\frac{G_y}{G_x})$$
If the two formula above confuses you, read on as we unpack these ideas one at a time.
In computer vision literature, you'll often hear about "taking the derivative" and this may erve as a source of confusion for beginning practitioners since "derivatives" is often thought of in the context of a continuous function. Images are a 2D matrix of discrete values, so how do we wrap our head around the idea of finding derivative?
But why do we even bother with derivatives when this course is suppopsed to be about edge detection in images?
Among the many ways to answer the question, my favorite being that image is really just a function. When it treat an image as a function, the utility of taking derivatives become a little more obvious. In the image below, supposed you want to count the number of windows in this area of Venezia Sestiere Cannaregio, your program can look for large derivatives since there are sharp changes in pixel intensity from the windows to the surrounding wall:
The code to generate the surface plot above is in img2surface.py.
Going back to our x-direction kernel in the Sobel Operator.
This kernel has all 0 in the middle, which is quite easy to intuit about. Essentially, for each pixel in our image, we want to compute its derivative in the x-direction by approximating a formula that you may have come across in your calculus class:
$$f'(x) = \lim_{h\to0}\frac{f(x+h)-f(x)}{h}$$
This approximation is also called 'forward difference', because we're taking a value of $x$, and computing the difference in $f(x)$ as we increment it by a small amount forward, denoted as $h$.
And as it turns out, using the 'central difference' to compute the derivative of our discrete signal can deliver better results:
$$f'(x) = \lim_{h\to0}\frac{f(x+0.5h)-f(x-0.5h)}{h}$$
To make this more concrete, we can plug the formula into an actual array of pixels:
$$[0, 255, 65, \underline{180}, 255, 255, 255]$$
when we set $h=2$ at the center pixel (index of value 180), we have the following:
$$\begin{aligned} f'(x) & = \lim_{h\to0}\frac{f(x+0.5h)-f(x-0.5h)}{h}\\ & = \frac{f(x+1)-f(x-1)}{2} \\ & = \frac{255-65}{2} \\ & = 95 \end{aligned}$$
Notice that a large part of the calculation we just perform is synonymous to a 1D convolution operation using a $\begin{bmatrix} -1 & 0 & 1 \end{bmatrix}$ kernel.
When the same 1x3 kernel $\begin{bmatrix} -1 & 0 & 1 \end{bmatrix}$ is applied on the right-most part of the image where its just white space ([..., 255, 255, 255]) the kernel would evaluate to 0. In other words, our derivative filter returns no response where it can't detect a sharp change in pixel intensity.
As a reminder, the x-direction kernel in our Sobel Operator is the following:
$$G_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix}$$
This takes our 1x3 kernel and instead of convolving one row of pixels at a time, extends it to convolve at 3x3 neighborhoods at a time using a weighted average approach.
The two kernels (one for horizontal and another for vertical edge detection) can be constructed, respectively, like the following:
sobel_x = np.array([[1, 0, -1],
[2, 0, -2],
[1, 0, -1]])
sobel_y = np.array([[1, 2, 1],
[0, 0, 0],
[-1, -2, -1]])
You may have guessed that, given its role in digital image processing, opencv have included a method that performs our Sobel Operator for us, and thankfully there is. Here's an example of using the cv2.Sobel(src, ddepth, dx, dy, dst=None, ksize) method:
gradient_x = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=3)
gradient_y = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=3)
print(f"Range: {np.min(gradient_x)} | {np.max(gradient_x)}")
gradient_x = np.uint8(np.absolute(gradient_x))
gradient_y = np.uint8(np.absolute(gradient_y))
print(f"Range uint8: {np.min(gradient_x)} | {np.max(gradient_x)}")
cv2.imshow("Gradient X", gradient_x)
cv2.imshow("Gradient Y", gradient_y)
The code above, extracted from sobel_01.py reinforces a couple of ideas that we've been working on. It shows that:
- the $G_x$ and $G_y$, gradients of the image, are computed separately through the convolution of two different Sobel kernels
- $G_x$ and $G_y$ responded to the change in pixel values along the x-direction and y-direction respectively, as visualized in the illustration above
- convolution using the two Sobel filters may, and often will, produce a value outside the range of 0 and 255. Given the presence of [-1, -2, -1] in one side of our kernel, mathematically this may lead to an output value of -1020. To store the values from these convolutions we use a 64-bit floating point (
cv2.CV_64F). OpenCV suggests to "keep the output datatype to some higher form such as cv2.CV_64F, take its absolute value and then convert back to cv2.CV_8U."
While the code above certainly works, OpenCV also has a method that scales, calculates absolute values and converts the result to 8-bit. cv2.convertScaleAbs(src, dst, alpha=1, beta=0) performs the following:
$$dst(I) = cast<uchar>(|src(I) * \alpha + \beta|)$$
gradient_x = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=3)
gradient_y = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=3)
gradient_x = cv2.convertScaleAbs(gradient_x)
gradient_y = cv2.convertScaleAbs(gradient_y)
print(f"Range: {np.min(gradient_x)} | {np.max(gradient_x)}")
At the beginning of this course I said that images are really just 2d functions before showing you the intricacies of our Sobel kernels. We saw the clever design of both the x- and y-direction kernels, by borrowing from the concept of "taking the derivatives" you often see in calculus text books.
But on a really basic level, these kernels only return the x and y edge responses. These are not the image gradient, just pure arithmetic values from following the convolution process. To get to the final form (where the edges in our image are emphasized) we still need to compute the gradient direction and magnitude for each point in our image.
This brings us back to our original formula. Recall that the x-direction and y-direction kernels are:
$$G_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix} G_y = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 0 & 0 \\ -1 & -2 & -1 \end{bmatrix}$$
We understand that each kernel is applied separately to obtain the gradient component in each orientation, $G_x$ and $G_y$. What is the significance of this? Well as it turns out if we know the shift in the x-direction and the corresponding change in value in the y-direction, then we can use the pythagorean theorem to approximate the "length of the slope", a concept that many of you are familiar with.
Expressed in formula, the gradient magnitude is hence:
$$|G| = \sqrt{G^2_x + G^2_y}$$
Along with the well-known mathematical formula that is Pythagorean theorem, some of you may also have some familiarity with the three trigonometric functions. Particularly, the tangent function tells us that in a right triangle, the tangent of an angle is the length of the opposite side divided by the length of the adjacent side.
This leads us to the following expression:
$$tan(\theta_{(x,y)})=\frac{G_y}{G_x}$$
To rewrite the expression above, we arrive at the formula to capture the gradient's direction:
$$\theta_{(x,y)}=tan^{-1}(\frac{G_y}{G_x})$$
This whole idea is also illustrated in code, and the script is provided to you:
gradient.py to generate the vector field in the picture above (right)img2surface.py on the penguin image in the assets folder generates the surface plot
Succinctly, supposed the two 3x3 kernels do not fire a response (for example when no edges are detected in the white background of our penguin), both $G_x$ and $G_y$ will be 0, which leads to a gradient magnitude of 0. You can compute these by hand, let OpenCV's implementation handle that for you, or use numpy as illustrated in gradient.py:
dY, dX = np.gradient(img)
Image segmentation is the process of decomposing an image into parts for further analysis. This has many utility:
- Background subtraction in human motion analysis
- Multi-object classification
- Find region of interest for OCR (optical character recognition)
- Count pedestrians from a streamed video source
- Isolating vehicle registration plates (license plate) and vehicle models from a busy highway scene
Current literature on image segmentation techniques can be classified into:
- Intensity-based segmentation
- Edge-based segmentation
- Region-based semantic segmentation
It's important to note, however, that the rise in popularity of deep learning framework and techniques has ushered a proliferation of new methods to perform what was once a highly difficult task. In future lectures, we'll explore image segmentation in far greater details. In this course, we'll study intensity-based segmentation and edge-based segmentation methods.
Intensity-based method is perhaps the simplest as intensity is the simplest property that pixels can share.
To make a more concrete case of this, let's assume you're working with a team of researchers to build an AI-based "sudoku solver" that, unimaginatively, will compete against human sudoku players in an attempt to further stake the claim in an ongoing debate of AI superiority.
While your teammates work on the algorithmic design for the actual solver, your task is comparatively straightforward: write a script to scan newspaper images (or print magazines), binarize them to discard everything except the digits in the sudoku puzzle.
This presents a great opportunity to use an intensity-based segmentation technique we spoke about earlier.
In intensitytresholding_01.py, you'll find a code demonstration of the numerous thresholding methods provided by OpenCV. In total, there are 5 simple thresholding methods: THRESH_BINARY, THRESH_BINARY_INV, THRESH_TRUNC, THRESH_TOZERO and THRESH_TOZERO_INV.
The method call between all of them are identical:
cv2.threshold(img, thresh, maxval, type)
We specify our source image img (usually in grayscale), a threshold value thresh used to binarize the image pixels, and a max value maxval for the pixel value to use for any pixel that crosses our threshold.
The mathematical functions for each one of them:
They're collectively known as simple thresholding in OpenCV because they use a global threshold value; Any pixels smaller than the threshold is set to 0 otherwise it is set to the maxval value.
The probably sound too simplistic for anything beyond the simplest of real-world images, and for the majority of cases they are. They call for proper judgment of the task at hand.
Applying the various types of simple thresholding method on our sudoku image, we observe that the digits are for the most part extracted successfully while the background information are greatly reduced:
Refer tointensitythresholding_01.py for the full code.
As a simple homework, try to practice simple thresholding on the car2.png located in your homework folder. To reduce noise, you may have to combine a blurring operation prior to thresholding. As you practice, pay attention to the interaction between your threshold values and the output. Later in the course, you'll learn how to draw contours, which would come in handy in producing the final output:
As you work on your homework, you will notice that given the varying lighting condition across the different region of our image, regardless of the global value we pick we either have a threshold value that is too low or too high.
Using a global value as an intensity threshold may work in particular cases but may be overly naive to perform well when, say, an image has different lighting conditions in different areas. A great example of this case is the object extraction exercise you performed using car2.png.
Adaptive thresholding is not a lot different from the aforementioned thresholding techniques, except it determines the threshold for each pixel based on its neighborhood. This in effect mans that the image is assigned different thresholds across the different regions, leading to a cleaner output when our image has different degrees of illumination.
The method is called with the source image (src), a max value (maxValue), the method (adaptiveMethod), a threshold type (thresholdType), the size of the neighborhood (blockSize) and a constant (C) that is subtracted from the mean or the weightted sum of the neighborhood pixels.
mean_adaptive = cv2.adaptiveThreshold(
img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 11, 2
)
gaussian_adaptive = cv2.adaptiveThreshold(
img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2
)
The code, taken from adaptivethresholding_01.py produces the following:
Edge-based segmentation separates foreground objects by first identifying all edges in our image. Sobel Operator and other gradient-based filter function are good and well-known candidates for such an operation.
Once we obtain the edges, we perform the contour approximation operation using the findContours method in OpenCV. But what exactly are contours?
In OpenCV's words,
Contours can be explained simply as a curve joining all the continuous points (along the boundary), having same color or intensity. The contours are a useful tool for shape analysis and object detection and recognition.
If we have "a curve joining all the continuous points along the boundary", then we are able to extract this object. If we wish to count the number of contours in our image, the method also convenient return a list of all the found contours, making it easy to perform len() on the list to retrieve the final value.
There are three arguments to the findContours() function, first being the source image, second is the retrieval mode and last is the contour approximation method. Both the contour retrieval mode and approximation method is discussed in the next sub-section.
(cnts, hierarchy) = cv2.findContours(
img,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE,
)
The function returns the contours and hierarchy, with contours being a list of all the contours in the image. Each contour is a Numpy array of (x,y) coordinates of boundary points of the object, giving each contour a shape of (n, x, y).
What this allow us to do, is to combine the contours we retrieved with the cv2.drawContours() function either individually, exhaustively in a for-loop fashion, or just everything in one go.
Assuming img being the image we want to draw our contours on, the following code demonstrates these different methods:
cv2.drawContours(img, cnts, -1, (0,255,0), 3)
cv2.drawContours(img, cnts, 2, (0,255,0), 3)
cnt_selected = [cnts[0], cnts[3], cnts[4]]
cv2.drawContours(canvas, cnt_selected, -1, (0, 255, 255), 1)
cv2.drawContours(img, contours, 3, (0,255,0), 3)
The first argument to this function being the source image, the second is the contours as a Python list, the third is the index of contours and remaining arguments are color and thickness of contour lines respectively.
One common problem beginners can run into is to perform the findContours operation on the grayscale image instead of the binary image, leading to poorer accuracy.
When we execute contour_01.py, we notice that the drawContour operation yields the following output:
There are 5 occurrences where our findContours function incorrectly approximated the wrong contour because two penguins were too close to each other. When we execute len(cnts), we will find that the returned value is 5 less than the actual count.
Try to fix contour_01.py by performing the contour approximation on our binary image using the thresholding technique you've learned in previous section.
Contour Retrieval and Approximation
In the findContours() function call, we passed our image to src in the first argumet. The second argument is the contour retrieval mode, and there are documentation for 4 of them:
RETR_EXTERNAL: retrieves only the extreme outer contours (see image below for reference)RETR_LIST: retrieves all contours without establishing any hierarchical relationshipsRETR_CCOMP: retrieves all contours and organize them into a two-level hierarchy (external boundary + boundaries of the holes)RETR_TREE: retrieves all of the contours and reconstructs a full hierarchy of nested contours
In our case, we don't particularly care about the hierarchy, and so the second to fourth method all has the same effect. In other cases, you may experiment with a different contour retrieval method to obtain both the contours and the hierarchy for further processing.
What about the last parameter passed to our findContours method?
Recall that contours are just boundaries of a shape? In a sense, it is an array of (x,y) coordinates used to "record" the boundary of a shape. Given this collection of coordinates, we can then recreate the boundary of our shape. This begs the next question: how many set of coordinates do we need to store to recreate our boundary?
Supposed we perform the findContour operation on an image of two rectangles, one method it may use to achieve that is to store as many points around these rectangle boxes as possible? When we set cv2.CHAIN_APPROX_NONE, that is in fact what the algorithm would do, resulting in 658 points around the border of the top rectangle:
However, notice the more efficient solution would have been to store only the 4 coordinates at each corner of the rectangle. The contour is perfectly represented and recreated using just 4 points for each rectangle, resulting in a total number of 8 points compared to 1,316 points. cv2.CHAIN_APPROX_SIMPLE is an implementation of this, and you can find the sample code below:
cnts, _ = cv2.findContours(
edged,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE,
)
print(f"Cnts Simple Shape (1): {cnts[0].shape}")
cnts2, _ = cv2.findContours(
edged,
cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_NONE,
)
print(f"Cnts NoApprox Shape:{cnts2[0].shape}")
The full script for the experiment above is in contourapprox.py.
You may, at this point, hop to the Learn By Building section to attempt your homework.
John Canny developed a multi-stage procedure that, some 30 years later, is "still a state-of-the-art edge detector". Better edge detection algorithms usually require greater computational resources -- and consequently -- longer processing times -- or a greater number of parameters, in an area where algorithm speed is oftentimes the most important criteria. For the reasons above along with its general robustness, the canny edge algorithm has become one of the "most important methods to find edges" even in modern literature.
I said it's a multi-stage procedure, because the technique as described in his original paper, computational theory of edge detection, works as follow:
- Gaussian smoothing
- Noise reduction using a 5x5 Gaussian filter
- Compute gradient magnitudes and angles
- Apply non-maximum suppression (NMS)
- Suppress close-by edges that are non-maximal, leaving only local maxima as edges
- Track edge by hysterisis
- Suppress all other edges that are weak and not connected to strong edges and link the edges
Step (1) and (2) in the procedure above can be achieved using code we've written so far in our Sobel Operator scripts. We use the Sobel mask filters to compute $G_x$ and $G_y$, respectively the gradient component in each orientation. We then compute the gradient magnitude and the angle $\theta$:
Gradient magnitude:
$$|G| = \sqrt{G^2_x + G^2_y}$$
And recall that the slope $\theta$ of the gradient is calculated as follow:
$$\theta(x,y)=tan^{-1}(\frac{G_y}{G_x})$$
Step (3) in the procedure is another common technique in computer vision known as the non-maximum suppression (NMS). Let's begin by taking a look at the output of our Sobel edge detector from earlier exercises:
Notice as we zoom in on the output image, we can see the gradient-based method did create our strong edges, but it also created "weak" edges it find in our image. Because it is not a parameterized function -- the edge is computed using values of the gradient magnitude and direction -- we have to rely on an additional mechanism for the edge thinning operation with the criterion being one accurate response to any given edge.
Non-maximum suppression help us obtain the strongest edge by suppressing all the gradient values, i.e. setting them to 0 except for the local maxima, which indicate locations with the sharpest change of intensity value. In the words of OpenCV:
After getting gradient magnitude and direction, a full scan of image is done to remove any unwanted pixels which may not constitute the edge. For this, at every pixel, pixel is checked if it is a local maximum in its neighborhood in the direction of gradient. If point A is on the edge, and point B and C are in gradient directions, point A is checked with point B and C to see if it forms a local maximum. If so, it is considered for next stage, otherwise, it is suppressed (put to zero).
The output of step (3) is a binary image with thin edges.
The code demonstrates how you would code such an NMS for the purpose of canny edge detection.
The final step of this multi-stage algorithm decides which among all edges are really edges and which of them are not. It accomplishes this using two threshold values, specified when we call the cv2.Canny() function:
canny = cv2.Canny(img, threshold1=50, threshold2=180)
Any edges with an intensity gradient above threshold2 are considered edges and any edges below threshold1 are considered non-edges and so are suppressed.
The edges that lie between these two values (in our code above, edges with intensity gradient between 50 and 180) are classified as edges if they are connected to sure-edge pixels (the ones above 180) otherwise they are also discarded.
This stage also removes small pixels ("noises") on the assumption that edges are long lines ("connected").
The full procedure is implemented in a single function, cv2.Canny() and the first three parameters are required, respectively being the input image, the first and second threshold value. canny_01.py implements this and compare that to the Sobel Edge detector we developed earlier:
In the homework directory, you'll find a picture of scattered lego bricks lego.jpg. Exactly the kind of stuff you don't want on your bedroom floor, as anyone living with kids at home would testify.
Your job is to apply what you've learned in this lesson to combine what you've learned from the class in kernel convolutions and Edge Detection (kernel.md) to build a lego brick counter.
Note that there are many ways you can build an edge detection. Given what you've learned so far, there are at least 3 equally adequate routines you can apply for this particular problem set.
For the sake of this exercise, your script should feature the use of a Sobel Operator (or a similar gradient-based edge detection method) since this is the main topic of this chapter.
================================================
FILE: edgedetect/edgedetect.md
================================================
# Definition
An edge can be defined as boundary between regions in an image[^1]. Edge detection techniques we'll learn in this course builds upon what we've learned from our lessons in kernel convolution. It is the process of using kernels to reduce the information in our data and preserving only the necessary structural properties in our image[^1].
# Gradient-based Edge Detection
Gradient points in the direction of the most rapid increase in intensity. When we apply a gradient based edge detection method, we are searching for the maximum and minimum in the first derivative of the image.
When we apply our convolution onto the image, we are finding for regions in the image where there's a sharp change in intensity or color. Arguably the most common edge detection method using this approach is the Sobel Operator.
## Sobel Operator
The `Sobel` operator applies a filtering operation to produce an image output where the edge is emphasized. It convolves our original image using two 3x3 kernels to capture approximations of the derivatives in both the horizontal and vertical directions.
The x-direction and y-direction kernels would be:
$$G_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix}
G_y = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 0 & 0 \\ -1 & -2 & -1 \end{bmatrix}
$$
Each kernel is applied separately to obtain the gradient component in each orientation, $G_x$ and $G_y$. Expressed in formula, the gradient magnitude is:
$$|G| = \sqrt{G^2_x + G^2_y} $$
Where the slope $\theta$ of the gradient is calculated as follow:
$$\theta(x,y)=tan^{-1}(\frac{G_y}{G_x})$$
If the two formula above confuses you, read on as we unpack these ideas one at a time.
### Intuition: Discrete Derivative
In computer vision literature, you'll often hear about "taking the derivative" and this may erve as a source of confusion for beginning practitioners since "derivatives" is often thought of in the context of a continuous function. Images are a 2D matrix of discrete values, so how do we wrap our head around the idea of finding derivative?
But why do we even bother with derivatives when this course is suppopsed to be about edge detection in images?
Among the many ways to answer the question, my favorite being that image is really just a function. When it treat an image as a function, the utility of taking derivatives become a little more obvious. In the image below, supposed you want to count the number of windows in this area of Venezia Sestiere Cannaregio, your program can look for large derivatives since there are sharp changes in pixel intensity from the windows to the surrounding wall:
The code to generate the surface plot above is in `img2surface.py`.
Going back to our x-direction kernel in the Sobel Operator.
This kernel has all 0 in the middle, which is quite easy to intuit about. Essentially, for each pixel in our image, we want to compute its derivative in the x-direction by approximating a formula that you may have come across in your calculus class:
$$f'(x) = \lim_{h\to0}\frac{f(x+h)-f(x)}{h}$$
This approximation is also called 'forward difference', because we're taking a value of $x$, and computing the difference in $f(x)$ as we increment it by a small amount forward, denoted as $h$.
And as it turns out, using the 'central difference' to compute the derivative of our discrete signal can deliver better results[^2]:
$$f'(x) = \lim_{h\to0}\frac{f(x+0.5h)-f(x-0.5h)}{h}$$
To make this more concrete, we can plug the formula into an actual array of pixels:
$$[0, 255, 65, \underline{180}, 255, 255, 255]$$
when we set $h=2$ at the center pixel (index of value 180), we have the following:
$$\begin{aligned}
f'(x) & = \lim_{h\to0}\frac{f(x+0.5h)-f(x-0.5h)}{h}\\
& = \frac{f(x+1)-f(x-1)}{2} \\
& = \frac{255-65}{2} \\
& = 95 \end{aligned}$$
Notice that a large part of the calculation we just perform is synonymous to a 1D convolution operation using a $\begin{bmatrix} -1 & 0 & 1 \end{bmatrix}$ kernel.
When the same 1x3 kernel $\begin{bmatrix} -1 & 0 & 1 \end{bmatrix}$ is applied on the right-most part of the image where its just white space ([..., 255, 255, 255]) the kernel would evaluate to 0. In other words, our derivative filter returns no response where it can't detect a sharp change in pixel intensity.
As a reminder, the x-direction kernel in our Sobel Operator is the following:
$$G_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix}$$
This takes our 1x3 kernel and instead of convolving one row of pixels at a time, extends it to convolve at 3x3 neighborhoods at a time using a weighted average approach.
### Code Illustrations: Sobel Operator
The two kernels (one for horizontal and another for vertical edge detection) can be constructed, respectively, like the following:
```py
sobel_x = np.array([[1, 0, -1],
[2, 0, -2],
[1, 0, -1]])
sobel_y = np.array([[1, 2, 1],
[0, 0, 0],
[-1, -2, -1]])
```
You may have guessed that, given its role in digital image processing, `opencv` have included a method that performs our Sobel Operator for us, and thankfully there is. Here's an example of using the `cv2.Sobel(src, ddepth, dx, dy, dst=None, ksize)` method:
```py
gradient_x = cv2.Sobel(img, cv2.CV_64F, 1, 0, ksize=3)
gradient_y = cv2.Sobel(img, cv2.CV_64F, 0, 1, ksize=3)
print(f"Range: {np.min(gradient_x)} | {np.max(gradient_x)}")
# Range: -177.0 | 204.0
gradient_x = np.uint8(np.absolute(gradient_x))
gradient_y = np.uint8(np.absolute(gradient_y))
print(f"Range uint8: {np.min(gradient_x)} | {np.max(gradient_x)}")
# Range uint8: 0 | 204
cv2.imshow("Gradient X", gradient_x)
cv2.imshow("Gradient Y", gradient_y)
```
The code above, extracted from `sobel_01.py` reinforces a couple of ideas that we've been working on. It shows that:
- the $G_x$ and $G_y$, gradients of the image, are computed separately through the convolution of two different Sobel kernels
- $G_x$ and $G_y$ responded to the change in pixel values along the x-direction and y-direction respectively, as visualized in the illustration above
- convolution using the two Sobel filters may, and often will, produce a value outside the range of 0 and 255. Given the presence of [-1, -2, -1] in one side of our kernel, mathematically this may lead to an output value of -1020. To store the values from these convolutions we use a 64-bit floating point (`cv2.CV_64F`). OpenCV suggests to "keep the output datatype to some higher form such as `cv2.CV_64F`, take its absolute value and then convert back to `cv2.CV_8U`.[^3]"
While the code above certainly works, OpenCV also has a method that scales, calculates absolute values and converts the result to 8-bit. `cv2.convertScaleAbs(src, dst, alpha=1, beta=0)` performs the following:
$$dst(I) = cast
