KR102489884B1 - Image processing apparatus for improving license plate recognition rate and image processing method using the same - Google Patents

Abstract

Disclosed is an image processing device and an image processing method using the same for improving license plate recognition rate. An image processing apparatus for improving a vehicle license plate recognition rate of the present invention identifies a license plate in an input vehicle image, extracts a number area from the identified license plate, and recognizes character information in the extracted number area to obtain vehicle number information. If the license plate is not recognized among the vehicle images input from the license plate recognition device and the vehicle license plate recognition device, the license plate is re-recognized through image reprocessing of the vehicle image, and the license plate is recognized by extracting the number area from the recognized license plate. It includes a license plate re-recognition device provided as a device.

Description

Image processing apparatus for improving license plate recognition rate and image processing method using the same

The present invention relates to an image processing apparatus and an image processing method using the same, and more particularly, to an image processing apparatus for improving a license plate recognition rate of a vehicle and an image processing method using the same.

Conventionally, a vehicle number has been used as information for identifying a vehicle. Identification of license plate numbers is very important for tracking criminal vehicles or enforcing traffic violations. The problem is that in order to secure such a license plate number, CCTV images installed on the road must be used. Currently, CCTVs installed on the road are of different types depending on the installation time, purpose, location, etc. such as low, medium, and high quality. The quality of the image is also different.

Although the supply of high-definition CCTV is gradually increasing according to the recent development of technology, there are still low-quality CCTVs installed on most roads, so the vehicle number is identified by analyzing the video image obtained by these low-quality crime prevention CCTVs.

The problem lies in that, according to the existing image processing method, it is very difficult or impossible to identify the license plate number from the video image obtained from such a low-quality CCTV. Most CCTVs for general methods of low quality are of poor quality, making it difficult to identify the license plate number of the vehicle. In particular, in many cases, it is difficult to identify an image captured at night, an image captured in bad weather, or an image with severe motion blur even with the naked eye.

A plan to solve these problems can be considered by replacing low-definition CCTV with high-definition CCTV, but the need to develop image processing technology based on deep learning to an identifiable level by reprocessing the video image obtained from low-quality CCTV is required. .

Prior Patent 1: Korean Registered Patent No. 10-2251799 (Announcement Date: May 07, 2021) Prior Patent 2: Korean Patent Publication No. 10-2020-0119409 (published date: October 20, 2020)

The object of the present invention, which has been made by the above-described need, is to provide an image processing apparatus for improving license plate recognition rate and an image processing method using the same, which increases the recognition accuracy of license plates through image reprocessing of unrecognized license plates. to be

In order to achieve the above object, an image processing apparatus for improving a license plate recognition rate according to an embodiment of the present invention identifies a license plate from an input vehicle image, extracts a number area from the identified license plate, and character from the extracted number area. License plate recognition device for obtaining vehicle number information by recognizing the information; And when the vehicle license plate recognition device does not recognize the license plate of the input vehicle image, the vehicle license plate is re-recognized through image reprocessing of the corresponding vehicle image, and the license plate is recognized by extracting a number region from the recognized license plate. It includes; license plate re-recognition device provided as a device.

In this case, the license plate recognition device transmits the vehicle image information to the license plate recognition device when the license plate identification unit identifies the license plate in the input vehicle image and the license plate identification unit does not recognize the license plate, and retrieves the number plate. If recognized, a number area extraction unit extracts the number area from the license plate image, a character recognition unit extracts text information by recognizing a character in the number area extracted by the number area extraction unit, and the character recognition is performed by the character recognition unit. Upon completion, a layout identification unit detects the layout of the license plate and finally calculates license plate information.

In this case, the license plate re-recognition device receives the unrecognized vehicle image when the vehicle license plate is not recognized among the vehicle images input from the license plate identification unit, and identifies the vehicle license plate using a deep learning method for the vehicle image. It further includes a deep learning module that does.

In this case, the license plate re-recognition device further includes a reprocessing module for obtaining an image including a license plate area by improving brightness or image quality by a reprocessing process for the vehicle license plate recognized by the deep learning module.

In this case, the license plate re-recognition apparatus further includes a segmentation module that extracts only the license plate region from the image including the vehicle license plate region processed by the reprocessing module using a segmentation technique.

In this case, the license plate re-recognition device extracts a character area from the license plate area processed by the segmentation module, and processes the extracted character area to change the size, shape, reference coordinates, etc. of the character according to the STN method It further includes the STN module.

In this case, the license plate re-recognition device performs a primary resolution conversion process for changing the license plate area image processed by the STN module to a high resolution, and uses the high-resolution and low-resolution images by adversarial training. Character information included in the license plate area may be extracted.

An image processing method using an image processing device for improving a license plate recognition rate including a license plate recognition device and a license plate re-recognition device according to another embodiment of the present invention, the license plate from the vehicle image input by the license plate recognition device A license plate recognition process step including identifying, extracting a number area from the identified license plate, and acquiring vehicle number information by recognizing character information in the extracted number area, wherein the license plate re-recognition device In the step of identifying the license plate, if the corresponding license plate is not identified by the license plate recognition device, re-recognizing the license plate through image reprocessing of the corresponding vehicle image and extracting the number area from the recognized license plate A license plate re-recognition processing step including acquiring license plate information and providing the obtained license plate information to the vehicle license plate recognizing device may be further executed.

In this case, the license plate recognition processing step may include: identifying a license plate from an input vehicle image by a license plate identification unit; Transferring corresponding vehicle image information to the license plate re-recognition device when the license plate identification unit does not recognize the license plate or transmitting the license plate image to the number area extraction unit when the license plate is recognized; extracting a number area from the license plate image transmitted by the number area extraction unit; extracting character information by recognizing characters in the number area extracted by the number area extraction unit by a character recognition unit; and finally calculating license plate information by detecting the layout of the license plate when the character recognition is completed in the character recognition unit by the layout identification unit.

In this case, in the license plate re-recognition step, when the deep learning module does not recognize the license plate of the vehicle image input from the license plate identification unit, the deep learning method for the vehicle image is received by receiving the unrecognized vehicle image. Identifying the vehicle license plate with; Acquiring, by a reprocessing module, an image including a license plate area by improving brightness or image quality of the vehicle license plate recognized by the deep learning module through a reprocessing process; extracting, by a segmentation module, only the license plate region from the image including the vehicle license plate region processed by the reprocessing module using a segmentation technique; extracting, by the STN module, a character area from among the license plate areas processed by the segmentation module, and processing the extracted character area to change the size, shape, reference coordinates, etc. of the character according to the STN method; The resolution processing module performs primary resolution conversion processing to change the license plate area image processed by the STN module to a high resolution, and includes it in the corresponding license plate area by adversarial training using the high-resolution and low-resolution images. Extracting the character information; includes.

According to various embodiments of the present invention, image reprocessing of a vehicle image obtained by a low-quality camera or a low-quality or distorted vehicle image obtained in a poor shooting condition by a method such as resolution adjustment, shape deformation control, etc. It has the effect of improving the recognition rate.

1 is a view illustrating the operation of a vehicle license plate recognition device according to an embodiment of the present invention by way of example;
2 is a diagram illustratively illustrating an operation of an image processing apparatus for improving a license plate recognition rate according to an embodiment of the present invention;
3 is a block diagram for exemplarily explaining the operation of the deep learning module shown in FIG. 2;
4 is a block diagram for illustratively explaining the operation of the reprocessing module shown in FIG. 2;
5 is a diagram for exemplarily explaining the operation of the segmentation module shown in FIG. 2;
Fig. 6 shows an exemplary license plate area image processed by the segmentation module shown in Fig. 5;
7 is a diagram for exemplarily explaining the operation of the STN module shown in FIG. 2;
8 is a diagram for exemplarily explaining primary processing of the resolution processing module shown in FIG. 2;
9 is a diagram for illustratively explaining secondary processing of the resolution processing module shown in FIG. 2, and
10 illustratively illustrates a result of recognizing a reprocessed license plate according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings. Components of the invention described below may be implemented individually or may be implemented in the form of an integrated module. The operation and function of each component can be variously modified by a person having ordinary knowledge in the related technology distribution within the scope of the technical idea of the present invention.

<Vehicle License Plate Identification>

1 is a diagram illustrating an operation of a vehicle license plate recognition device according to an embodiment of the present invention by way of example. Referring to FIG. 1 , the license plate recognizing apparatus 100 includes a license plate identification unit 110, a license plate region extraction unit 120, a character recognition unit 130, and a layout identification unit 140.

The license plate recognition apparatus 100 of the present invention identifies the license plate by processing the input image by the license plate identification unit 110. If the license plate is not identified, the vehicle image is transferred to the license plate re-recognition device 200 to be described below, and the license plate re-recognition process is performed. The vehicle license plate recognizing apparatus 100 may extract the license plate region by cropping the license plate region of the license plate identified image by the license plate region extractor 120 . The vehicle license plate recognizing apparatus 100 processes the extracted license plate area to identify text areas by the text recognition unit 130 and obtains text information included in each identified text area. The license plate recognizing apparatus 100 may finally acquire license plate information by using the identified character information and the layout information recognized by the layout recognition unit 140 . A distinctive feature of the present invention is that its action and effect can be exerted in a situation where a vehicle license plate is not recognized. This will be described with reference to separate drawings below.

2 is a diagram exemplarily illustrating an operation of an image processing apparatus for improving a license plate recognition rate according to an embodiment of the present invention. Referring to FIG. 2 , the present invention has a distinctive feature in that images are processed by being divided into a license plate identification processor 100 and a license plate reprocessing processor 200 .

Since the vehicle license plate recognition processor 100 has been described in detail above, a detailed description thereof will be omitted. However, when an unrecognized situation occurs in which the vehicle license plate cannot be identified by the license plate identification unit 110 of the license plate recognition processor 100, it is emphasized that there is a technical feature in performing a reprocessing operation for the unrecognized license plate. .

That is, when the license plate identification unit 110 does not recognize the vehicle license plate included in the vehicle image, it determines that the license plate is not recognized and transfers the vehicle image to the license plate re-recognition device 200. The situation in which the license plate is not recognized means a situation in which the vehicle license plate is not recognized due to physical damage or contamination of the license plate, low resolution, contrast ratio, noise, etc. of the input vehicle image.

The description is centered on the case where the license plate is not identified by the license plate identification unit 110, but even when the number region cannot be extracted by the number region extraction unit 120, the license plate reprocessing processor 200 converts the number region image. By providing an image reprocessing process, it is possible to operate to obtain number information more accurately.

The license plate reprocessing processor 200 includes a deep learning module 210, a reprocessing module 220, a segmentation module 230, an STN module 240, and a resolution processing module 250. Although each module is individually and sequentially shown in FIG. 2 , it may be implemented as one integrated module, and the order of each reprocessing may be changed.

FIG. 3 is a block diagram for explaining the operation of the deep learning module shown in FIG. 2 by way of example. The deep learning module 210 of the present invention includes an input layer receiving an input image, an average pooling layer for pooling by average pooling, a convolution layer, Optimum parameters can be estimated by passing through a layer composed of a continuous layer, a dropout, a first pulley connect, and a second pulley connect. This parameter estimation module 211 is not an essential component, and may be selectively configured in consideration of hardware resources and training time.

The deep learning module 210 is an input layer, a convolutional layer, a max pooling layer, a first basic block layer, a max pooling layer, a second basic block layer, a third basic block layer, a max pooling layer, a dropout, It can be composed of a convolutional layer, a dropout, and a convolutional layer. The composition and order of each layer can be added or deleted by considering the input image and hardware resources.

In addition, the deep learning module 210 is configured to include an input layer, a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer, and an output layer as basic blocks constituting the backbone network architecture. can

In this way, the deep learning module 210 of the license plate re-recognition processor 200 of the present invention can receive an untranslated vehicle image and identify a license plate area in the input vehicle image.

FIG. 4 is a block diagram illustrating the operation of the reprocessing module shown in FIG. 2 by way of example. Referring to FIG. 4 , the reprocessing module 220 includes first reprocessing processing 221 and second reprocessing processing 222 . The first reprocessing processing 221 calculates image brightness values for all pixels or some pixels of interest in the vehicle image including the license plate area, and compares them with a reference value by averaging them locally or globally. When the brightness value is smaller than the reference value, the brightness value is increased for all pixels of the vehicle image or for some pixels of interest. Alternatively, if the brightness value is greater than the reference value, the brightness value of all pixels or some pixels of interest in the vehicle image is lowered and processed.

When the first reprocessing is complete, a second reprocessing process 222 may be executed. The contrast ratio value is measured for the image whose brightness value is adjusted. When the contrast ratio value is greater than the reference contrast ratio, processing may be performed to lower the contrast ratio for all pixels or partial pixels of interest in the corresponding vehicle image. Alternatively, when the contrast ratio value is smaller than the reference contrast ratio, processing may be performed to increase the contrast ratio for all pixels or partial pixels of interest in the corresponding vehicle image. Since the contrast ratio adjustment of pixel values for adjusting the contrast ratio is a well-known technique, a detailed description thereof will be omitted.

Also, in the second reprocessing processing 222, if the contrast ratio is adjusted, an image erosion process is executed. When image erosion is performed, a boundary of a foreground image is eroded and converted into a background image, so that the foreground image can be thinned. Then, the second reprocessing processing 222 executes image dilation processing. When image expansion is performed, the foreground image may be stretched by expanding a boundary portion of the foreground image.

FIG. 5 is a diagram for exemplarily explaining the operation of the segmentation module shown in FIG. 2 . Referring to FIG. 5, if a ground truth box image is set, assuming four boxes (boxes 1 to 4) including some or all of the ground truth box, in the case of the first box and the second box Since the ratio with the region of interest exceeds 0.5, these two boxes are left, and in the case of the third box and the fourth box, since the ratio with the region of interest is less than 0.5, processing is performed to ignore them.

Looking specifically, a fast R-CNN first extracts a feature map from an image using a ConvNet. These feature maps are then passed through the Region Proposal Network (RPN), which returns candidate bounding boxes. A RoI pooling layer is then applied to these candidate bounding boxes to bring all candidates to the same size. Finally, it is passed to a fully connected layer to classify and output the object's bounding box.

Similar to the ConvNet used by Faster R-CNN to extract feature maps from images, we use the ResNet 101 architecture to extract features from images in Mask R-CNN. So, the first step uses the ResNet 101 architecture to fetch images and extract features. These features serve as inputs to the next layer.

We take the functional map obtained in the previous step and apply the regional proposal network (RPM). It basically predicts whether an object is in that area. In this step, we get a map of regions or features that the model predicts to contain certain objects.

Areas obtained from RPN may have different shapes. Therefore, apply a pooling layer and transform all areas into the same shape. Next, these regions are propagated through a fully connected network so that class labels and bounding boxes are predicted.

The steps so far work pretty much the way Faster R-CNN works, but there are differences between the two frameworks. In addition, Mask R-CNN also generates segmentation masks. To this end, it is possible to reduce the computation time by first calculating the region of interest. Calculate IoU (Intersection over Union) using the answer boxes for all prediction regions. You can calculate IoU as follows.

IoU = Cross Area / Combination Area

Now, only if IoU is greater than or equal to 0.5, we consider it a region of interest. Otherwise, the specific region is ignored. Do this for all regions, then select only the set of regions with an IoU greater than 0.5.

As shown in Figure 5, the IoUs of boxes 1 and 2 are greater than 0.5, but the IoUs of boxes 3 and 4 are less than 0.5. After that. Box 1 and Box 2 are regions of interest for this particular image and Box 3 and Box 4 can be ignored.

Next, we look at the last step of Mask R-CNN. Having a RoI based on the IoU value allows adding mask branches to existing architectures. This returns a segmentation mask for each region containing the object. Returns a 28 X 28 mask for each region and then expands it for inference. Here, all objects in the input vehicle image can be segmented. This is the final step of Mask R-CNN, which predicts the mask for every object in the image.

FIG. 6 is a diagram illustrating an exemplary license plate area image processed by the segmentation module shown in FIG. 5 . Referring to FIG. 6 , the segmentation module 230 may accurately recognize a vehicle area even in a vehicle image without a license plate recognition (see FIG. 6(a) ), and may crop only the recognized vehicle area (FIG. 6(a) ). see (b)). The problem lies in adding processing to deal with the fact that the license plate is twisted or its shape is distorted in the process of recognizing the characters included in the cropped license plate area, which lowers the recognition rate.

FIG. 7 is a diagram for exemplarily explaining the operation of the STN module shown in FIG. 2 . Referring to FIG. 7 , it can be confirmed that the cropped license plate image is not in a rectangular shape but slightly distorted in shape (FIG. 7(a)). If the shape is distorted in this way, it is possible to misrecognize or not recognize the character in the case of character recognition processing, so the distortion caused by the deformation of the license plate by the STN module 240 must be corrected.

To this end, it is possible to perform processing to correct shape distortion individually for each character included in the license plate area using a spatial transformer network (STN) method.

First, a character area is recognized on the license plate, and a process of correcting shape distortion is executed for each individual character. First, referring to FIG. 7(a), the number '5' area among the character areas of the license plate is identified. The number '5' is cropped, and correction processing for shape deformation is performed on the image including the number '5'. For the image including the number '5', identity transformation is executed by changing the coordinate values from source coordinates to target coordinates (refer to FIG. 7(c)). Then, an object is recognized in the image including the number '5', and the tilt distortion is correctly corrected by changing the reference coordinates of the recognized object using an affine transformation method (see FIG. 7(d)).

FIG. 8 is a diagram for illustratively explaining primary processing of the resolution processing module shown in FIG. 2 . Referring to FIG. 8 , the resolution processing module 250 inputs a plurality of vehicle images having different resolutions to Lprnet, inputs vehicle license plate information extracted from Lprnet to a decoder of a generator, and A vehicle image with improved resolution may be acquired by inputting the image to an encoder.

FIG. 9 is a diagram for illustratively explaining secondary processing of the resolution processing module shown in FIG. 2 . Referring to FIG. 9 , a high-resolution image is downsampled to generate a low-resolution image, and a medium-resolution image with improved resolution is acquired by inputting the low-resolution image as an input to adversarial training. The medium-resolution image obtained in this way is input to the discriminator as a first input, and the high-resolution image is input as a second input to calculate the resulting value. The calculated result value can be input as an input to the output terminal (1-fold scale) of the license plate recognition processing module that processes the medium resolution image as input to finally acquire the license plate information of the corresponding vehicle image.

10 is a diagram illustratively illustrating a result of recognizing a reprocessed license plate according to an embodiment of the present invention. Referring to FIG. 10 , when a vehicle image including a slightly damaged license plate is not recognized by the license plate recognition device 100, when the vehicle image is input to the license plate recognition device 200, the corresponding vehicle image By reprocessing the image of the license plate area in , resolution, brightness, and contrast ratio are adjusted, so that the recognition rate can be improved.

100: vehicle license plate recognition device
110: license plate identification unit
120: number area extraction unit
130: character recognition unit
140: layout recognition unit
200: license plate re-recognition device
210: deep learning module
220: reprocessing module
230: segmentation module
240: STN module
250: resolution processing module

Claims (10)

In the image processing apparatus for improving the license plate recognition rate,
a license plate recognizing device that identifies a license plate in an input vehicle image, extracts a number area from the identified license plate, and recognizes character information in the extracted number area to obtain vehicle number information; and
When the license plate recognition device does not recognize the license plate of the input vehicle image, the vehicle license plate recognition device re-recognizes the license plate through image reprocessing of the corresponding vehicle image, and extracts the number area from the recognized license plate to obtain the license plate recognition device. Including; license plate re-recognition device provided by
The vehicle license plate recognition device transmits the vehicle image information to the license plate re-recognition device when the license plate identification unit identifies the license plate in the input vehicle image and the license plate identification unit does not recognize the license plate, and recognizes the license plate. A number area extraction unit for extracting a number area from a license plate image, a character recognition unit for recognizing a character in the number area extracted by the number area extraction unit and extracting text information, and when character recognition is completed in the character recognition unit, the corresponding A layout identification unit for detecting the layout of the license plate and finally calculating license plate information;
The license plate re-recognition device is a deep learning module for receiving an unrecognized vehicle image and identifying the vehicle license plate by a deep learning method for the vehicle image when the vehicle license plate is not recognized among the vehicle images input from the license plate identification unit. ; a reprocessing module for obtaining an image including a license plate area by improving brightness or image quality of the vehicle license plate recognized by the deep learning module through a reprocessing process; a segmentation module for extracting only the license plate area from the image including the license plate area processed by the reprocessing module using a segmentation technique; And an STN module that extracts a character area from the license plate area processed by the segmentation module and processes the extracted character area to change the size, shape, reference coordinates, etc. of the character according to the STN method,
For the license plate area image processed by the STN module, a first resolution conversion process is performed to change the image to a high resolution, and character information included in the license plate area is extracted by adversarial training using high and low resolution images. characterized in that,
Image processing device for improving license plate recognition rate.
delete delete delete delete delete delete In the image processing method using an image processing device for improving the license plate recognition rate including a license plate recognition device and a license plate re-recognition device,
A license plate comprising identifying a license plate from a vehicle image input by the license plate recognizing device, extracting a number area from the identified license plate, and acquiring vehicle number information by recognizing character information in the extracted number area. Including a recognition processing step,
In the step of identifying the license plate by the license plate re-recognition device, if the corresponding vehicle license plate is not identified by the license plate recognition device, re-recognizing the vehicle license plate through image reprocessing of the corresponding vehicle image and the recognized vehicle Further executing a license plate re-recognition processing step including obtaining license plate information by extracting a license plate area from a license plate and providing the obtained license plate information to the vehicle license plate recognition device,
The license plate recognition processing step may include: identifying a license plate from a vehicle image input by a license plate identification unit; Transferring corresponding vehicle image information to the license plate re-recognition device when the license plate identification unit does not recognize the license plate or transmitting the license plate image to the number area extraction unit when the license plate is recognized; extracting a number area from the license plate image transmitted by the number area extraction unit; extracting character information by recognizing characters in the number area extracted by the number area extraction unit by a character recognition unit; And when the character recognition is completed in the character recognition unit by the layout identification unit, detecting the layout of the license plate and finally calculating license plate information;
In the license plate re-recognition processing step, when the deep learning module does not recognize the vehicle license plate of the vehicle image input from the license plate identification unit, the vehicle license plate is received by receiving the unrecognized vehicle image and using a deep learning method for the vehicle image. identifying; Acquiring, by a reprocessing module, an image including a license plate area by improving brightness or image quality of the vehicle license plate recognized by the deep learning module through a reprocessing process; extracting, by a segmentation module, only the license plate region from the image including the vehicle license plate region processed by the reprocessing module using a segmentation technique; extracting, by the STN module, a character area from among the license plate areas processed by the segmentation module, and processing the extracted character area to change the size, shape, reference coordinates, etc. of the character according to the STN method; The resolution processing module performs primary resolution conversion processing to change the license plate area image processed by the STN module to a high resolution, and uses the high-resolution and low-resolution images to perform adversarial training, which is included in the corresponding license plate area Characterized in that executing including; extracting text information;
Image processing method for improving license plate recognition rate.
delete delete

Images