JP6829575B2 - Image processing equipment, image processing system and image processing method - Google Patents

Description

The present invention is an image process capable of efficiently and accurately converting an image of a display board captured in various different situations into an image corresponding to the case where the image is captured from the front (hereinafter referred to as "front image"). The present invention relates to an apparatus, an image processing system, and an image processing method.

Conventionally, there is known a technique of capturing and reading a display board such as a license plate attached to a vehicle or a road sign arranged on a road. For example, vehicles passing through roads that form a skeletal road network (hereinafter referred to as "main roads"), vehicles passing through toll gates on toll roads, and entering and exiting parking lots nationwide, locally, or within cities. In order to read the vehicle number of the vehicle or the like, a device is installed that images the license plate attached to the vehicle and reads the vehicle number indicated in the image data including the imaged license plate.

For example, in Patent Document 1, coordinate conversion parameters are stored in advance according to the position of the license plate in the image, and if the position of the license plate in the image is detected, the coordinate conversion parameter corresponding to the position is used. A technique for obtaining a front image of a license plate by converting the coordinates of the image of the license plate portion is disclosed. By performing the recognition process after converting the license plate portion into the front image in this way, the accuracy of the recognition process can be improved.

Further, Patent Document 2 discloses a technique for recognizing a license plate number by rotating an image captured by the license plate and performing enlargement correction according to the traveling direction of the vehicle with respect to the image pickup device. Further, in Patent Document 3, four points including two plate stoppers and the rightmost number are detected from the image obtained by capturing the license plate, coordinate conversion is performed using these four points, and the number viewed from the front. A technique for acquiring an image of a plate is disclosed.

Further, Patent Document 4 adds a learning function by using a neural network when the image is not captured from the front or when a part of the road sign is hidden by a tree or the like and the graphic component is missing. However, it is mentioned that learning is performed so that road signs can be correctly extracted for missing or deformed graphic components.

Japanese Unexamined Patent Publication No. 7-114688 Japanese Patent No. 4670721 JP 2015-32087 Japanese Patent No. 4762026

However, the above-mentioned Patent Document 1 has a limited approach direction of the vehicle, and cannot be applied when the approach direction changes. Further, unlike the above-mentioned Patent Document 1, the above-mentioned Patent Document 2 can correspond to a plurality of approach directions, but is not suitable for high-speed processing because it requires two images taken at time intervals. Is.

Further, the one of Patent Document 3 cannot deal with the case where the image does not include the four points required for converting into the front image. Further, in Patent Document 4, when the road sign is photographed at an angle, or when a part of the road sign is hidden by a tree or the like and the graphic component is missing, the graphic component is deformed or missing. However, it is mentioned that the neural network is trained so that the road sign can be extracted correctly, but the extracted road sign image cannot be converted into the front image because there is no information for correction in the front image.

For these reasons, the prior arts represented by Patent Documents 1 to 4 have a situation in which the image of the display board cannot be converted into a front image, so that the application situation is limited or the recognition process is performed assuming an actual environment. There is a problem that the accuracy of

Therefore, when converting an image including a license plate imaged by an imaging device into a front image, how to efficiently deal with various different situations has become an important issue. Such a problem occurs not only in the case of an image including a license plate but also in the case of an image including a road sign and the like.

The present invention has been made to solve the above-mentioned problems of the prior art, and is an image processing apparatus and image processing capable of efficiently and accurately converting images captured in various different situations into front images. It is an object of the present invention to provide a system and an image processing method.

To solve the above problems, the present invention includes a learning image generation unit for generating a learning image with missing a region including a portion of the reference point display panel from the reference image having the unrealized four reference points , The multi-layer neural network in which supervised learning was performed using the learning image and the four reference points in the learning image, the input image receiving unit for acquiring the input image including the display board, and the multi-layer neural network. put an image based on the input image, the four and the corresponding point specifying unit for specifying the information relating to corresponding points, four corresponding points identified by the corresponding point specifying section for said input image corresponding to the reference point It is characterized by including a front image generation unit that generates a front image corresponding to the input image by performing projection conversion based on information related to at least three corresponding points .

Further, the present invention is characterized in that, in the above invention, a character recognition unit that recognizes characters included in the front image generated by the front image generation unit is further provided.

The present invention, in the above invention, the multi-layer neural network, the first learning image image is missing some of the information of the real image obtained by imaging the display panel to projective transformation by a predetermined imaging apparatus and other studies習用image, and performing supervised learning and four reference points in the learning image as the input information.

Further, the present invention is characterized in that, in the above invention, a type identification unit for identifying the type of the display board is further provided.

Further, in the above invention, the corresponding point specifying unit inputs an analysis image, which is a grayscale image generated from an input image acquired by the input image receiving unit, into the multilayer neural network. It is a feature.

Further, the present invention is characterized in that, in the above invention, the display board is a license plate or a road sign indicating a vehicle registration number.

Further, the present invention provides an image processing system for processing an image including a display panel, for learning is missing a region including a portion of the reference point the panel from the reference image having the unrealized four reference points An image generation unit for learning that generates an image, a multi-layer neural network that performs supervised learning using the learning image and four reference points in the learning image, and an input that acquires an input image including a display board. An image receiving unit, a corresponding point specifying unit that inputs an image based on the input image to the multilayer neural network, and specifies information related to four corresponding points related to the input image corresponding to the reference point, and the corresponding point. It is provided with a front image generation unit that generates a front image corresponding to the input image by performing projection conversion based on information related to at least three correspondence points among the four correspondence points specified by the specific unit. It is a feature.

Further, the present invention includes a learning image generation step of generating a learning image with missing a region including a portion of the reference point display panel from the reference image having the unrealized four reference points, and the learning image a step of executing a supervised learning multi-layer neural network using the four reference points in the learning image, the input image acquiring step of acquiring an input image including a display panel, the input image into the multi-layer neural network A corresponding point specifying step for inputting an image based on the image and specifying information related to four corresponding points related to the input image corresponding to the reference point, and at least four corresponding points specified by the corresponding point specifying step. It is characterized by including a front image generation step of performing projection conversion based on information related to the three corresponding points to generate a front image corresponding to the input image .

According to the present invention, it is possible to obtain a front image of a display board from an image including a display board captured in various different situations.

FIG. 1 is an explanatory diagram for explaining the concept of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram showing a system configuration of the image processing system according to the first embodiment. FIG. 3 is a functional block diagram showing a configuration of a learning stage of the image processing apparatus shown in FIG. FIG. 4 is a diagram for explaining the creation of a learning image. FIG. 5 is a functional block diagram showing a configuration of a recognition stage of the image processing apparatus shown in FIG. FIG. 6 is a flowchart showing a processing procedure in the learning stage of the image processing apparatus shown in FIG. FIG. 7 is a flowchart showing a processing procedure in the recognition stage of the image processing apparatus shown in FIG. FIG. 8 is an example of an image described by the flowchart shown in FIG. FIG. 9 is an explanatory diagram for explaining the concept of the image processing apparatus according to the second embodiment. FIG. 10 is a diagram showing a system configuration of the image processing system according to the second embodiment. FIG. 11 is a functional block diagram showing a configuration of the image processing apparatus shown in FIG. 10 at the learning stage. FIG. 12 is a diagram illustrating a reference point of a sign to be processed by the image processing apparatus shown in FIG. FIG. 13 is a functional block diagram showing a configuration of the image processing apparatus shown in FIG. 10 at the recognition stage. FIG. 14 is a flowchart showing a processing procedure in the recognition stage of the image processing apparatus shown in FIG. FIG. 15 is an example of an image described by the flowchart shown in FIG.

Hereinafter, examples of the image processing apparatus, the image processing system, and the image processing method according to the first embodiment will be described with reference to the attached drawings. In the first embodiment, the case of reading the characters and numbers of the license plate that displays the registration number of the vehicle, which is one of the display boards, will be mainly described.

<Concept of image processing device according to Example 1>
First, the concept of the image processing apparatus according to the first embodiment will be described. FIG. 1 is an explanatory diagram for explaining the concept of the image processing apparatus according to the first embodiment. Here, a multi-gradation image (for example, a 256-gradation black-and-white image, a 24-bit color image, etc.) obtained by imaging the license plate used for learning (hereinafter referred to as a “reference image”) and a generation from this reference image. It is assumed that the created learning image has already been acquired.

The image processing device 3 according to the first embodiment causes the multi-layer neural network 36 and the network model update processing unit 37 to perform supervised learning in advance at the learning stage, and weights synapses and the like connecting the nodes of the multi-layer neural network 36. Store as network model parameter G. Then, in the recognition stage, the multi-layer neural network 36 is reconstructed using the network model parameter G, and the front image F is generated by the multi-layer neural network 36 using the position information of the four corners of the image. After that, the front image F of the license plate is recognized as characters, and the characters and numbers included in the license plate are output.

Specifically, in the first embodiment, the functional unit (network model update processing unit 37) that updates the network model is separated from the multi-layer neural network 36. When performing supervised learning in advance, the network model update processing unit 37 generates a network model parameter G to be learning data (synapse weights, etc.). By doing so, the network model update processing unit 37 becomes unnecessary at the authentication stage, and a large number of different multi-layer neural networks 36 can be reconstructed using the network model parameter G.

Here, the multi-layer neural network 36 is a multi-layered neural network used in deep learning (deep learning), and in this embodiment, a convolutional neural network is used. This convolutional neural network is a kind of forward propagation type artificial neural network, and individual neurons are arranged so as to correspond to the visual cortex. This convolutional network is biologically influenced and is a type of multi-layer perceptron designed for less pretreatment. Since this convolutional neural network itself is a known technique, detailed description thereof will be omitted here.

Further, the multi-layer neural network 36 and the network model update processing unit 37 are made to perform supervised learning using the learning image B and the four reference points. Specifically, the network model update processing unit 37 updates (learns) the network model parameter G so that the four corresponding points output from the multi-layer neural network 36 match the position coordinates of the four reference points. The multi-layer neural network 36 is reconstructed by the updated network model parameter G, and learning is performed in the same manner using the four corresponding points output from the multi-layer neural network 36. The learning process is repeated to fix the network model parameter G. For example, the network model parameter G can be adjusted using an error backpropagation algorithm or the like.

As shown in FIG. 1A, in the learning stage, the learning image B is input to the multilayer neural network 36, and learning data such as four reference points, which are teacher data, is input to the network model update processing unit 37 ( Step S11). Then, the multilayer neural network 36 is trained based on the learning image B and the four reference points, and the network model update processing unit 37 updates the network model parameter G (step S12). Since the supervised learning itself by the multi-layer neural network 36 and the network model update processing unit 37 is an existing technique, detailed description thereof will be omitted here.

The learning image B is image data generated from a reference image which is an image of a license plate attached to a vehicle. Specifically, the learning image B is a reference image A which is an actual image of the license plate imaged by the imaging device 1, a part of the reference image A on the assumption that a part of the license plate is hidden or the like. Image (first learning image), image obtained by projecting and transforming reference image A assuming the relative angle between the vehicle and the image pickup device 1 (second learning image) or reference image A. This is an image obtained by projecting and converting an image lacking some information (third learning image) or the like. Further, it is desirable that the learning image B is an image cut out so as to include only the license plate and its surroundings in order to improve the processing efficiency.

The four reference points are correct answer data (teacher data) for updating the network model parameter G and causing the multi-layer neural network 36 to perform supervised learning. Specifically, the position coordinates of the points at the four corners of the license plate included in the learning image B are designated as reference points.

As described above, the reason why the multilayer neural network 36 according to the first embodiment is trained by using the training image B and the four reference points is that the input has various distortions due to the difference in the imaging direction in the actual environment. This is to obtain a front image F without distortion even when the image C is input. To explain this point concretely, when the license plate of the vehicle is imaged by the image pickup device 1, various distortions occur in the license plate portion in the input image C depending on the relative angle between the vehicle and the image pickup device 1. For example, even though a rectangular license plate is imaged, the license plate may be reflected in a parallelogram or trapezoidal shape. In addition, a part of the license plate may be hidden due to the presence of other vehicles or trees. As a result, a situation may occur in which the four reference points required for the three-axis projective transformation cannot be specified. Therefore, the learning image B having various distortions and the four reference points serving as the correct answer data (teacher data) are input in advance to train the multi-layer neural network 36, and the updated network model parameter G I remember. The four reference points can be specified by instructing and inputting points on the learning image B displayed on the display unit 32 by the operator, and the four reference points designated by the operator can be specified. It can also be specified by a projective transformation similar to an image.

Next, the processing at the recognition stage will be described. Here, as preprocessing, it is assumed that a recognition image D including the license plate and its peripheral portion is generated from the input image C obtained by capturing the license plate of the vehicle. In this preprocessing, edge detection, smoothing, and the like can also be performed.

As shown in FIG. 1 (b), the analysis image E is generated from the recognition image D. When the analysis image E is input to the multilayer neural network 36 (step S21), the multilayer neural network 36 outputs information relating to the positions of the four corresponding points corresponding to the four reference points (step S22). .. The "information relating to the position of the corresponding point" may be the position coordinates of the corresponding point itself or may be information for specifying the corresponding point. For example, it may be the position coordinates of one corresponding point and a vector from this corresponding point to another corresponding point.

After that, the recognition image D is projected and transformed using the four corresponding points (step S23), and the front image F of the license plate is acquired. After that, the characters (including numbers) included in the front image F are recognized (step S24), and the characters included in the license plate are output. A well-known technique is used when performing projective conversion and character recognition.

By performing the above series of processes, even when the input image C obtained by capturing the license plate of the vehicle at various angles is input, the input image C is efficiently converted into the front image F and the accuracy is improved. The characters in the license plate can be extracted well. In particular, even if some information related to the license plate included in the input image C is lost and it is difficult to extract all four reference points by normal processing, learning processing is performed in preparation for this case. Therefore, it is possible to specify the corresponding points corresponding to the four reference points and perform the three-axis projective conversion.

<System configuration of image processing system>
Next, the system configuration of the image processing system according to the first embodiment will be described. FIG. 2 is a diagram showing a system configuration of the image processing system according to the first embodiment. As shown in the figure, this image processing system has a configuration in which an image pickup device 1 and an image processing device 3 are connected via a network 2.

The image pickup device 1 comprises an image pickup device such as a CCD camera capable of capturing a black-and-white image of 256 gradations, and transmits image data to the image processing device 3 via a network 2. The imaging device 1 includes a surveillance camera or the like that images the license plates of vehicles passing on the road. Here, a case where the image pickup device 1 captures an image and transmits the image to the image processing device 3 will be described. However, the image pickup device 1 captures a moving image and transmits the moving image to the image processing device 3, and the image processing device 3 transmits the moving image. It can also be applied when cropping an image. Further, although only one imaging device 1 is shown here for convenience of explanation, a plurality of imaging devices 1 may be provided. Further, as the image pickup device 1, an image pickup device capable of capturing a color image instead of a black-and-white image of 256 gradations can also be used. Alternatively, the image pickup device 1 may use an image pickup device capable of capturing an image by infrared light or the like instead of an image by visible light.

The network 2 is formed by a wired network such as Ethernet (registered trademark) and is used for image transfer. The network 2 is not limited to Ethernet, and may be directly connected by a cable using a single communication cable to transmit an image signal, or may be a wireless network.

The image processing device 3 is a device that outputs characters included in the license plate in the input image C received from the image pickup device 1. Specifically, four corresponding points are specified by using the multi-layer neural network 36 reconstructed using the network model parameter G, and the four corresponding points are used to perform the projective transformation of the recognition image D to perform the frontal surface. The image F is acquired, the front image F is recognized as a character, and the characters in the license plate are output. A detailed description of the image processing device 3 will be described later.

<Structure when learning processing of the image processing device 3>
Next, a configuration in the case of performing the learning process of the image processing device 3 shown in FIG. 2 will be described. FIG. 3 is a functional block diagram showing a configuration when the learning process of the image processing device 3 shown in FIG. 2 is performed. As shown in the figure, the image processing device 3 includes an input unit 31, a display unit 32, a communication interface unit 33, a storage unit 34, and a learning processing control unit 35.

The input unit 31 is an input device such as a keyboard and a mouse, and the display unit 32 is a display device such as a liquid crystal panel. The communication interface unit 33 is a communication device for receiving an image captured by the image pickup device 1 by using Ethernet communication or the like.

The storage unit 34 is a storage device composed of a non-volatile memory such as a flash memory or a hard disk, and stores a reference image A, a learning image B, a network model parameter G, and the like. This reference image A is a basic image used when learning the multi-layer neural network 36, and at least four reference points (upper left, upper right, lower left, and lower right) of the license plate are specified. It is a front image with the license plate to be obtained facing the front.

The learning processing control unit 35 is a control unit that controls the learning processing by using the multi-layer neural network 36 and the network model update processing unit 37 to generate the network model parameter G. The reference image generation unit 38 and the learning image It has a generation unit 39 and a learning processing unit 40.

The reference image generation unit 38 associates the front image of the license plate to be processed (assuming that it is normalized to a predetermined size) with the four reference points input from the operator from the input unit 31. This is a processing unit that generates a reference image. Here, a case where the reference image A is generated using the front image input via the communication interface unit 33 will be described, but the reference image will be generated using the front image stored in a portable recording medium such as a USB memory. It is also possible to generate A.

The learning image generation unit 39 generates the learning image B by performing various geometric transformations on the reference image A in consideration of the mode of the input image C captured by the image pickup device 1, and the generated learning. This is a processing unit that stores the image B for storage in the storage unit 34. The learning image B is a learning image generated based on the reference image A, and is an image that has undergone a projective transformation so as to give distortion to the reference image A, and a part of information on the license plate ( For example, an image or the like in which the upper right part) is lost is included.

Specifically, the learning image generation unit 39 performs a process of generating one or a plurality of learning images B having a distortion obtained by geometrically deforming the reference image A received from the reference image generation unit 38. .. For example, the reference image A or the learning image B is projected and transformed using the four reference points to newly generate the learning image B. When a plurality of learning images B are generated, the reference images A are geometrically transformed into different modes. At that time, since the four reference points also exist at the converted positions, they are included in the learning image B. However, the learning image generation unit 39 accepts the learning image B manually deformed by the operator or the learning image B generated by the operator applying the image processing of geometric transformation to the reference image A. You can also.

When the operator uses the input unit 31 to instruct on the learning image B displayed on the display unit 32, the instructed position coordinates serve as a reference point. Here, it is assumed that the instructions of the four points (upper left, upper right, lower left, lower right) of the four corners of the license plate are accepted, and these four points are used as reference points. It is also possible to automatically detect the four corner points (upper left, upper right, lower left, lower right) of the license plate by using an image processing technique such as template matching. If a part of the image is lost and some of the points at the four corners of the license plate (for example, the upper right) cannot be automatically detected, the operator can additionally instruct the reference point. The specific learning image B will be described later.

The learning processing unit 40 is a processing unit that processes a portion related to learning of the image processing device 3, and has a corresponding point learning reception unit 41, a multi-layer neural network 36, and a network model update processing unit 37. Actually, the program and data corresponding to the learning processing unit 40 are loaded into the CPU and executed, and the functions of the corresponding point learning reception unit 41, the multi-layer neural network 36, and the network model update processing unit 37 are executed. become. Then, the network model parameter G of the storage unit 34 is updated with the learning by the learning processing unit 40.

The correspondence point learning reception unit 41 receives the learning image B to be learned, generates the analysis image E from the learning image B, outputs it to the multilayer neural network 36, and has four learning images B. This is a processing unit that outputs a reference point to the network model update processing unit 37. The analysis image E may be a shade image such as a black-and-white image having 256 gradations or an RGB brightness image, or may be image data such as a differential image or HSV.

The multi-layer neural network 36 operates based on the input of the input analysis image E while reconstructing the synaptic weights between the nodes by the network model parameter G, and outputs four corresponding points.

The network model update processing unit 37 calculates the difference between the four corresponding points output from the multi-layer neural network 36 and the four reference points input from the corresponding point learning reception unit 41, and the network model so that the difference is small. This is a processing unit that updates the parameter G. The updated network model parameter G is applied to the multi-layer neural network 36, and learning is repeated.

<Example of learning image B>
Next, four types of learning images will be described as an example of the learning image B. FIG. 4 is a diagram showing an example of the learning image B. As shown in FIG. 4A, the reference image A, which is an actual image obtained by capturing the license plate, includes four reference points P1 to P4 and is used as a learning image B. As shown in FIG. 4B, an image lacking some information of the reference image A becomes a learning image B (first learning image). In FIG. 4, the kana characters, which are the business identification numbers, are not displayed in order to prevent the identification of individual vehicles, but in the actual processing, the kana characters may be included in the processing.

Further, as shown in FIG. 4 (c), an image obtained by projective conversion using the four reference points P1 to P4 of the reference image A (image after projective conversion indicated by a broken line arrow in the figure) is used for learning. Let it be image B (second learning image). A new learning image B (third learning image) can be obtained by projecting and transforming the learning image B (first learning image) in which some information of the reference image A is missing. Of these reference images A and the first to third learning images, one type of image may be used as the learning image B, or a plurality of types of images may be used as the learning image B. Although not shown, if the position coordinates of the converted reference points P1'to P4' are included in the learning image B, the learning can be automatically advanced.

<Configuration when performing recognition processing of the image processing device 3>
Next, a configuration when the image processing device 3 shown in FIG. 2 performs recognition processing will be described. FIG. 5 is a functional block diagram when the image processing device 3 shown in FIG. 2 performs recognition processing.

The recognition processing control unit 44 uses the multi-layer neural network 36 to which the trained network model parameter G is applied to identify four corresponding points corresponding to the four reference points of the recognition image D and generate a front image F. It is a control unit that recognizes characters based on the front image F. The recognition processing control unit 44 includes an input image reception unit 45 that receives an input image C including a background from an external image pickup device 1 or the like via a communication interface unit 33, and a recognition processing unit 46 that includes a multilayer neural network 36. Consists of. In FIG. 5, the same reference numerals as those in FIG. 3 are omitted because they overlap with the description of FIG.

The input image receiving unit 45 receives the input image C and performs preprocessing such as cutting out the license plate portion, removing noise from the image, and scaling processing to align the image to a predetermined size to generate the recognition image D. Do. Here, the input image C is an image of the license plate to be processed, which is captured by the imaging device 1. This input image C includes not only the license plate but also a part of the vehicle body. Further, in the recognition image D, the learning image B shown in FIG. 3, and the recognition image D, the learning image B has information on four reference points, whereas the recognition image D Is different in that it does not have information on the reference point or its corresponding point.

The recognition processing unit 46 outputs four corresponding points of the license plate image of the recognition image D using the multi-layer neural network 36 to which the learned network model parameter G is applied, and based on the output four corresponding points. This is a processing unit that performs projective conversion of the recognition image D, generates a front image F, performs character recognition, and outputs the characters (including numbers) of the license plate. It has a corresponding point identification unit 47, a multi-layer neural network 36, a front image generation unit 48, and a character recognition unit 49.

The corresponding point specifying unit 47 receives the recognition image D, generates the analysis image E, and outputs the recognition image D to the front image generation unit 48. The image E for analysis may be a gray image or RGB luminance pixels, or may be a differential image or an image such as HSV. The corresponding point specifying unit 47 and the corresponding point learning reception unit 41 are different in that the corresponding point specifying unit 47 outputs the recognition image D to the front image generation unit 48.

The front image generation unit 48 performs a process of projecting and transforming the recognition image D using the four corresponding points output from the multilayer neural network 36 to generate the front image F. Here, this projective transformation is a transformation used when projecting a certain plane onto another plane, and is used when converting a recognition image D viewed from an angle into a front image F. Since this projective transformation is a well-known technique, detailed description thereof will be omitted here.

The character recognition unit 49 performs a process of character recognition and output of characters (including numbers) included in the license plate portion of the front image F. For example, general character recognition can be performed by normalization, feature extraction, matching, and knowledge processing, but since this character recognition is a well-known technique, detailed description thereof will be omitted here.

The storage unit 50 at the time of recognition processing is composed of the same device as the storage unit 34 at the time of learning, temporarily stores the input image C, and the learned network model parameter G and the character recognition unit 49 are characters. The character recognition dictionary H used at the time of recognition is stored.

<Learning process procedure>
Next, the processing procedure in the learning stage of the image processing apparatus 3 shown in FIG. 2 will be described. FIG. 6 is a flowchart showing a processing procedure in the learning stage of the image processing apparatus 3 shown in FIG. Here, it is assumed that the reference image A is already stored in the storage unit 34.

As shown in FIG. 6, the image processing device 3 accepts the designation of four reference points in the learning image B including the reference image A (step S101). These four reference points serve as teacher data when the multi-layer neural network 36 is made to perform supervised learning.

After that, the learning image B is generated from the reference image A (step S102). Specifically, the reference image A is geometrically transformed to generate a learning image B having a distortion obtained by geometrically deforming the reference image A. The learning image B includes a reference image A obtained by capturing the number plate from the front, an image in which some information of the reference image A is omitted, an image obtained by projecting and converting the reference image A, and a part of the reference image A. An image obtained by projecting and converting an image lacking information is included. Further, the same geometrical deformation is applied to the reference point coordinates with respect to the reference image A to obtain new reference point coordinates in the learning image B. Even if the pixel information near the reference point is omitted in the reference image A, the position coordinates converted by the same geometric transformation as the image are used as the position coordinates of the reference point in the learning image B. If it is used, the generation of the learning image B can be automated. The learning image B has both image data and position coordinate information of four reference points (hereinafter, reference point information).

Next, an analysis image E is generated by converting the learning image B into a black-and-white image of 256 gradations, an RGB brightness image, or image data such as a differential image or HSV (step S103). However, in this step, the geometric transformation that makes it impossible to maintain the relationship between the image data and the reference point information in the learning image B is not performed.

Next, the initial value of the network model parameter G, the plurality of analysis images E, and the reference point information corresponding to each analysis image E are input to the multi-layer neural network 36 (step S104), and the multi-layer neural network 36 4 Two corresponding points are calculated (step S105).

Next, the difference between the calculated corresponding point and the corresponding reference point is calculated (step S106), and if the difference is equal to or less than a predetermined value (step S107; No), the network model parameter G at that time has been updated. Is stored in the storage unit 34.

In step S107, when the difference exceeds a predetermined value (step S107; Yes), and if the number of repetitions is equal to or greater than the predetermined value (step S108; No), the process ends. In this case, it becomes abnormal. On the other hand, if the number of repetitions is less than a predetermined value (step S108; Yes), the network model parameter G is updated so that the difference is close (step S109), and the multi-layer neural network 36 to which the updated network model parameter G is applied is applied. Step S105 and subsequent steps are repeated using.

<Recognition processing procedure>
Next, the processing procedure in the recognition stage of the image processing apparatus 3 shown in FIG. 2 will be described. FIG. 7 is a flowchart showing a processing procedure in the recognition stage of the image processing apparatus 3 shown in FIG. Further, the description will be given with reference to FIGS. 8 (a) to 8 (d).

As shown in FIG. 7, the image processing device 3 receives the input image C (FIG. 8A) (step S201), preprocesses the input image C including the license plate, and generates the recognition image D. (Step S202). Then, an analysis image E, which is an image to be input to the multilayer neural network 36, is generated by converting the image into a black-and-white image having 256 gradations, an RGB luminance image, or the like (step S203).

The learned network model parameter G is read from the storage unit 50 and input to the multi-layer neural network 36 to form the multi-layer neural network 36, and the analysis image E is input to this (step S204). Corresponding points are calculated by the multi-layer neural network 36 (step S205). Here, as shown in FIG. 8B, the recognition image D to which the corresponding points are added can be obtained.

Then, the recognition image D is projected and transformed using these four corresponding points (step S206), and the front image F as shown in FIG. 8C is acquired. Then, character recognition processing is performed on the front image F to acquire and output the characters of the license plate as shown in FIG. 8 (d) (step S207).

As described above, in the first embodiment, the input image C is applied to the multi-layer neural network 36 in which the network model parameter G is updated by performing supervised learning using the learning image B and the reference point information. The analysis image E that has undergone the predetermined processing is input from the preprocessed recognition image D, and the four corresponding points on the recognition image D corresponding to the four reference points are specified. Using these four corresponding points, the recognition image D is projected and transformed to obtain the front image F, and the front image F is subjected to character recognition to output the characters on the license plate. As a result, the input image C captured under various conditions such as the imaging distance and the angle can be efficiently and accurately converted into the front image F, and the character recognition result of the license plate can be output. Therefore, it is possible to efficiently recognize the characters on the display board included in the image.

In the first embodiment, a case where an image processing device which is a computer is used has been shown, but the present invention is not limited to this, and may be applied to a case where distributed computing is performed by a plurality of computers. it can. It can also be applied when processing is performed on the cloud.

By the way, in the above-described first embodiment, the case where the present invention is used for character recognition of a license plate is shown, but the present invention is not limited to this, and a road sign or the like is used by an image processing device mounted on a vehicle. It can also be applied when performing character recognition of characters included in the display board of. The display board such as a road sign is specifically a guide sign, a warning sign, a regulation sign, an instruction sign, an auxiliary sign, etc., and in particular, a road sign on which characters are displayed is targeted for processing. Therefore, in the second embodiment, the case where the character recognition of the characters included in the road sign or the like is performed by the image processing device mounted on the vehicle will be shown. A detailed description of the same parts as in the first embodiment will be omitted.

<Concept of image processing device according to Example 2>
First, the concept of the image processing apparatus according to the second embodiment will be described. FIG. 9 is an explanatory diagram for explaining the concept of the image processing apparatus according to the second embodiment. Here, the learning target of the supervised learning described in the first embodiment, that is, the reference image A which is one type of license plate, is the processing target of the road sign having a plurality of shape types. Is different. In the second embodiment, the learning process after the type identification of the reference image A is the same as the learning process unit 40 described in the first embodiment, and therefore most of the details are omitted. In the second embodiment, learning is performed for each type of the reference image A, and the network model parameter G is created for each type of the reference image A. The type of the reference image A referred to here is a type of a shape such as a rectangular license plate, a circular shape, a polygonal shape, or the like.

The image processing device 4 according to the second embodiment has the multi-layer neural network 36 trained in advance in the learning stage, and is reconfigured with the trained network model parameter G according to the type of the shape of the road sign in the recognition stage. Four corresponding points are acquired using the multi-layer neural network 36. Then, character recognition is performed on the front image F that has been projected and transformed using these four corresponding points. This multi-layer neural network 36 is a convolutional neural network used for the same deep learning (deep learning) used in the first embodiment.

As shown in FIG. 9A, in the learning stage, learning data such as a learning image B and four reference points are input to the multi-layer neural network 36 (step S31), and these learning images B and The multilayer neural network 36 is made to perform supervised learning based on the four reference points, and the network model parameter G is updated by the network model update processing unit 37 (step S32). This learning is performed for each type of road sign shape, and network model parameter G is generated for each type of object (step S33).

The learning image B is image data generated from the reference image A obtained by capturing the road sign. Specifically, image data having distortion obtained by geometrically deforming a reference image is generated by using a well-known image processing technique, and the generated image data is used as a learning image B. The concept is the same as that described in the first embodiment and will be omitted.

The four reference points are correct answer data (teacher data) for causing the multi-layer neural network 36 to perform supervised learning. Specifically, the position coordinates of four points that specify the object included in the learning image B are designated as reference points. This supervised learning is performed according to the type of road sign shape, and the network model parameter G is prepared for each type.

In the second embodiment, the points manually instructed and the true reference points do not necessarily have to coincide with each other with respect to the four reference points. With round or triangular road signs, it is difficult to visually input the four points corresponding to the four corners of the license plate, so set more characteristic points as points to be manually specified. Four points serving as reference points are obtained by the multilayer neural network 36. For example, in manual input, three vertices of a triangle are designated, four corners of the circumscribed rectangle of the triangle are obtained by calculation from the designated three points, and the four corners are used as reference points.
This has the effect of improving the accuracy of the reference point and making it possible to handle display boards of various shapes.

Next, the processing at the recognition stage will be described. As shown in FIG. 9B, when the input image C that captures the road sign is input, preprocessing is performed and the recognition image D is output (step S41). Based on this recognition image D, the type of road sign shape is identified (step S42), and thereafter, the analysis image E is input to the multi-layer neural network 36 to which the network model parameter G according to the type is applied. (Step S43) Information relating to the positions of the four corresponding points corresponding to the four reference points is output (step S44). Next, the recognition image D is projected and transformed (step S45) using the four corresponding points to generate the front image F. Then, the front image F is recognized as a character (step S46). At this time, the character recognition dictionary H according to the type is used. This is because the fonts of the road signs are different. Since the processing contents of steps S43 to S46 of FIG. 9 are the same as the processing contents of steps S21 to S24 shown in FIG. 1, duplicate description will be omitted.

By performing the above-mentioned series of processes, even when the input image C obtained by capturing the road sign at various angles is input, the input image C is efficiently converted into the front image F, and the road is accurately converted. The characters of the sign can be output. In particular, even if some information related to the road sign included in the input image C is lost and it is difficult to extract all four reference points by normal processing, learning processing is performed in preparation for this case. Therefore, it is possible to specify four corresponding points and perform a three-axis projective conversion. Then, the characters of road signs of various shapes can be recognized.

<System configuration of image processing system>
Next, the system configuration of the image processing system according to the second embodiment will be described. FIG. 10 is a diagram showing a system configuration of the image processing system according to the second embodiment. As shown in the figure, this image processing system has a configuration in which an image pickup device 1 mounted on a vehicle and an image processing device 4 are connected. Since the image pickup apparatus 1 is the same as that shown in the first embodiment, the description thereof will be omitted here.

The image processing device 4 is a device that identifies the type of shape of the road sign in the input image C received from the image pickup device 1 and recognizes the characters included in the road sign. Specifically, the type of the shape of the road sign is identified from the recognition image D, and four corresponding points are specified by the multi-layer neural network 36 using the network model parameter G in which supervised learning is performed according to the type. , The front image F is acquired by performing the projective conversion of the recognition image D using these four corresponding points, and the front image F is character-recognized using the character recognition dictionary H prepared according to the type. Output the characters included in the road sign.

<Configuration of image processing device 4>
Next, the configuration of the image processing device 4 shown in FIG. 10 will be described. FIG. 11 is a functional block diagram showing a configuration of the image processing device 4 shown in FIG. 10 at the time of learning. As shown in the figure, the image processing device 4 includes an input unit 31, a display unit 32, a communication interface unit 33, a storage unit 34, and a learning processing control unit 35. The same parts as those of the image processing apparatus 3 of the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted. The input unit 31 is used by the operator to input the type of the reference image.

The storage unit 34 is a storage device composed of a non-volatile memory such as a flash memory or a hard disk, and stores a reference image A, a learning image B, a network model parameter G (a plurality of sets), and the like. This reference image A is a basic image used when the multilayer neural network 36 and the network model update processing unit 37 perform supervised learning to update the network model parameter G, and is at least four criteria for specifying a road sign. It is a front image that can identify a point.

The learning image B is a learning image generated based on the reference image A, and the reference image A is geometrically transformed so as to have distortion, and some information of the road sign is lost. Images etc. are included. Various geometric transformations and the like are performed on the reference image A in consideration of the mode of the input image C that can be captured by the image pickup device 1. The principle of how to make the learning image B is the same as that described in the first embodiment. Further, also in the second embodiment, the learning image B has reference point information which is the position information of the four reference points.

Here, the recognition image D is an image of a road sign portion obtained by performing preprocessing such as trimming, noise removal, and size adjustment on the image captured by the image pickup device 1.

The learning processing unit 40 of the second embodiment shown in FIG. 11 has the same configuration as the learning processing unit 40 of the first embodiment of FIG. However, in the second embodiment, there are a plurality of types of the reference image A, and all the types of the reference image A are individually learned.

Here, the reference point when the object to be processed is a road sign will be described with reference to the figure of FIG. FIG. 12A shows direction and distance signs, and since the shape is the same quadrangle as the license plate, the four corners serve as reference points. FIG. 12B shows a non-quadrangular sign with a circular shape and a polygonal shape. In this case, at the beginning of learning, four points (P1 to P4) of a quadrangle containing the corresponding marker are input as reference points, and a point on the outer circumference or a corner point P1'on the outer circumference is used by the multilayer neural network 36. Get ~ P4'.

Next, a case where the image processing device 4 of FIG. 10 performs the recognition process will be described. FIG. 13 is a functional block diagram when performing recognition processing. The recognition processing control unit 51 includes an input image receiving unit 45 and a recognition processing unit 46. The recognition processing control unit 51 has an input image reception unit 45 that receives an input image C including a background from the image pickup device 1 via a communication interface unit 33, and a type identification unit 52 that identifies the type of the shape of the road sign. It is composed of a recognition processing unit 46 including a multi-layer neural network 36 reconstructed with the network model parameter G learned for each type. In FIG. 13, the same reference numerals as those in FIG. 5 are omitted because they overlap with the description of FIG.

The input image C is an image of a road sign to be processed. Specifically, the recognition processing unit 46 obtains four corresponding points or approximate corresponding points on the recognition image D corresponding to the four reference points by the corresponding point specifying unit 47, and determines the type of the shape of the road sign. The four corresponding points of the road sign of the recognition image D are extracted using the trained multi-layer neural network 36 with the supervision reconstructed by the trained network model parameter G, and the position information of these four corresponding points is obtained. The recognition image D is projected and converted to obtain the front image F based on the above, the obtained front image F is recognized by using the character recognition dictionary H prepared according to the type, and the characters of the road sign are output. To do.

The type identification unit 52 performs a process of identifying the type of the shape of the road sign using the recognition image D generated by the input image reception unit 45. At this time, the template prepared for each type to be identified in advance is applied, and the type of the nearest template is set as the type of the shape of the processing target.

The recognition processing unit 46 is composed of hardware shapes (boards) for each type, is indicated by reference numerals 46 (1) to 46 (n), and the recognition processing unit 46 (i) is selected according to the type. (I indicates any of 1 to n). In this respect, the recognition processing control unit 51 of the second embodiment is different from the recognition processing control unit 44 of the first embodiment.

<Recognition processing procedure>
Next, the processing procedure in the recognition stage of the image processing apparatus 4 shown in FIG. 13 will be described. FIG. 14 is a flowchart showing a processing procedure in the recognition stage of the image processing apparatus 4 shown in FIG. Briefly, the input image reception unit 45 receives the input image C (step S301), generates the recognition image D (step S302), and the type identification unit 52 uses the recognition image D to generate the shape of the road sign. Is determined (step S303). Then, the recognition processing unit 46 (i) according to the type is selected (step S304). Next, an analysis image E similar to that described in the first embodiment is generated (step S305), and the generated analysis image E is reconstructed with the network model parameter G of the type identified by the type identification unit 52. Input to the neural network 36 (step S306). The multi-layer neural network 36 calculates the corresponding points (step S307).

The recognition image D is projected and transformed using the obtained corresponding point position coordinates to generate the front image F (step S308). Then, character recognition is performed from the obtained front image F using the character recognition dictionary H according to the type, and the characters of the road sign to be processed are obtained (step S309).

Here, the image handled in the second embodiment will be described. FIG. 15 shows a triangular national highway number sign as an example. FIG. 15A is an input image C, which shows an image taken from diagonally above. FIG. 15B is a recognition image D, which shows an image having corresponding point information. FIG. 15C shows the front image F after the projective transformation. FIG. 15 (d) is a character-recognized national highway number.

In the second embodiment, a plurality of recognition processing units 46 are prepared according to the type of the shape of the road sign to be processed. For example, using FPGA (Field-Programmable Gate Array), hardware is used each time the type is identified. There is no need to prepare multiple hardware boards if you try to build.

In the second embodiment, the case of specifying the type of the road sign is shown, but the present invention is not limited to this, and the same applies to the case of outputting characters included in a display board such as a signboard. Can be applied. It is also possible to combine Examples 1 and 2.

Further, in Examples 1 and 2 above, the case where the corresponding points corresponding to the four reference points are used is shown, but the corresponding points corresponding to five or more reference points are set, and four of them are used. It is also possible to perform projective transformation. Further, when the distortion of the display board in the above input image can be approximated by a parallelogram, the three corresponding points are detected and detected by the multi-layer neural network 36 trained so that the three corresponding points can be output. It is also possible to convert to a front image by an affine transformation instead of a three-axis projective transformation using the three corresponding points.

Further, in the above-mentioned Examples 1 and 2, learning is not performed in the recognition stage, but unsupervised learning may be performed in the recognition stage. Further, in the above Examples 1 and 2, the case where the black-and-white image is the processing target is shown, but the color image and the infrared image can also be the processing target. Infrared images can be used at night. The analysis image E may be a differential image using a differential operator such as a Sobel operator or a Roberts operator, or another image to which a Gabor filter or the like is applied.

The image processing apparatus, image processing system, and image processing method of the present invention are suitable for efficiently and accurately converting images captured in various different situations into front images.

A Reference image B Learning image C Input image D Recognition image E Analysis image F Front image G Network model parameter H Character recognition dictionary 1 Image pickup device 2 Network 3, 4 Image processing device 31 Input section 32 Display section 33 Communication interface Unit 34 Storage unit 35 Learning processing control unit 36 Multi-layer neural network 37 Network model update processing unit 38 Reference image generation unit 39 Learning image generation unit 40 Learning processing unit 41 Corresponding point learning reception unit 44 Recognition processing control unit 45 Input image reception unit 46 Recognition processing unit 47 Corresponding point identification unit 48 Front image generation unit 49 Character recognition unit 50 Storage unit 51 Recognition processing control unit 52 Type identification unit

Claims (8)

A learning image generation unit for generating a learning image with missing a region including a portion of the reference point display panel from the reference image having the unrealized four reference points,
A multi-layer neural network in which supervised learning was performed using the learning image and four reference points in the learning image, and
An input image reception unit that acquires an input image including a display board,
A correspondence point specifying unit that inputs an image based on the input image to the multilayer neural network and specifies information related to four corresponding points related to the input image corresponding to the reference point .
It is provided with a front image generation unit that generates a front image corresponding to the input image by performing a projective transformation based on information related to at least three correspondence points out of the four correspondence points specified by the correspondence point identification unit. An image processing device characterized by the fact that. The image processing apparatus according to claim 1 , further comprising a character recognition unit that recognizes characters included in the front image generated by the front image generation unit. Said multilayer neural networks, and Manabu習用image a first learning image image projective transformation is missing some of the information of the real image obtained by imaging the display panel by a predetermined image pickup device, in the learning image The image processing apparatus according to claim 1 or 2 , wherein supervised learning is performed using four reference points as input information. The image processing apparatus according to any one of claims 1 to 3, further comprising a type identification unit for identifying the type of the display board. Any one of claims 1 to 4 , wherein the corresponding point specifying unit inputs an analysis image, which is a shade image generated from the input image acquired by the input image receiving unit, into the multilayer neural network. The image processing apparatus according to one. The image processing apparatus according to any one of claims 1 to 5 , wherein the display board is a license plate or a road sign indicating a vehicle registration number. An image processing system that processes images including a display board.
A learning image generation unit for generating a learning image with missing a region including a portion of the reference point the panel from the reference image having the unrealized four reference points,
A multi-layer neural network in which supervised learning was performed using the learning image and four reference points in the learning image, and
An input image reception unit that acquires an input image including a display board,
A correspondence point specifying unit that inputs an image based on the input image to the multilayer neural network and specifies information related to four corresponding points related to the input image corresponding to the reference point .
It is provided with a front image generation unit that generates a front image corresponding to the input image by performing a projective transformation based on information related to at least three correspondence points out of the four correspondence points specified by the correspondence point identification unit. An image processing system characterized by the fact that. A learning image generation step of generating a learning image with missing a region including a portion of the reference points from the reference image having the unrealized four reference points display panel,
A step of causing a multi-layer neural network to perform supervised learning using the learning image and four reference points in the learning image, and
An input image acquisition step to acquire an input image including a display board, and
A correspondence point identification step of inputting an image based on the input image into the multilayer neural network and specifying information related to four corresponding points related to the input image corresponding to the reference point ,
The front image generation step includes a front image generation step of generating a front image corresponding to the input image by performing a projective transformation based on information relating to at least three correspondence points out of the four correspondence points specified by the correspondence point identification step. An image processing method characterized by being.

Images