JP6858101B2 - Coordinate detector and trained model - Google Patents

Description

The present invention relates to a coordinate detector and a trained model.

With the development of machine vision technology and the widespread use of camera-mounted devices such as smart devices, it is required to accurately detect the shape of a photographed object.

As a method of detecting the shape of a photographed object, there is a detection method using feature point extraction. However, in the detection method using feature point extraction, it is necessary to review the feature points to be extracted and adjust the threshold value used for extraction for each object to be detected, which increases the workload of the operator. ..

Japanese Unexamined Patent Publication No. 2007-085937

Therefore, in recent years, a technique for detecting the shape of a photographed object by using machine learning has been studied. However, the conventional detection method using machine learning has low detection accuracy.

The disclosed technique has been made in view of the above, and an object thereof is to accurately detect the shape of a photographed object.

In the disclosed aspect, the coordinate detection device has a storage unit and a detection unit. The storage unit outputs information on a region in which the specified point of the object is located in the center in an input image in which the object is captured, which is generated by machine learning using an image in which the specified point of the object is located in the center. Memorize the trained model. The detection unit detects a region in which the defined point of the object is located at the center of the input image in which the object is photographed by using the learned model, and determines the defined point of the object in the input image. Detect coordinates.

According to the disclosed aspect, the shape of the object can be detected with high accuracy.

FIG. 1 is a diagram showing a configuration example of the object shape detection system of the first embodiment. FIG. 2 is a diagram showing a configuration example of the learning model generator of the first embodiment. FIG. 3 is a diagram showing a configuration example of the coordinate detection device of the first embodiment. FIG. 4 is a flowchart provided for explaining the processing of the learning model generator of the first embodiment. FIG. 5 is a diagram for explaining the operation of the learning model generator of the first embodiment. FIG. 6 is a diagram for explaining the operation of the learning model generator of the first embodiment. FIG. 7 is a diagram for explaining the operation of the coordinate detection device of the first embodiment. FIG. 8 is a diagram for explaining the operation of the classification model of the first embodiment. FIG. 9 is a flowchart provided for explaining the processing of the learning model generator of the second embodiment. FIG. 10 is a diagram for explaining the operation of the learning model generator of the second embodiment. FIG. 11 is a diagram for explaining the operation of the coordinate detection device according to the second embodiment. FIG. 12 is a diagram for explaining the operation of the detection classification model of the second embodiment. FIG. 13 is a diagram for explaining the operation of the detection classification model of the second embodiment. FIG. 14 is a diagram showing an example of the coordinate transformation of the second embodiment. FIG. 15 is a diagram showing a configuration example of the character recognition device of the third embodiment. FIG. 16 is a diagram for explaining the operation of the correction unit and the recognition unit according to the third embodiment. FIG. 17 is a diagram showing an example of the perspective projection conversion of the third embodiment. FIG. 18 is a diagram showing a configuration example of the image processing device of the fourth embodiment. FIG. 19 is a diagram for explaining the operation of the image processing apparatus of the fourth embodiment. FIG. 20 is a diagram showing an example of the object to be detected in the fifth embodiment. FIG. 21 is a diagram showing an example of the object to be detected in the fifth embodiment.

Hereinafter, examples of the coordinate detection device and the trained model disclosed in the present application will be described with reference to the drawings. It should be noted that this embodiment does not limit the coordinate detection device and the trained model disclosed in the present application. Further, in the embodiment, the same reference numerals are given to the configurations having the same functions and the steps for performing the same processing.

[Example 1]
<Configuration of object shape detection system>
FIG. 1 is a diagram showing a configuration example of the object shape detection system of the first embodiment. In FIG. 1, the object shape detection system 1 includes a learning model generation device 10 and a coordinate detection device 20.

The original image is input to the learning model generation device 10, the learning model generation device 10 generates a "learned model" using the input original image, and outputs the generated learned model to the coordinate detection device 20. ..

An image (hereinafter sometimes referred to as "detection target image") obtained by capturing an object (hereinafter sometimes referred to as "detection target object") to be shape-detected is input to the coordinate detection device 20. .. The coordinate detection device 20 detects the shape of the detection target object captured in the detection target image by using the trained model generated by the learning model generation device 10, and outputs the detection result. The detection target image corresponds to the "input image" to the coordinate detection device 20.

<Configuration of learning model generator>
FIG. 2 is a diagram showing a configuration example of the learning model generator of the first embodiment. In FIG. 2, the learning model generation device 10 includes a data set generation unit 11, a learning model generation unit 12, a storage unit 13, and an output unit 14.

The original image is input to the data set generation unit 11. The data set generation unit 11 generates a "data set" used for generating a trained model from the original image, and outputs the generated data set to the training model generation unit 12.

The learning model generation unit 12 generates a trained model using the data set generated by the data set generation unit 11, and outputs the generated trained model to the storage unit 13. That is, the data set generated by the data set generation unit 11 becomes the teacher data when the trained model is generated.

The storage unit 13 stores the learned model generated by the learning model generation unit 12.

The output unit 14 acquires the trained model stored in the storage unit 13 and outputs the acquired trained model to the coordinate detection device 20. The output of the trained model from the learning model generation device 10 to the coordinate detection device 20 is performed according to, for example, an operator's instruction to the training model generation device 10.

<Configuration of coordinate detection device>
FIG. 3 is a diagram showing a configuration example of the coordinate detection device of the first embodiment. In FIG. 3, the coordinate detection device 20 includes an acquisition unit 21, a storage unit 22, and a detection unit 23.

The acquisition unit 21 acquires the learned model output from the learning model generation device 10, and outputs the acquired learned model to the storage unit 22.

The storage unit 22 stores the learned model acquired by the acquisition unit 21.

The detection target image is input to the detection unit 23, and the detection unit 23 detects the shape of the detection target object using the learned model stored in the storage unit 22 and outputs the detection result.

<Processing of learning model generator>
FIG. 4 is a flowchart provided for explaining the processing of the learning model generator of the first embodiment.

In FIG. 4, in step S11, the data set generation unit 11 generates the learning data set A as teacher data from the original image.

In step S13, the learning model generation unit 12 learns the “detection model” as the first trained model.

Further, in step S15, the learning model generation unit 12 learns the “classification model” as the second learned model.

The learning model generation unit 12 may simultaneously perform the processing of step S13 and the processing of step S15 in parallel, or may perform the other processing after one processing is completed.

<Operation of learning model generator>
5 and 6 are diagrams for explaining the operation of the learning model generator of the first embodiment. FIG. 5 illustrates an operation example of learning of the detection model, and FIG. 6 illustrates an operation example of learning of the classification model. Hereinafter, an operation example of learning the detection model and an operation example of learning the classification model will be described separately. Further, in the following, as an example of the detection target object captured in the detection target image, a rectangular license plate of an automobile will be described. Further, in the following, as an example of the "specified points" existing on the object to be detected, the "corner points" existing at the four corners of the license plate will be described. Corner points are sometimes called "vertices".

<Example of learning operation of detection model: Fig. 5>
As shown in FIG. 5, a plurality of images of a vehicle having a number plate NP are input to the data set generation unit 11 as original images, and the data set generation unit 11 uses these plurality of original images as first teacher data. Data set A1 and data set A2 as the second teacher data are generated.

As shown in FIG. 5, the data set A1 is formed by a plurality of "negative images" in which the four corner points of the license plate NP are unclear in the image to be detected.

On the other hand, as shown in FIG. 5, the data set A2 is formed by a plurality of "positive images" including only one of the four corner points of the license plate NP.

Here, in each positive image PI, the corner point CP is located at the center of the positive image. That is, for example, when the aspect ratio of the positive image PI is "x: y = 1: 1", the positive image PI is generated so that the corner points CP are arranged at the positions of x = 1/2 and y = 1/2. Let me. In other words, of the boundary lines formed by the four sides of the license plate NP, the boundary lines of the two tangent sides are a straight line parallel to the y direction at x = 1/2 and the x direction at y = 1/2. The positive image PI is generated so that it almost overlaps the parallel straight lines. In the example shown in FIG. 5, the lower left corner point CP of the four corner points of the license plate NP is located at the center of the positive image PI. That is, in the example shown in FIG. 5, in the positive image PI, of the four sides of the license plate NP, the left side substantially overlaps the straight line parallel to the y direction at x = 1/2, and the lower side in contact with the left side is y. It almost overlaps with a straight line parallel to the x direction at = 1/2.

The learning model generation unit 12 performs machine learning using the data sets A1 and A2 generated by the data set generation unit 11 as teacher data, and generates a detection model as the first trained model. Machine learning when generating a detection model is performed by, for example, Boosting using LBP (Local Binary Pattern) features.

<Example of learning operation of classification model: Fig. 6>
As shown in FIG. 6, a plurality of images of a vehicle having a number plate NP are input to the data set generation unit 11 as original images, and the data set generation unit 11 uses these plurality of original images as third teacher data. Data set A3 is generated. The data set A1, the data set A2, and the data set A3 form the learning data set A in step ST11 of FIG.

As shown in FIG. 6, the data set A3 includes a plurality of images including only the upper left corner point among the four corner points of the license plate NP (hereinafter, may be referred to as “upper left corner point image”). , Multiple images containing only the upper right corner point (hereinafter sometimes referred to as "upper right corner point image") and multiple images containing only the lower right corner point (hereinafter referred to as "lower right corner point image"). It is formed by (sometimes called) and a plurality of images (hereinafter sometimes referred to as "lower left corner point image") including only the lower left corner point. The upper left corner point image is labeled "Corner 1" corresponding to the upper left corner point, the upper right corner point image is labeled "Corner 2" corresponding to the upper right corner point, and the lower right corner point is labeled. The image is labeled "Corner 3" corresponding to the lower right corner point, and the lower left corner point image is labeled "Corner 4" corresponding to the lower left corner point. In each image forming the data set A3, the corner point does not have to be located at the center of the image.

The learning model generation unit 12 performs machine learning using the data set A3 generated by the data set generation unit 11 as teacher data, and generates a classification model as a second trained model. Deep learning is used as machine learning when generating a classification model. The classification model is generated by, for example, a four-layer CNN (Convolutional Neural Network).

<Operation of coordinate detection device>
FIG. 7 is a diagram for explaining the operation of the coordinate detection device of the first embodiment.

The detection model generated as shown in FIG. 5 and the classification model generated as shown in FIG. 6 are acquired from the learning model generation device 10 by the acquisition unit 21 of the coordinate detection device 20 and stored in the storage unit 22. Is remembered in.

As shown in FIG. 7, when the detection target image on which the license plate is photographed is input to the coordinate detection device 20, the detection unit 23 first uses the detection model for the detection target image in the detection target image. A candidate for a region in which the corner point of the license plate is located at the center (hereinafter, may be referred to as a "corner point existence region candidate") is detected. That is, in the detection model generated by the learning model generation device 10, the detection target image is input, and information indicating a corner point existence region candidate (hereinafter, may be referred to as "candidate information") is detected in the detection target image. This is a trained model to be output to unit 23. Further, the detection model is a first element (first node) that is any layer from the input layer, the output layer, and the input layer to the output layer to which the image to be detected is input and belongs to a layer other than the output layer. And a second element (second node) whose value is calculated based on the first element and the weight of the first element. Then, the detection model performs an operation based on the first element and the weight of the first element with each element belonging to each layer other than the output layer as the first element for the detection target image input to the input layer. Output candidate information. In the example shown in FIG. 7, when the detection unit 23 uses the detection model for the image to be detected, for example, seven corner point existence region candidates of CA1 to CA7 are detected. Here, the size and aspect ratio of the area of the corner point existence area candidate are the same as those of the positive image PI in FIG.

Next, the detection unit 23 uses a classification model for the candidate information, and the corner point existence region candidate indicated by the candidate information is an region including any of the four corner points of the license plate. To identify. That is, in the classification model generated by the learning model generation device 10, candidate information is input, and the corner point existence area candidate indicated by the candidate information is an area including any of the four corner points of the license plate. This is a trained model that outputs information indicating whether or not the information (hereinafter, may be referred to as “corner point first specific information”) to the detection unit 23. Further, the classification model includes an input layer into which candidate information is input, an output layer, a first element which is any layer from the input layer to the output layer and belongs to a layer other than the output layer, and a first element. It has a second element whose value is calculated based on the weight of the first element. Then, the classification model corners the candidate information input to the input layer by performing an operation based on the first element and the weight of the first element, with each element belonging to each layer other than the output layer as the first element. The point first specific information is output.

Here, an operation example of the classification model will be described. FIG. 8 is a diagram for explaining the operation of the classification model of the first embodiment. As shown in FIG. 8, the classification model first calculates the certainty of each of the corner point existence region candidates CA1 to CA7 as to which of the corners 1 to 4 corresponds to the region including the corner point. In the example shown in FIG. 8, the corner point existence region candidate CA1 is classified as a region including a corner 4 (that is, a lower left corner) having a probability of 20% according to the classification model. Similarly, the corner point existence area candidate CA2 is an area including a corner 1 with a probability of 5% (that is, the upper left corner), and the corner point existence area candidate CA3 is an area including a corner 4 (that is, a lower left corner) with a probability of 15%. The point existence area candidate CA4 is an area including a corner 1 (that is, the upper left corner) with a probability of 92%, and the corner point existence area candidate CA5 is an area including a corner 2 (that is, the upper right corner) with an accuracy of 86%. CA6 is classified as an area including a corner 4 with an accuracy of 88% (that is, a lower left corner), and a corner point existence area candidate CA7 is classified as an area including a corner 3 with an accuracy of 91% (that is, a lower right corner) by a classification model. ..

Then, the classification model selects the corner with the highest accuracy for each of the corners 1 to 4 and outputs the corner point first specific information. That is, in the examples shown in FIGS. 7 and 8, the classification model selects the corner point existence area candidate CA4 as the area including the upper left corner point, and associates the selected corner point existence area candidate CA4 with the “corner 1”. The first specific information of the corner point is output. Further, the classification model selects the corner point existence area candidate CA5 as the area including the upper right corner point, and outputs the corner point first specific information called "corner 2" in association with the selected corner point existence area candidate CA5. .. Further, the classification model selects the corner point existence area candidate CA7 as the area including the lower right corner point, and outputs the corner point first specific information called "corner 3" in association with the selected corner point existence area candidate CA7. To do. Further, the classification model selects the corner point existence area candidate CA6 as the area including the lower left corner point, and outputs the corner point first specific information called "corner 4" in association with the selected corner point existence area candidate CA6. .. As a result, the detection unit 23 can detect a region in which the four corner points of the license plate exist (hereinafter, may be referred to as a "corner point existence region") in the detection target image.

Next, as shown in FIG. 7, the detection unit 23 extracts the center coordinates of each of the four corner point existing regions. That is, the detection unit 23 extracts the center coordinates of the corner point existence region associated with the corner point first specific information called "corner 1" as the coordinates CO1 of the upper left corner point of the license plate. Further, the detection unit 23 extracts the center coordinates of the corner point existence region associated with the corner point first specific information called "corner 2" as the coordinates CO2 of the upper right corner point of the license plate. Further, the detection unit 23 extracts the center coordinates of the corner point existence region associated with the corner point first specific information called "corner 3" as the coordinates CO3 of the lower right corner point of the license plate. Further, the detection unit 23 extracts the center coordinates of the corner point existence region associated with the corner point first specific information called "corner 4" as the coordinates CO4 of the lower left corner point of the license plate. In this way, the detection unit 23 detects the center coordinates of each of the four corner point existence regions detected using the classification model as the coordinates of each of the four corner points of the license plate in the detection target image. , Output as detection result. By connecting the outer edges of the coordinates of the four corner points detected in this way with a straight line, the rectangular shape of the license plate captured in the image to be detected can be detected.

As described above, in the first embodiment, the coordinate detection device 20 has a storage unit 22 and a detection unit 23. The storage unit 22 stores the detection model as the first trained model and the classification model as the second trained model. The detection model is a trained model generated by machine learning using a positive image PI in which the corner points of the license plate are located in the center. Further, the detection model and the classification model output information (candidate information, corner point first specific information) of the region where the corner point of the license plate is located at the center in the detection target image. The detection unit 23 uses a detection model and a classification model to detect a region in which the corner point of the license plate is located at the center of the detection target image captured by the license plate, and the detection unit 23 detects the corner point of the license plate in the detection target image. Detect the coordinates of. That is, the detection model and the classification model output the information (candidate information, corner point first specific information) of the region where the corner point of the license plate is located at the center in the detection target image to the detection unit 23, thereby detecting the detection target image. The detection unit 23 that detects the coordinates of the corner points of the license plate in the above is made to detect the region where the corner points of the license plate are located at the center.

More specifically, the detection model inputs the detection target image and outputs information (candidate information) indicating a candidate of a region in which the corner point of the license plate is located at the center of the detection target image. In the classification model, candidate information is input, and information indicating which corner point of the four corner points of the license plate is included in the corner point existence area candidate indicated by the candidate information (corner point first). Specific information) is output. The detection unit 23 detects the corner point existence region candidate using the detection model for the detection target image, and detects the region including the corner point of the license plate using the classification model for the detected corner point existence region candidate. Then, the center coordinates of the region detected by using the classification model are detected as the coordinates of the corner points of the license plate in the detection target image.

In this way, detection is performed by detecting the coordinates of the license plate corner points in the detection target image using the trained model generated by machine learning using the positive image PI in which the license plate corner points are located in the center. The rectangular shape of the license plate captured in the target image can be detected with high accuracy.

[Example 2]
<Configuration of object shape detection system, configuration of learning model generator, configuration of coordinate detection device>
Since each configuration of the object shape detection system, the learning model generation device, and the coordinate detection device of the second embodiment is the same as that of the first embodiment (FIGS. 1 to 3), the description thereof will be omitted.

<Processing of learning model generator>
FIG. 9 is a flowchart provided for explaining the processing of the learning model generator of the second embodiment.

In FIG. 9, in step S21, the data set generation unit 11 generates the learning data set B as teacher data from the original image.

In step S23, the learning model generation unit 12 learns the “detection classification model” as the learned model.

<Operation of learning model generator>
10 to 12 are diagrams provided for explaining the operation of the learning model generator of the second embodiment.

As shown in FIG. 10, a plurality of images of a vehicle having a number plate NP are input to the data set generation unit 11 as original images, and the data set generation unit 11 uses these plurality of original images as first teacher data. Data set B1 and data set B2 as the second teacher data are generated. The data set B1 and the data set B2 form the training data set B in step ST21 of FIG.

As shown in FIG. 10, the data set B1 is formed by an upper left corner point image, an upper right corner point image, a lower right corner point image, and a lower left corner point image. That is, the data set B1 is formed by a plurality of "positive images" including only one of the four corner points of the license plate NP. The upper left corner point image is labeled "Corner 1" corresponding to the upper left corner point, the upper right corner point image is labeled "Corner 2" corresponding to the upper right corner point, and the lower right corner point is labeled. The image is labeled "Corner 3" corresponding to the lower right corner point, and the lower left corner point image is labeled "Corner 4" corresponding to the lower left corner point.

Here, in each of the positive image PIs labeled with the corners 1 to 4, the corner point CP is located at the center of the positive image. That is, for example, when the aspect ratio of the positive image PI is "x: y = 1: 1", the positive image PI is generated so that the corner points CP are arranged at the positions of x = 1/2 and y = 1/2. Let me. In other words, of the boundary lines formed by the four sides of the license plate NP, the boundary lines of the two tangent sides are a straight line parallel to the y direction at x = 1/2 and the x direction at y = 1/2. The positive image PI is generated so that it almost overlaps the parallel straight lines. In the example shown in FIG. 10, the lower left corner point CP of the four corner points of the license plate NP is located at the center of the positive image PI. That is, in the example shown in FIG. 10, in the positive image PI, of the four sides of the license plate NP, the left side substantially overlaps the straight line parallel to the y direction at x = 1/2, and the lower side in contact with the left side is y. It almost overlaps with a straight line parallel to the x direction at = 1/2.

Further, in the data set B2, for each original image, the coordinate corners 1 (x, y) and the corners 2 (x) of the four corner points of the license plate NP corners 1 to 4 captured in each original image are taken. , Y), corner 3 (x, y), and corner 4 (x, y) are associated.

The learning model generation unit 12 generates a detection classification model by performing machine learning using the data sets B1 and B2 generated by the data set generation unit 11 as teacher data. Deep learning is used as machine learning when generating a detection classification model. The detection classification model is generated by, for example, an 11-layer FCN (Fully Convolutional Networks) based on YOLOv2.

Machine learning of the detection classification model is performed in two stages, the first stage and the second stage, as shown below.

That is, in the first step, the learning model generation unit 12 performs machine learning using the data set B1 generated by the data set generation unit 11 as the first teacher data, and learns the initial value of the weight. The learning of the initial value corresponds to the learning of the classification model of the first embodiment.

Next, in the second stage, the learning model generation unit 12 sets the initial values learned in the first stage as the weights of each element of the filter groups F1 to F8, and then sets the data set B2 generated by the data set generation unit 11. Machine learning is performed using it as the second teacher data to generate a detection classification model.

<Operation of coordinate detection device>
FIG. 11 is a diagram for explaining the operation of the coordinate detection device according to the second embodiment.

The detection classification model generated as shown in FIGS. 10 to 12 is acquired from the learning model generation device 10 by the acquisition unit 21 of the coordinate detection device 20 and stored in the storage unit 22.

As shown in FIG. 11, when the detection target image in which the license plate is captured is input to the coordinate detection device 20, the detection unit 23 first uses the detection classification model for the detection target image in the license plate. Four corner point existence areas CB1, CB2, CB3, CB4 are detected, and corners 1S (x, y), corners 2S (x, y), and corners 3S (corners 3S), which are the center coordinates of the detected corner point existence areas, are detected. x, y) and corner 4S (x, y) are detected. Here, the corner 1S (x, y) is the coordinates corresponding to the upper left corner point of the license plate, the corner 2S (x, y) is the coordinates corresponding to the upper right corner point of the license plate, and the corner 3S ( x, y) is the coordinates corresponding to the lower right corner point of the license plate, and corner 4S (x, y) is the coordinates corresponding to the lower left corner point of the license plate. However, the coordinates of the corner 1S (x, y), the corner 2S (x, y), the corner 3S (x, y), and the corner 4S (x, y) are local coordinates of relative coordinates.

Therefore, the detection unit 23 then detects the local coordinates of the corner 1S (x, y), the corner 2S (x, y), the corner 3S (x, y), and the corner 4S (x, y) in the image to be detected. Convert to absolute coordinates (hereinafter sometimes referred to as "image coordinates"). Then, the detection unit 23 detects the image coordinates after the corner 1S (x, y) is coordinate-converted as the coordinates CO1 of the upper left corner point of the license plate. Further, the detection unit 23 detects the image coordinates after the corner 2S (x, y) is coordinate-converted as the coordinates CO2 of the upper right corner point of the license plate. Further, the detection unit 23 detects the image coordinates after the corner 3S (x, y) are coordinate-converted as the coordinates CO3 of the lower right corner point of the license plate. Further, the detection unit 23 detects the image coordinates after the corner 4S (x, y) are coordinate-converted as the coordinates CO4 of the lower left corner point of the license plate. In this way, the detection unit 23 has the center coordinate corner 1S (x, y), corner 2S (x, y), and corner 3S (x) of each of the four corner point existence regions detected using the detection classification model. , Y) and corner 4S (x, y) are coordinate-converted and then detected as the coordinates of the four corner points of the license plate in the detection target image. By connecting the outer edges of the image coordinates of the four corner points detected in this way with a straight line, the rectangular shape of the license plate captured in the image to be detected can be detected.

Here, in the detection classification model generated by the learning model generation device 10, the detection target image is input, the corner point existence region is specified in the detection target image, and the specified corner point existence region is four number plates. Information indicating which of the corner points includes the corner point (hereinafter, may be referred to as "corner point second specific information") and the corner point existence area indicated by the corner point second specific information. This is a trained model that outputs information indicating the center coordinates of the above (hereinafter, may be referred to as “center coordinate information”) to the detection unit 23. Then, the detection unit 23 detects the four corner point existence areas on the license plate based on the corner point second specific information and the center coordinate information, and also detects the corners which are the center coordinates of each of the detected corner point existence areas. 1S (x, y), corner 2S (x, y), corner 3S (x, y), and corner 4S (x, y) are detected. Further, the detection classification model includes an input layer into which an image to be detected is input, an output layer, a first element which is any layer from the input layer to the output layer and belongs to a layer other than the output layer, and the first element. It has a second element whose value is calculated based on the element and the weight of the first element. Then, the detection classification model performs an operation based on the first element and the weight of the first element with each element belonging to each layer other than the output layer as the first element for the detection target image input to the input layer. Outputs the center coordinate information by.

<Operation of detection classification model>
12 and 13 are diagrams for explaining the operation of the detection classification model of the second embodiment.

As shown in FIG. 12, in the detection classification model, the detection target image input to the detection classification model is divided into a plurality of "M × N" grids, and each grid is sequentially set to the "attention grid GR". , Works as follows. That is, as shown in FIG. 12, the detection classification model includes the center coordinates of each of the rectangular regions a, b, and c (hereinafter, may be referred to as "rectangular region center coordinates") in the attention grid GR. Under the condition, the rectangular regions a, b, and c are detected so that the corner points of the license plate are found in the rectangular regions a, b, and c. The number of rectangular regions detected for one attention grid GR is preset as the “number of anchors”. Here, as an example, a case where three rectangular regions a, b, and c are detected for one attention grid GR will be described with “the number of anchors = 3”. The detected rectangular area (hereinafter, may be referred to as “detected rectangular area”) is expressed by the following equation (1).

Detection rectangular area =
(X coordinate, y coordinate, width w, height h, accuracy,
Class probability (corner 1, corner 2, corner 3, corner 4)) ... Equation (1)

In the formula (1), the corners 1 to 4 correspond to the upper left corner point, the upper right corner point, the lower right corner point, and the lower left corner point of the license plate, respectively, as described above. Further, in the equation (1), the "width w" and the "height h" are predetermined values. Further, the "x coordinate" and the "y coordinate" in the equation (1) are relative coordinates (that is, local coordinates) with respect to the origin of the grid GR of interest, as shown in FIG. Further, the "probability" in the equation (1) represents the probability that any corner point is included in the detection rectangular area. Further, the "class probability" in the equation (1) represents the existence probability of each of the four corner points in the detection rectangular area.

For example, the detection rectangular regions a, b, and c for the attention grid GR shown in FIG. 12 are expressed by the following equations (2) to (4).

Detection rectangular area a
= (0.12,0.87,0.49,0.64,0.589, (0,0,0.9999,0))… Equation (2)
Detection rectangular area b
= (0.16,0.77,1.77,2.11,0.010, (0,0.0001,0.9998,0.0001))… Equation (3)
Detection rectangular area c
= (0.33,0.73,5.45,6.56,0.016, (0.54,0.23,0.05,0.19))… Equation (4)

Therefore, for example, assuming that the threshold value of "accuracy" is "0.5", it is the detection rectangular area a that has an accuracy of 0.5 or more, and the accuracy of the detection rectangular areas b and c is less than 0.5, which is shown in FIG. For the grid GR of interest, the detection rectangle areas b and c are excluded, and the detection rectangle area a is selected as a corner point existence area candidate for the lower right corner point.

The detection classification model performs the above operation while sequentially setting all the grids to the attention grid GR. Then, the detection classification model determines the detection rectangular region having the highest accuracy as the final corner point existence region for each corner point. Then, the detection classification model outputs the information indicating the detection rectangular area having the highest accuracy as the second specific information of the corner points for each corner point, and outputs the information indicating the center coordinates of the rectangular area as the center coordinate information.

<Conversion from local coordinates to image coordinates>
For example, the detection unit 23 has corners 1S (x, y), corners 2S (x, y), corners 3S (x, y), and corners 4S (x), which are local coordinates of the detection rectangular region, according to the following equation (5). , Y) is converted into corner 1S (x', y'), corner 2S (x', y'), corner 3S (x', y'), and corner 4S (x', y'), which are image coordinates. To do. Therefore, the corner 1S (x', y'), the corner 2S (x', y'), the corner 3S (x', y'), and the corner 4S (x', y'), which are the image coordinates, are the detection units, respectively. It corresponds to the coordinates CO1, CO2, CO3, and CO4 (FIG. 11) output from 23. In the formula (5), "W" represents the width of the detection target image, and "H" represents the height of the detection target image. Further, in the equation (5), "u" represents the position of the attention grid GR in the x direction, and "v" represents the position of the attention grid GR in the y direction.

x'= W (u + x) / M
y'= H (v + y) / N ... Equation (5)

For example, if the size of the image to be detected is (W, H) = (1322,902) and the total number of grids is (M × N) = (13 × 13), the coordinate values before and after the coordinate conversion are As shown in FIG. FIG. 14 is a diagram showing an example of the coordinate transformation of the second embodiment.

As described above, in the second embodiment, the storage unit 22 stores the detection classification model as the learned model. The detection classification model is a trained model generated by machine learning using a positive image PI in which the corner points of the license plate are located in the center. In addition, the detection classification model outputs information on a region in which the corner point of the license plate is located at the center in the image to be detected (corner point second specific information, center coordinate information). The detection unit 23 detects the region where the corner point of the license plate is located at the center of the detection target image on which the license plate is captured by using the detection classification model, and the coordinates of the corner point of the license plate in the detection target image. Is detected. That is, the detection classification model outputs the information of the region where the license plate corner point is located at the center in the detection target image (corner point second specific information, center coordinate information), so that the license plate corner in the detection target image is output. The detection unit 23 that detects the coordinates of the points is made to detect the region where the corner point of the license plate is located at the center.

More specifically, in the detection classification model, the detection target image is input, the region where the corner point of the license plate is located at the center in the detection target image is specified, and the specified region is a plurality of four corners of the license plate. Information indicating which of the points the area includes the corner point (corner point second specific information) and information indicating the center coordinates of the area indicated by the corner point second specific information (center coordinate information). ) And is output. The detection unit 23 detects the corner point existence region using the detection classification model for the detection target image, detects the center coordinates of the corner point existence region, converts the detected center coordinates into coordinates, and then detects the detection target. It is detected as the coordinates of the corner points of the license plate in the image.

In this way, detection is performed by detecting the coordinates of the license plate corner points in the detection target image using the trained model generated by machine learning using the positive image PI in which the license plate corner points are located in the center. The rectangular shape of the license plate captured in the target image can be detected with high accuracy.

Further, since the function of the detection classification model of the second embodiment corresponds to the one in which the functions of both the detection model and the classification model of the first embodiment are integrated, the calculation scale when detecting the coordinates of the corner points is carried out. Example 2 is smaller than Example 1.

[Example 3]
<Configuration of character recognition device>
FIG. 15 is a diagram showing a configuration example of the character recognition device of the third embodiment. In FIG. 15, the character recognition device 30 includes a coordinate detection device 20, a correction unit 31, and a recognition unit 32.

The detection target image input to the character recognition device 30 is input to the coordinate detection device 20 and the correction unit 31. For example, in the image to be detected, a rectangular license plate of an automobile is photographed as in the first and second embodiments.

As described in the first or second embodiment, the coordinate detection device 20 detects the coordinates CO1, CO2, CO3, and CO4 of the four corner points of the license plate in the image to be detected, and corrects them as the detection result. Output to unit 31.

The correction unit 31 corrects the rectangular distortion of the license plate captured in the detection target image together with the characters of the license plate based on the coordinates CO1, CO2, CO3, and CO4 of the corner points, and the distortion is corrected. The image of the license plate having the latter rectangle is output to the recognition unit 32. That is, the correction unit 31 corrects the distortion of the rectangle formed by connecting the outer edges of the coordinates CO1, CO2, CO3, and CO4 of the four corner points with a straight line.

The recognition unit 32 recognizes the characters existing in the rectangle after the distortion is corrected, and outputs the recognition result. Character recognition is performed by, for example, OCR (Optical Character Recognition).

<Operation of correction unit and recognition unit>
FIG. 16 is a diagram for explaining the operation of the correction unit and the recognition unit according to the third embodiment.

As shown in FIG. 16, the correction unit 31 in which the detection target image and the coordinates CO1, CO2, CO3, CO4 of the four corner points of the license plate are input is based on the coordinates CO1, CO2, CO3, CO4. , The distortion of the rectangular shape of the license plate captured in the image to be detected is corrected by using the well-known technique "perspective projection conversion". That is, the correction unit 31 corrects the distortion of the shape of the region surrounded by the plurality of corner points corresponding to the coordinates CO1, CO2, CO3, and CO4 by using the perspective projection transformation. By this correction, even if the shape of the license plate captured in the detection target image is, for example, a trapezoid, the shape of the license plate is corrected to a rectangle. In addition, along with the correction of the distortion of the rectangle of the license plate, the distortion of the characters in the license plate is also corrected at the same time.

FIG. 17 is a diagram showing an example of the perspective projection conversion of the third embodiment. FIG. 17 is an example of coordinate conversion when the actual size of the rectangle of the license plate is 330 × 165 [mm]. By fluoroscopic projection conversion, the coordinates CO1 (x', y'), CO2 (x', y'), CO3 (x', y'), and CO4 (x', y') are converted to the coordinates CO1 (x'', y'). It is corrected to y''), CO2 (x'', y''), CO3 (x'', y''), CO4 (x'', y'').

As described above, in the third embodiment, the character recognition device 30 includes the coordinate detection device 20, the correction unit 31, and the recognition unit 32. The coordinate detection device 20 detects the coordinates CO1, CO2, CO3, and CO4 of the four corner points of the license plate in the image to be detected. The correction unit 31 corrects the distortion of the shape of the region surrounded by the plurality of corner points corresponding to the coordinates CO1, CO2, CO3, and CO4, respectively. The recognition unit 32 recognizes the characters existing in the region after the distortion of the shape is corrected.

By doing so, the recognition accuracy of the characters in the license plate can be improved. Further, in the third embodiment, the shape of the license plate in the image to be detected is detected by using machine learning, while the shape is corrected and the character is recognized separately from the shape detection by machine learning. Therefore, the number of teacher data to be prepared can be reduced, the learning time of machine learning can be shortened, and the calculation can be performed, as compared with the case where the distorted characters in the number plate are recognized as they are by using machine learning. Since the amount can be reduced, machine learning can be performed using a processor with lower specifications.

[Example 4]
<Configuration of image processing device>
FIG. 18 is a diagram showing a configuration example of the image processing device of the fourth embodiment. In FIG. 18, the image processing device 40 includes a coordinate detection device 20, a correction unit 41, a storage unit 42, a superposition unit 43, and an inverse conversion unit 44.

The detection target image input to the image processing device 40 is input to the coordinate detection device 20 and the correction unit 41.

As described in the first or second embodiment, the coordinate detection device 20 detects the coordinates of the corner points of the detection target object in the detection target image, and outputs the detection result to the correction unit 41.

The correction unit 41 uses the well-known technique "perspective projection conversion" based on the coordinates of the corner points detected by the coordinate detection device 20 to form the shape of the detection target object captured in the detection target image. The distortion is corrected, and an image of the object to be detected having the shape after the distortion is corrected (hereinafter, may be referred to as a “shape-corrected image”) is output to the superimposing unit 43. That is, the correction unit 41 corrects the distortion of the shape formed by connecting the outer edges of the coordinates of the plurality of corner points with a straight line.

The storage unit 42 stores in advance content that is superimposed on the image of the object to be detected (hereinafter, may be referred to as “superimposed content”). The superimposed content stored in the storage unit 42 is content without distortion.

The superimposition unit 43 acquires the superimposition content from the storage unit 42, superimposes the acquired superimposition content on the shape-corrected image, and superimposes the superimposition content on the shape-corrected image (hereinafter, “superimposition post-image”). Is sometimes called) is output to the inverse conversion unit 44.

The inverse conversion unit 44 performs a transformation opposite to the perspective projection conversion performed by the correction unit 41 on the superimposed image to distort the shape of the superimposed image, and superimposes the superimposed image after the shape is distorted. Output as a result.

<Operation of image processing device>
FIG. 19 is a diagram for explaining the operation of the image processing apparatus of the fourth embodiment.

As shown in FIG. 19, the coordinate detection device 20 detects the coordinates of the corner points of the detection target object in the detection target image. In FIG. 19, as an example, it is assumed that the shape of the object to be detected is a “star”.

The correction unit 41 corrects the distortion of the star-shaped image by performing perspective projection conversion based on the coordinates of the corner points detected by the coordinate detection device 20. That is, the correction unit 41 corrects the distortion of the shape of the region surrounded by the plurality of corner points corresponding to the plurality of coordinates detected by the coordinate detection device 20 by using the perspective projection conversion. This correction also corrects the distortion of the markers set in the star-shaped image.

The superimposing unit 43 superimposes the superimposing content acquired from the storage unit 42 on the distortion-corrected star-shaped image with reference to the marker set in the star-shaped image.

The inverse conversion unit 44 performs a transformation opposite to the perspective projection conversion performed by the correction unit 41 on the superimposed image to distort the shape of the superimposed image.

As described above, in the fourth embodiment, the image processing device 40 includes the coordinate detection device 20, the correction unit 41, and the superimposition unit 43. The coordinate detection device 20 detects the coordinates of a plurality of corner points of the detection target object in the detection target image. The correction unit 41 corrects the distortion of the shape of the region surrounded by the plurality of corner points corresponding to the plurality of coordinates detected by the coordinate detection device 20. The superimposing unit 43 superimposes a predetermined content on the shape after the distortion is corrected.

By doing so, the content can be superimposed on the image of the object to be detected after the distortion is corrected, so that accurate representation in AR (Augmented Reality) is possible, for example.

[Example 5]
The object to be detected by the object shape detection system 1 is not limited to the license plate. For example, the object to be detected may be a road marking or the like. 20 and 21 are diagrams showing an example of the object to be detected in the fifth embodiment. The road sign shown in FIG. 20 has three corner points. Further, the road sign shown in FIG. 21 has eight corner points.

Further, the corner point is an example of a "specified point" existing on the object to be detected, and the specified point targeted by the object shape detection system 1 is not limited to the corner point. For example, an arbitrary defining point may be set on the circumference of a circular road sign. That is, the specified point targeted by the object shape detection system 1 may be any point existing on the outer edge of the object to be detected in the image to be detected.

Further, as the specified points, the screw of the license plate in the image to be detected, the center of the seal of the license plate, the characters "・" and "-" in the license plate, and the like may be adopted.

[Other Examples]
[1] The storage units 13, 22, and 42 are realized as hardware by, for example, a memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. Examples of the memory that realizes the storage units 13, 22, and 42 include RAM (Random Access Memory) such as SDRAM (Synchronous Dynamic Random Access Memory), ROM (Read Only Memory), and flash memory. The data set generation unit 11, the learning model generation unit 12, the detection unit 23, the correction units 31, 41, the recognition unit 32, the superimposition unit 43, and the inverse conversion unit 44 can be realized as hardware, for example, by a processor. As an example of a processor that realizes a data set generation unit 11, a learning model generation unit 12, a detection unit 23, a correction unit 31, 41, a recognition unit 32, a superimposition unit 43, and an inverse conversion unit 44, a CPU (Central Processing Unit) and a DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) and the like. Further, the data set generation unit 11, the learning model generation unit 12, the detection unit 23, the correction units 31, 41, the recognition unit 32, the superimposition unit 43, and the inverse conversion unit 44 are LSIs (Large Scale Integrated) including a processor and peripheral circuits. It may be realized by circuit). The output unit 14 and the acquisition unit 21 are realized as hardware by, for example, a wireless communication module or a network interface module. Therefore, for example, the learning model generation device 10 is realized as a computer device such as a personal computer or a server. Further, for example, the coordinate detection device 20, the character recognition device 30, or the image processing device 40 is realized as a smart device such as a smartphone or a tablet terminal.

[2] All or part of each process in the above description in the object shape detection system 1 may be realized by causing a processor included in the object shape detection system 1 to execute a program corresponding to each process. For example, the program corresponding to each process in the above description may be stored in the memory, and the program may be read from the memory by the processor and executed. Further, the program is stored in a program server connected to the object shape detection system 1 via an arbitrary network, downloaded from the program server to the object shape detection system 1 and executed, or read by the object shape detection system 1. It may be stored in a possible recording medium, read from the recording medium, and executed. Recording media that can be read by the object shape detection system 1 include, for example, memory cards, USB memory, SD cards, flexible discs, magneto-optical discs, CD-ROMs, DVDs, Blu-ray (registered trademark) discs, and the like. Includes portable storage media. The program is a data processing method described in an arbitrary language or an arbitrary description method, and may be in any format such as source code or binary code. In addition, the program is not necessarily limited to a single program, but is distributed as multiple modules or multiple libraries, or cooperates with a separate program represented by the OS to achieve its function. Including things.

[3] The specific form of dispersion / integration of the object shape detection system 1 is not limited to the one shown in the drawing, and all or a part of the object shape detection system 1 can be added according to various additions or functional loads. Therefore, it can be functionally or physically distributed / integrated in any unit.

[4] The correction units 31 and 41 in Examples 3 and 4 may correct the distortion of the shape by using a technique other than the perspective projection conversion.

1 Object shape detection system 10 Learning model generation device 20 Coordinate detection device 11 Data set generation unit 12 Learning model generation unit 23 Detection unit 31, 41 Correction unit 32 Recognition unit 43 Superimposition unit

Claims (4)

A first trained model and a second trained model generated by machine learning using an image including a specified point of an object, and an input image in which the object is photographed is input, and the input image is described as described above. The first trained model that outputs the first information that is information indicating the candidate of the region including the specified point of the object and the first information are input, and the candidate indicated by the first information is the object. A storage unit that stores a second trained model that outputs second information, which is information indicating which of the plurality of specified points is included in the specified point.
Using a pre-Symbol first learned model to the input image in which the object was photographed to detect the candidate, a defined point of the object using the relative detected the candidate second trained model A detection unit that detects a region including the region and detects the coordinates of a specified point of the object in the input image based on the region detected by using the second trained model.
A coordinate detection device comprising. The coordinate detection device according to claim 1 and
A correction unit that corrects the distortion of the shape of the region surrounded by the plurality of specified points corresponding to the plurality of coordinates detected by the coordinate detection device, and the correction unit.
A recognition unit that recognizes characters existing in the region after the distortion of the shape is corrected, and
A character recognition device comprising. The coordinate detection device according to claim 1 and
A correction unit that corrects the distortion of the shape of the region surrounded by the plurality of specified points corresponding to the plurality of coordinates detected by the coordinate detection device, and the correction unit.
A superimposing portion that superimposes a predetermined content on the shape after the distortion is corrected, and
An image processing device comprising. A trained model formed from a first trained model and a second trained model generated by machine learning using an image containing a defined point of an object.
The first trained model is input with an input image in which the object is photographed, and outputs first information which is information indicating a candidate for a region including a specified point of the object in the input image.
The second trained model is information indicating which of the plurality of specified points of the object is included in the candidate indicated by the first information when the first information is input. Output some second information,
Trained model.

Images