JP7138158B2 - OBJECT CLASSIFIER, OBJECT CLASSIFICATION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an object classification device, an object classification method, and a program.

2. Description of the Related Art Conventionally, there has been known a technique capable of accumulating images captured by an in-vehicle camera and identifying metal objects such as manholes and road markings included in the images. For example, the object recognition device described in Patent Document 1 generates a bird's-eye view image from accumulated images from an in-vehicle camera, and identifies a metal object existing on the road surface by using the brightness difference occurring in the bird's-eye view image. . Specifically, the object recognition device sets the conditions for digital image processing, converts the image from the in-vehicle camera into a bird's-eye view image, detects metal objects included in the bird's-eye view image, and detects the relative position of the vehicle from the metal object. is located.

Japanese Patent Application Laid-Open No. 2009-266003

As in the object recognition device described above, car-mounted cameras are used to detect manholes and public facilities. It is rare.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an object classification device, an object classification method, and a program capable of classifying objects with high accuracy.

(1) One aspect of the present invention includes a detection unit that detects an object region from a captured image, a first process that classifies an object included in the object region based on a first element, and the first process. a classifying unit that classifies objects classified by the second element based on the second element, the classifying unit classifying the first classification result based on the first element as teacher data and the first processing using a first model that outputs a first classification result based on the first element included in the image of the object region when the image of the object region is input. is machine-learned using the second classification result based on the second element as teacher data, and when the image of the object area is input, the second classification based on the second element included in the image of the object area is performed. executing the second process using a second model that outputs 2 classification results, the first element being at least one of a pattern and a symbol of the object, and the second element is at least one of shape, location and size of a hole in said object .

( 2 ) An aspect of the present invention is the above-described object classification device, wherein the classification unit trapezoidally corrects the object region detected by the detection unit, and performs the trapezoidal correction based on the object region. 1 and the second process may be performed.

( 3 ) An aspect of the present invention is a detection unit that detects an object region from a captured image, and a first classification unit that classifies objects included in the object region based on at least one of patterns and symbols. and a second classification unit that classifies the objects classified by the first classification unit based on the shape of the hole; and the objects classified by the first classification unit based on the position of the hole. a third classifying unit that classifies the objects classified by the second classifying unit based on size; and a third classifying unit that classifies the objects classified by the third classifying unit based on size. 4 classification units, wherein the first classification unit performs machine learning using a first classification result based on at least one of the pattern and the symbol as teacher data, and is included in the image of the object region. A process based on a first model that outputs a first classification result when at least one of a pattern and a symbol is input, and the second classification unit performs a second classification based on the shape of the hole. performing machine learning using the classification result as teacher data, performing processing based on a second model for outputting a second classification result when an image of the object region based on the classification result by the first classification unit is input, A third classification unit performs machine learning using a third classification result based on the position of the hole as teacher data, and when an image of the object region based on the classification result of the first classification unit is input, a third classification result is obtained. The fourth classification unit classifies the objects classified by the second classification unit based on the size of the objects, It is an object classification device that classifies objects classified by three classification units .

( 4 ) One aspect of the present invention is a step of detecting an object region from a captured image, a first process of classifying an object included in the object region based on a first element, and and a second process of classifying the classified objects based on a second element, wherein machine learning is performed using a first classification result based on the first element as teacher data, and the object is executing the first processing using a first model that outputs a first classification result based on the first element contained in the image of the object region when an image of the region is input; Machine learning is performed using a second classification result based on the elements of as teacher data, and when an image of the object area is input, a second classification result based on the second element contained in the image of the object area is output. performing the second process using a second model, wherein the first element is at least one of a pattern and a symbol of the object; and the second element is a hole portion of the object; is at least one of the shape, position, and size of the object classification method.

( 5 ) An aspect of the present invention is a step of detecting an object region from a captured image, a first classification step of classifying an object included in the object region based on at least one of a pattern and a symbol, a second classification step of classifying the objects classified by the first classification step based on the shape of the hole; and a third classification step of classifying the objects classified by the first classification step based on the position of the hole. and a fourth classification step of classifying the objects classified by the second classification step based on size, and classifying the objects classified by the third classification step based on size. , in the first classification step, machine learning is performed using a first classification result based on at least one of the pattern and the symbol as teacher data, and at least one of the pattern and the symbol included in the image of the object region is performed based on a first model for outputting a first classification result when is input, and in the second classification step, machine learning is performed using the second classification result based on the shape of the hole as teacher data, Processing is performed based on a second model for outputting a second classification result when an image of the object region based on the classification result of the first classification step is input, and the third classification step includes: A process based on a third model, which is machine-learned using a third classification result based on the above as teacher data, and outputs a third classification result when an image of the object region based on the classification result of the first classification step is input. wherein the fourth classification step classifies the objects classified by the second classification step and classifies the objects classified by the third classification step based on the size of the objects. is.

( 6 ) An aspect of the present invention provides a computer with a step of detecting an object region from a captured image, a first process of classifying an object included in the object region based on a first element, the first a second process of classifying the objects classified by the process of (1) based on a second element, and the first classification result based on the first element is converted to teacher data and the first processing using a first model that outputs a first classification result based on the first element included in the image of the object region when the image of the object region is input. is performed, machine learning is performed using the second classification result based on the second element as teacher data, and when the image of the object area is input, the second classification based on the second element included in the image of the object area is performed. executing the second process using a second model that outputs two classification results, the first element being at least one of a pattern and a symbol of the object, and the second element is at least one of shape, position and size of a hole in said object .

( 7 ) An aspect of the present invention provides a computer with a step of detecting an object area from a captured image, and classifying an object included in the object area based on at least one of a pattern and a symbol. a second classification step of classifying the objects classified by the first classification step based on the shape of the hole; and classifying the objects classified by the first classification step based on the position of the hole. a fourth classification step of classifying the objects classified by the second classification step based on size, and classifying the objects classified by the third classification step based on size; wherein the first classification step is machine-learned using a first classification result based on at least one of the pattern and the symbol as teacher data, and is included in the image of the object region. performing processing based on a first model for outputting a first classification result when at least one of a pattern and a symbol is input, wherein the second classification step includes a second classification based on the shape of the hole; performing machine learning using the classification result as teacher data, performing processing based on a second model for outputting a second classification result when an image of the object region based on the classification result of the first classification step is input, and In the third classification step, machine learning is performed using the third classification result based on the position of the hole as teacher data, and when the image of the object region based on the classification result in the first classification step is input, the third classification result is and the fourth classification step classifies the objects classified by the second classification step based on the size of the objects, and classifies the objects classified by the third classification step It is a program that classifies objects that have been drawn .

According to one aspect of the present invention, objects can be classified with high accuracy.

1 is a block diagram showing one configuration example of an object detection system according to an embodiment; FIG. 4 is a flow chart showing an example of a processing procedure in the object detection device of the embodiment; It is a figure which shows an example of the captured image in embodiment. It is a block diagram which shows an example of the detection part of embodiment. It is a figure which shows an example of the hiding determination process in embodiment. It is a figure for explaining keystone correction in an embodiment, (a) shows a field before correction, and (b) shows a field after correction. FIG. 4 is a diagram illustrating correcting (converting) an object region to a region by a function in an embodiment; FIG. 4 is a diagram for explaining the process of converting from a trapezoidally corrected area to a perfect circular area in the embodiment, where (a) shows an object area A, (b) shows an area A′, and (c) shows a true circular area. A circular area A'' is shown. It is a figure which shows an example of the captured image in embodiment. It is a figure which shows an example of the captured image in embodiment, (a) shows the object area|region containing the whole object, (b) shows the object area|region which was cut off. FIG. 10A is a diagram for explaining a problem when obtaining the center point of an object, FIG. 1A is an example of a captured image, and FIG. 1B is a plan view of an object region; It is a figure for demonstrating conversion from the rectangle to a trapezoid in embodiment, (a) shows a rectangle and (b) shows a trapezoid. It is a figure for explaining keystone correction in an embodiment, (a) shows a rectangle before correction, and (b) shows a trapezoid after correction. FIG. 10 is a diagram illustrating transforming the center point of a rectangle to the center point of a trapezoid in an embodiment; It is a figure which shows an example of a right side image and a left side image. FIG. 10 is a diagram for explaining processing for determining matching between an object region in a right image and an object region in a left image in the embodiment; It is a top view showing an example of a manhole in an embodiment. It is a block diagram which shows one structural example of the classification|category part in embodiment.

An object classification device, an object classification method, and a program to which the present invention is applied will be described below with reference to the drawings.

[Overview of embodiment]
The object detection system 1 of the embodiment collects captured images transmitted from one or more in-vehicle devices 10 and detects an object included in the collected captured images. The object detection system 1 performs a process of estimating the position of a detected object and a process of classifying the detected object. The object detection system 1 creates information including object position information and object classification information, and provides the created information to other systems (not shown). In the following embodiments, captured images include, for example, a right image captured by a camera installed in the front right portion of the vehicle and a left image captured by a camera installed in the front left portion of the vehicle. The captured image is not limited to the right side image and the left side image, and may be an image captured by a single camera mounted on the vehicle, or may be an image captured behind the vehicle. Images captured by various cameras that are not mounted on the vehicle may also be used. In the following embodiments, the object is a manhole, but the object detection system 1 is not limited to this, and can be applied to various objects that can be targets for position estimation and object classification. For example, the object detection system 1 can also be applied to utility poles and the like provided along roads.

[Configuration of object detection system 1]
FIG. 1 is a block diagram showing a configuration example of an object detection system 1 according to an embodiment. The object detection system 1 includes, for example, one or more in-vehicle devices 10, a data collection device 20, and an object detection device 100. The in-vehicle device 10, the data collection device 20, and the object detection device 100 are connected to, for example, a communication network. Each device connected to the communication network has a communication interface such as a NIC (Network Interface Card) or a wireless communication module (not shown in FIG. 1). Communication networks include, for example, the Internet, WANs (Wide Area Networks), LANs (Local Area Networks), cellular networks, and the like.

The in-vehicle device 10 includes a camera, a data transmission device, and the like, and transmits left and right images including a right image and a left image to the data collection device 20 . The data collection device 20 collects the right image and the left image, and stores the right image and the left image in association with the captured position information. In the following description, the right image and the left image are collectively referred to as captured images.

The object detection device 100 includes, for example, a detection unit 110, a position estimation unit 120, a classification unit 130, and an information provision unit 140. Functional units such as the detection unit 110, the position estimation unit 120, the classification unit 130, and the information provision unit 140 are implemented by a processor such as a CPU (Central Processing Unit) executing a program stored in a program memory. Some or all of these functional units may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array), It may be realized by cooperation of software and hardware. The program is stored in advance in a storage device such as the HDD or flash memory of the object detection device 100, for example.

The detection unit 110 detects an object area included in the captured image. The object area is information indicating an area representing an object in the captured image. The position estimator 120 estimates the position of the object. Classification unit 130 classifies objects. The information provider 140 provides information including the position of the object estimated by the position estimator 120 and the classification of the object determined by the classifier 130 . Note that the information providing unit 140 provides information including the estimated positions and classifications of each of the right and left images.

[Processing of object detection system 1]
FIG. 2 is a flow chart showing an example of a processing procedure in the object detection device 100 of the embodiment. The object detection device 100 first acquires a captured image to be processed from the data acquisition device 20 (step S100). Next, the object detection apparatus 100 determines whether or not part of the object detected from the captured image is hidden by another object (step S102). Object detection apparatus 100 excludes the detected object from subsequent processing when a part of the object detected from the captured image is hidden by another object. Next, the object detection device 100 estimates the position of the object to be processed (step S104). Position estimation processing of the embodiment estimates the position of an object within a captured image. Next, the object detection apparatus 100 determines whether or not the object whose position is estimated using the right image matches the object whose position is estimated using the left image (step S106). Next, the object detection device 100 classifies the objects detected from the captured image (step S108). Object detection device 100 then provides information including the position and classification of the object (step S110). The object detection device 100 provides information including, for example, the location and classification for each object for which matching is determined. The position information providing process may provide position information including the position in the captured image estimated by the position estimation process and the latitude and longitude corresponding to the captured image. may provide latitude and longitude information for the object calculated based on the location of the object. In the embodiment, the processes are executed in the order of the hidden determination process (step S102), the position estimation process (step S104), and the match determination process (step S106). You may

Note that the object detection apparatus 100 may estimate the position of the object and not classify the object. In this case, the object detection device 100 provides information that includes the position of the object and does not include the classification. Further, the object detection apparatus 100 may classify the objects and not estimate the position of the objects. In this case, the object detection device 100 may provide information that includes the classification of the object and not the location.

[Hidden determination process (step S102)]
The hiding determination process (step S102) will be described below. FIG. 3 is a diagram showing an example of a captured image according to the embodiment. The captured image shown in FIG. 3 is an image captured by a camera provided on the right side of the vehicle. The captured image includes two object areas A1 and A2 corresponding to two manholes.

FIG. 4 is a block diagram showing an example of the detection unit 110 of the embodiment. The detection unit 110 includes a detection processing unit 112 and a model construction unit 114 . The detection processing unit 112 inputs the captured image to the detection model 112A, and outputs the estimation result of the object region from the detection model 112A. The object region estimation result includes information indicating the shape and position of the object region. The detection model is a machine learning model, such as a convolutional neural network (CNN).

The model construction unit 114 constructs the detection model 112A. The model construction unit 114 acquires information indicating a captured image and an object region as teacher data. The model construction unit 114 inputs a captured image to the detection model 112A during learning. When a captured image is input, the detection model 112A outputs an estimation result indicating the object area of the object area A included in the captured image. The model construction unit 114 recursively updates the processing parameters of the detection model 112A so that the information indicating the object region output from the detection model 112A matches the information indicating the object region as teacher data. The processing parameters are, for example, the number of layers, the number of nodes in each layer, the method of connecting nodes between each layer, the activation function, the error function, the gradient descent algorithm, the pooling area, the kernel, the weight coefficient, and the weight in the convolutional neural network. At least one of the matrices. Thereby, the model building unit 114 performs deep learning, for example, in order to acquire the processing parameters. Deep learning is machine learning using a multi-layered structure, particularly a neural network with three or more layers. Note that the model construction unit 114 may not be included in the detection unit 110 as long as the detection model 112A can be introduced into the detection processing unit 112 at the time of initial setting or maintenance of the object detection device 100 . At the time of estimation, the detection model 112A receives a photographed image and outputs an estimation result indicating the shape and position of the object region. The detection processing unit 112 outputs a photographed image to which a tag corresponding to the estimation result is added, for example.

FIG. 5 is a diagram illustrating an example of hidden determination processing in the embodiment. The detection unit 110 uses the object area acquired by the detection processing unit 112 to perform the hidden determination process (step S102). For example, as shown in FIG. 5(a), a part of the object area A corresponding to the manhole may be hidden by the obstacle area B corresponding to the obstacle. The detection unit 110 performs the following processing in order to exclude the object region A partially hidden by the obstacle region B from the processing target.

First, the detection unit 110 detects an object area A shown in FIG. 5(b). Next, the detection unit 110 converts the object area A into an area A' as shown in FIG. Next, the detection unit 110 transforms the area A' into a perfect circular area A'' shown in FIG. 5(d). Next, the detection unit 110 calculates the area ratio between the area A' and the perfect circular area A''. Next, the detection unit 110 determines whether or not the object area A is hidden by the obstacle area B based on the comparison between the calculated area ratio and a predetermined value. Each process will be described below.

(Conversion processing from object area A to area A')
FIG. 6 is a diagram for explaining trapezoidal correction according to the embodiment, in which (a) shows area A before correction and (b) shows area A′ after correction. In FIG. 6A, (x0, y0) are the coordinates of the convergence point of the captured image. (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are the edge coordinates of a rectangle enclosing object region A. w is the width of the object area A connecting the two longitudinal midpoints (xc1, yc1) and (xc2, yc2). (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4') are the edge coordinates of the trapezoidal region in contact with the object region A. The trapezoidal area has two bases parallel to the X direction, a straight line connecting the vertical midpoint (xc1, yc1) and the convergence point, and a straight line connecting the vertical midpoint (xc2, yc2) and the convergence point. defined by In FIG. 6B, the corrected area A' is a circular area with a diameter w. The area A' after correction touches the square area. The square regions have (x1'', y1''), (x2'', y2''), (x3'', y3''), and (x4'', y4'').

Using the edge coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the object region A and the convergence point (x0, y0), the detection unit 110 Find the edge coordinates (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4') of the trapezoidal region. The detection unit 110 uses, for example, Equation 1 below.

Next, the detection unit 110 detects the coordinates (x1'', y1''), (x2'', y2''), (x3'', y3''), and (x4'', y4'') of the square area. ). The detection unit 110 uses, for example, Equation 2 below.

Next, the detection unit 110 obtains a trapezoidal correction function P1 from a formula for projectively transforming a trapezoidal area into a square area. The detection unit 110 obtains a function P1 defined, for example, by Equations 3 and 4 below. Equation 3 is a formula for obtaining the coordinates (xi'', yi'') of the square area by transforming the coordinates (xi', yi') of the trapezoidal area using the function P1. Equation 4 is the matrix that defines P1.

Next, the detection unit 110 can obtain a keystone-corrected area A' by substituting the object area A into the function P1. FIG. 7 is a diagram showing correction (conversion) of an object area A to an area A' by a function P1 in the embodiment.

(Conversion processing from area A' to perfect circle area A'')
8A and 8B are diagrams for explaining the process of converting from a trapezoidally corrected area A' to a perfect circle area A'' in the embodiment, where (a) shows the object area A and (b) shows the area A ', and (c) shows a perfect circular region A''.
First, the detection unit 110 detects the X-direction minimum point z1, the X-direction maximum point z3, the Y-direction minimum point z4, and the Y-direction maximum point z2 of the area A. FIG. These detected points z1 to z4 are the coordinates of the outer peripheral points of the area A'. Next, the detection unit 110 obtains the X direction middle point (the center in the width direction) and the Y direction middle point (the center in the height direction) using the outer peripheral points, and determines the area that is the center in the width direction and the center in the height direction. Find the center point z0 of A. Next, the detection unit 110 trapezoidally corrects the X-direction minimum point z1, the X-direction maximum point z3, the Y-direction minimum point z2, the Y-direction maximum point z4, and the center point z0 using the function P1. That is, the detection unit 110 uses zn'=P1(zn) to obtain zn' after trapezoidal correction. Next, the detection unit 110 detects the distance d1 between the center point z0′ and the outer peripheral point z1′, the distance d2 between the central point z0′ and the outer peripheral point z2′, the distance d3 between the central point z0′ and the outer peripheral point z3′, the center A distance d4 between the point z0' and the outer peripheral point z4' is obtained (dn=d(z0'-zn')). The detection unit 110 sets the maximum value among the distances d1, d2, d3, and d4 as the radius r of the perfect circular area A'' (r=max(d1, d2, d3, d4)).

(Hidden judgment processing)
The detection unit 110 counts the number of pixels N1 included in the area A'. The detection unit 110 counts the number of pixels N2 included in the perfect circular area A''. The detection unit 110 may count ^N2 using π×r2. The detection unit 110 determines whether or not the ratio between the number of pixels N1 and the number of pixels N2 is equal to or less than a predetermined number. judge. The detection unit 110 determines that the object region A corresponds to an object that is not hidden by an obstacle when the ratio is not equal to or less than the predetermined number. The detection unit 110 excludes the object area A corresponding to the object hidden by the obstacle from the subsequent processing target, and selects the object area A corresponding to the object not hidden by the obstacle as the subsequent processing target. The predetermined value is a value set in advance, for example, a value set so as to select the object region A such that subsequent position estimation can be performed with high accuracy. It is set to pick objects that are not obscured by obstacles.

(Selection processing of other object regions A)
FIG. 9 is a diagram showing an example of a captured image in the embodiment. The detection unit 110 may select the object corresponding to the object area A based on the distance from the imaging position based on the position of the object area A in the image to the object. The detection unit 110 sets a predetermined range in the captured image as the object area A detection area. The detection unit 110 sets, for example, an area included from the position Rmin to the position Rmax in the Y direction of the captured image as the detection area. The position Rmin and the position Rmax are set based on, for example, a range in which a clear and appropriately sized image can be captured in the imaging range of the camera, so that subsequent position estimation can be performed with high accuracy. . If the object region A is included in the detection region, the detection unit 110 sets the object region A as a target for subsequent processing. Exclude from coverage.

FIG. 10 is a diagram showing an example of a captured image according to the embodiment, (a) showing an object area A1 including the entire object, and (b) showing a cut-out object area A2. The detection unit 110 may select an object corresponding to the object area based on comparison between the width of the object area and a predetermined object area width. For example, the detection unit 110 determines whether or not the difference between the width W1 of the object area A1 detected from the left image and the width W2 of the object area A2 detected from the right image is equal to or greater than a predetermined value. The width of the object region is the difference between the minimum value (rightmost coordinate) and the maximum value (leftmost coordinate) in the X direction (horizontal direction) of the captured image. By determining that the width W2 is smaller than the width W1 by a predetermined value or more, the detection unit 110 excludes the object region A2 from subsequent processing targets. The predetermined value is, for example, set to a value that excludes manholes (objects) that are not entirely included in the captured image and that are cut off from the captured image.

[Position estimation process (step S104)]
The position estimation process (step S104) will be described below. FIGS. 11A and 11B are diagrams for explaining a problem when obtaining the center point of an object. FIG. 11A is an example of a captured image, and FIG. 11B is a plan view of an object area. When the object area A detected by the detection unit 110 is transformed into a plane viewed from above, the center point C calculated from the object area A shifts to the center point C#. That is, from the midpoint between the leftmost coordinate a and the rightmost coordinate b in the horizontal direction of the object region A shown in FIG. Even if the center point C is calculated, the center point C# is the middle point between the leftmost coordinate a# and the rightmost coordinate b# of the object area A# in the horizontal direction, and the uppermost coordinate c of the object area A# in the vertical direction. It deviates from the point calculated from # and the middle point of the lowest coordinate d#.

(Conversion processing from rectangle 200 to trapezoid 300)
12A and 12B are diagrams for explaining conversion from a rectangle to a trapezoid in the embodiment, where (a) shows a rectangle and (b) shows a trapezoid. The position estimating unit 120 converts a rectangle 200 having a shape based on the position in the image of the object region A to a trapezoid 300 having a shape based on the position in the image of the object region A, based on the conversion function. Transform the center point C. The rectangle 200 is, for example, a square whose center of gravity is the center point C of the object region A and whose one side has the same length as the width of the object region A in the X direction. The width of the object area A is the distance from the leftmost coordinate a of the object area A to the rightmost coordinate. A trapezoid 300 is a figure transformed by the rectangle 200, and is a figure whose sides are in contact with the leftmost coordinate a, the rightmost coordinate b, the uppermost coordinate c, and the lowermost coordinate d.

13A and 13B are diagrams for explaining trapezoidal correction in the embodiment, in which FIG. 13A shows a rectangle 200 before correction and FIG. 13B shows a trapezoid 300 after correction. In FIG. 13(a), a rectangle 200 is a square with sides that are the width w of the object region A. In FIG. The width w is the distance between the vertical midpoint of the object region A (xc1, yc1) and the vertical midpoint of the object region A (xc2, yc2). Rectangle 200 has (x1'', y1''), (x2'', y2''), (x3'', y3''), and (x4'', y4'') as end coordinates. . In FIG. 13B, (x0, y0) are the coordinates of the convergence point of the captured image. (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are the edge coordinates of a rectangle enclosing object region A. (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4') are the edge coordinates of trapezoid 300; The trapezoid 300 has two bases parallel to the X direction, a straight line connecting the vertical midpoint (xc1, yc1) and the convergence point, and a straight line connecting the vertical midpoint (xc2, yc2) and the convergence point. defined by

The position estimation unit 120 obtains a rectangle (square) 200 assuming a plane. For example, the position estimation unit 120 substitutes xc1, yc1, and w into Equation 5 below to obtain end coordinates (x1'', y1''), (x2'', y2'') of the rectangle 200. ), (x3'', y3''), and (x4'', y4'').

Using the edge coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) of the object region A and the convergence point (x0, y0), the position estimation unit 120 , the edge coordinates (x1′, y1′), (x2′, y2′), (x3′, y3′), and (x4′, y4′) of the trapezoid 300 are obtained. The position estimating unit 120 uses, for example, Equation 6 below to determine the end coordinates (x1', y1'), (x2', y2'), (x3', y3'), and (x4') of the trapezoid 300. , y4′).

Next, the position estimation unit 120 calculates the coordinates (x1'', y1''), (x2'', y2''), (x3'', y3''), and (x4'', y4') of the rectangle 200. ') and the edge coordinates (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4') of the trapezoid 300, the keystone correction Find the function P2. The position estimation unit 120 uses, for example, Equation 7 below. Equation 8 is the matrix that defines P2.

Next, the position estimation unit 120 can obtain the center point C# of the trapezoid 300 by transforming the center point C of the rectangle 200 using the function P2. FIG. 14 is a diagram illustrating transforming the center point C of the rectangle 200 to the center point C# of the trapezoid 300 in the embodiment. Position estimation section 120 estimates center point C# as the position of object region A. FIG.

[Consistency determination processing (step S106)]
FIG. 15 is a diagram showing an example of a right image and a left image. For example, when a plurality of manholes are included in the captured image, there is a possibility of erroneously determining that the manhole included in the right image and the manhole included in the left image are the same. In order to suppress this, the object detection system 1 performs matching determination processing for determining whether or not the object area included in the right image and the object area included in the left image are the same.

16A and 16B are diagrams for explaining the process of determining the matching between the object area of the right image and the object area of the left image in the embodiment. For example, the position estimation unit 120 calculates the distance between the center point CR of the object area AR of the right image and the center point CL of the object area AL of the left image in a state in which the right image and the left image are superimposed. If the calculated distance between the center points is within the first predetermined value and is greater than or equal to the second predetermined value, the position estimation unit 120 determines whether the object corresponding to the object area AR having the center point CR and the object having the center point CL It is determined that the object corresponding to the area AL is the same.

The first predetermined value is, for example, the maximum value of positional deviation between the right image and the left image described above. The second predetermined value is, for example, the minimum value of positional deviation between the right image and the left image described above. The maximum positional deviation is, for example, the distance Da between the position x10# in the right image corresponding to the extreme point x10 on the position Rmin of the left image and the extreme point x20 on the position Rmin of the left image. The minimum positional deviation point is, for example, the distance Db between the position x11# in the right image corresponding to the central point x11 on the position Rmax of the left image and the center x21 on the position Rmax of the left image.

Note that the position estimating unit 120 determines the distance between the center point CR of the object area AR of the right image and the center point CL of the object area AL of the left image in the determination of matching, but is not limited to this. Predetermined values for matching determination are set for each of the distance in the perspective direction (Y direction) of the image and the distance in the horizontal direction (X direction) of the captured image. Matching may be determined based on a comparison with a predetermined value, and a comparison between the X-direction distance of the right and left images and a predetermined value. The predetermined value in this case may be set based on the positional deviation of the right and left images in the Y direction, and may be set based on the positional deviation of the right and left images in the X direction.

[Classification process (step S108)]
Drawing 17 is a top view showing an example of a manhole in an embodiment. As shown in Figures 17(a), (b), (c), and (d), manholes differ in pattern, symbol or mark, keyhole shape, keyhole position, and size, and are characterized by these elements. attached. These elements are, for example, the standards applied to manholes, the patterns and symbols that represent the manhole owners and managers, the shape and position of the keyholes used to open the manholes, the types, types and uses of the manholes, the size of the manholes and their use. Examples include a shape, but the present invention is not limited to this, and any element for distinguishing the manhole from other manholes may be used.

FIG. 18 is a block diagram showing a configuration example of the classification section 130 in the embodiment. The classification unit 130 includes, for example, a conversion unit 132, a plurality of classification processing units 134A, 134B and 134C, a diameter classification unit 136, and a model construction unit 138. Functional units such as the conversion unit 132, the plurality of classification processing units 134A, 134B and 134C, and the diameter classification unit 136 are implemented by, for example, executing a program using a CPU or the like. The conversion unit 132 converts the image representing the object region A into a plane image. The transformation unit 132 may acquire the planar image transformed by the position estimation unit 120 . A plurality of classification processors 134A, 134B and 134C classify objects based on mutually different factors. A diameter classifier 136 classifies objects based on diameter.

The classification processing unit 134A performs a first process of classifying the objects included in the object area A based on the first element. The classification processing unit 134B, the classification processing unit 134C, and the diameter classification unit 136 perform a second process of classifying the objects classified by the first process based on the second element. The first element is at least one of the pattern and symbol of the object, and the second element is at least one of the shape, position and size of the hole in the object. The classification unit 130 may include any one of the classification processing unit 134A, the classification processing unit 134B, the classification processing unit 134C, and the diameter classification unit 136.

The classification processing unit 134A performs classification processing according to the pattern/mark model 1341 . The pattern/mark model 1341 is a model machine-learned using, for example, a picked-up image to which a tag indicating the pattern of an object is attached as teacher data. The machine learning model in the embodiment is, for example, a convolutional neural network (CNN: Convolutional Neural Network), but is not limited thereto, and includes a perceptron neural network, a recurrent neural network (RNN: Recurrent Neural Network), a residual network ( Other neural networks such as ResNet (Residual Network) may be set. The pattern/mark model 1341 is partly or partly based on supervised learning estimation models such as decision trees, regression trees, random forests, gradient boosting trees, linear regressions, logistic regressions, or SVMs (support vector machines). Can be set to all.

The pattern/mark model 1341 outputs an estimation result indicating the pattern or mark of the object region A included in the captured image when the captured image is input during learning. The classification processing unit 134A updates the processing parameters of the pattern/mark model 1341 so that the estimation result matches the tag as teacher data. The processing parameters are, for example, the number of layers, the number of nodes in each layer, the method of connecting nodes between each layer, the activation function, the error function, the gradient descent algorithm, the pooling area, the kernel, the weight coefficient, and the weight in the convolutional neural network. At least one of the matrices. At the time of estimation, the classification processing unit 134A receives the image of the object region A supplied from the conversion unit 132 and outputs a classification result. The classification processing unit 134A outputs, for example, the image of the object region A to which the tag corresponding to the classification result is added to the classification processing unit 134B and the classification processing unit 134C.

The classification processing unit 134B performs classification processing according to the shape model 1342 . The shape model 1342 is a model machine-learned using, for example, a captured image with a tag indicating the shape of a keyhole as teacher data. The machine learning model in the embodiment is, for example, a convolutional neural network (CNN), but is not limited to this as in the pattern/mark model 1341, and may be a perceptron neural network, a recurrent neural network (RNN), or a residual network. Other neural networks such as (ResNet) may be set. Also, the shape model 1342 may be partly or wholly a supervised learning estimation model such as a decision tree, regression tree, random forest, gradient boosting tree, linear regression, logistic regression, or SVM (support vector machine). May be set. When a captured image is input during learning, the shape model 1342 outputs an estimation result indicating the shape of the keyhole included in the captured image. The classification processing unit 134B updates the processing parameters of the shape model 1341 so that the estimation result matches the tag as teacher data. The processing parameters are, for example, the number of layers, the number of nodes in each layer, the method of connecting nodes between each layer, the activation function, the error function, the gradient descent algorithm, the pooling area, the kernel, the weight coefficient, and the weight in the convolutional neural network. At least one of the matrices. At the time of estimation, the classification processing unit 134B receives the image of the object region A supplied from the classification processing unit 134A and outputs a classification result. The classification processing unit 134B outputs to the diameter classification unit 136, for example, the image of the object area A to which the tag corresponding to the classification result is added. The classification processing unit 134B classifies into two types, for example, the elliptical keyhole shape shown in FIGS. 17(a) and 17(b) and the circular keyhole shape shown in FIG. 17(c).

The classification processing unit 134C performs classification processing according to the position model 1343. FIG. The position model 1343 is a model machine-learned using, for example, a captured image with a tag indicating the position of the keyhole as teacher data. The machine learning model in the embodiment is, for example, a convolutional neural network, but is not limited to this as in the pattern/mark model 1341, and may be a perceptron neural network, a recurrent neural network (RNN), or a residual network (ResNet). You may set other neural networks, such as. Also, the shape model 1342 may be partly or wholly a supervised learning estimation model such as a decision tree, regression tree, random forest, gradient boosting tree, linear regression, logistic regression, or SVM (support vector machine). May be set. During learning, when a captured image is input, the position model 1343 outputs an estimation result indicating the position of the keyhole included in the captured image. The classification processing unit 134B updates the processing parameters of the shape model 1341 so that the estimation result matches the tag as teacher data. The processing parameters are, for example, the number of layers, the number of nodes in each layer, the method of connecting nodes between each layer, the activation function, the error function, the gradient descent algorithm, the pooling area, the kernel, the weight coefficient, and the weight in the convolutional neural network. At least one of the matrices. At the time of estimation, the classification processing unit 134C receives the image of the object region A supplied from the classification processing unit 134A and outputs a classification result. The classification processing unit 134C outputs to the diameter classification unit 136, for example, the image of the object area A to which the tag corresponding to the classification result is added. The classification processing unit 134C classifies into two types, for example, the positions of the keyholes shown in FIGS. 17A and 17B and the positions of the keyholes shown in FIG. 17C.

The diameter classification unit 136 performs classification processing according to the diameter of the object indicated by the object region A. FIG. The diameter classifier 136 acquires information indicating the width w of the object region A obtained by the position estimator 120 . The diameter classification unit 136 determines whether or not the width w of the object region A is equal to or larger than a predetermined value, and assigns a tag corresponding to the classification result according to the determination result. The predetermined value is, for example, the diameter set by manhole standards. The diameter classification unit 136 outputs the tagged object region A image as a classification result. Information about the width w of the object region A acquired by the object detection system 1 is provided to another system, and the diameter classification unit 136 performs classification using the width w of the object region A acquired by the other system. may This allows the object detection system 1 to perform classification with high accuracy using the accurate width w of the object region A calculated by another system.

The model constructing unit 138 constructs a pattern/mark model 1341 , a shape model 1342 , and a position model 1343 . The model construction unit 138 inputs teacher data to the pattern/mark model 1341, and processes the pattern/mark model 1341 so that the classification results output from the pattern/mark model 1341 become tags as teacher data. Update parameters recursively. The model construction unit 138 inputs teacher data to the shape model 1342, and recursively updates the processing parameters of the shape model 1342 so that the classification results output from the shape model 1342 become tags as teacher data. do. The model construction unit 138 inputs teacher data to the position model 1343, and recursively updates the processing parameters of the position model 1343 so that the classification result output from the position model 1343 becomes a tag as teacher data. do. Note that the model construction unit 138 does not have to be included in the classification unit 130 as long as each model can be introduced into the classification unit 130 at the time of initial setting or maintenance of the object detection device 100 .

<Effects of Embodiment>
As described above, according to the object detection system 1 of the embodiment, the detection unit 110 that detects the object area A from the captured image and the position estimation unit 120 that estimates the position of the object corresponding to the object area A are provided. The position estimation unit 120 generates a rectangle 200 based on a conversion function from a rectangle 200 having a shape based on the position in the image of the object region A to a trapezoid 300 having a shape based on the position in the image of the object region A. , and estimates the transformed center point C# as the position of the object. The conversion function is, for example, a function that converts a rectangle 200 having sides with a length corresponding to the width w of the object region A into a trapezoid 300 based on the end coordinates of the object region A and the convergence point of the captured image. The object detection device 100 of the embodiment can detect the position of an object with high accuracy. According to the object detection apparatus 100 of the embodiment, for example, it is possible to suppress the displacement of the position that occurs during conversion, as in the case where the position of the object is estimated by converting the object area A included in the captured image into plane coordinates. can.

According to the object detection system 1 of the embodiment, the position estimation unit 120 trapezoidally corrects the object area A, converts the trapezoidally corrected object area A′ into a circular area A″, and converts the trapezoidally corrected object area A ' and the circular area A'', the object corresponding to the object area A can be selected. As a result, according to the object detection system 1, when an object is hidden by an obstacle, the object area A corresponding to the object can be excluded from the target of position estimation. As a result, according to the object detection system 1, the accuracy of position estimation can be improved.

According to the object detection system 1 of the embodiment, the position estimation unit 120 detects an object corresponding to the object area A based on the distance (Rmin to Rmax) from the imaging position to the object based on the position of the object area A in the image. can be selected. Thus, according to the object detection system 1 of the embodiment, it is possible to avoid the process of estimating the position of the object area A corresponding to the object located far from the in-vehicle device 10 . As a result, according to the object detection system 1, the accuracy of position estimation can be improved.

According to the object detection system 1 of the embodiment, the object corresponding to the object area A is detected by the position estimation unit 120 based on the comparison between the width of the object area A detected by the detection unit 110 and the predetermined object area width. can be selected. As a result, according to the object detection system 1, for example, the object region A that is cut off at the edge of the captured image can be excluded from the position estimation processing target. As a result, according to the object detection system 1, the accuracy of position estimation can be improved.

According to the object detection system 1 of the embodiment, the position estimation unit detects the first center point CL detected based on the left side image of the captured image and the second center point CL detected based on the right side image of the captured image. When the distance from the center point CR is within the first predetermined value Da and is equal to or greater than the second predetermined value Db, the object corresponding to the object area A having the first center point CL and the second center point CR can be determined to be the same as the object corresponding to the object region A having . As a result, according to the object detection system 1, it is possible to suppress an error in the identity of the object due to the deviation of the object between the right image and the left image. As a result, the object detection system 1 can provide information in which the position estimated using the left image and the position estimated using the right image are associated with each other for objects determined to be identical. .

According to the object detection system 1 of the embodiment, the detection unit 110 that detects the object area A from the captured image and the classification that executes the first process of classifying the objects included in the object area A based on the first element. Classification unit 130 (object classifier) can be realized. The first element is at least one of the pattern and symbol of the object, and the second element is at least one of the shape, position and size of the hole in the object. According to the object detection system 1 of the embodiment, the classification accuracy can be improved because the objects are classified in a plurality of stages. Further, according to the object detection system 1 of the embodiment, objects are classified in a plurality of stages, so classification can be made more detailed.

According to the object detection system 1 of the embodiment, the classification unit 130 performs machine learning using the first element and the first classification result (tag) as teacher data, and inputs the first element included in the object region A. A first model (1341) that outputs the first classification result in the case, and machine learning is performed using the second element and the second classification result as teacher data, and the second element included in the object region A is input. and a second model (1342 or 1343) that outputs a second classification result in the case. According to the object detection system 1 of the embodiment, since the machine learning model is constructed in a plurality of stages, the classification accuracy of each machine learning model can be improved.

According to the object detection system 1 of the embodiment, the classification unit 130 trapezoidally corrects the object region A detected by the detection unit 110, and performs the first processing and the second processing based on the trapezoidally corrected object region A. is executed, the classification accuracy can be improved.

According to the object detection system 1 of the embodiment, the detection unit 110 detects an object region from a captured image, and the classification processing unit classifies objects included in the object region A based on at least one of patterns and symbols. 134A (first classification unit), a classification processing unit 134B (second classification unit) that classifies the objects classified by the classification processing unit 134A based on the shape of the hole, and the objects classified by the classification processing unit 134B. The classification processing unit 134C (third classification unit) classifies the objects based on the position of the hole, and the objects classified by the classification processing unit 134B are classified based on the size, and the classification processing unit 134C classifies them. and a diameter sorter 136 that sorts the objects that are picked up based on size. According to the object detection system 1 of the embodiment, classification processing can be performed in multiple stages by classifying the object into patterns and symbols of the object, the shape of the hole, the position of the hole, and the size of the object. Accuracy can be increased.

According to the object detection system 1 of the embodiment, the classification processing unit 134A performs machine learning using at least one of the patterns and symbols and the first classification result as teacher data, and the patterns and symbols included in the object region A are determined. Processing is performed based on the pattern/mark model 1341 (first model) that outputs the first classification result when at least one of the patterns is input, and the classification processing unit 134B performs processing based on the hole shape and the second classification result. A shape model 1342 (second model) that outputs a second classification result when a shape of a hole corresponding to an object that has been machine-learned using the classification result as teacher data and has been classified by the classification processing unit 134A is input. machine learning is performed by the classification processing unit 134C using the position of the hole and the result of the third classification as teacher data, and when the position of the hole of the object classified by the classification processing unit 134A is input, the third The object classified by the classification processing unit 134B is classified by the diameter classification unit 136 based on the size of the object. The objects classified by the processing unit 134C are classified. As a result, according to the object detection system 1 of the embodiment, the machine learning model is constructed by dividing the pattern and symbol of the object, the shape of the hole, and the position of the hole. can be raised.

Although each embodiment and modifications have been described, these are only examples and are not limited to these. A section may be combined with one or more other embodiments or one or more other modifications to realize one aspect of the present invention.

Note that a program for executing each process of the object detection apparatus 100 in this embodiment may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system. , the above-described various processes related to the object detection device 100 may be performed.

Note that the “computer system” referred to here may include hardware such as an OS and peripheral devices. The "computer system" also includes the home page providing environment (or display environment) if the WWW system is used. In addition, "computer-readable recording medium" means writable non-volatile memory such as flexible disk, magneto-optical disk, ROM, flash memory, portable medium such as CD-ROM, hard disk built in computer system, etc. storage device.

Furthermore, "computer-readable recording medium" means a volatile memory (e.g., DRAM (Dynamic
Random Access Memory)), which holds a program for a certain period of time. Also, the program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium.

Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

1 object detection system 10 in-vehicle device 100 object detection device 110 detection unit 120 position estimation unit 130 classification unit 132 conversion units 134A, 134B, 134C classification processing unit 1341, pattern/mark model 1342 shape model 1343 position model 136 diameter classification Part 138 Model building part 140 Information providing part

Claims (7)

a detection unit that detects an object area from a captured image;
executing a first process of classifying the objects contained in the object region based on a first element and a second process of classifying the objects classified by the first process based on a second element; and a classifier for
The classification unit is machine-learned using a first classification result based on the first element as teacher data, and based on the first element included in the image of the object area when the image of the object area is input. The first process is executed using a first model that outputs a first classification result, machine learning is performed using a second classification result based on the second element as teacher data, and an image of the object region is obtained. performing the second processing using a second model that, when input, outputs a second classification result based on the second element contained in the image of the object region;
the first element is at least one of a pattern and a symbol of the object;
the second element is at least one of the shape, position, and size of a hole in the object;
Object classifier. The classification unit
trapezoidally correcting the object region detected by the detection unit;
2. The object classification device according to claim 1 , wherein said first process and said second process are performed based on a keystone-corrected object region. a detection unit that detects an object area from a captured image;
a first classification unit that classifies objects included in the object region based on at least one of patterns and symbols;
a second classification unit that classifies the objects classified by the first classification unit based on the shape of the hole;
a third classification unit that classifies the objects classified by the first classification unit based on the positions of the holes;
a fourth classification unit that classifies the objects classified by the second classification unit based on size, and classifies the objects classified by the third classification unit based on size;
The first classification unit is machine-learned using a first classification result based on at least one of the pattern and the symbol as teacher data, and performs at least one of the pattern and the symbol included in the image of the object region. Perform processing based on a first model that outputs a first classification result when inputting
The second classification unit is machine-learned using a second classification result based on the shape of the hole as teacher data. Perform processing based on a second model that outputs the classification result of 2,
The third classification unit is machine-learned using a third classification result based on the position of the hole as teacher data. Perform processing based on a third model that outputs the classification result of 3,
The fourth classification unit classifies the objects classified by the second classification unit and classifies the objects classified by the third classification unit based on the size of the objects.
Object classifier. detecting an object region from the captured image;
executing a first process of classifying the objects contained in the object region based on a first element and a second process of classifying the objects classified by the first process based on a second element; and
Machine learning is performed using a first classification result based on the first element as teacher data, and a first classification result based on the first element included in the image of the object area when the image of the object area is input. machine learning is performed using a first model that outputs a second classification result based on the second element as teacher data, and when an image of the object region is input, the performing the second processing using a second model that outputs a second classification result based on the second element included in the image of the object region;
the first element is at least one of a pattern and a symbol of the object;
the second element is at least one of the shape, position, and size of a hole in the object;
Object classification method. detecting an object region from the captured image;
a first classification step of classifying objects included in the object region based on at least one of patterns and symbols;
a second classification step of classifying the objects classified by the first classification step based on the shape of the hole;
a third classification step of classifying the objects classified by the first classification step based on the positions of the holes;
a fourth classification step of classifying the objects classified by the second classification step based on size, and classifying the objects classified by the third classification step based on size;
including
In the first classification step, machine learning is performed using a first classification result based on at least one of the pattern and the symbol as teacher data, and at least one of the pattern and the symbol included in the image of the object region is determined. perform processing based on a first model that outputs a first classification result when input;
In the second classification step, machine learning is performed using a second classification result based on the shape of the hole as teacher data. performing processing based on a second model that outputs a classification result;
In the third classification step, machine learning is performed using the third classification result based on the position of the hole as teacher data, and when the image of the object region based on the classification result in the first classification step is input, the third classification is performed. perform processing based on a third model that outputs a classification result;
The fourth classification step classifies the objects classified by the second classification step and classifies the objects classified by the third classification step based on the size of the objects.
Object classification method. to the computer,
detecting an object region from the captured image;
executing a first process of classifying the objects contained in the object region based on a first element and a second process of classifying the objects classified by the first process based on a second element; causing a process to be performed including the step of
Machine learning is performed using a first classification result based on the first element as teacher data, and a first classification result based on the first element included in the image of the object area when the image of the object area is input. machine learning is performed using a first model that outputs a second classification result based on the second element as teacher data, and when an image of the object region is input, the executing the second process using a second model that outputs a second classification result based on the second element included in the image of the object region;
the first element is at least one of a pattern and a symbol of the object;
the second element is at least one of the shape, position, and size of a hole in the object;
program. to the computer,
detecting an object region from the captured image;
a first classification step of classifying objects included in the object region based on at least one of patterns and symbols;
a second classification step of classifying the objects classified by the first classification step based on the shape of the hole;
a third classification step of classifying the objects classified by the first classification step based on the positions of the holes;
a fourth classification step of classifying the objects classified by the second classification step based on size, and classifying the objects classified by the third classification step based on size;
A program that executes a process including
In the first classification step, machine learning is performed using a first classification result based on at least one of the pattern and the symbol as teacher data, and at least one of the pattern and the symbol included in the image of the object region is determined. perform processing based on a first model that outputs a first classification result when input;
In the second classification step, machine learning is performed using a second classification result based on the shape of the hole as teacher data. performing processing based on a second model that outputs a classification result;
In the third classification step, machine learning is performed using the third classification result based on the position of the hole as teacher data, and when the image of the object region based on the classification result in the first classification step is input, the third classification is performed. perform processing based on a third model that outputs a classification result;
The fourth classification step classifies the objects classified by the second classification step and classifies the objects classified by the third classification step based on the size of the objects.
program .

Images