JP2021105820A - Object detection device and object detection method - Google Patents

Abstract

To improve accuracy in object detection and position estimation.SOLUTION: An object detection device comprises: detection units 12A and 12B for detecting a target object from respective images obtained by photographing the same area using a plurality of cameras with different elevation/depression angles and calculating a plurality of position coordinate groups for each of the plurality of cameras; and a matching unit 20 for integrating the plurality of position coordinate groups and determining a position coordinate of the same object. The detection units 12A and 12B detect a region where the target object including an object related to the target object is present, and detect the target object and a portion including a shadow of the target object from inside the region where the target object is present.SELECTED DRAWING: Figure 1

Description

The present invention relates to an object detection device and an object detection method.

Public infrastructure is the foundation of industry and livelihood. An example of public infrastructure is a structure such as a utility pole. Inspection and maintenance of public infrastructure is important because if a utility pole collapses, a power outage will occur or communication will be cut off. In order to efficiently inspect and maintain public infrastructure, it is necessary to know the exact location of the structure to be inspected.

In recent years, attention has been focused on technology for automatically detecting an object in an image using machine learning. By using techniques such as deep learning, it is possible to detect roads, buildings, etc. from images with high accuracy.

JP-A-2018-97588

A wide range of information can be obtained at once by using images taken from the sky such as aerial photographs and satellite photographs. However, due to factors such as the size of the target object and the shooting time, it may not be possible to detect the structure using only the image taken from the sky.

Using images taken with an in-vehicle camera has the advantage that detailed analysis can be performed easily, but the cost for shooting is high, and the target area tends to be narrower than that of aerial / satellite photographs. Images taken with a camera installed on the upper floors of a building can be used to obtain a wide range of images from the bird's-eye view, but the shooting direction is fixed and some structures cannot be detected.

Regardless of whether an image in the vertical direction, a horizontal direction, or a bird's-eye view direction is used, it is required to detect an object from the image with higher accuracy.

The present invention has been made in view of the above, and an object of the present invention is to improve the accuracy of object detection and position estimation.

The object detection device according to one aspect of the present invention includes a detection unit that detects a target object from each of images taken by a plurality of cameras having different elevation / depression angles in the same area and obtains a plurality of position coordinate groups for each of the plurality of cameras. A matching unit that integrates the plurality of position coordinate groups and determines the position coordinates of the same object is provided, and the detection unit detects a region in which the target object exists, including an object related to the target object. It is characterized in that the target object and the portion including the shadow of the target object are detected in the region where the target object exists.

According to the present invention, it is possible to improve the accuracy of object detection and position estimation.

It is a figure which shows the structural example of the object detection apparatus of this embodiment. It is a figure which shows an example of the image taken from the vertical direction input to the object detection device. It is a figure which shows an example of the image taken from the horizontal direction input to the object detection device. It is a figure which shows an example of the area information to input to an object detection apparatus. It is a figure which shows an example of the utility pole and the accessory of the utility pole detected by the object detection device. It is a figure which shows an example which detected a utility pole and a shadow together. It is a figure which showed the position of the target object detected from each of the images taken by a plurality of cameras with different elevation / depression angles. It is a figure for demonstrating the determination process of the same object. It is a figure which shows the state of correcting the position coordinate of a target object. It is a flowchart which shows the processing flow of the object detection apparatus.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

A configuration example of the object detection device of the present embodiment will be described with reference to FIG. The object detection device of the present embodiment is a device that integrates the position coordinates obtained from each of the images taken by a plurality of cameras having different elevation / depression angles in the same area and outputs the position information of the detected object. Specifically, the object detection device detects the target object from an aerial photograph or a satellite photograph taken from the sky and obtains the position coordinates, and also detects the target object from the image taken by the in-vehicle camera and obtains the position coordinates. , Integrate the position coordinates obtained from each image. At this time, the same object is determined, and the position coordinates of the same object are put together. In the following, an example using an aerial photograph and an image taken by an in-vehicle camera will be described, but the description is not limited to this. An image in the bird's-eye view taken from the upper floors of the building may be used. It suffices if the position coordinates can be obtained from each of the images taken by a plurality of cameras having different elevation / depression angles and integrated.

The object detection device of FIG. 1 includes position detection units 10A and 10B, a matching unit 20, and a position correction unit 30. Each part of the object detection device may be configured by a computer provided with an arithmetic processing unit, a storage device, and the like, and the processing of each part may be executed by a program. This program is stored in a storage device included in the object detection device, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network.

The position detection units 10A and 10B include input units 11A and 11B for inputting an image and area information, and detection units 12A and 12B for detecting the target object from the image and obtaining the position of the target object.

Each of the input units 11A and 11B inputs images taken by a plurality of cameras having different elevation / depression angles from the image storage units 51A and 51B, respectively. For example, the input unit 11A inputs an image taken from the vertical direction such as the aerial photograph or the satellite photograph shown in FIG. 2 from the image storage unit 51A, and the input unit 11B inputs the image taken from the image storage unit 51B to FIG. Input an image taken from the horizontal direction, such as a picture taken with the in-vehicle camera shown. As the image, a still image cut out from the moving image for each frame may be used. The input units 11A and 11B also input shooting information such as the shooting location, shooting direction, and shooting time of the image together with the image. The input units 11A and 11B may perform preprocessing such as filter processing.

Further, the input units 11A and 11B input the area information of the area where the image was taken from the area information storage unit 52, and set the search target area or the non-search target area of the target object. Regional information is geometric information that represents a figure by points, lines, and polygons, and quantitative attribute information such as qualitative or elevation such as land use and geology.

An outline of setting the search target area from the area information will be described with reference to FIG. In the example of FIG. 4, a road map, a topographic map, and a vegetation distribution map were input to set the search target area. A road map is a map showing roads with line segments. The topographic map is a digital elevation model (DEM), and is data obtained by dividing the ground surface into squares and giving them elevation values. The vegetation distribution map is a diagram showing the vegetation distribution of whether or not it is a natural forest. For example, if the target object is a signal, the signal exists only along the road, so the area near the road is the search target area, the area with few undulations is the search target area, and the area that is not a natural forest is the search target area. .. In the example of FIG. 4, the search target area is shown in gray in each of the road map, topographic map, and vegetation distribution map. The search target area for searching the target object is set by superimposing the search target areas specified from these area information. The area information is not limited to this, and for example, attribute information such as a river or a pond may be input, and the river or the river may be excluded from the search target area.

The input units 11A and 11B may mask the non-search target area of the input image, notify the search target area or the non-search target area to the detection units 12A and 12B, and the detection units 12A and 12B are in the image. You may process by excluding the part corresponding to the area not to be searched by.

Each of the detection units 12A and 12B automatically detects the target object from the image and obtains the position coordinate group (including one or more position coordinates) of the target object. The detection units 12A and 12B detect the target object from the input image by using the determination model generated by learning the image of the target object by machine learning, and on the map of the target object detected from the captured information of the image. Find the position coordinates of. The detection unit 12A obtains the position coordinate group of the target object from the image taken over a wide area from the sky, and the detection unit 12B obtains the position of the target object from the image group taken by the in-vehicle camera in the area corresponding to the image taken from the sky. Find the coordinate group. That is, a plurality of position coordinate groups for each image group (for each camera) input by the input units 11A and 11B are obtained.

The detection units 12A and 12B detect not only the target object itself but also related objects. That is, the detection units 12A and 12B learn including the object related to the target object, and at the time of detection, detect the target object including the object related to the target object. For example, when the target object is a utility pole, as shown in FIG. 5, the shadow 110 of the utility pole 100 and the overhead wire 120 to be held are also detected. As a result, even if the detection accuracy is low only with the utility pole, the accuracy can be expected to be improved by collectively detecting the related objects.

The detection units 12A and 12B can accurately obtain the position coordinates of the target object by detecting the target object again from the detected area including the related object. For example, in the example shown in FIG. 5, the utility pole 100, which is the target object, is included, but in order to accurately obtain the position coordinates of the utility pole 100, it is preferable that the position of the utility pole 100 itself in the image can be specified. The detection units 12A and 12B obtain the position coordinates of the utility pole 100 based on the position of the root portion of the utility pole 100 detected in the region.

As shown in FIGS. 6A and 6B, the detection units 12A and 12B collectively detect the utility pole 100 and the shadow 110, and position the root portion 200A of the utility pole 100 based on the positions of the utility pole 100 and the shadow 110. It may be a coordinate. Alternatively, the corner 200B on the side where the shadow does not extend may be the position coordinate of the utility pole 100 in the detection frame in which the utility pole 100 and the shadow 110 are detected together. In FIG. 6A, since the shadow 110 extends to the right side of the utility pole 100, the lower left corner of the detection frame is used as the position coordinates of the utility pole 100. In FIG. 6B, since the shadow 110 extends to the left side of the utility pole 100, the lower right corner of the detection frame is used as the position coordinate of the utility pole 100. As a result, the processing can be speeded up while maintaining sufficient accuracy. The detection units 12A and 12B may use the points on the sides of the detection frame as the position coordinates of the utility pole 100 or the center in the detection frame as the position coordinates of the utility pole 100, depending on the positional relationship between the utility pole 100 and the shadow 110. good.

The detection units 12A and 12B may be divided into classes according to how the related objects are attached, including the objects related to the target object. Specifically, the classification is performed according to the direction of the shadow extending from the utility pole. The direction in which the shadow extends from the utility pole in the image can be specified by the shooting location, shooting direction, and shooting time of the image. For example, in an image taken facing north around 10 am in spring or autumn, the shadow of the target object extends in the upper left direction on the image. Detecting the target object Since the shooting location, shooting direction, and shooting time of the image are known, when detecting the target object from the image, the target object and related objects of the class that matches the image conditions are detected. .. From the images taken facing north around 10 am in spring or autumn, the target object classified into the class with a shadow extending in the upper left direction is detected.

The detection units 12A and 12B may generate data obtained by synthesizing an object related to the target object in order to supplement the learning data that is lacking at the time of learning. For example, data obtained by synthesizing shadows extending in various directions on an image of a telephone pole is generated and used for learning. Data may be generated by adding an object attached to a utility pole such as an overhead wire or a closure.

The detection units 12A and 12B obtain the position coordinates for mapping on the map for each of the target objects detected from the image. The position coordinates of the target object detected from the vertical and horizontal images are the position coordinates of the camera, the elevation / depression angle of the camera, the position coordinates associated with the image (for example, the position coordinates of the four corners), and the position coordinates of the target object in the image. It can be estimated from the detected coordinates. For example, from a horizontal image, the position coordinates can be estimated by performing triangulation from two images for the same target object. Alternatively, a depth map may be created from a moving image in the horizontal direction to estimate the position coordinates. The information necessary for estimating the position coordinates is input together with the image when the input units 11A and 11B input the image. The method of estimating the position coordinates is not limited to this. The position coordinate group of the target object is output from each of the detection units 12A and 12B. Since these position coordinate groups include position coordinates indicating the same object, the matching unit 20 in the subsequent stage determines the same object.

Although FIG. 1 shows two detection units 12A and 12B, the present invention is not limited to this. One detection unit may process each of the images taken by a plurality of cameras to obtain a position coordinate group for each image. Further, a detection unit may be added to process images taken by three or more different cameras (for example, vertical, horizontal, and bird's-eye view cameras). In this case, a position coordinate group is obtained for each image group having a different elevation / depression angle (for example, an image group in the vertical direction, a horizontal direction, and a bird's-eye view direction).

The matching unit 20 matches the position coordinate group obtained by the detection unit 12A with the position coordinate group obtained by the detection unit 12B to determine the same object, and integrates the position coordinate groups.

FIG. 7 shows the positions of the target objects detected by the detection units 12A and 12B, respectively. In FIG. 7, the position of the target object detected by the detection unit 12A is indicated by ◯, and the position of the target object detected by the detection unit 12B is indicated by Δ. The detection rate of the target object can be improved by detecting the target object from each of the images taken by a plurality of cameras having different elevation / depression angles and integrating the detection results.

When integrating the position coordinate groups, the matching unit 20 determines the position coordinates of the same object output by each of the detection units 12A and 12B.

For example, as a first method, as shown by the following equation, the matching unit 20 uses the position coordinates of the same object when the deviation of the position coordinates of the detected target objects of the detection units 12A and 12B is a certain distance or less. Judge that there is.

Here, x _{i1, j1} , x _{i2, j2} are the position coordinates obtained from the images taken by the j1st and j2nd cameras. d (x _{i1, j1} , x _{i2, j2} ) is the error between the position coordinates x _{i1, j1} , x _{i2, j2.} ε is a parameter for determining whether or not they are the same object.

As a second method, as shown by the following equation, the matching unit 20 is based on the detected position coordinates of the detection units 12A and 12B and the parameters set for each camera (may be each of the detection units 12A and 12B). The defined area is calculated, and if there is a common area, it is determined that the coordinates are the position coordinates of the same object.

Here, x _i is an estimation target (searching for whether or not there is a condition that satisfies the condition), and ε _j1 and ε _j2 are parameters set for each camera.

As shown in FIG. 8, when there is a common region 300 in the region of radius ε _{j1 centered on the position coordinates x i1 and j1 and the region of ε j2} centered on the position coordinates x _{i2 and j2, the position coordinates x It is determined that i1, j1} , x _{i2, j2} are the position coordinates of the same object.

Even if the common area 300 exists, if the area 300 is a place where the target object cannot exist, such as in a river, the matching unit 20 does not determine that the object is the same. Specifically, the matching unit 20 does not determine the same object when the area 300 does not include the search target area set by the input units 11A and 11B from the area information, or when the area 300 is included in the non-search target area. .. As a result, the same object can be determined more accurately than when the determination is made based only on the distance between the position coordinates.

As a third method, the matching unit 20 calculates a penalty value using the detected position coordinates of each of the detection units 12A and 12B and an error function set for each camera (may be for each of the detection units 12A and 12B). It is determined whether or not the objects are the same based on the calculated penalty value. Specifically, as shown in the following equation, the estimation target is estimated from _{the position coordinates x i1, j1} , x _{i2, j2} detected by the detection units 12A and 12B using _{the error functions c j1} and c _{j2 set for each camera. Find} the penalty values P j1 and P _j2 according to the distance to x _i . _{If there is an estimation target x i} such that the sum of the penalty values P _j1 and P _j2 is divided by 2 (= the number of cameras) and the value is less than or equal to the parameter ε, the position coordinates x _{i1, j1} , x _{i2, j2} Is determined to be the position coordinates of the same object.

The error functions c _j1 and c _j2 are, for example, functions in which the values increase as the _{distance from the position coordinates x i1, j1} , x _{i2, j2} detected by the detection units 12A and 12B increases. For example, for the detection units 12A and 12B having a small detection error, an error function is set so that the value suddenly increases according to the distance from the detected position coordinates, and for the detection units 12A and 12B having a large detection error, an error function is set. Set an error function so that the value gradually increases according to the distance from the detected position coordinates.

Even if there are three or more detection units, it can be dealt with by extending the above equation.

In the third method as well, the area information may be used to determine the same object.

The position correction unit 30 collects the position coordinates determined to be the same object detected by the detection units 12A and 12B into one, and corrects the position coordinates based on the position coordinates determined to be the same object.

For example, the position coordinates detected from the vertical image in which the position coordinates can be obtained more accurately are regarded as positive, and the correction amount of the position coordinates of the same object detected from the horizontal image is obtained. Then, the position coordinates that can be detected only from the image in the horizontal direction are corrected by the obtained correction amount. In FIG. 9, the position coordinates detected from the vertical image are indicated by ◯, and the position coordinates detected from the horizontal image are indicated by Δ. The position correction unit 30 obtains a correction amount for adjusting the position coordinates (Δ) to the position coordinates (〇). The position correction unit 30 corrects the position coordinates detected only from the image in the horizontal direction with the obtained correction amount.

If there is an object whose accurate position coordinates are known by surveying, etc., the correction amount to match the position coordinates with the accurate position coordinates is obtained in each of the vertical and horizontal directions, and based on each of the obtained correction amounts. The position coordinates obtained from the vertical direction and the position coordinates obtained from the horizontal direction may be corrected. Specifically, the position correction unit 30 obtains the correction amount for adjusting the position coordinates obtained from the vertical direction to the accurate position coordinates for the target object whose exact position coordinates are known, and obtains the correction amount from the vertical direction. The position coordinates of the target object are corrected by the obtained correction amount. Similarly, in the horizontal direction, the position correction unit 30 obtains the correction amount for adjusting the position coordinates obtained from the horizontal direction to the accurate position coordinates for the target object whose exact position coordinates are known, and obtains the correction amount from the horizontal direction. The position coordinates of other target objects are corrected by the obtained correction amount.

Alternatively, the position correction unit 30 may correct the position coordinates by searching for a combination of position coordinates that minimizes the objective function L represented by the following equation in consideration of the camera error.

Here, x _i is the optimization target and is the position coordinate of the i-th target object. x _{i, j} are the position coordinates of the i-th target object obtained from the image of the j-th camera. If the i-th target object is not detected from the image of the j-th camera, it is regarded as _{x i} = x _{i, j.} c _j is an error function set for each camera. It may be the same as the error function used in the third method of the matching unit 20. N is the total number of target objects. The position coordinates of the same object are collectively counted as one. K is the total number of cameras. K _i is the total number of cameras that have detected _{x i.}

The operation of the object detection device of the present embodiment will be described with reference to the flowchart of FIG.

In step S11, the position detection unit 10A inputs an image taken from the vertical direction and detects the position coordinates of the target object, and in step S12, the position detection unit 10B inputs an image taken from the horizontal direction. , Detects the position coordinates of the target object. Steps S11 and S12 may be performed in parallel or in order.

In step S13, the matching unit 20 integrates the position coordinates detected from the vertical image and the position coordinates detected from the horizontal image, and determines the position coordinates of the same object.

In step S14, the position correction unit 30 includes a position correction unit that collects the position coordinates of the same object into one and corrects the position coordinates.

By the above processing, the object detection device outputs the position coordinate group of the target object. For example, the object detection device displays the position coordinates at which the target object is detected on the map.

As described above, the object detection device of the present embodiment detects the target object from each of the images taken by a plurality of cameras having different elevation / depression angles in the same area, and obtains a plurality of position coordinate groups for each of the plurality of cameras. The detection units 12A and 12B and a matching unit 20 that integrates a plurality of position coordinate groups and determines the position coordinates of the same object are provided. As a result, the detection rate of the target object can be improved.

The detection units 12A and 12B of the present embodiment detect the area where the target object exists including the object related to the target object, and then detect the target object and the part including the shadow thereof from the area where the target object exists. do. Further, the detection units 12A and 12B obtain the position coordinates of the target object based on the position of the shadow of the target object detected from the image. As a result, the position coordinates of the target object can be obtained more accurately.

The object detection device of the present embodiment realizes improvement in detection accuracy of the target object and speeding up of processing by limiting the area for detecting the target object based on the geometric information and attribute information of the area where the target object is detected. can.

10A, 10B ... Position detection unit 11A, 11B ... Input unit 12A, 12B ... Detection unit 20 ... Matching unit 30 ... Position correction unit

Claims (5)

A detection unit that detects a target object from each of images taken by a plurality of cameras with different elevation / depression angles in the same area and obtains a plurality of position coordinate groups for each of the plurality of cameras.
A matching unit that integrates the plurality of position coordinate groups and determines the position coordinates of the same object is provided.
The detection unit detects the region where the target object exists, including the object related to the target object, and then detects the target object and the portion including the shadow of the target object from the region where the target object exists. An object detection device characterized by detecting. The detection unit learns so that the target object can be classified according to the direction of the shadow, and when detecting the target object from the image, inputs information that can specify the direction of the shadow in the image and detects the target object. The object detection device according to claim 1, wherein the object detection device is characterized by the above. The object detection device according to claim 1 or 2, wherein the detection unit obtains the position coordinates of the target object based on the position of the shadow of the target object detected from the image. The object detection device according to any one of claims 1 to 3, wherein the area for detecting the target object is limited based on the geometric information and the attribute information of the area for detecting the target object. A computer-executed object detection method
A step of detecting a target object from each of images taken by a plurality of cameras having different elevation / depression angles in the same area and obtaining a plurality of position coordinate groups for each of the plurality of cameras.
It has a step of integrating the plurality of position coordinate groups and determining the position coordinates of the same object.
In the step of detecting the target object, after detecting the region where the target object exists including the target object and the accessory, the target object and the shadow of the target object are cast from within the region where the target object exists. An object detection method characterized by detecting a portion including a part.

Images