JP7271413B2 - Object detection device and object detection method - Google Patents

Description

Application of Article 30, paragraph 2 of the Patent Law The Society of Automatic Control Engineers, Systems and Information Division Academic Lecture 2019 (SSI2019) The Society of Instrument and Control Engineers, Systems and Information Division <Publication> Supplementary Material (Known Fact 1)_20190912_SSI2019_Submission Summary <Publication> Supplementary Material (Known Fact) 1)_20191123_SSI2019_Posted manuscript <Publication> Supplementary material (publicly known fact 1)_20191125_SSI2019_Short presentation manuscript <Publication> Supplementary material (publicly known fact 1)_20191125_SSI2019_Poster manuscript [2] [meeting] ▲1 ▼Date Reiwa 1 September 24, 2018 (2) Name of meeting: “Meeting (NTTFT)” <Publication> Supplementary material (public knowledge 2)_20190924_For NTTFT_FT_Detection of utility poles from horizontal image <Publication> Supplementary material (public knowledge 2) _20190924_For NTTFT_Introduction of initiatives related to utility pole position detection research [3] [Meeting] ▲1 ▼ Date September 27, 2019 ▲2 ▼ Meeting name “Meeting (NTT East Japan-Tohoku)” <Publication> Supplementary Materials (Public Facts 3)_20190927_NTT East Japan - For Tohoku_FT_Detection of utility poles from horizontal images <Publication> Supplementary Materials (Public Facts 3)_20190927_NTT East Japan - For Tohoku_Introduction of initiatives related to research on utility pole position detection [4] [Meeting] ▲1 ▼Date October 25, 2019 ▲2▼Meeting name “Meeting (NTT West)” Introduction [5] [Exhibition announcement] ▲1 ▼ Date November 21-22, 2019 ▲2 ▼ Name of the meeting "NTT COMWARE'SDAY 2019" <Publication> Supplementary material (Public knowledge 5) _20191121_CWsDAY_Explanation slide_Object position detection <Publication> Supplementary material (Public knowledge 5)_20191121_CWsDAY_Exhibition panel [6] [Announcement within NTT Group] ▲1 ▼ Date November 26, 2019 ▲2 ▼ Meeting name Deep Learning Liaison Committee” <Publication> Supplementary Materials (Known Facts 6)_20191126_DL Liaison Committee_SSI2019 Summary, etc.

The present invention relates to an object detection device and an object detection method.

Public infrastructure is the foundation of industry and life. An example of public infrastructure is a structure such as a utility pole. Inspection and maintenance of public infrastructure is important because power outages and communication interruptions occur when utility poles collapse. In order to efficiently inspect and maintain public infrastructure, it is necessary to know the exact position of the structure to be inspected.

In recent years, attention has been focused on techniques for automatically detecting objects in images using machine learning. By using techniques such as deep learning, it is possible to detect roads and buildings from images with high accuracy.

JP 2018-97588 A

A wide range of information can be obtained at once by using an image taken from the sky, such as an aerial photograph or a satellite photograph. However, due to factors such as the size of the target object and the time of photographing, it may not be possible to detect the structure using only an image photographed from above.

The use of images taken by an in-vehicle camera has the advantage of facilitating detailed analysis. Using images captured by a camera installed on the upper floors of a building can provide a wide range of images from a bird's-eye view, but there are structures that cannot be detected because the shooting direction is fixed.

Regardless of which image is used in the vertical direction, horizontal direction, or bird's-eye view direction, it is required to detect an object from the image with higher accuracy.

The present invention has been made in view of the above, and an object of the present invention is to improve the accuracy of object detection and position estimation.

An object detection device according to one aspect of the present invention detects a target object from each of images captured by a first camera that captures an image of the same area from a vertical direction and a second camera mounted on a moving object that moves on the ground, a detection unit that obtains a first position coordinate group from the image of the first camera and obtains a second position coordinate group from the image of the second camera; and the first position coordinate group and the second position. A matching unit that matches coordinate groups and determines the position coordinates of the same object, and obtains a correction amount for correcting each position coordinate of the second position coordinate group to each position coordinate of the first position coordinate group, each position coordinate of the second position coordinate group is corrected to each position coordinate of the first position coordinate group, and position coordinates corresponding to the position coordinates of the second position coordinate group are corrected to the position coordinates of the first position coordinate group; a position correction unit that corrects the position coordinates of the second position coordinate group by the correction amount to determine the position coordinates of the target object when the position coordinates do not exist in the target object.

According to the present invention, it is possible to improve the accuracy of object detection and position estimation.

It is a figure which shows the structural example of the object detection apparatus of this embodiment. It is a figure which shows an example of the image image|photographed from the vertical direction input into an object detection apparatus. It is a figure which shows an example of the image image|photographed from the horizontal direction input into an object detection apparatus. It is a figure which shows an example of the area information input into an object detection apparatus. It is a figure which shows an example of the utility pole and the attachment of a utility pole which an object detection apparatus detects. It is a figure which shows an example which collectively detected the utility pole and the shadow. FIG. 4 is a diagram showing positions of target objects detected from images captured by a plurality of cameras having different elevation/depression angles; It is a figure for demonstrating the determination process of the same object. FIG. 10 is a diagram showing how the position coordinates of a target object are corrected; 4 is a flow chart showing the flow of processing of the object detection device;

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

A configuration example of the object detection device according to the present embodiment will be described with reference to FIG. The object detection apparatus of this embodiment is an apparatus that integrates position coordinates obtained from images of the same area captured by a plurality of cameras with different elevation/depression angles and outputs position information of a detected object. Specifically, an object detection device detects a target object from an aerial photograph or a satellite photograph taken from the sky and obtains the position coordinates, and also detects the target object from an image taken by an in-vehicle camera and obtains the position coordinates. , integrate the position coordinates obtained from each image. At this time, the same object is determined and the position coordinates of the same object are put together. Note that although an example using an aerial photograph and an image captured by an in-vehicle camera will be described below, the present invention is not limited to this. A bird's-eye view image captured from a high floor of a building may be used. It suffices if position coordinates can be obtained from each of the images captured by a plurality of cameras with different elevation/depression angles and integrated.

The object detection device in FIG. 1 includes position detection units 10A and 10B, a matching unit 20, and a position correction unit 30. FIG. Each unit included in the object detection device may be configured by a computer including an arithmetic processing unit, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the object detection device, and can be recorded on a recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be provided through a network.

The position detection units 10A and 10B are provided with input units 11A and 11B for inputting images and area information, and detection units 12A and 12B for detecting target objects from images and obtaining the positions of the target objects.

The input units 11A and 11B input images captured by a plurality of cameras having different elevation/depression angles from the image storage units 51A and 51B, respectively. For example, the input unit 11A receives from the image storage unit 51A an image taken from a vertical direction, such as the aerial photograph or satellite photograph shown in FIG. Input an image taken from the horizontal direction, such as a photo taken with an in-vehicle camera shown. The image may be a still image clipped frame by frame from a moving image. The input units 11A and 11B also input shooting information such as the shooting location, shooting direction, and shooting time of the image together with the image. The input units 11A and 11B may perform preprocessing such as filtering.

Also, the input units 11A and 11B input the area information of the area where the image was captured from the area information accumulation unit 52, and set the search target area or non-search target area of the target object. Regional information is geometric information representing figures by points, lines, and polygons, and qualitative information such as land use and geology, or quantitative attribute information such as elevation.

With reference to FIG. 4, an outline of setting a search target area from area information will be described. In the example of FIG. 4, a road map, a topographic map, and a vegetation distribution map are input to set the search target area. A road map is a map in which roads are indicated by line segments. A topographic map is a digital elevation model (DEM), and is data in which the ground surface is divided into squares and given elevation values. The vegetation distribution map is a diagram showing the distribution of vegetation whether it is a natural forest or not. For example, if the target object is a signal, the signal exists only along the road, so the area near the road is set as the search target area, the area with few ups and downs is set as the search target area, and the area that is not a natural forest is set as the search target area. . In the example of FIG. 4, the search target area is shown in gray in each of the road map, topographic map, and vegetation distribution map. A search target area for searching for a target object is set by superimposing the search target areas specified from the area information. The area information is not limited to this, and for example, attribute information such as rivers and ponds may be input and the rivers and rivers may be set as non-search areas.

The input units 11A and 11B may mask the non-search target area of the input image, or notify the detection unit 12A and 12B of the search target area or non-search target area so that the detection units 12A and 12B can may be processed by excluding the portion corresponding to the non-search target area.

Each of the detection units 12A and 12B automatically detects the target object from the image and obtains the position coordinate group (including one or more position coordinates) of the target object. The detection units 12A and 12B detect the target object from the input image using a judgment model generated by learning the image of the target object by machine learning, and detect the target object detected from the shooting information of the image on the map. Find the position coordinates of The detection unit 12A obtains the position coordinate group of the target object from an image of a wide range photographed from the sky, and the detection unit 12B obtains the position of the target object from the image group of the area corresponding to the image photographed from the sky, which is photographed by an in-vehicle camera. Find the coordinate group. That is, a plurality of position coordinate groups are obtained for each image group (for each camera) input by the input units 11A and 11B.

The detection units 12A and 12B detect not only the target object itself but also related objects. That is, the detection units 12A and 12B learn including objects related to the target object, and detect the target object including objects related to the target object at the time of detection. For example, when the target object is a utility pole, as shown in FIG. 5, the shadow 110 of the utility pole 100 and the overhead wire 120 to be held are also detected. As a result, even if the detection accuracy is low with only utility poles, it can be expected to improve accuracy by collectively detecting related objects.

The detection units 12A and 12B can obtain the position coordinates of the target object with high accuracy by detecting the target object again from the detected area including the related object. For example, in the example shown in FIG. 5, the utility pole 100, which is the target object, is included, but in order to obtain the position coordinates of the utility pole 100 with high accuracy, it is preferable to specify the position of the utility pole 100 itself within the image. The detection units 12A and 12B obtain the position coordinates of the utility pole 100 based on the position of the root portion of the utility pole 100 detected within the area.

As shown in FIGS. 6A and 6B, the detection units 12A and 12B collectively detect the utility pole 100 and the shadow 110, and locate the base portion 200A of the utility pole 100 based on the positions of the utility pole 100 and the shadow 110. Coordinates may be used. Alternatively, the corner 200B of the detection frame where the utility pole 100 and the shadow 110 are collectively detected, the corner 200B on the side where the shadow does not extend may be used as the position coordinates of the utility pole 100 . In FIG. 6A, since the shadow 110 extends to the right side of the utility pole 100, the position coordinates of the utility pole 100 are set to the lower left corner of the detection frame. In FIG. 6B, since the shadow 110 extends to the left of the utility pole 100, the lower right corner of the detection frame is the position coordinate of the utility pole 100. In FIG. This makes it possible to speed up the processing while maintaining sufficient accuracy. Depending on the positional relationship between the utility pole 100 and the shadow 110, the detection units 12A and 12B may use a point on the side of the detection frame as the position coordinates of the utility pole 100, or the center of the detection frame as the position coordinates of the utility pole 100. good.

The detection units 12A and 12B may learn by classifying objects, including objects related to the target object, into classes according to how the related objects are attached. Specifically, classification is performed according to the direction of the shadow extending from the utility pole. The direction in which the shadow extends from the utility pole in the image can be specified by the imaging location, imaging direction, and imaging time of the image. For example, in an image taken facing north at around 10:00 am in spring or autumn, the shadow of the target object extends toward the upper left on the image. Since the shooting location, shooting direction, and shooting time of the image to detect the target object are known, when detecting the target object from the image, include objects related to the target object of the class that meets the image conditions. . A target object classified into a class having a shadow extending in the upper left direction is detected from an image taken facing north at around 10:00 am in spring or autumn.

During learning, the detection units 12A and 12B may generate data obtained by synthesizing objects related to the target object in order to supplement missing learning data. For example, data obtained by synthesizing shadows extending in various directions with an image of a utility pole is generated and used for learning. You may generate the data which added the object attached to the utility pole, such as an overhead wire and a closure.

The detection units 12A and 12B obtain position coordinates for mapping on the map for each target object detected from the image. The position coordinates of the target object detected from the vertical and horizontal images are the camera position coordinates, the camera elevation/depression angle, the position coordinates attached to the image (for example, the position coordinates of the four corners), and the position coordinates of the target object in the image. It can be estimated from the detected coordinates. For example, from horizontal images, position coordinates can be estimated by triangulation from two images of the same target object. Alternatively, position coordinates may be estimated by creating a depth map from a horizontal moving image. Information necessary for estimating the position coordinates is input together with the image when the input units 11A and 11B input the image. Note that the method of estimating the position coordinates is not limited to this. A position coordinate group of the target object is output from each of the detection units 12A and 12B. Since these position coordinate groups include position coordinates indicating the same object, the subsequent matching unit 20 determines the same object.

In addition, although two detection units 12A and 12B are illustrated in FIG. 1, the present invention is not limited to this. A single detection unit may process each of images captured by a plurality of cameras to obtain a group of position coordinates for each image. Additional detectors may be added to process images captured by three or more different cameras (eg, vertical, horizontal, and overhead cameras). In this case, a group of position coordinates is obtained for each group of images with different elevation/depression angles (for example, group of images in the vertical direction, horizontal direction, and bird's-eye view direction).

The matching unit 20 matches the position coordinate group obtained by the detection unit 12A and the position coordinate group obtained by the detection unit 12B, determines the same object, and integrates the position coordinate groups.

FIG. 7 shows the positions of target objects detected by the detection units 12A and 12B. In FIG. 7, the position of the target object detected by the detection unit 12A is indicated by ◯, and the position of the target object detected by the detection unit 12B is indicated by Δ. By detecting a target object from each of images captured by a plurality of cameras with different elevation/depression angles and integrating the detection results, the detection rate of the target object can be improved.

When integrating the position coordinate group, the matching unit 20 determines the position coordinates of the same object output by the detection units 12A and 12B.

For example, as a first method, the matching unit 20 uses the position coordinates of the same object when the deviation of the position coordinates of the target objects detected by the detection units 12A and 12B is less than a certain distance, as shown in the following formula. Determine that there is.

Here, x _i1,j1 and x _i2,j2 are the position coordinates obtained from the images taken by the j1-th and j2-th cameras. d(x _i1,j1 , x _i2,j2 ) is the error between the position coordinates x _i1,j1 and x _i2,j2 . ε is a parameter for determining whether or not the objects are the same.

As a second method, the matching unit 20 uses position coordinates detected by each of the detection units 12A and 12B and parameters set for each camera (or for each detection unit 12A and 12B) as shown in the following formula. Defined areas are calculated, and if there is a common area, it is determined that the coordinates are for the same object.

Here, x _i is an estimation target (a search is made as to whether or not there exists an object that satisfies the conditions), and ε _j1 and ε _j2 are parameters set for each camera.

As shown in FIG. 8, when there is an area 300 having a radius ε _j1 centered at the position coordinates x _i1,j1 and an area ε _{j2 centered at the position coordinates x i2, j2} , a common area 300 exists. It is determined that _i1,j1 and x _i2,j2 are the position coordinates of the same object.

Even if there is a common area 300, if the area 300 is in a place where the target object cannot exist, such as in a river, the matching unit 20 does not determine it as the same object. Specifically, when the area 300 does not include the search target area set by the input units 11A and 11B based on the area information, or when the area 300 is included in the non-search area, the matching unit 20 does not determine that the object is the same object. . This makes it possible to determine the same object with higher accuracy than when determining only the distance between position coordinates.

As a third method, the matching unit 20 calculates the penalty value using the position coordinates detected by the detection units 12A and 12B and the error function set for each camera (may be for each detection unit 12A and 12B), Whether or not the objects are the same is determined based on the calculated penalty value. Specifically, as shown in the following equation, using the error functions c _j1 and c _j2 set for each camera, the position coordinates x _i1,j1 and x _i2,j2 detected by the detection units 12A and 12B are used to estimate the object Penalty values P _j1 and P _j2 corresponding to the distance to x _i are obtained. If there is an estimation target x _i such that the value obtained by dividing the sum of the penalty values P _j1 and P _j2 by 2 (=the number of cameras) is equal to or less than the parameter ε, the position coordinates x _i1,j1 , x _i2,j2 are the position coordinates of the same object.

The error functions c _j1 and c _j2 are, for example, functions whose values increase with distance from the position coordinates x _i1,j1 and x _i2,j2 detected by the detection units 12A and 12B. For example, for the detection units 12A and 12B with small detection errors, an error function is set such that the value increases sharply according to the distance from the detected position coordinates, and for the detection units 12A and 12B with large detection errors, An error function is set such that the value increases gradually according to the distance from the detected position coordinates.

Even when there are three or more detection units, it can be handled by extending the above formula.

Also in the third method, area information may be used for determination of the same object.

The position correction unit 30 combines the position coordinates determined as the same object detected by the detection units 12A and 12B into one, and corrects the position coordinates based on the position coordinates determined as the same object.

For example, the position coordinates detected from the vertical image, for which the position coordinates can be obtained more accurately, are assumed to be positive, and the correction amount of the position coordinates of the same object detected from the horizontal image is obtained. Then, the position coordinates detected only from the image in the horizontal direction are corrected by the obtained correction amount. In FIG. 9, the position coordinates detected from the vertical image are indicated by ◯, and the position coordinates detected from the horizontal image are indicated by Δ. The position correction unit 30 obtains a correction amount for matching the position coordinates (Δ) with the position coordinates (◯). The position correction unit 30 corrects the position coordinates detected only from the image in the horizontal direction with the obtained correction amount.

If there is a target object for which the exact position coordinates are known by surveying, etc., calculate the correction amount to match the position coordinates to the correct position coordinates for each of the vertical and horizontal directions, and The position coordinates obtained in the vertical direction and the position coordinates obtained in the horizontal direction may be corrected. Specifically, the position correction unit 30 obtains a correction amount for adjusting the position coordinates obtained in the vertical direction to the correct position coordinates for the target object whose accurate position coordinates are known, and obtains the correction amount in the vertical direction. The position coordinates of the target object are corrected by the obtained correction amount. Similarly for the horizontal direction, the position correction unit 30 obtains a correction amount for adjusting the position coordinates obtained in the horizontal direction to the correct position coordinates for the target object whose accurate position coordinates are known, and calculates the correction amount in the horizontal direction. Then, the position coordinates of the other target object are corrected by the obtained correction amount.

As another method, the position correction unit 30 may correct the position coordinates by searching for a combination of position coordinates that minimizes the objective function L expressed by the following equation that takes camera errors into consideration.

Here, x _i is the position coordinate of the i-th object to be optimized. x _i,j is the position coordinates of the i-th target object obtained from the j-th camera image. If the i-th target object is not detected from the j-th camera image, it is assumed that x _i =x _i,j . c _j is an error function set for each camera. The same error function as used in the third method of the matching unit 20 may be used. N is the total number of target objects. Position coordinates of the same object are collectively counted as one. K is the total number of cameras. K _i is the total number of cameras that detected x _i .

The operation of the object detection device of this embodiment will be described with reference to the flowchart of FIG.

In step S11, the position detection unit 10A receives an image photographed in the vertical direction and detects the position coordinates of the target object. In step S12, the position detection unit 10B receives the image photographed in the horizontal direction. , to detect the position coordinates of the target object. Steps S11 and S12 may be performed in parallel or sequentially.

In step S13, the matching unit 20 integrates the position coordinates detected from the vertical image and the position coordinates detected from the horizontal image, and determines the position coordinates of the same object.

In step S<b>14 , the position correction unit 30 combines the position coordinates of the same object into one and includes a position correction unit that corrects the position coordinates.

Through the above processing, the object detection device outputs the position coordinate group of the target object. For example, the object detection device indicates and displays the position coordinates of the detected target object on a map.

As described above, the object detection apparatus of this embodiment detects a target object from each of images of the same area captured by a plurality of cameras with different elevation/depression angles, and obtains a plurality of position coordinate groups for each of the plurality of cameras. It comprises detection units 12A and 12B and a matching unit 20 that integrates a plurality of position coordinate groups and determines the position coordinates of the same object. Thereby, the detection rate of the target object can be improved.

The matching unit 20 of this embodiment obtains an area based on the parameters ε _j1 and ε _j2 set for each camera for each of the position coordinates included in the position coordinate group. The position coordinates having the area 300 are determined as the position coordinates of the same object. Alternatively, the matching unit 20 of the present embodiment uses error functions c _j1 and c _j2 set for each camera to calculate the position coordinates x _{i1 , j1 , x i2} , x _{i2 ,} Penalty values P _j1 and P _j2 corresponding to the distance from _j2 are calculated, and among a plurality of position coordinate groups, there is a position coordinate x _i for which the sum of the penalty values P _j1 and P _j2 is equal to or less than the parameter ε. are the position coordinates of the same object. This makes it possible to determine the position coordinates of the same object in consideration of the characteristics of the cameras and the detection units 12A and 12B.

The _object detection apparatus of this embodiment uses penalty values P _j1 , P A position correction unit 30 is provided for obtaining _j2 and correcting the position coordinates so that the sum of the penalty values P _j1 and P _j2 is minimized. As a result, it is possible to improve the accuracy of the position coordinates of the target object.

DESCRIPTION OF SYMBOLS 10A, 10B... Position detection part 11A, 11B... Input part 12A, 12B... Detection part 20... Matching part 30... Position correction part

Claims (6)

A target object is detected from each of images captured by a first camera that captures an image of the same area from a vertical direction and by a second camera mounted on a moving object that moves on the ground, and a first image is captured from the image of the first camera. a detection unit that obtains a group of position coordinates and obtains a second group of position coordinates from the image of the second camera ;
a matching unit that matches the first group of position coordinates and the second group of position coordinates and determines the position coordinates of the same object;
A correction amount for correcting each position coordinate of the second position coordinate group to each position coordinate of the first position coordinate group is obtained, and each position coordinate of the second position coordinate group is corrected to the first position coordinate. When the position coordinates corresponding to the position coordinates of the second position coordinate group do not exist in the first position coordinate group, the position coordinates of the second position coordinate group are corrected to the position coordinates of the second position coordinate group. An object detection device comprising a position correction unit that corrects with a correction amount to obtain position coordinates of a target object . The matching unit obtains a region based on parameters set for each camera for each of the position coordinates included in the position coordinate group , and finds position coordinates having a common region among the plurality of position coordinate groups. 2. The object detection device according to claim 1, wherein position coordinates of the same object are determined. The matching unit obtains a penalty value corresponding to a distance from the position coordinates for each of the position coordinates included in the position coordinate group based on an error function set for each camera , and calculates a plurality of position coordinate groups. 2. The object detection device according to claim 1, wherein the position coordinates at which the sum of the penalty values is equal to or less than a threshold value exists between and is determined as the position coordinates of the same object. 4. The object detection apparatus according to any one of claims 1 to 3, wherein the matching unit uses geometric information and attribute information of an area where the target object is detected to determine the position coordinates of the same object. The position correction unit obtains a penalty value corresponding to a distance from the position coordinates based on an error function set for each camera for each position coordinate determined as the position coordinates of the same object, and determines the penalty value. 5. The object detection device according to any one of claims 1 to 4, wherein the position coordinates of the same object are corrected so that the total is minimized. A computer implemented object detection method comprising:
A target object is detected from each of images captured by a first camera that captures an image of the same area from a vertical direction and by a second camera mounted on a moving object that moves on the ground, and a first image is captured from the image of the first camera. determining a group of position coordinates, and determining a second group of position coordinates from the image of the second camera ;
a step of matching the first group of position coordinates and the second group of position coordinates to determine the position coordinates of the same object;
A correction amount for correcting each position coordinate of the second position coordinate group to each position coordinate of the first position coordinate group is obtained, and each position coordinate of the second position coordinate group is corrected to the first position coordinate. When the position coordinates corresponding to the position coordinates of the second position coordinate group do not exist in the first position coordinate group, the position coordinates of the second position coordinate group are corrected to the position coordinates of the second position coordinate group. An object detection method comprising the step of correcting with a correction amount to obtain position coordinates of a target object .

Images