JP6464706B2 - Object detection method, object detection program, and object detection apparatus - Google Patents

Description

  The present invention relates to an object detection method and the like.

  As a method for detecting an object from a photographed image by attaching a camera that reflects the outside world to a moving body such as a vehicle, there is a related art 1 using an optical flow. FIG. 20 is a diagram for explaining the related art 1. In this prior art 1, feature points are extracted from images at each time included in a photographed image, and the amount of positional deviation of the feature points with time change is calculated. In the following description, the amount of misregistration of feature points is expressed as a flow amount. In the example illustrated in FIG. 20, the related art 1 detects the feature point 10 a from the image 10 at time t and detects the feature point 11 a from the image 11 at time t + 1. Prior art 1 calculates the amount of displacement between the feature point 10a and the feature point 11a as a flow amount.

  Prior art 1 determines whether an object included in images 10 and 11 is a three-dimensional object according to whether or not the flow amount is equal to or greater than a threshold value. For example, the related art 1 determines that the object 1 is a three-dimensional object when the flow amount is equal to or greater than a threshold value. Prior art 1 determines that object 1 is a road surface pattern when the flow amount is less than the threshold value. The threshold value used by the prior art 1 is calculated based on the moving amount of the moving body and the image position.

  Here, in the prior art 1, a three-dimensional object is identified from the magnitude of the flow amount. However, identifying a three-dimensional object and a road surface with only one feature point is affected by noise. For this reason, the prior art 1 extracts a cluster of feature points from an object, and determines that the object is a three-dimensional object if the number of feature points with a flow amount equal to or greater than a threshold value is greater than or equal to a certain number. Reduce the impact.

  However, as a fundamental problem of Prior Art 1, it is not possible to uniquely associate a feature point at a certain time with a feature point at a subsequent time. For example, in the image 12, an object at a certain time is defined as an object 2, and an object at a subsequent time is defined as an object 3. Further, it is assumed that the conventional technique 1 extracts the feature point 12a from the object 2 and extracts the feature points 12b, 12c, and 12d from the object 3. In this case, since it is not possible to specify which of the feature points 12b to 12d corresponds to the feature point 12a, the flow amount cannot be calculated. In general, since the shape of an object is a vertical direction, it is difficult to detect the object at the center of the image where the direction of the shape of the object matches the moving direction of the feature point.

  Conventional technique 2 exists as a conventional technique for solving the problems of conventional technique 1. Prior art 2 performs collation using the end points of the object. FIG. 21 is a diagram (1) for explaining the related art 2. If it is an end point, the feature point at each time can be uniquely associated even if the moving direction of the feature point is the same as the shape of the object. In the example illustrated in FIG. 21, the related art 2 detects the end points 20 a and 20 b from the image 20 at time t, and detects the end points 21 a and 21 b from the image 21 at time t + 1. In addition, since the end points are used, in the related art 2, the end points 20a and 21a can be uniquely associated, and the end points 20b and 21b can be uniquely associated.

  In the prior art 2, when an end point is obtained, a contour line is generated by connecting a plurality of feature points in the vertical direction, and the upper end and the lower end of the generated contour line are detected as end points. FIG. 22 is a diagram (2) for explaining the prior art 2. In the example illustrated in FIG. 22, the related art 2 detects the contour 3 </ b> A by detecting the feature points 30 to 39 of the object 4 and connecting the feature points 30 to 39 adjacent in the vertical direction.

  In addition, since the outline of a horizontal direction is dependent on the width | variety of a solid object, it is difficult to detect. For example, a narrow three-dimensional object such as a pole cannot detect a horizontal outline. In addition, it is difficult to extract a horizontal outline of a three-dimensional object even when a white line of a road surface pattern extends across the three-dimensional object to a distance. For this reason, in the prior art 2, the outline in the vertical direction is detected, and the end point is detected.

JP 2007-334859 A JP 2008-77624 A

  However, the above-described conventional technique has a problem that a three-dimensional object cannot be detected stably.

  FIG. 23 is a diagram for explaining the problems of the prior art. For example, the brightness difference may become small due to changes in brightness, the way the object looks in the background, and the like, and there may be places where feature points are not detected even on the outline of the object. If a feature point is not detected, an accurate end point may not be obtained due to, for example, the occurrence of a break in the contour line or the feature point not being detected at the end point position.

  In the example shown in FIG. 23, the feature points 41 to 47 are detected from the object 4, and the feature points are not detected from some parts. For example, no feature point is detected between the feature points 43 to 44. When the contour line is generated according to the conventional technique 2, the distance between the feature point 43 and the feature point 44 is long, so that no contour line is generated between the feature point 43 and the feature point 44, and the contour lines 4A and 4B. Is generated, the outline is interrupted, and an accurate end point cannot be detected. As described above, if the end points cannot be accurately detected, the end points detected from the images at the respective times cannot be accurately associated, and an accurate flow amount cannot be calculated. It cannot be detected.

  An object of one aspect is to provide an object detection method, an object detection program, and an object detection apparatus that can detect a three-dimensional object stably.

  In the first plan, the computer executes the following processing. The computer extracts edge features from a plurality of images taken at different shooting times by a camera installed on the moving body. The computer determines, based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time, and the determined second edge feature is determined. Based on this, the first edge feature is linearly interpolated. The computer compares a plurality of first edge features subjected to linear interpolation to determine a set of first edge features having symmetry. When the end point position of the first edge feature is not the same end point position, the computer performs second-order interpolation on the other end point position based on the end point position of one of the first edge features. . The computer determines whether or not the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to primary interpolation or secondary interpolation.

  According to one embodiment of the present invention, a three-dimensional object can be detected stably.

FIG. 1 is a diagram for explaining primary interpolation processing performed by the object detection apparatus according to the present embodiment. FIG. 2 is a diagram illustrating an example of contour lines at a plurality of times at which discontinuities cannot be interpolated. FIG. 3 is a diagram for explaining secondary interpolation processing performed by the object detection apparatus according to the present embodiment. FIG. 4 is a functional block diagram illustrating the configuration of the object detection apparatus according to the present embodiment. FIG. 5 is a diagram illustrating an example of a data structure of the contour line information table. FIG. 6 is a flowchart illustrating a processing procedure of the contour line extraction unit. FIG. 7 is a diagram for explaining adjacent pixels on one line of the edge point A. FIG. FIG. 8 is a flowchart showing the processing procedure of the primary interpolation processing of the contour line integration unit. FIG. 9 is a flowchart showing a processing procedure of similarity determination processing. FIG. 10 is a diagram for supplementarily explaining processing in which the contour line integration unit calculates similarity. FIG. 11 is a diagram illustrating a primary interpolation process executed by the contour line integration unit. FIG. 12 is a flowchart illustrating a processing procedure of the secondary interpolation processing of the symmetry determining unit. FIG. 13 is a flowchart illustrating a processing procedure for calculating the degree of symmetry. FIG. 14 is a diagram (1) for supplementarily explaining the process of calculating the degree of symmetry. FIG. 15 is a diagram (2) for supplementarily explaining the process of calculating the degree of symmetry. FIG. 16 is a diagram (3) for supplementarily explaining the process of calculating the degree of symmetry. FIG. 17 is a diagram illustrating a secondary interpolation process executed by the symmetry determining unit. FIG. 18 is a flowchart illustrating the processing procedure of the object detection apparatus according to the present embodiment. FIG. 19 is a diagram illustrating an example of a computer that executes an object detection program. FIG. 20 is a diagram for explaining the related art 1. FIG. 21 is a diagram (1) for explaining the related art 2. FIG. 22 is a diagram (2) for explaining the prior art 2. FIG. 23 is a diagram for explaining the problems of the prior art.

  Hereinafter, embodiments of an object detection method, an object detection program, and an object detection device disclosed in the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  Processing of the object detection apparatus according to the present embodiment will be described. The object detection apparatus performs a primary interpolation process for integrating the contour lines of the same object from the contour lines at a plurality of times, and a secondary interpolation process for interpolating the contour lines using the symmetry of the object.

  The primary interpolation process will be described. FIG. 1 is a diagram for explaining primary interpolation processing performed by the object detection apparatus according to the present embodiment. It is difficult to interpolate a contour line that has been interrupted from the contour line at a certain time, but the brightness and the appearance of the object change depending on the movement of the moving object at another time, so the appearance of the object differs and the extracted contour Lines vary with time. The object detection device can interpolate a portion that has been interrupted at a certain time by integrating contour lines at a plurality of times.

  In the example shown in FIG. 1, the object detection apparatus extracts contour lines 51, 52, and 53 from the object 5 included in the frame at time t-2, and the discontinuity 6a occurs. The object detection device extracts the contour lines 54, 55, and 56 from the object 5 included in the frame at time t-1, and the discontinuity 6b is generated. The object detection device extracts the contour lines 57 and 58 from the object 5 included in the frame at time t, and the discontinuity 6c occurs.

  The object detection device generates the integrated contour 60 by integrating the contours 51 to 58 at a plurality of times. The object detection device compares the integrated contour line 60 with the contour lines 51 to 53 of the frame at time t-2, and generates the contour line 61 by interpolating the discontinuity 6a. The object detection device compares the integrated contour line 60 with the contour lines 54 to 56 of the frame at time t-1 and interpolates the discontinuity portion 6b to generate the contour line 62. The object detection device compares the integrated contour 60 with the contours 57 and 58 of the frame at time t, and generates the contour 63 by interpolating the discontinuity 6c.

  However, even if a plurality of contour lines are integrated, there is a case where the interrupted portion cannot be interpolated. FIG. 2 is a diagram illustrating an example of contour lines at a plurality of times at which discontinuities cannot be interpolated. In the example illustrated in FIG. 2, the object detection device extracts contour lines 71 and 72 from the object 7 included in the frame at time t−2, and the discontinuity 8a occurs. The object detection device extracts the contour lines 73 and 74 from the object 7 included in the frame at time t-1, and a discontinuity 8b occurs. The object detection device extracts the contour lines 75 and 76 from the object 7 included in the frame at time t, and a discontinuity 8c occurs.

  The object detection device generates an integrated contour 80 by integrating the contours 71 to 76 at a plurality of times. However, interrupted portions 80a and 80b also occur in the integrated contour line 80. For this reason, due to the influence of the discontinuous points 80a and 80b, the contour line at each time cannot be interpolated even if the integrated contour line 80 is used. In such a case, the object detection device executes a secondary interpolation process.

  FIG. 3 is a diagram for explaining secondary interpolation processing performed by the object detection apparatus according to the present embodiment. The object detection device determines whether or not the two contour lines are symmetric within the frame at the same time. When the two contour lines are symmetric, the object detection apparatus interpolates the discontinuity portion of each contour line because the shape of each symmetric contour and the position of the end point are the same.

  In the example illustrated in FIG. 3, the object detection device extracts the contour line 85a from the object 8 included in the frame at a certain time, and a discontinuity point 86a is generated. The object detection device extracts the contour line 85b from the object 8, and a discontinuous portion 86b is generated. Here, when it is determined that the contour line 85a and the contour line 85b are symmetric, the object detection device generates the contour line 87a by interpolating the discontinuity portion 86a with the contour line 85b. If the object detection device determines that the contour line 85a and the contour line 85b are symmetric, the object detection device generates the contour line 87b by interpolating the discontinuity portion 86b with the contour line 85a. Note that the object detection apparatus does not execute the secondary interpolation process for a set of contour lines in which each contour line has no symmetry.

  As described above, the object detection device executes the primary interpolation process and the secondary interpolation process, so that the contour line can be correctly extracted even if there is an error in the discontinuity point or the end point position in the contour line. It becomes possible to detect a solid object stably.

  Next, the configuration of the object detection apparatus according to the present embodiment will be described. FIG. 4 is a functional block diagram illustrating the configuration of the object detection apparatus according to the present embodiment. As shown in FIG. 4, the object detection device 100 is connected to a camera 90. The object detection apparatus 100 includes a storage unit 110, a video input unit 120, a feature point extraction unit 130, a contour line extraction unit 140, a contour line integration unit 150, a symmetry determination unit 160, a three-dimensional object determination unit 170, and a result output unit 180. Have.

  The camera 90 is a monocular imaging device installed on a moving body such as a vehicle. The camera 90 outputs video information of the shooting range to the object detection device 100. The video information includes a frame for each time.

  The storage unit 110 stores an outline information table 110a. The storage unit 110 corresponds to a storage device such as a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), and a flash memory.

  The contour line information table 110a is a table that holds information on contour lines detected from a frame. FIG. 5 is a diagram illustrating an example of a data structure of the contour line information table. As shown in FIG. 5, this outline information table associates frame identification numbers, outline numbers, start point coordinates, and end point coordinates. The frame identification number is information that uniquely identifies a frame included in the video information. The contour line number is information for uniquely identifying the contour line. The start point coordinates and end point coordinates are information indicating the start point coordinates and end point coordinates of the contour line. Note that the contour line information table 110a may hold other information related to the contour line in association with each other.

  The video input unit 120 is an interface that receives input of video information from the camera 90. The video input unit 120 outputs the video information to the feature point extraction unit 130. The video input unit 120 performs digital conversion when the video information is an analog video. If the video information is a color image, the video input unit 120 converts it to a monochrome image. The video input unit 120 outputs a digital video of a monochrome image to the feature point extraction unit 130.

  The feature point extraction unit 130 is a processing unit that extracts feature points from each image frame included in the video information. The feature point extraction unit 130 is an example of an extraction unit. For example, the feature point extraction unit 130 performs edge extraction from the image frame and generates a feature point image. The feature point extraction unit 130 outputs the feature point image information to the contour line extraction unit 140. For example, the feature point image is associated with edge point coordinates and edge point intensity information for each edge point.

  The feature point extraction unit 130 operates a general differential operator such as Sobel, compares the edge strength obtained by the differential processing with a predetermined strength threshold, and extracts an edge that is equal to or greater than the threshold. The feature point extraction unit 130 may compare the edge strengths of two adjacent horizontal pixels with respect to the edge extracted from the image frame, and may perform thinning as an edge point if the peak value is reached.

  The contour line extraction unit 140 is a processing unit that extracts a contour line by connecting edge points on feature point coordinates based on a feature point image. FIG. 6 is a flowchart illustrating a processing procedure of the contour line extraction unit. As shown in FIG. 6, the contour line extraction unit 140 selects an unprocessed edge point on the feature point image (step S101). The edge point selected in step S101 is denoted as edge point A for convenience.

  The contour line extraction unit 140 determines whether there is an edge point that can be connected to an adjacent pixel on one line of the edge point A (step S102). The connectable edge point in step S101 is referred to as an edge point B for convenience. FIG. 7 is a diagram for explaining adjacent pixels on one line of the edge point A. FIG. For example, with the edge point A as a reference, adjacent pixels on one line correspond to the region 89a in FIG. In the example shown in FIG. 7, the edge point B exists in the area 89a.

  Returning to the description of FIG. If there is no edge point that can be connected to an adjacent pixel on one line of the edge point A (No in step S102), the contour line extraction unit 140 proceeds to step S110. On the other hand, when there is an edge point B that can be connected to an adjacent pixel on one line of the edge point A (step S102, Yes), the contour line extraction unit 140 determines whether there are a plurality of edge points B. Determination is made (step S103).

  The contour line extraction unit 140 determines whether or not there are a plurality of edge points B (step S103). If there are not a plurality of edge points B (No at Step S103), the contour line extracting unit 140 proceeds to Step S105. On the other hand, when there are a plurality of edge points B (Yes in step S103), the contour line extraction unit 140 selects an optimum edge point B. In step S103, for example, the contour line extraction unit 140 compares the strength and inclination of the plurality of edge points B with the strength and inclination of the edge point A, and the edge point that is most similar to the strength and inclination of the edge point A B is selected as the optimal edge point. For example, the contour line extraction unit 140 determines the slope of the edge point B as the slope of the line segment connecting the edge point A and the edge point B, and the slope of the edge point A is a point on the contour line having the edge point A. And the slope of the line segment connecting the edge points.

  The contour line extraction unit 140 determines whether or not the contour number of the edge point B is unregistered (step S105). If the contour line number of the edge point B is not yet registered (No at Step S105), the contour line extracting unit 140 proceeds to Step S110.

  On the other hand, when the contour line number of the edge point B is not registered (step S105, Yes), it is determined whether or not there is an edge point C that can be connected to the edge point B on the same line as the edge point A. Determination is made (step S106). For example, in the example illustrated in FIG. 7, the edge point C exists on the same line as the edge point A. An adjacent image on one line of the edge point C is an area 89b, and the edge point B exists in the area 89b. In this case, an edge point C that can be connected to the edge point B exists on the same line as the edge point A.

  Returning to the description of FIG. If there is an edge point C that can be connected to the edge point B on the same line as the edge point A (step S106, Yes), the contour line extraction unit 140 proceeds to step S107. On the other hand, when there is no edge point C that can be connected to the edge point B on the same line as the edge point A (No in step S106), the contour line extraction unit 140 proceeds to step S109.

  Step S107 will be described. The contour line extraction unit 140 determines whether or not the edge point C is more optimal as the connection target of the edge point B (step S107). For example, the contour line extraction unit 140 compares the strength and inclination of the edge point A, the strength and inclination of the edge point B, and the strength and inclination of the edge point C. The contour line extraction unit 140 determines that the edge point C is stronger when the strength and inclination of the edge point C are more similar to the strength and inclination of the edge point A. Is determined to be more optimal.

  When the edge point C is more optimal as the connection target of the edge point B (Yes in step S107), the contour line extraction unit 140 updates the information on the edge point B and the edge point C (step S108). ), The process proceeds to step S110. In step S <b> 108, the contour line extraction unit 140 registers the contour line number registered in the edge point C in the edge point B when the contour line number is already registered in the connection source edge point C. . If no contour line number is registered for the edge point C, the contour line extraction unit 140 registers a unique contour line number for the edge point B and the edge point C. On the other hand, when the edge point C is not more optimal as the connection target of the edge point B (No in step S107), the contour line extraction unit 140 proceeds to step S109.

  Step S109 will be described. The contour line extraction unit 140 updates the information of the edge point B and the edge point A (step S109), and proceeds to step S110. In step S <b> 109, the contour line extraction unit 140 registers the contour line number registered in the edge point A in the edge point B when the contour line number is already registered in the connection source edge point A. . The contour line extraction unit 140 registers the unique contour line number in the edge point A and the edge point C when the contour line number is not registered in the edge point A.

  Step S110 will be described. The contour line extraction unit 140 determines whether or not to end the process (step S110). If the contour extraction unit 140 does not terminate the process (No at Step S110), the contour line extraction unit 140 proceeds to Step S101. On the other hand, when the process ends (step S110, Yes), the contour extraction unit 140 ends the process of extracting the contour.

  The contour line extraction unit 140 performs the processing shown in FIG. 6 for each edge point on the feature point image, and extracts a line segment having a final connection number of edge points equal to or greater than a threshold value as a contour line. The contour extraction unit 140 registers the extracted contour information in the contour information table 110a. For example, the contour line extraction unit 140 associates the frame identification number, the contour line number, and the start point coordinates and end point coordinates of the contour line, and registers them in the contour line information table 110a. Note that the contour line extraction unit 140 may associate the identification information of each edge point included in the contour line with the coordinates of each edge point and register them in the contour line information table 110a. In addition, the contour extraction unit 140 outputs information on the contour extracted from the feature point image at the current time to the contour integration unit 150.

  The contour integration unit 150 is a processing unit that executes the primary interpolation processing described with reference to FIG. The contour line integration unit 150 corresponds to a primary interpolation unit. The contour line integration unit 150 compares the current contour line with the contour line registered in the contour line information table 110a, and performs interpolation for integrating the contour lines extracted from the same object. The contour line integration unit 150 does not perform the primary interpolation process because the contour line extracted from the initial frame is registered in the contour line information table 110a as a time-series line segment. In the following description, the information on the contour line registered in the contour line information table 110a is appropriately expressed as a time-series line segment. Further, the contour line extracted from the frame at the current time is appropriately described as a current line segment.

  The contour integration unit 150 extracts the contour lines extracted from the second and subsequent frames from the same object by comparing the current line segment with the time-series line segments registered in the contour line information table 110a. The contours thus formed are integrated and linear interpolation is performed.

  FIG. 8 is a flowchart showing the processing procedure of the primary interpolation processing of the contour line integration unit. As illustrated in FIG. 8, the contour line integration unit 150 determines whether or not the current time frame is two or more frames (step S <b> 201). If the current time frame is not two frames or more (step S201, No), the contour line integration unit 150 proceeds to step S208.

  On the other hand, when there are two or more frames at the current time (step S201, Yes), the contour line integration unit 150 calculates the moving direction of the time series line segment (step S202). In step S202, the contour line integration unit 150 calculates the moving direction of the time-series line segment using a geometric condition called epipolar constraint from the translation amount, rotation amount, and camera installation parameter of the moving body. The contour line integration unit 150 acquires information on the translation amount and the rotation amount of the moving body from the moving body. The camera installation parameters are stored in the storage unit 110.

  The contour line integration unit 150 determines whether or not a current line segment exists in the moving direction of the time-series line segment (step S203). If the current line segment does not exist in the moving direction of the time-series line segment (No at Step S203), the contour line integration unit 150 proceeds to Step S210.

  On the other hand, when there is a current line segment in the moving direction of the time-series line segment (Yes in step S203), the contour line integration unit 150 executes similarity determination processing (step S204). Specific processing of the similarity determination processing in step S204 will be described later.

  The contour integration unit 150 determines whether or not the similarity is equal to or greater than a threshold value (step S205). If the similarity is not greater than or equal to the threshold (No at Step S205), the contour integration unit 150 proceeds to Step S210.

  If the similarity is greater than or equal to the threshold (Yes in step S205), the contour line integration unit 150 generates an integrated contour line that integrates the time-series line segment and the current line segment, and interpolates the current line segment ( Step S206). The contour line integration unit 150 determines whether or not the loop processing of the time series line segment has been completed (step S207). If the loop processing of the time-series line segment has not been completed (No at Step S207), the contour line integration unit 150 proceeds to Step S202.

  On the other hand, when the time series line segment loop processing is completed (Yes in step S207), the contour line integration unit 150 determines whether there is a current line segment that is not integrated (step S208). When there is no current line segment that is not integrated (No in step S208), the contour line integration unit 150 ends the process. On the other hand, when there is a current line segment that is not integrated (Yes in step S208), the contour line integration unit 150 adds the current line segment to the time-series line segment (step S209), and ends the process. In step S209, for example, the contour line integration unit 150 registers information on the current line segment in the contour line information table 110a.

  The process proceeds to step S210. The contour line integration unit 150 updates the number of lost times for the time series line segment (step S210). In step S210, the contour line integration unit 150 adds 1 to the number of lost times for the time-series line segment. The contour line integration unit 150 determines whether or not the lost count of the time series line segment is equal to or greater than a threshold value (step S211).

  The contour line integration unit 150 removes the time series line segment from the contour line information table 110a (step S212) when the number of lost times of the time series line is greater than or equal to the threshold (step S211, Yes). Transition. On the other hand, when the number of lost times of the time series line segment is less than the threshold value (No at Step S211), the process proceeds to Step S207.

  Next, the processing procedure of the similarity determination process shown in step S204 of FIG. 8 will be described. FIG. 9 is a flowchart showing a processing procedure of similarity determination processing. As shown in FIG. 9, the contour line integration unit 150 calculates the overlapping position of the time-series line segment and the current line segment (step S301). The contour line integration unit 150 adjusts the scale of the time series line segment and the current line segment (step S302). In step S <b> 302, the contour line integration unit 150 compares the end point of the time-series line segment with the end point of the current line segment, and performs scale adjustment to match the shorter line segment with the longer line segment.

  The contour line integration unit 150 calculates the centroid position of the time-series line segment and the current line segment on which the scale adjustment has been performed, and matches the centroid position of the time-series line segment and the centroid position of the current line segment (step S303). The contour line integration unit 150 selects the y coordinate and calculates the amount of lateral displacement (step S304). The contour integration unit 150 determines whether or not the y-coordinate processing has been completed (step S305).

  If the y-coordinate processing has not ended (No at Step S305), the contour integration unit 150 proceeds to Step S304. On the other hand, when the y-coordinate processing is completed (Yes in step S305), the contour integration unit 150 calculates the similarity based on the amount of deviation of each y-coordinate (step S306).

  An example of processing in which the contour line integration unit 150 calculates the similarity will be described. FIG. 10 is a diagram for supplementarily explaining processing in which the contour line integration unit calculates similarity. In FIG. 10, a line segment 88a corresponds to a time series line segment, and a line segment 88b corresponds to a current line segment. The contour line integration unit 150 adjusts the scale of the time-series line segment 88a and the current line segment 88b so that the end points match each other, and overlaps them at the center of gravity position 88c. The contour line integration unit 150 calculates the shift amount indicated by the shaded portion 89. For example, the contour line integration unit 150 calculates the similarity between the time-series line segment and the current line segment using a calculation formula in which the similarity increases as the deviation amount decreases.

  An example of primary interpolation processing executed by the contour line integration unit 150 will be described. FIG. 11 is a diagram illustrating a primary interpolation process executed by the contour line integration unit. A line segment 90a shown in FIG. 11 indicates a time-series line segment, and a line segment 90b indicates a current line segment. The similarity between the time-series line segment 90a and the current line segment 90b is set to be equal to or greater than a threshold value. The contour integration unit 150 interpolates the points at the uppermost end and the lowermost end of the points where the movement direction lines of the end points of both line segments 90a, 90b and the extended lines of both line segments 90a, 90b intersect. As the end points, the line segments 90a and 90b are interpolated. For example, the contour line integration unit 150 performs interpolation by extending the end point of the time series line segment 90a to the point 92a, and generates a time series line segment 91a after interpolation. The contour line integration unit 150 performs interpolation by extending the end point of the current line segment 90b to the point 91b, and generates a current line segment 92b after the interpolation.

  Returning to the description of FIG. The symmetry determination unit 160 is a processing unit that executes the secondary interpolation processing described with reference to FIG. The symmetry determination unit 160 corresponds to a secondary interpolation unit. The symmetry determination unit 160 refers to the information on the contour line registered in the contour line information table 110a, and determines a set of contour lines with high symmetry for each contour line extracted from the frame at the same time. The symmetry determination unit 160 performs interpolation by matching each end point position with the longer end point position for a set of contour lines having high symmetry.

  FIG. 12 is a flowchart illustrating a processing procedure of the secondary interpolation processing of the symmetry determining unit. As shown in FIG. 12, the symmetry determination unit 160 calculates the degree of symmetry of a certain contour line A with another contour line B (step S401). Specific processing for calculating the degree of symmetry in step S401 will be described later.

  The symmetry determination unit 160 determines whether or not there is a contour line B whose degree of symmetry with respect to the contour line A is greater than or equal to a threshold value (step S402). The symmetry determination unit 160 proceeds to step S404 when there is no contour line B having a degree of symmetry equal to or greater than the threshold (No in step S402). On the other hand, when there is a contour line B whose degree of symmetry is equal to or greater than the threshold (Yes in step S402), the symmetry determining unit 160 records the contour line number of the contour line A on the contour line B side (step S403). ). In step S <b> 403, for example, the symmetry determining unit 160 records the contour line number of the contour line A in the extended region associated with the contour line B.

  The symmetry determination unit 160 determines whether or not the contour loop processing has ended (step S404). When the contour loop processing has not been completed (No at Step S404), the symmetry determining unit 160 proceeds to Step S401.

  On the other hand, when the contour loop processing is completed (Yes in step S404), the symmetry determining unit 160 refers to the contour number recorded in the extension region of each contour (step S405). The symmetry determination unit 160 determines whether or not there is a contour line number (step S406). If the contour line number does not exist (No at Step S406), the symmetry determining unit 160 proceeds to Step S411.

  When the contour line number exists (step S406, Yes), the symmetry determining unit 160 determines whether there are a plurality of contour line numbers (step S407). When there is no plurality of contour line numbers (No at Step S407), the symmetry determining unit 160 proceeds to Step S411.

  On the other hand, when there are a plurality of contour line numbers (step S407, Yes), the symmetry determination unit 160 determines whether the contour line A and the contour line B can be connected to each other (step S408). ). For step S408, for example, the symmetry determining unit 160 can connect the contour line A and the contour line B when the difference between the tilt of the contour line A and the tilt of the contour line B is less than a predetermined tilt. It is determined that Further, the symmetry determining unit 160 determines that the difference between the average value of the edge strengths of the edge points included in the contour line A and the average value of the edge strengths of the edge points included in the contour line B is less than a predetermined threshold value. In this case, the contour line A and the contour line B may be determined to be connectable.

  If the symmetry determining unit 160 is not connectable to each other (No at Step S408), the symmetry determining unit 160 proceeds to Step S411. On the other hand, the symmetry determination unit 160 integrates the contour line A and the contour line B (step S409) when they can be connected to each other (step S408, Yes). The symmetry determination unit 160 performs interpolation by aligning the end point positions of all the contour lines that are symmetric (step S410).

  The symmetry determination unit 160 determines whether or not the contour loop processing has been completed (step S411). If the contour loop processing has not ended (No at Step S411), the symmetry determining unit 160 proceeds to Step S405. On the other hand, the symmetry determination unit 160 ends the processing when the contour loop processing has ended (step S411, Yes).

  Next, the process procedure of the process in which the symmetry determination unit 160 shown in step S401 in FIG. 12 calculates the degree of symmetry will be described. FIG. 13 is a flowchart illustrating a processing procedure for calculating the degree of symmetry. As illustrated in FIG. 13, the symmetry determination unit 160 calculates the overlapping position and the overlapping rate for the two contour lines of the contour line A and the contour line B (step S501).

  FIG. 14 is a diagram (1) for supplementarily explaining the process of calculating the degree of symmetry. In the example illustrated in FIG. 14, the contour line 200 a is the contour line A, and the contour line 200 b is the contour line B. In this case, the overlapping position is a position corresponding to 201. Further, the overlapping rate indicates a ratio of the overlapping position 201 in the length of the contour line 200b, based on the contour line 200b.

  Returning to the description of FIG. The symmetry determining unit 160 determines whether or not there is an overlapping position on the two contour lines (step S502). When there is no overlapping position on the two contour lines (No in step S502), the symmetry determining unit 160 ends the process of calculating the degree of symmetry.

  On the other hand, when the overlapping position exists in the two contour lines (Yes in step S502), the symmetry determining unit 160 calculates the similarity between the two contour lines (step S503). In step S503, the symmetry determination unit 160 calculates the similarity between the two contour lines, for example, in the same manner as the process illustrated in FIG.

  The symmetry determination unit 160 selects the y coordinate of the overlapping portion (step S504), and calculates the horizontal midpoint (step S505). The symmetry determination unit 160 calculates the brightness difference integral value for the laterally symmetrical position with the midpoint as a reference (step S506).

  FIG. 15 is a diagram (2) for supplementarily explaining the process of calculating the degree of symmetry. In the example illustrated in FIG. 15, the contour line 200 a is the contour line A, and the contour line 200 b is the contour line B. The overlapping part is 201. Further, a line corresponding to the selected y coordinate is set to 205, and a midpoint is set to 210. The symmetry determination unit 160 calculates the difference between the brightness of each pixel from the midpoint 210 to the contour line 200a and the brightness of each pixel from the midpoint 210 to the contour line 200b, and the calculated difference. Are integrated to calculate the brightness difference integral value. The symmetry determination unit 160 calculates a brightness difference integral value for each line, and sums each brightness difference integral value as a final brightness difference integral value.

  Returning to the description of FIG. The symmetry determination unit 160 determines whether or not the horizontal processing has been completed (step S507). For example, in step S507, the symmetry determination unit 160 determines that the processing in the horizontal direction has ended when all the y coordinates included in the overlapping portion have been selected. The symmetry determination unit 160 proceeds to step S506 when the horizontal processing has not been completed (No in step S507).

  On the other hand, when the process in the horizontal direction is completed (Yes in step S507), the symmetry determining unit 160 calculates the average value of the brightness of the y coordinate (step S508).

  FIG. 16 is a diagram (3) for supplementarily explaining the process of calculating the degree of symmetry. In the example shown in FIG. 16, the contour line 200 a is the contour line A, and the contour line 200 b is the contour line B. The overlapping part is 201. In addition, y1 to y5 exist as the y coordinate in the overlapping portion 201, and the lines corresponding to y1 to y5 are defined as lines 211 to 215, respectively. The symmetry determination unit 160 calculates the first average value of the brightness of the pixels included in the line 211. Similarly, the symmetry determination unit 160 calculates the first average value of brightness for the lines 212 to 215. In addition, the symmetry determining unit 160 calculates the second average value of the entire line by adding the first average values of the lines 211 to 215 and dividing by the number of lines. Let this 2nd average value be the average value of the brightness of ay coordinate.

  Returning to the description of FIG. The symmetry determination unit 160 determines whether or not the processing of all the y-coordinates at the overlapping portion has been completed (step S509). The symmetry determination unit 160 proceeds to step S504 when the processing of all the y-coordinates of the overlapping portion has not been completed (No in step S509).

  On the other hand, the symmetry determination unit 160 calculates the degree of symmetry when the processing of all the y-coordinates of the overlapping portion is completed (Yes in step S509) (step S510). For step S510, for example, the symmetry determination unit 160 calculates the degree of symmetry based on Expression (1). For example, in Equation (1), the brightness continuity deviation value corresponds to the value obtained by summing the difference values between the first average value and the second average value of each line described in FIG.

  Degree of symmetry = overlap rate-2 similarity between contour lines-brightness difference integral value-brightness continuity deviation amount (1)

  An example of the secondary interpolation process performed by the symmetry determination unit 160 will be described. FIG. 17 is a diagram illustrating a secondary interpolation process executed by the symmetry determining unit. In the example illustrated in FIG. 17, it is assumed that the contour line 220a and the contour line 230 have symmetry, and the contour line 220b and the contour line 230 have symmetry. The symmetry determination unit 160 generates a contour 240 by supplementing the discontinuity between the contour 220a and the contour 220b with the contour 230. The symmetry determination unit 160 deletes the information on the contour line 220a and the contour line 220b registered in the contour line information table 110a, and instead registers the information on the contour line 240 in the contour line information table 110a.

  Note that, after the symmetry determination unit 160 performs the secondary interpolation processing of the contour line, the contour integration unit 150 executes the primary interpolation processing again based on the contour line interpolated by the secondary interpolation processing. May be.

  Returning to the description of FIG. The three-dimensional object determination unit 170 is a processing unit that determines whether a three-dimensional object exists based on the contour line information table 110a. The three-dimensional object determination unit 170 outputs information on the determination result to the result output unit 180. For example, the three-dimensional object determination unit 170 calculates the flow amount by comparing the end point position of the current line segment with the end point position of the time series line segment. The three-dimensional object determination unit 170 determines that a three-dimensional object exists at the extraction source of the current line segment when the flow amount is equal to or greater than the threshold value. On the other hand, the three-dimensional object determination unit 170 determines that there is no three-dimensional object when the flow amount is less than the threshold value.

  The result output unit 180 is a processing unit that outputs information on a contour line determined to be a three-dimensional object out of contour lines based on the determination result from the three-dimensional object determination unit 170 to a display device or the like.

  Next, a processing procedure of the object detection apparatus according to the present embodiment will be described. FIG. 18 is a flowchart illustrating the processing procedure of the object detection apparatus according to the present embodiment. As shown in FIG. 18, the video input unit 120 of the object detection apparatus 100 accepts video input (step S601).

  The feature point extraction unit 130 of the object detection apparatus 100 performs edge extraction from the image frame to generate a feature point image (step S602). The contour extraction unit 140 of the object detection device 100 executes contour extraction based on the feature point image (step S603). The processing procedure for contour extraction in step S603 corresponds to the processing procedure in FIG.

  The contour line integration unit 150 of the object detection apparatus 100 executes primary interpolation processing (step S604). The processing procedure of the primary interpolation processing in step S604 corresponds to the processing procedure of FIG.

  The symmetry determination unit 160 of the object detection apparatus 100 determines symmetry and executes a secondary interpolation process (step S605). The processing procedure of the secondary interpolation processing in step S605 corresponds to the processing procedure of FIG.

  The three-dimensional object determination unit 170 of the object detection apparatus 100 executes the three-dimensional object determination based on the contour line information table 110a (step S606). The result output unit 180 of the object detection apparatus 100 outputs the determination result (step S607).

  Next, the effect of the object detection apparatus 100 according to the present embodiment will be described. After the object detection apparatus 100 executes a primary interpolation process for integrating the contour lines of the same subject photographed at a plurality of times and a secondary interpolation process for interpolating the discontinuity of the contour line based on a set of symmetrical contour lines. The solid object determination is executed based on the contour line after the interpolation. For this reason, according to the object detection apparatus 100, it is possible to stably detect a three-dimensional object even when the outline is interrupted or the feature point is not detected at the end point position.

  The object detection device 100 calculates the midpoint of the contour line, and based on the difference value between the brightness of the image from the midpoint to one contour line and the brightness of the image from the midpoint to the other contour line, It is determined whether the set of lines has symmetry. For this reason, it can be accurately determined whether or not the set of contour lines has symmetry.

  The object detection device 100 determines whether or not each contour line has symmetry based on the continuity of brightness in the vertical direction of the image between the contour lines. For this reason, it can be accurately determined whether or not the set of contour lines has symmetry.

  Next, an example of a computer that executes an object detection program that realizes the same function as that of the object detection apparatus 100 described in the above embodiment will be described. FIG. 19 is a diagram illustrating an example of a computer that executes an object detection program.

  As illustrated in FIG. 19, the computer 300 includes a CPU 301 that executes various arithmetic processes, an input device 302 that receives input of data from a user, and a display 303. The computer 300 includes a reading device 304 that reads a program and the like from a storage medium, an interface device 305 that exchanges data with other computers via a network, and a camera 306. The computer 300 also includes a RAM 307 that temporarily stores various types of information and a hard disk device 308. The devices 301 to 308 are connected to the bus 309.

  The hard disk device 308 has an extraction program 308a, a primary interpolation program 308b, a secondary interpolation program 308c, and a determination program 308d. The CPU 301 reads the extraction program 308 a, the primary interpolation program 308 b, the secondary interpolation program 308 c, and the determination program 308 d of the hard disk device 308 and develops them in the RAM 307.

  The extraction program 308a functions as an extraction process 307a. The primary interpolation program 308b functions as a primary interpolation process 307b. The secondary interpolation program 308c functions as a secondary interpolation process 307c. The determination program 308d functions as a determination process 307d.

  For example, the extraction process 307 a corresponds to the feature point extraction unit 130. The primary interpolation process 307 b corresponds to the contour line integration unit 150. The secondary interpolation process 307 c corresponds to the symmetry determination unit 160. The determination process 307d corresponds to the three-dimensional object determination unit 170.

  The programs 308a to 307d are not necessarily stored in the hard disk device 308 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, and an IC card inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 308a to 307d.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1) An object detection method executed by a computer,
Extract edge features from multiple images taken by different cameras at different shooting times.
Based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time is determined, and based on the determined second edge feature. Linearly interpolating the first edge feature,
A plurality of first edge features subjected to the primary interpolation are compared to determine a set of first edge features having symmetry;
For the determined set of first edge features, when the end point position of the first edge feature is not the same end point position, the other end point position is secondarily interpolated based on the end point position of one first edge feature;
An object that executes a process of determining whether or not the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to the primary interpolation or the secondary interpolation. Detection method.

(Supplementary Note 2) The process of determining the set of first edge features having the symmetry calculates the midpoint of each first edge feature, the brightness of the image from the midpoint to one first edge feature, The supplementary note 1, wherein it is determined whether or not each of the first edge features is symmetric based on a difference value with respect to an image brightness from the midpoint to the other first edge feature. Object detection method.

(Additional remark 3) The process which determines the 1st edge feature set which has the said symmetry is based on the continuity of the brightness of the vertical direction of this image about the image between each 1st edge feature, said each said 1st The object detection method according to appendix 1 or 2, wherein it is determined whether or not one edge feature has symmetry.

(Appendix 4)
Extract edge features from multiple images taken by different cameras at different shooting times.
Based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time is determined, and based on the determined second edge feature. Linearly interpolating the first edge feature,
A plurality of first edge features subjected to the primary interpolation are compared to determine a set of first edge features having symmetry;
For the determined set of first edge features, when the end point position of the first edge feature is not the same end point position, the other end point position is secondarily interpolated based on the end point position of one first edge feature;
An object for executing a process of determining whether or not the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to the primary interpolation or the secondary interpolation. Detection program.

(Supplementary Note 5) The process of determining the first edge feature group having the symmetry calculates the midpoint of each first edge feature, and the brightness of the image from the midpoint to one first edge feature, The supplementary note 4 is characterized in that it is determined whether or not each of the first edge features is symmetric based on a difference value with respect to an image brightness from the midpoint to the other first edge feature. Object detection program.

(Additional remark 6) The process which determines the 1st edge feature set which has the said symmetry is based on the continuity of the brightness of the vertical direction of this image about the image between each 1st edge feature, said each said 1st 6. The object detection program according to appendix 4 or 5, wherein it is determined whether or not one edge feature has symmetry.

(Supplementary note 7) An extraction unit that extracts edge features from a plurality of images taken by a camera installed on a moving body at different shooting times;
Based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time is determined, and based on the determined second edge feature. A primary interpolation unit for linearly interpolating the first edge feature;
A plurality of first-interpolated first edge features are compared to determine a set of first edge features having symmetry, and for the determined first edge feature set, end points having the same end point position of the first edge feature A second-order interpolation unit that performs second-order interpolation on the other end point position based on the end point position of one of the first edge features when it is not a position;
A determination unit that determines whether the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to the primary interpolation or the quadratic interpolation. Detection device.

(Supplementary Note 8) The second interpolation unit calculates the midpoint of each first edge feature, the brightness of the image from the midpoint to one first edge feature, and the other first edge feature from the midpoint The object detection apparatus according to appendix 7, wherein the first edge feature is determined based on a difference value with respect to the brightness of the image that reaches the first point.

(Supplementary Note 9) For the image between the first edge features, the second interpolation unit determines whether the first edge features have symmetry based on the continuity of brightness in the vertical direction of the images. 9. The object detection device according to appendix 7 or 8, wherein the object detection device determines whether or not.

DESCRIPTION OF SYMBOLS 100 Object detection apparatus 110 Memory | storage part 120 Image | video input part 130 Feature point extraction part 140 Contour line extraction part 150 Contour line integration part 160 Symmetry determination part 170 Three-dimensional object determination part 180 Result output part

Claims (4)

An object detection method executed by a computer,
Extract edge features from multiple images taken by different cameras at different shooting times.
Based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time is determined, and based on the determined second edge feature. Linearly interpolating the first edge feature,
The plurality of first edge features subjected to the primary interpolation are compared , the midpoint of each first edge feature is calculated, the brightness of the image from the midpoint to one first edge feature, and the midpoint to the other A set of first edge features having symmetry is determined by determining whether or not each of the first edge features has symmetry based on a difference value from the brightness of the image reaching the first edge feature. And
For the determined set of first edge features, when the end point position of the first edge feature is not the same end point position, the other end point position is secondarily interpolated based on the end point position of one first edge feature;
An object that executes a process of determining whether or not the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to the primary interpolation or the secondary interpolation. Detection method. The process of determining the set of first edge features having the symmetry is performed on the images between the first edge features based on the continuity of brightness in the vertical direction of the images. The object detection method according to claim 1 , wherein it is determined whether or not there is symmetry. On the computer,
Extract edge features from multiple images taken by different cameras at different shooting times.
Based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time is determined, and based on the determined second edge feature. Linearly interpolating the first edge feature,
The plurality of first edge features subjected to the primary interpolation are compared , the midpoint of each first edge feature is calculated, the brightness of the image from the midpoint to one first edge feature, and the midpoint to the other A set of first edge features having symmetry is determined by determining whether or not each of the first edge features has symmetry based on a difference value from the brightness of the image reaching the first edge feature. And
For the determined set of first edge features, when the end point position of the first edge feature is not the same end point position, the other end point position is secondarily interpolated based on the end point position of one first edge feature;
An object for executing a process of determining whether or not the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to the primary interpolation or the secondary interpolation. Detection program. An extraction unit that extracts edge features from a plurality of images with different shooting times taken by a camera installed on a moving body;
Based on the extracted edge feature, the second edge feature photographed at the second photographing time corresponding to the first edge feature photographed at the first photographing time is determined, and based on the determined second edge feature. A primary interpolation unit for linearly interpolating the first edge feature;
The plurality of first-interpolated first edge features are compared , the midpoint of each first edge feature is calculated, the brightness of the image from the midpoint to one first edge feature, and the midpoint to the other By determining whether or not each of the first edge features is symmetric based on a difference value from the brightness of the image reaching the first edge feature, a set of symmetric first edge features is determined. In the determined first edge feature set, when the end point position of the first edge feature is not the same end point position, a secondary that interpolates the other end point position based on the end point position of one of the first edge features An interpolation unit;
A determination unit that determines whether the first edge feature is an edge feature of a three-dimensional object based on the set of first edge features subjected to the primary interpolation or the quadratic interpolation. Detection device.