JP6620607B2 - Image analysis program, image analysis apparatus, and image analysis method - Google Patents

Description

  The present invention relates to an image analysis program, an image analysis apparatus, and an image analysis method.

  An object detection method based on an optical flow at an end point of a contour line is known as a method of detecting an object from an image captured by a camera by installing a camera that captures the outside world on a moving body such as a vehicle. In this detection method, contour lines are extracted from a plurality of time-series images included in a video, end points of the contour lines are associated between the images, and a positional deviation amount of the end points is calculated. This displacement amount is called a flow amount.

  FIG. 1 shows an example of association of objects between images. FIG. 1A shows a case where points other than the end points on the contour line of the object are associated. An object 111 is shown in the image 101 at time 1, and the object 111 moves in the direction indicated by the arrow 113 as the moving body moves. Then, the object 112 is shown in the image 102 at time 2 after time 1.

  Assuming that the object 111 in the image 101 and the object 112 in the image 102 are the same object, the point 121 on the contour line of the object 111 moves to the position of the point 122 on the contour line of the object 112. However, since the point 121 is not an end point of the contour line, the correspondence between the point 121 and the point on the contour line of the object 112 is not uniquely determined.

  FIG. 1B shows a case where the end points of the contour line of the object are associated with each other. The end points 131 and 132 of the contour line of the object 111 in the image 101 move to the positions of the end points 133 and 134 of the contour line of the object 112 in the image 102. In this case, the correspondence between the end point 131 and the end point 132 and the end point 133 and the end point 134 is uniquely determined without being affected by the moving direction of the moving body and the shape of the contour line.

  FIG. 2 shows an example of a pattern and a three-dimensional object on the road surface taken by the in-vehicle camera. The three-dimensional object 203 is stationary on the road surface 201, and the vehicle 205 on which the camera 206 is mounted is moving on the road surface 201 in the direction indicated by the arrow 211. In the image taken by the camera 206 at time 1, the point of interest 204 on the three-dimensional object 203 and the point 207 included in the pattern 202 on the road surface 201 overlap each other.

  At this time, the difference between the direction 212 connecting the position of the camera 206 and the point of interest 204 at time 1 and the direction 213 connecting the position of the camera 206 and the point of interest 204 at time 2 appears as the parallax of the three-dimensional object 203. On the other hand, the difference between the direction 212 connecting the position of the camera 206 and the point 207 at time 1 and the direction 214 connecting the position of the camera 206 and the point 207 at time 2 appears as the parallax of the pattern 202. In this case, it can be seen that the parallax of the three-dimensional object 203 is larger than the parallax of the pattern 202.

  Due to the difference in parallax between the three-dimensional object 203 and the pattern 202, the flow amount of points on the three-dimensional object 203 and the flow amount of points included in the pattern 202 between the image at time 1 and the image at time 2 are as follows. Are compared, the flow amount of points on the three-dimensional object 203 becomes larger. Using this property, if the flow amount of a point in the image is greater than or equal to the threshold value, the point is determined to be a point on the three-dimensional object 203. If the flow amount is less than the threshold value, the point becomes a pattern 202. It can be determined that the point is included. The threshold for the flow amount can be obtained from the position of the point in the image and the movement amount of the vehicle 205.

  In the calculation of the flow amount of the three-dimensional object, a contour line whose angle with respect to the vertical direction of the image is within a predetermined value is used. The reason is that in the image, the angle of the contour line in the height direction of the three-dimensional object is often close to the vertical, the contour line close to the vertical direction is dominant, and the correspondence between the contour lines between the images is unique. It is because it is easy to settle.

  There is also known a technique for determining a pedestrian who has a high risk of jumping into a course based on a contour line in an image photographed from a vehicle (see, for example, Patent Document 1). A technique for calculating a three-dimensional position of a surrounding object from an image taken from a vehicle and a moving amount of the vehicle is also known (see, for example, Patent Document 2).

JP 2012-203484 A JP 2014-240753 A

  In a conventional object detection method based on an optical flow, there is a case where a solid object region is erroneously detected in an image in which a contour line included in a pattern on a road surface and a contour line of a solid object are connected.

  Such a problem is not limited to the image in which the contour line included in the pattern on the road surface and the contour line of the three-dimensional object are connected, but also occurs in other images where the specific region on the surface and the three-dimensional object region are in contact with each other. It is.

  In one aspect, an object of the present invention is to detect a three-dimensional object region with high accuracy from an image in which a specific region on a surface and a three-dimensional object region are in contact with each other.

In one plan, the image processing program causes the computer to execute the following processing.
(1) The computer extracts the image feature of the first region from the first image captured at the first time by the imaging device installed on the moving body moving on the surface, and the first time is defined by the imaging device. Image features of the second region are extracted from second images taken at different second times.
(2) The computer obtains a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface based on the image feature of the first area.
(3) The computer obtains a second position that is a candidate for a boundary in the second region corresponding to the first position, based on the moving amount of the moving body between the first time and the second time.
(4) The computer uses an image feature of the first area when the first position is used as a boundary in the first area, and an image feature of the second area when the second position is used as a boundary in the second area. The first position is determined as a boundary in the first region based on the feature similarity.
(5) The computer specifies the region of the three-dimensional object in the first region based on the determined boundary.

  According to one embodiment, the region of the three-dimensional object can be detected with high accuracy from the image in which the specific region on the surface is in contact with the region of the three-dimensional object.

It is a figure which shows matching of the object between images. It is a figure which shows the pattern and three-dimensional object on a road surface. It is a figure which shows the image which the outline of two objects connected. It is a figure which shows the calculation method of the evaluation value with respect to a division | segmentation point. It is a figure which shows the example of failure by an individual process. It is a figure which shows the example of failure by local processing. It is a block diagram of an image analysis apparatus. It is a flowchart of an image analysis process. It is a figure which shows an image analysis process. It is a block diagram which shows the specific example of an image analyzer. It is a flowchart which shows the specific example of an image analysis process. It is a flowchart of an outline extraction process. 3 is a diagram illustrating three pixels adjacent to an edge point A. FIG. It is a figure which shows the edge point C which can be connected with the edge point B. FIG. It is a figure which shows outline information. It is a flowchart of an outline collation process. It is a flowchart of a difference degree calculation process. It is a figure which shows an overlap part. It is a figure which shows the superimposition of two overlapping parts. It is a flowchart of a candidate list generation process. It is a figure which shows a candidate list production | generation process. It is a flowchart of a division | segmentation score calculation process. It is a flowchart of a feature similarity calculation process. It is a flowchart of a dividing point determination process. It is a figure which shows a dividing point determination process. It is a flowchart of a solid object detection process. It is a block diagram of information processing apparatus.

Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 3 shows an example of an image in which the contour lines of two objects are connected. When the pattern 311 on the road surface and the three-dimensional object 312 are shown in the image 301 and the contour line of the pattern 311 and the contour line of the three-dimensional object 312 are connected and one contour line 321 is extracted, Since the upper end point 322 is a part of the pattern 311, it is located on the road surface. Also, the lower end point 323 of the contour line 321 corresponds to the lower end of the three-dimensional object 312, and thus is located on the road surface. Therefore, the contour line 321 is determined to be the contour line of the pattern on the road surface, and detection of the three-dimensional object 312 fails.

  In order to detect the three-dimensional object 312 from such an image 301, it is conceivable to divide the outline 321 into two. When two regions of an object touch each other in an image, the pixel value and the shape often change near the boundary between the regions. Thus, for each point on the contour line, an evaluation value based on a change in pixel value and shape is calculated, and a threshold value of the calculated evaluation value is determined, whereby a division point indicating a boundary between objects can be determined.

  FIG. 4 shows an example of an evaluation value calculation method for the division points. When the outline 401 is divided into an upper partial outline 402 and a lower partial outline 403 by a point 401 that is a candidate for a division point on the outline to be divided, as an evaluation value for the point 401, for example, A division score S of the following formula can be used.

S = S1 + S2 (1)
S1 = | I (A1) −I (A2) | (2)
S2 = | θ | (3)

  I (A1) represents the pixel value of the local region A1 on the partial contour 402, and I (A2) represents the pixel value of the local region A2 on the partial contour 403. θ represents an angle between the partial outline 402 and the partial outline 403.

  For each point on the contour line 321 in FIG. 3, by calculating the division score S of Expression (1), the contour line 321 is the contour line 325 and the contour line 326 at the division point 324 where the division score S is maximum. Can be divided into Based on the contour line 326, the three-dimensional object 312 is detected. Thus, when the contour line satisfies a predetermined division condition, the three-dimensional object detection failure is suppressed by dividing the contour line into two contour lines.

  However, even if the contour line is divided into two contour lines based on the division score S, the division point may deviate from the correct boundary and the detection of the three-dimensional object may fail. The cause is considered to be individual processing for each image and local processing in the image.

  FIG. 5 shows an example of failure due to individual processing. In a region 501 in the image at time 1, a dividing point 502 on the contour line is determined, and in a region 503 in the image at time 2, a dividing point 504 on the contour line is determined. In this way, when the dividing point is determined independently for each image at each time included in the video, the dividing point is freely determined without imposing a constraint between two times. For this reason, there is no consistency between the dividing point 502 at time 1 and the dividing point 504 at time 2.

  FIG. 6 shows an example of failure due to local processing. Since S1 in Expression (2) is calculated based on a change in the pixel value between the local regions, a point having a large local change is likely to be a division point. In the image of FIG. 6, although the point 602 in the region 601 is a correct boundary, the point 603 is determined as a division point by being pulled by a local change in pixel value.

  For these reasons, if the positions of the two division points at two times are shifted and the combination is inconsistent, a flow amount different from the correct flow amount is calculated, and detection of the three-dimensional object fails.

  FIG. 7 shows a configuration example of the image analysis apparatus of the embodiment. The image analysis device 701 includes an extraction unit 711, a determination unit 712, and a specification unit 713. The extraction unit 711 extracts an image feature of a region from an image captured by an imaging device installed on a moving body that moves on the surface. The determination unit 712 determines the boundary between the specific region on the surface and the three-dimensional object existing on the surface based on the image feature extracted by the extraction unit 711, and the specification unit 713 determines the boundary determined by the determination unit 712. Based on this, the region of the three-dimensional object in the image is specified.

  FIG. 8 is a flowchart illustrating an example of image analysis processing performed by the image analysis apparatus 701 in FIG. First, the extraction unit 711 extracts the image feature of the first area from the first image photographed at the first time, and extracts the second area from the second image photographed at the second time different from the first time. Are extracted (step 801).

  Next, the determination unit 712 obtains a first position that is a candidate for a boundary between the specific region and the three-dimensional object included in the first region based on the image feature of the first region (step 802). Then, the determination unit 712 obtains a second position that is a candidate for a boundary in the second region corresponding to the first position, based on the moving amount of the moving body between the first time and the second time (step) 803).

  Next, the determination unit 712 uses the first area as the boundary in the first area and the second area using the second position as the boundary in the second area. Based on the feature similarity with the image feature, the first position is determined as the boundary in the first region (step 804).

  Then, the specifying unit 713 specifies the region of the three-dimensional object in the first region based on the boundary determined by the determining unit 712 (Step 805).

  According to the image analysis apparatus 701 in FIG. 7, the three-dimensional object region can be detected with high accuracy from the image in which the specific region on the surface is in contact with the three-dimensional object region.

  FIG. 9 shows an example of image analysis processing performed by the image analysis apparatus 701. In this image analysis process, an edge line segment is used as an image feature of each region extracted from an image, and a constraint between two times and a global evaluation for a division point are adopted.

  The edge line segment extracted from the image corresponds to the edge point connection portion generated by connecting the edge points detected from the image, and is a finite-length curve portion or straight line portion having a start point and an end point. Since the edge line segment often represents the contour line of the object, the edge line segment is hereinafter sometimes referred to as a contour line.

  By adopting constraints between two times, candidates for physically possible division points are obtained from a three-dimensional object model. For example, when the contour line in the image at time 1 is composed of N (N is an integer of 1 or more) points, these points are used as the division point candidates P1 to P1. In this case, by applying a virtual three-dimensional object model to the contour line, the division point candidate Qi on the contour line at time 2 corresponding to the division point candidate Pi (i = 1 to N) on the contour line at time 1. The position of is calculated. As a result, it is possible to obtain a combination 901-i of division point candidates that is consistent between two times.

  Further, by adopting global evaluation for the dividing points, N combinations 901-1 to 901-of the dividing point candidates are not affected by local pixel value changes based on the evaluation values of the entire contour line. An optimum combination is required from N. In this case, the evaluation value for the combination 901-i of the dividing point candidate Pi and the dividing point candidate Qi is calculated using the characteristics of the entire contour line at time 1 and the characteristics of the entire contour line at time 2.

  Specifically, the characteristics of the entire partial contour line 902 on the upper side of Pi, the characteristics of the entire partial contour line 903 on the lower side of Pi, the characteristics of the entire partial contour line 904 on the upper side of Pi, and the lower side of Qi An evaluation value for the combination 901-i is calculated using the characteristics of the entire partial contour 905. This evaluation value is calculated based on, for example, the degree of similarity between partial contour lines between two times and the degree of difference between partial contour lines at the same time.

  FIG. 10 shows a specific example of the image analysis apparatus 701 in FIG. 10 includes an extraction unit 711, a determination unit 712, a specification unit 713, a video input unit 1011, an output unit 1012, and a storage unit 1013. The extraction unit 711 includes a feature point extraction unit 1021 and a contour line extraction unit 1022, and the determination unit 712 includes a matching unit 1031, a generation unit 1032, a dissimilarity calculation unit 1033, a similarity calculation unit 1034, and a division point determination unit 1035. including.

  For example, in a safety support system that detects a stationary object such as a pedestrian, a surrounding vehicle that may come into contact immediately before the vehicle departs from a parking lot, and notifies the driver of the presence of the object, An image analysis device 701 can be used. In this case, the imaging device 1001 may be a single monocular camera mounted on a vehicle. It is also possible to install the imaging devices 1001 at a plurality of locations such as the front and back of the vehicle.

  The image analysis device 701 can be implemented as a hardware circuit, for example. In this case, each component of the image analysis device 701 may be mounted as an individual circuit or may be mounted as one integrated circuit.

  FIG. 11 is a flowchart illustrating a specific example of image analysis processing performed by the image analysis apparatus 701 in FIG. In the image analysis processing of FIG. 11, the imaging device 1001 is mounted on a vehicle moving on a road surface, and an outline representing the contour of each region is provided as an image feature of each region extracted from an image captured by the imaging device 1001. Used.

  The specific area is, for example, a pattern area on the road surface, and the three-dimensional object on the road surface is, for example, another vehicle or a pedestrian. As the boundary between the specific region and the three-dimensional object, the boundary between the contour line of the specific region and the contour line of the three-dimensional object region is used.

  First, the video input unit 1011 acquires the image 1041 at the current time from the video captured by the imaging device 1001 mounted on the vehicle, and stores the acquired image 1041 in the storage unit 1013 (step 1101).

  When the video output from the imaging apparatus 1001 is an analog video, the video input unit 1011 converts an analog image at each time into a digital image, and stores the converted digital image as an image 1041. When the video output from the imaging apparatus 1001 is a color video, the video input unit 1011 may convert a color image at each time into a monochrome image and store the converted monochrome image as an image 1041. When a plurality of imaging devices 1001 are installed, the image 1041 is acquired from the video captured by each imaging device 1001.

  Next, the feature point extraction unit 1021 detects an edge from the image 1041, generates a feature point image 1042 from the detected edge, and stores it in the storage unit 1013 (step 1102). At this time, the feature point extraction unit 1021 applies, for example, a differential operator such as a Sobel filter to the image 1041, and obtains the edge strength of each pixel by differential processing. Next, the feature point extraction unit 1021 compares the obtained edge strength with a threshold value, and detects a region having an edge strength equal to or higher than the threshold value as an edge.

  Then, the feature point extraction unit 1021 performs thinning processing on the detected edges to determine edge points, and generates a feature point image 1042 including the determined edge points. In the thinning process, for example, the edge strength of each pixel is compared with the edge strength of two pixels adjacent in the horizontal direction, and if the central edge strength is a peak value, that pixel is determined as an edge point.

  The contour line extraction unit 1022 extracts a contour line whose angle with respect to the vertical direction of the feature point image 1042 is within a predetermined value from the feature point image 1042, and stores the contour line information 1043 representing the extracted contour line in the storage unit 1013. (Step 1103). The contour line information 1043 includes information on contour lines in the plurality of feature point images 1042 at each of a plurality of times from the start time of the video to the current time.

  Next, the collation unit 1031 collates the contour line at the current time with the past contour line included in the contour line information 1043 using the constraint between two times, and the contours of the same object with the two contour lines It is determined whether or not the line is a line (step 1104). Then, the collation unit 1031 sets, in the contour line information 1043, information that associates two contour lines that are determined to be the contour line of the same object.

  Next, the generation unit 1032 determines the boundary on the current contour from the position of the candidate for the dividing point that is a candidate for the boundary on the past contour among the two contours associated in the contour information 1043. The position of the candidate for the dividing point that is a candidate for is predicted (step 1105). Then, the generation unit 1032 generates a candidate list 1044 including a plurality of combinations of past division point candidates and current division point candidates, and stores the generated candidate list 1044 in the storage unit 1013.

  Next, for each combination included in the candidate list 1044, the dissimilarity calculation unit 1033 calculates the dissimilarity between the partial contour lines on both sides of the division point candidate on the past contour line and the both sides of the division point candidate on the current contour line. The difference between the partial contour lines is calculated (step 1106). Then, the dissimilarity calculation unit 1033 calculates a division score 1045 from the two dissimilarities, and stores the calculated division score 1045 in the storage unit 1013.

  Next, the similarity calculation unit 1034, for each combination included in the candidate list 1044, the partial contour lines on both sides of the division point candidate on the past contour line and the partial contour lines on both sides of the division point candidate on the current contour line. The feature similarity 1046 is calculated (step 1107). Then, the similarity calculation unit 1034 stores the calculated feature similarity 1046 in the storage unit 1013.

  Next, the division point determination unit 1035 performs global evaluation on the division point based on the division score 1045 and the feature similarity 1046, and obtains an optimal combination from the combinations included in the candidate list 1044 (step 1108). Then, the dividing point determination unit 1035 determines the position of the past dividing point candidate represented by the combination as the dividing point on the past outline, and determines the position of the current dividing point candidate as the dividing point on the current outline. .

  Next, the specifying unit 713 divides the past and current contour lines at the position of the division point determined by the division point determination unit 1035, and specifies the region of the three-dimensional object in the corresponding image 1041 (step 1109). The specifying unit 713 stores the detection result 1047 including the specified region in the storage unit 1013, and the output unit 1012 outputs the detection result 1047.

  Next, the video input unit 1011 checks whether or not video is continuously input from the imaging device 1001 (step 1110). If video is input (YES in step 1110), the image at the next time is checked. The processing from step 1101 onward is repeated. Then, when the video input is stopped (step 1110, NO), the image analysis device 701 ends the process.

  According to such image analysis processing, even when the contour line included in the pattern on the road surface and the contour line of the three-dimensional object are connected in the image, the boundary between the pattern and the three-dimensional object is correctly identified. Therefore, the region of the three-dimensional object can be detected with high accuracy.

  FIG. 12 is a flowchart showing an example of the contour line extraction process in step 1103 of FIG. In this contour line extraction processing, connection determination is performed for each edge point included in each line from the lower end to the upper end of the feature point image 1042, and the edge point is connected to generate a contour line. And the same outline ID is provided with respect to the some edge point contained in each outline.

  First, the contour line extraction unit 1022 selects one edge point in the feature point image 1042 as the edge point A (step 1201). Next, the contour line extraction unit 1022 checks whether or not there is an edge point in three pixels that are included in the line one line higher than the line including the edge point A and are adjacent to the edge point A (step). 1202).

  FIG. 13 shows an example of three pixels adjacent to the edge point A. The pixel 1301 at the upper left of the edge point A, the pixel 1302 above the edge point A, and the pixel 1303 at the upper right of the edge point A correspond to three pixels adjacent to the edge point A.

  When an edge point exists in any of the three pixels (step 1202, YES), the contour line extraction unit 1022 checks whether or not an edge point exists in a plurality of pixels among the three pixels (step 1203). When an edge point exists in a plurality of pixels (step 1203, YES), the contour line extraction unit 1022 selects an optimum edge point among the edge points as the edge point B (step 1204). As the optimum edge point, for example, an edge point having an edge strength closer to the edge strength of the edge point A may be selected, and the edge point A can be connected to the edge point A with a slope closer to the slope of the contour line including the edge point A. An edge point may be selected.

  Next, the contour line extraction unit 1022 checks whether or not the edge point B has been registered as a contour line (step 1205). When the contour line ID is already assigned to the edge point B, the edge point B is determined to be registered, and when the contour line ID is not assigned to the edge point B, the edge point B is not registered. Determined.

  When the edge point B is not registered (step 1205, NO), the contour line extraction unit 1022 is included in the same line as the edge point A and whether there is another edge point C that can be connected to the edge point B. Is checked (step 1206).

  FIG. 14 shows an example of an edge point C that can be connected to the edge point B. The edge point B in FIG. 14A is adjacent to the edge point A. Since the edge point C adjacent to the left of the edge point A is also adjacent to the edge point B, it can be connected to the edge point B. Further, the edge point B in FIG. 14B is adjacent to the upper left of the edge point A. Since the edge point C on the left side of the edge point A is adjacent to the edge point B, it can be connected to the edge point B.

  When the edge point C exists (step 1206, YES), the contour line extraction unit 1022 compares the edge point A and the edge point C to determine which is the optimum edge point (step 1207). . As the optimum edge point, for example, an edge point having an edge strength closer to the edge strength of the edge point B may be selected, and the edge point B can be connected to the edge point B with an inclination closer to the inclination of the contour line including the edge point B. An edge point may be selected.

  When it is determined that the edge point C is the optimum edge point (step 1207, YES), the contour line extraction unit 1022 assigns the same contour line ID to the edge point B and the edge point C, so that the edge point B The edge point C is connected (step 1208). When the edge point C has been registered, the contour line ID of the edge point C is given to the edge point B, and when the edge point C has not been registered, the new contour line IDs are the edge point B and the edge point C. Is granted to.

  Next, the contour line extraction unit 1022 checks whether or not all edge points in the feature point image 1042 have been processed (step 1210). When an unprocessed edge point remains (step 1210, NO), the contour line extraction unit 1022 repeats the processing after step 1201 for the next edge point.

  On the other hand, when the edge point A is determined as the optimum edge point (step 1207, NO), the contour line extraction unit 1022 assigns the same contour line ID to the edge point A and the edge point B, thereby A and edge point B are connected (step 1209). When the edge point A has been registered, the contour line ID of the edge point A is given to the edge point B, and when the edge point A has not been registered, the new contour line IDs are the edge point A and the edge point B. Is granted to. Then, the contour line extraction unit 1022 performs the processing after step 1210.

  When no edge point exists in any of the three pixels (step 1202, NO), or when the edge point B has already been registered (step 1205, YES), the contour line extraction unit 1022 performs the processing from step 1210 onward. . When an edge point exists only in one pixel (step 1203, NO), the contour line extraction unit 1022 selects the edge point as the edge point B, and performs the processing after step 1205. When the edge point C does not exist (step 1206, NO), the contour line extraction unit 1022 performs the processing after step 1209.

  When all edge points have been processed (step 1210, YES), the contour line extraction unit 1022 generates contour line information 1043 including edge point information for each contour line ID (step 1211). At this time, the contour line extraction unit 1022 may extract, as a contour line, an edge line segment to which a predetermined number or more of edge points are connected among edge line segments to which a plurality of edge points are connected.

  FIG. 15 shows an example of the contour line information 1043. The contour line information 1043 in FIG. 15 includes an image ID, a contour line ID, a length, an edge position, the number of lost times, a division state, and a division point position. The image ID is identification information of the image 1041, and the contour line ID is contour line identification information. The length represents the number of edge points included in the contour line, and the edge position represents the position of each edge point included in the contour line. If the horizontal direction and the vertical direction of the feature point image 1042 are the x-axis and the y-axis, respectively, the position of the edge point is represented by the x-coordinate and the y-coordinate.

  The number of lost times represents the number of times the tracking of the contour line has failed in the video, and its initial value is zero. The collation unit 1031 increments the lost count. The division state represents the presence or absence of division points on the contour line, and the initial value is no division. The division point position represents the position of the division point. The division state is changed and the division point position is set by the division point determination unit 1035.

  FIG. 16 is a flowchart showing an example of the contour line matching process in step 1104 of FIG. First, the collation unit 1031 checks what number of frames in the video the image 1041 at the current time is (step 1601).

  When the image 1041 at the current time is the image of the first frame (step 1601, NO), the corresponding past image 1041 does not exist, and all the contour lines in the image 1041 at the current time are newly extracted contours. Since it is a line, the collation part 1031 complete | finishes a process.

  On the other hand, when the image 1041 at the current time is an image of the second and subsequent frames (step 1601, YES), the collation unit 1031 includes the contour line in the image 1041 at the current time and the past information included in the contour line information 1043. The contour line in the image 1041 is collated.

  Hereinafter, for the sake of simplicity, the past image 1041 is referred to as an image F1, and the image 1041 at the current time is referred to as an image F2. As the image F1, a predetermined number of images before the image F2 is used. The image F1 may be an image of a frame immediately before the image F2.

  First, the matching unit 1031 selects one contour line included in the image F1, and uses the amount of movement of the vehicle between the shooting time and the current time of the image F1 and the installation information of the imaging device 1001. The movement destination position of the selected contour line in the image F2 is calculated (step 1602). For example, as described in Patent Document 2, the movement destination position is a geometric shape called epipolar constraint based on the translation amount and rotation amount of the vehicle between two times and the relative position and angle of the imaging device 1001 with respect to the vehicle. Calculated using conditions.

  Next, the collation unit 1031 checks whether or not an outline exists at the movement destination position in the image F2 (step 1603). When the contour line exists at the movement destination position (step 1603, YES), the collation unit 1031 calculates the degree of difference between the contour line in the image F1 and the contour line of the movement destination position (step 1604), and the calculated difference. The degree is compared with a threshold (step 1605).

  When the degree of difference is equal to or less than the threshold (step 1605, YES), the collation unit 1031 determines that the contour line in the image F1 and the contour line at the movement destination position are the same contour line. Therefore, the matching unit 1031 changes the outline ID indicating the outline of the destination position in the image F2 to the same ID as the outline ID indicating the outline in the image F1 in the outline information 1043 (step 1606). ). As a result, the two contour lines are registered as the same contour line.

  Next, the collation unit 1031 checks whether or not all the contour lines in the image F1 have been selected (step 1610). If an unselected contour line remains (step 1610, NO), the next contour is checked. The process after step 1602 is repeated for the line.

  When no contour line exists at the movement destination position (step 1603, NO), or when the dissimilarity is larger than the threshold value (step 1605, NO), the collation unit 1031 calculates the contour line between the contour line in the image F1 and the movement destination position. It is determined that the line is not the same contour line. In this case, the collation unit 1031 assumes that tracking of the contour line in the image F1 has failed, increments the number of lost contour lines (step 1607), and compares the incremented lost number with a threshold (step 1608). ).

  When the number of lost times is equal to or greater than the threshold (step 1608, YES), the collation unit 1031 deletes the contour information in the image F1 from the contour information 1043 (step 1609), and performs the processing from step 1610 onward. On the other hand, when the number of lost times is less than the threshold value (step 1608, NO), the collation unit 1031 performs the processing after step 1610 without deleting the information of the contour line.

  When all the contour lines in the image F1 are selected (step 1610, YES), the collation unit 1031 ends the process.

  FIG. 17 is a flowchart illustrating an example of the dissimilarity calculation process in step 1604 of FIG. First, the collation unit 1031 obtains an overlapping portion between the contour line in the image F1 and the contour line of the movement destination position in the image F2 based on the moving direction of the contour line in the image F1 (step 1701).

  FIG. 18 illustrates an example of an overlapping portion between the contour line in the image F1 and the contour line in the image F2. An outline 1802 exists at the movement destination position in the image F2 obtained from the outline 1801 in the image F1.

  In this case, a straight line 1811 indicating the moving direction of one end point 1821 of the outline 1801 and a straight line 1812 indicating the moving direction of the other end point 1822 are obtained based on the epipolar constraint. One end point 1831 of the contour line 1802 exists on the straight line 1811, and the other end point 1832 does not exist on the straight line 1812.

  Therefore, the collation unit 1031 obtains a straight line 1813 passing through the end point 1832 of the contour line 1802 based on the epipolar constraint, and obtains an intersection 1823 between the straight line 1813 and the contour line 1801. Then, the collation unit 1031 determines a partial contour 1803 between the end point 1821 and the intersection 1823 as an overlapping portion on the contour 1801 and determines the entire contour 1802 as an overlapping portion on the contour 1802.

  Next, the collation unit 1031 adjusts the scale so that the lengths of the overlapping portions of the two contour lines become the same in order to compensate for the change in the length of the contour line between two times (step 1702). The center-of-gravity position of each overlapping portion after adjustment is calculated (step 1703). The collation unit 1031 then superimposes the centroids of the two overlapping portions, selects points included in each overlapping portion, and calculates the amount of horizontal displacement between the selected points (step 1704).

  FIG. 19 shows an example of superposition of two overlapping portions. A line segment 1901 represents a scaled partial outline 1803, and a line segment 1902 represents a scaled outline 1802. When the center of gravity of the line segment 1901 and the center of gravity of the line segment 1902 are overlapped at a point 1903, the difference in x coordinate between each point on the line segment 1901 and a point on the line segment 1902 having the same y coordinate. Is calculated as the amount of horizontal displacement.

  Next, the collation unit 1031 checks whether or not all the points in the overlapping portion have been selected (step 1705), and if unselected points remain (step 1705, NO), the step is performed for the next point. The processing of 1704 is repeated.

  When all the points in the overlapping portion are selected (step 1705, YES), the collation unit 1031 calculates a plurality of statistical values of deviation amounts (step 1706). As the statistical value of the deviation amount, an average value, median value, maximum value, minimum value, or the like of a plurality of deviation amounts can be used, and the calculated statistical value is used as the degree of difference in step 1604.

  According to the contour line matching process in FIG. 16, the contour line in the image 1041 at the current time corresponding to the contour line in the past image 1041 is obtained from the movement amount of the vehicle between the two times and the installation information of the imaging device 1001. It is possible to identify and associate those contour lines. Further, by deleting the contour line when the number of lost times is equal to or greater than the threshold value, a region such as a three-dimensional object that has disappeared from the imaging range can be excluded from the processing target.

  FIG. 20 is a flowchart showing an example of candidate list generation processing in step 1105 of FIG. First, the generation unit 1032 selects one contour line in the image F1 used in the contour line matching process, refers to the contour line information 1043, and has the same contour line ID as the selected contour line. Is present in the image F2 (step 2001).

  When the contour line having the same contour line ID exists in the image F2 (step 2001, YES), the generation unit 1032 uses the condition that the lower end point of the contour line in the image F1 exists on the road surface. Then, the position of the end point in the three-dimensional space is calculated (step 2002). When the end point exists on the road surface, the height is 0. Therefore, the position of the end point in the three-dimensional space can be calculated from the relative position and angle of the imaging device 1001 to the vehicle. Hereinafter, a position in the three-dimensional space may be referred to as a three-dimensional position.

  Next, the generation unit 1032 selects one edge point on the contour line in the image F1, and the condition that the contour line in the image F1 corresponds to the contour line of the three-dimensional object in the direction perpendicular to the road surface. Is used to calculate a three-dimensional position corresponding to the selected edge point (step 2003).

  Next, the generation unit 1032 uses the amount of movement of the vehicle between the shooting time of the image F1 and the current time and the installation information of the imaging device 1001 to calculate the current time from the three-dimensional position corresponding to the edge point. A three-dimensional position is calculated (step 2004). Then, the generation unit 1032 calculates a position in the image F2 corresponding to the three-dimensional position at the current time (step 2005).

  Each position in step 2003 to step 2005 is calculated by using the translation amount and rotation amount of the vehicle between two times and the relative position and angle of the imaging device 1001 with respect to the vehicle, as described in Patent Document 2, for example. Can be calculated.

  Next, the generation unit 1032 obtains an edge point corresponding to the calculated position on the contour line in the image F2 having the same contour ID as the contour line in the image F1 (step 2006). Then, the generation unit 1032 records the selected edge point in the image F1 and the obtained edge point in the image F2 in the candidate list 1044 as a combination of division point candidates.

  Next, the generation unit 1032 calculates a three-dimensional position corresponding to the selected edge point using a condition that the contour line in the image F1 corresponds to the contour line of the pattern on the road surface (step 2007).

  Next, the generation unit 1032 calculates the three-dimensional position at the current time from the three-dimensional position corresponding to the edge point in the same manner as in step 2004 (step 2008), and in the same manner as in step 2005, The position is calculated (step 2009). Then, the generation unit 1032 obtains an edge point on the contour line in the image F2 in the same manner as in step 2006, and the selected edge point in the image F1 and the obtained edge point in the image F2 are obtained as candidate division points. The combinations are recorded in the candidate list 1044 (step 2010).

  Next, the generation unit 1032 checks whether or not all edge points on the contour line in the image F1 have been selected (step 2011), and if unselected edge points remain (step 2011, NO), The processing after step 2003 is repeated for the next edge point.

  If all edge points have been selected (step 2011, YES), the generation unit 1032 checks whether all the contour lines in the image F1 have been selected (step 2012). When an unselected contour line remains (step 2012, NO), the generation unit 1032 repeats the processing after step 2001 for the next contour line. If all the contour lines have been selected (step 2012, YES), the generation unit 1032 ends the process.

  According to such candidate list generation processing, a combination of division point candidates when each contour line is assumed to be a contour line of a solid object, and a division point when each contour line is assumed to be a contour line of a pattern on the road surface A candidate list 1044 is generated that includes candidate combinations.

  Note that the generation unit 1032 may perform the processing from step 2003 to step 2010 on a small number of edge points selected at a predetermined interval, not on all edge points on the contour line in the image F1. Thereby, the processing load of the image analysis apparatus 701 can be reduced.

  FIG. 21 shows an example of candidate list generation processing. Assuming that the edge point 2111 below the contour line 2101 in the image F1 exists on the road surface, the three-dimensional position of the edge point 2111 is calculated, and the corresponding contour in the image F2 is calculated from the constraint between two times. An edge point 2121 on the line 2102 is determined.

  Here, assuming that the contour line 2101 and the contour line 2102 are contour lines in the height direction of the three-dimensional object 2131 on the road surface, three-dimensional corresponding to the edge points 2112 to 2115 on the contour line 2101 respectively. The position is calculated. Next, the edge point 2122 to the edge point 2125 on the outline 2102 corresponding to the edge point 2112 to the edge point 2115 are obtained from the constraint between two times. A combination of each edge point on the contour line 2101 and each edge point on the contour line 2102 is recorded as a combination of division point candidates.

  In this way, by adopting the constraint between two times, the candidate for the dividing point on the contour line 2102 in the image F2 corresponding to the candidate for the dividing point on the contour line 2101 in the image F1 is uniquely obtained.

  FIG. 22 is a flowchart illustrating an example of the division score calculation process in step 1106 of FIG. First, the dissimilarity calculation unit 1033 selects one contour line in the image F1, refers to the candidate list 1044, selects one combination including division point candidates on the selected contour line, and the contour. The line is divided into two partial contour lines at the position of the dividing point candidate.

  Then, the dissimilarity calculation unit 1033 calculates the pixel value statistical value V1 of the edge point included in the upper partial contour line of the division point candidate and the slope U1 of the partial contour line (step 2201). For example, a luminance value can be used as the pixel value, and an average value, median value, maximum value, minimum value, or the like of a plurality of pixel values can be used as the statistical value of the pixel value.

  Next, the dissimilarity calculation unit 1033 calculates the pixel value statistical value V2 of the edge point included in the lower partial contour line of the division point candidate and the slope U2 of the partial contour line (step 2202). Then, the dissimilarity calculation unit 1033 calculates an evaluation value E1 indicating the dissimilarity between the upper partial contour line and the lower partial contour line by the following equation (step 2203).

E1 = w1 × D1 + w2 × D2 (11)
D1 = | V1-V2 | (12)
D2 = | θ1 | (13)

  W1 and w2 in equation (11) are positive real numbers representing adjustment coefficients, and θ1 in equation (13) is an angle between the gradient U1 and the gradient U2. The pixel value difference D1 indicates the difference between the pixel value of the upper partial outline and the pixel value of the lower partial outline, and the inclination difference D2 indicates the inclination of the upper partial outline and the lower part. The degree of difference from the inclination of the contour line is shown.

  Similarly, the dissimilarity calculation unit 1033 divides the contour line in the image F2 into two partial contour lines at the position of the division point candidate, and the edge point pixels included in the partial contour line above the division point candidate. The statistical value V3 of the value and the slope U3 of the partial contour line are calculated (step 2204).

  Next, the dissimilarity calculation unit 1033 calculates the pixel value statistical value V4 of the edge point included in the lower partial outline of the division point candidate and the inclination U4 of the partial outline (step 2205). Then, the dissimilarity calculation unit 1033 calculates an evaluation value E2 indicating the dissimilarity between the upper partial contour line and the lower partial contour line by the following equation (step 2206).

E2 = w1 × D3 + w2 × D4 (14)
D3 = | V3-V4 | (15)
D4 = | θ2 | (16)

  Θ2 in Expression (16) is an angle between the inclination U3 and the inclination U4. The dissimilarity calculation unit 1033 calculates a division score E3 indicating an evaluation value for the selected combination of division point candidates according to the following equation, and records the division score E3 as the division score 1045 (step 2207).

E3 = E1 + E2 (17)

  Next, the dissimilarity calculation unit 1033 checks whether or not all combinations including the dividing point candidates on the contour line in the image F1 have been selected (step 2208). When an unselected combination remains (step 2208, NO), the difference calculation unit 1033 repeats the processing after step 2201 for the next combination.

  When all combinations are selected (step 2208, YES), the dissimilarity calculation unit 1033 checks whether all the contour lines in the image F1 have been selected (step 2209). When an unselected contour line remains (step 2209, NO), the dissimilarity calculation unit 1033 repeats the processing after step 2201 for the next contour line. When all the contour lines are selected (step 2209, YES), the dissimilarity calculation unit 1033 ends the process.

  According to such a division score calculation process, the division score E3 of Expression (17) can be obtained for each division point candidate on each contour line in the image F1. As the division score E3 is larger, the corresponding division point candidate is more suitable as a division point on the contour line.

  FIG. 23 is a flowchart showing an example of the feature similarity calculation process in step 1107 of FIG. First, the similarity calculation unit 1034 selects one contour line in the image F1, refers to the candidate list 1044, and selects one combination including division point candidates on the selected contour line.

  Then, the similarity calculation unit 1034 divides each of the contour line in the image F1 and the contour line in the image F2 into two partial contour lines at the position of the division point candidate, and above the contour line in the image F1. And a partial outline above the outline in the image F2 are selected (step 2301).

  Next, the similarity calculation unit 1034 adjusts the scale so that the lengths of the two selected partial contour lines are the same (step 2302), and calculates the similarity G1 of the two partial contour lines after the adjustment. The calculation is performed according to the following equation (step 2303).

G1 = w3 × R1 + w4 × R2 (21)
R1 = 255−VD1 / M1 (22)
R2 = w5-w6 × (XD1 / M1) (23)

  W3 and w4 in Equation (21) and w5 and w6 in Equation (23) are positive real numbers representing adjustment coefficients, and M1 in Equation (22) and Equation (23) is on each adjusted partial contour line. Represents the total number of points. VD1 in Expression (22) represents the sum of differences between the pixel values at two points between the partial contour lines. As the pixel value, for example, a luminance value can be used.

  XD1 in the equation (23) represents the total sum of the horizontal shift amounts of two points between the partial contour lines. For example, similarly to the two overlapping portions shown in FIG. 19, the amount of deviation in the horizontal direction can be obtained by superimposing two adjusted partial contour lines at the center of gravity. In this case, the absolute value of the difference in the x coordinate between each point on one partial contour line and the point on the other partial contour line having the same y coordinate is calculated as a horizontal shift amount.

  The pixel value similarity R1 indicates the similarity between the pixel value of the upper partial contour line in the image F1 and the pixel value of the upper partial contour line in the image F2, and the inclination similarity R2 indicates the similarity in the image F1. The similarity between the inclination of the upper partial contour line and the inclination of the upper partial contour line in the image F2 is shown.

  Next, the similarity calculation unit 1034 selects a lower partial contour line in the image F1 and a lower partial contour line in the image F2 (step 2304). Then, the similarity calculation unit 1034 performs scale adjustment between the two selected partial contour lines (step 2305), and calculates the similarity G2 of the two partial contour lines after the adjustment using the following equation (step 2305). 2306).

G2 = w3 × R3 + w4 × R4 (24)
R3 = 255−VD2 / M2 (25)
R4 = w5-w6 × (XD2 / M2) (26)

  M2 in Expression (25) and Expression (26) represents the total number of points on each partial contour line after adjustment. VD2 in Expression (25) represents the sum of the differences between the two pixel values between the partial contour lines, and XD2 in Expression (26) represents the total sum of the horizontal deviation amounts of the two points between the partial contour lines. .

  The pixel value similarity R3 indicates the similarity between the pixel value of the lower partial contour line in the image F1 and the pixel value of the lower partial contour line in the image F2, and the inclination similarity R4 indicates the image similarity. The similarity between the inclination of the lower partial outline in the image and the inclination of the lower partial outline in the image F2 is shown.

  Next, the similarity calculation unit 1034 calculates the feature similarity G3 between the partial outlines on both sides in the image F1 and the partial outlines on both sides in the image F2 by the following formula, and the feature similarity G3 is the feature. It records as similarity 1046 (step 2307).

G3 = G1 + G2 (27)

  Next, the similarity calculation unit 1034 checks whether or not all combinations including division point candidates on the contour line in the image F1 have been selected (step 2308). When an unselected combination remains (step 2308, NO), the similarity calculation unit 1034 repeats the processing after step 2301 for the next combination.

  When all combinations are selected (step 2308, YES), the similarity calculation unit 1034 checks whether or not all the contour lines in the image F1 have been selected (step 2309). When an unselected contour line remains (step 2309, NO), the similarity calculation unit 1034 repeats the processing after step 2301 for the next contour line. If all the contour lines are selected (step 2309, YES), the similarity calculation unit 1034 ends the process.

  According to such feature similarity calculation processing, the feature similarity G3 of Expression (27) can be obtained for each division point candidate on each contour line in the image F1. The larger the feature similarity G3, the more appropriate the corresponding dividing point candidate is as a dividing point on the contour line.

  Note that when the video output from the imaging apparatus 1001 is a color video, the video input unit 1011 can store a color image at each time in the storage unit 1013. In this case, the dissimilarity calculation unit 1033 and the similarity calculation unit 1034 may calculate the division score E3 and the feature similarity G3 by using the RGB value of the color image, the color difference signal, or the like as the pixel value.

  FIG. 24 is a flowchart showing an example of the dividing point determination process in step 1108 of FIG. First, the division point determination unit 1035 selects one contour line in the image F1, refers to the division score 1045 and the feature similarity 1046, and calculates a feature evaluation value E4 for each division point candidate on the selected contour line. The calculation is performed according to the following equation (step 2401).

E4 = E3 + G3 (31)

  Then, the division point determination unit 1035 selects the maximum value among the feature evaluation values E4 for the plurality of division point candidates and compares it with a threshold value (step 2402). If the maximum value of the feature evaluation value E4 is equal to or greater than the threshold (step 2402, YES), the dividing point determination unit 1035 determines a dividing point candidate corresponding to the maximum value as a dividing point, and contours at the determined position of the dividing point. The line is divided (step 2403). Thereby, the outline in the image F1 and the outline in the image F2 are divided | segmented by each division | segmentation point.

  Then, the division point determination unit 1035 updates the division state and the division point position of the outline in the image F1 and the outline in the image F2 in the outline information 1043 (step 2404). When the dividing state of the contour line is not divided, the dividing point determination unit 1035 changes the dividing state to dividing and sets the determined dividing point position as the dividing point position.

  On the other hand, when the division state of the outline is “with division”, the division point determination unit 1035 updates the division point position without changing the division state. At this time, the division point determination unit 1035 can obtain a weighted average of the already recorded division point positions and the determined division point positions, and can set the obtained position as the division point position. When the weight of the recorded dividing point position is set to 0, the determined dividing point position is set as the dividing point position.

  Next, the dividing point determination unit 1035 checks whether or not all the contour lines in the image F1 have been selected (step 2405). When an unselected contour line remains (step 2405, NO), the dividing point determination unit 1035 repeats the processing after step 2401 for the next contour line.

  When the maximum value of the feature evaluation value E4 is less than the threshold value (step 2402, NO), the dividing point determination unit 1035 performs the processing after step 2405 without dividing the contour line. If all contour lines have been selected (step 2405, YES), the dividing point determination unit 1035 ends the process.

  According to such division point determination processing, each contour line in the image F1 and each contour line in the image F2 can be divided at the optimum division points. Note that it is also possible to determine the division points using only the feature similarity G3 as the feature evaluation value E4.

  FIG. 25 shows an example of the dividing point determination process. The luminance value distribution 2501 represents a change in the luminance value on the contour line in the image F1, and the luminance value distribution 2502 represents a change in the luminance value on the corresponding contour line in the image F2. Each luminance value distribution is obtained by plotting luminance values with respect to the y coordinate of the edge point on each contour line.

  For example, when the point 2511 on the luminance value distribution 2501 corresponds to a division point candidate in the image F1, the luminance value distribution 2501 includes the luminance value distribution 2503 of the upper partial outline and the luminance value of the lower partial outline. It is divided into distribution 2504. Similarly, when the point 2512 on the luminance value distribution 2502 corresponds to a candidate for a dividing point in the image F2, the luminance value distribution 2502 includes the luminance value distribution 2505 of the upper partial outline and the luminance of the lower partial outline. It is divided into a value distribution 2506.

  Next, the luminance value distribution 2503 and the luminance value distribution 2505 are adjusted by adjusting the scales of the upper partial contour line in the image F1 and the upper partial contour line in the image F2, so that the luminance value distribution 2507 and the luminance value distribution 2509 are adjusted. Respectively. Further, by adjusting the scale of the lower partial contour line in the image F1 and the lower partial contour line in the image F2, the luminance value distribution 2504 and the luminance value distribution 2506 become the luminance value distribution 2508 and the luminance value distribution. Respectively converted to 2510.

  In this case, the pixel value similarity R1 of Expression (22) is calculated from the difference between the luminance value distribution 2507 and the luminance value distribution 2509, and from the difference between the luminance value distribution 2508 and the luminance value distribution 2510, A pixel value similarity R3 is calculated. However, since the pixel value similarity R1 and the pixel value similarity R3 are small, as a result, the feature evaluation value E4 of Expression (31) is small, and the two division point candidates corresponding to the points 2511 and 2512 are division points. Not adopted as.

  On the other hand, when the point 2513 on the luminance value distribution 2501 corresponds to a division point candidate in the image F1, the luminance value distribution 2501 includes the luminance value distribution 2521 of the upper partial outline and the luminance value of the lower partial outline. It is divided into distribution 2522. Similarly, when the point 2514 on the luminance value distribution 2502 corresponds to a division point candidate in the image F2, the luminance value distribution 2502 includes the luminance value distribution 2523 of the upper partial outline and the luminance of the lower partial outline. It is divided into a value distribution 2524.

  Next, the luminance value distribution 2521 and the luminance value distribution 2523 are adjusted to the luminance value distribution 2525 and the luminance value distribution 2527 by adjusting the scale of the upper partial contour line in the image F1 and the upper partial contour line in the image F2. Respectively. Further, by adjusting the scale of the lower partial contour line in the image F1 and the lower partial contour line in the image F2, the luminance value distribution 2522 and the luminance value distribution 2524 become the luminance value distribution 2526 and the luminance value distribution. Respectively converted to 2528.

  In this case, the pixel value similarity R1 of Equation (22) is calculated from the difference between the luminance value distribution 2525 and the luminance value distribution 2527, and from the difference between the luminance value distribution 2526 and the luminance value distribution 2528, A pixel value similarity R3 is calculated. Since the pixel value similarity R1 and the pixel value similarity R3 are large, as a result, the feature evaluation value E4 of Expression (31) increases, and two division point candidates corresponding to the points 2513 and 2514 are divided points. Adopted as

  FIG. 26 is a flowchart illustrating an example of the three-dimensional object detection process in step 1109 of FIG. First, the specifying unit 713 selects one contour line in the image F1, and refers to the contour line information 1043 to determine whether a contour line having the same contour ID as the selected contour line exists in the image F2. It is checked whether or not (step 2601).

  When a contour line having the same contour ID exists in the image F2 (step 2601, YES), the specifying unit 713 refers to the contour line information 1043, and the contour line in the image F1 and the contour line in the image F2 Are divided at respective dividing point positions (step 2602). And the specific | specification part 713 calculates the flow amount of each partial outline based on the optical flow of the end point of the upper and lower partial outline.

  Next, the specifying unit 713 compares the flow amount of each partial outline with a threshold value (step 2603). When the flow amount is greater than or equal to the threshold (step 2603, YES), the specifying unit 713 determines that the region having the partial outline is a region of a three-dimensional object, and generates a detection result 1047 including that region (step). 2604). The threshold for the flow amount can be set based on, for example, the amount of movement of the vehicle and the position of the partial contour line in the images F1 and F2.

  On the other hand, when the flow amount is less than the threshold value (step 2603, YES), the specifying unit 713 determines that the region having the partial outline is a pattern region on the road surface (step 2605).

  Next, the specifying unit 713 checks whether or not all the contour lines in the image F1 have been selected (step 2606). When an unselected contour line remains (step 2606, NO), the specifying unit 713 repeats the processing after step 2601 for the next contour line. If all the contour lines are selected (step 2606, YES), the specifying unit 713 ends the process.

  Using the detection result 1047 generated by the specifying unit 713, the output unit 1012 may display on the screen an image 1041 in which the region of the three-dimensional object is emphasized, or may output a warning sound for the driver.

  The configuration of the image analysis device 701 in FIGS. 7 and 10 is merely an example, and some components may be omitted or changed according to the use or conditions of the image analysis device 701. For example, when the edge detection process is performed outside the image analysis apparatus 701, the video input unit 1011 and the feature point extraction unit 1021 can be omitted. When the division point is determined using only the feature similarity G3, the difference calculation unit 1033 can be omitted. The imaging device 1001 may be mounted on a mobile object such as a self-propelled robot or an animal.

  The flowcharts of FIGS. 8, 11, 12, 16, 17, 20, 22 to 24, and 26 are merely examples, and some processes are performed depending on the configuration or conditions of the image analysis apparatus 701. May be omitted or changed. For example, in the image analysis process of FIG. 11, when the edge detection process is performed outside the image analysis apparatus 701, the processes of step 1101 and step 1102 can be omitted. When the division point is determined using only the feature similarity G3, the process of step 1106 can be omitted.

  In step 1202 of FIG. 12, the contour line extraction unit 1022 searches not only three pixels included in the line one line above the line including the edge point A but also more pixels including pixels on both sides thereof. As an alternative, an edge point may be searched.

  In the contour line matching process of FIG. 16, when it is not necessary to delete a contour line that has failed to be tracked, the processes in steps 1607 to 1609 can be omitted. In step 1604 and step 1605, the collation unit 1031 determines whether or not the contour lines are the same contour line based on the similarity between the two contour lines instead of the dissimilarity between the two contour lines. You may judge.

  In the division score calculation process of FIG. 22, the difference degree calculation unit 1033 may calculate the division score E3 based on either the pixel value difference degree or the inclination difference degree. In the feature similarity calculation process of FIG. 23, the similarity calculation unit 1034 may calculate the feature similarity G3 based on either the pixel value similarity or the inclination similarity.

  In the division point determination process of FIG. 24, when the division point is determined using only the feature similarity G3, the division point determination unit 1035 uses the feature similarity G3 as the feature evaluation value E4 in Step 2401.

  1, 3, 5, 6, 9, 18, and 21 are merely examples, and other images may be taken according to the configuration or conditions of the image analysis device 701. is there. The pattern on the road surface and the three-dimensional object in FIG. 2 are merely examples, and another object may be photographed depending on the photographing environment. The edge points in FIGS. 13 and 14 are merely examples, and other edge points may be detected according to the captured image.

  The outline information in FIG. 15 is merely an example, and some information may be omitted or changed according to the configuration or conditions of the image analysis apparatus 701. For example, when it is not necessary to delete a contour line that has failed to be tracked, the number of lost times can be omitted. The luminance value distribution in FIG. 25 is merely an example, and the luminance value distribution changes according to the captured image.

  Expressions (1) to (31) are merely examples, and other calculation expressions may be used depending on the configuration or conditions of the image analysis apparatus 701.

  An image feature other than the contour line can be used as the image feature of each region extracted from the image. For example, when a pixel set included in each region is extracted as an image feature, a boundary line between the pixel set of the specific region and the pixel set of the three-dimensional object region is used as a boundary between the specific region and the three-dimensional object. .

  7 and 10 can be implemented as a hardware circuit, or can be implemented using an information processing apparatus (computer) as shown in FIG.

  27 includes a central processing unit (CPU) 2701, a memory 2702, an input device 2703, an output device 2704, an auxiliary storage device 2705, a medium driving device 2706, and a network connection device 2707. These components are connected to each other by a bus 2708. The imaging device 1001 in FIG. 10 may be connected to the bus 2708.

  The memory 2702 is a semiconductor memory such as a read only memory (ROM), a random access memory (RAM), or a flash memory, and stores programs and data used for processing. The memory 2702 can be used as the storage unit 1013 in FIG.

  The CPU 2701 (processor) operates as the extraction unit 711, the determination unit 712, and the specification unit 713 in FIGS. 7 and 10 by executing a program using the memory 2702, for example. The CPU 2701 is the video input unit 1011, the feature point extraction unit 1021, the contour line extraction unit 1022, the collation unit 1031, the generation unit 1032, the dissimilarity calculation unit 1033, the similarity calculation unit 1034, and the division point determination unit 1035 in FIG. 10. Also works.

  The input device 2703 is, for example, a keyboard, a pointing device, or the like, and is used for inputting an instruction or information from an operator or a user. The output device 2704 is, for example, a display device, a printer, a speaker, or the like, and is used to output an inquiry or instruction to an operator or user and a processing result. The output device 2704 can be used as the output unit 1012 in FIG. The processing result may be a detection result 1047.

  The auxiliary storage device 2705 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 2705 may be a hard disk drive or a flash memory. The auxiliary storage device 2705 can be used as the storage unit 1013 in FIG. The information processing apparatus can store programs and data in the auxiliary storage device 2705 and load them into the memory 2702 for use.

  The medium driving device 2706 drives a portable recording medium 2709 and accesses the recorded contents. The portable recording medium 2709 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 2709 may be a compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a universal serial bus (USB) memory, or the like. An operator or a user can store programs and data in the portable recording medium 2709 and load them into the memory 2702 for use.

  As described above, the computer-readable recording medium for storing the program and data used in the processing is a physical (non-transitory) recording medium such as the memory 2702, the auxiliary storage device 2705, or the portable recording medium 2709. It is a medium.

  The network connection device 2707 is a communication interface that is connected to a communication network such as a local area network and a wide area network, and performs data conversion accompanying communication. The network connection device 2707 can be used as the output unit 1012 in FIG. The information processing apparatus can receive programs and data from an external apparatus via the network connection apparatus 2707, and can use them by loading them into the memory 2702.

  The information processing apparatus can receive a processing request from the user terminal via the network connection apparatus 2707 and transmit the detection result 1047 to the user terminal.

  Note that the information processing apparatus does not have to include all the components illustrated in FIG. 27, and some of the components may be omitted depending on the application or conditions. For example, when the information processing apparatus receives a processing request from the user terminal via the communication network, the input device 2703 and the output device 2704 may be omitted. When the portable recording medium 2709 or the communication network is not used, the medium driving device 2706 or the network connection device 2707 may be omitted.

  Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art can make various changes, additions and omissions without departing from the scope of the present invention as set forth in the appended claims. Let's go.

With respect to the embodiment described with reference to FIGS. 3 to 27, the following additional notes are disclosed.
(Appendix 1)
An image feature of the first region is extracted from a first image taken at a first time by an imaging device installed on a moving body moving on the surface, and a second time different from the first time by the imaging device. Extracting the image features of the second region from the second image taken in
Based on the image characteristics of the first area, a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface is obtained,
Based on the amount of movement of the moving body between the first time and the second time, a second position that is a candidate for a boundary in the second region corresponding to the first position is obtained,
The image characteristics of the first area when the first position is used as the boundary in the first area, and the second area when the second position is used as the boundary in the second area. Determining the first position as the boundary in the first region based on a feature similarity with an image feature;
Identifying a region of the three-dimensional object in the first region based on the determined boundary;
An image analysis program that causes a computer to execute processing.
(Appendix 2)
The computer obtains a plurality of positions including the first position as the boundary candidates in the first area, and determines a plurality of positions including the second position as the boundary candidates in the second area. The image characteristics of the first area when each of the plurality of positions in the first area is used as the boundary in the first area, and each of the plurality of positions in the second area are obtained as the first position. A feature evaluation value including a feature similarity with the image feature of the second area when used as the boundary in two areas is obtained, and the first position is used as the boundary in the first area. Based on the result of comparing the feature evaluation value and the feature evaluation value when a position other than the first position is used as the boundary in the first region, the first position is determined in the first region. The appendix is characterized in that the boundary is determined. 1, wherein the image analysis program.
(Appendix 3)
The image feature of the first area is a third edge line segment including a first edge line segment of the specific area on the surface and a second edge line segment in the height direction of the three-dimensional object, The image feature of the area is a fourth edge line segment included in the second area, and the plurality of positions in the first area are the first edge line segment and the first edge line on the third edge line segment. A plurality of positions in the second area are candidates for boundaries on the fourth edge line segment, and the computer is configured to select the plurality of positions in the first area. Are used as the boundary on the third edge line segment, and each of the plurality of positions in the second region is used as the boundary on the fourth edge line segment, and the third edge line segment and The feature similarity with the fourth edge line segment is obtained. 2, wherein the image analysis program.
(Appendix 4)
The computer corresponds to the third edge line segment based on the movement amount of the moving body and installation information of the imaging device from among a plurality of edge line segments extracted from the second image. A fourth edge line segment is identified, and using the condition that the plurality of positions in the first area are on the second edge line segment, the first edge line segment is determined from each of the plurality of positions in the first area. The image analysis program according to supplementary note 3, wherein each of the plurality of positions in two regions is obtained.
(Appendix 5)
The computer includes a pixel value of a first partial edge line segment between one end point of the third edge line segment and the boundary, and a first value between the one end point of the fourth edge line segment and the boundary. A first pixel value similarity indicating a similarity with a pixel value of a two partial edge line segment; a pixel value of a third partial edge line segment between the other end point of the third edge line segment and the boundary; Based on the second pixel value similarity indicating the similarity between the other end point of the fourth edge line segment and the pixel value of the fourth partial edge line segment between the boundary and the third edge line segment, The image analysis program according to appendix 3 or 4, wherein the feature similarity with the fourth edge line segment is obtained.
(Appendix 6)
The computer includes a first inclination similarity indicating a similarity between the inclination of the first partial edge line segment and the inclination of the second partial edge line segment, the inclination of the third partial edge line segment, and the fourth part. The supplementary note 5 wherein the feature similarity between the third edge line segment and the fourth edge line segment is obtained based on a second slope similarity degree indicating a similarity with the slope of the edge line segment. Image analysis program.
(Appendix 7)
The computer is arranged on the third edge line segment based on a first pixel value difference indicating a difference between the pixel value of the first partial edge line segment and the pixel value of the third partial edge line segment. A first evaluation value for the boundary of the second partial edge line segment, and based on a second pixel value difference degree indicating a difference degree between the pixel value of the second partial edge line segment and the pixel value of the fourth partial edge line segment, A second evaluation value for the boundary on a four-edge line segment is obtained, the feature similarity between the third edge line segment and the fourth edge line segment, the first evaluation value, and the second evaluation value; 7. The image analysis program according to appendix 5 or 6, wherein the feature evaluation value is obtained using.
(Appendix 8)
The computer calculates the first evaluation value based on a first inclination difference indicating a difference between the inclination of the first partial edge line segment and the inclination of the third partial edge line segment, and the second part The image analysis program according to appendix 7, wherein the second evaluation value is obtained based on a second inclination dissimilarity indicating a dissimilarity between the inclination of an edge line segment and the inclination of the fourth partial edge line segment.
(Appendix 9)
9. The device according to claim 1, wherein the moving body is a vehicle, the imaging device is installed in front of or behind the vehicle, and the specific area is a pattern on the surface. Image analysis program.
(Appendix 10)
An image feature of the first region is extracted from a first image taken at a first time by an imaging device installed on a moving body moving on the surface, and a second time different from the first time by the imaging device. An extraction unit for extracting image features of the second region from the second image taken in
Based on the image characteristics of the first area, a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface is obtained, and the first time And a second position that is a candidate for a boundary in the second region corresponding to the first position based on a movement amount of the moving body between the second time and the second time, and Feature similarity between the image feature of the first region when used as the boundary in one region and the image feature of the second region when the second position is used as the boundary in the second region A determination unit for determining the first position as the boundary in the first region based on:
Based on the boundary determined by the determining unit, a specifying unit that specifies a region of the three-dimensional object in the first region;
An image analysis apparatus comprising:
(Appendix 11)
The determining unit obtains a plurality of positions including the first position as the boundary candidates in the first area, and a plurality of positions including the second position as the boundary candidates in the second area. Image characteristics of the first area when each of the plurality of positions in the first area is used as the boundary in the first area, and each of the plurality of positions in the second area is When a feature evaluation value including a feature similarity with the image feature of the second region when used as the boundary in the second region is obtained, and the first position is used as the boundary in the first region Based on a result of comparing the feature evaluation value and the feature evaluation value when a position other than the first position is used as the boundary in the first area, the first position is set in the first area. Note 10 wherein the boundary is determined as Placing an image analysis apparatus.
(Appendix 12)
The image feature of the first area is a third edge line segment including a first edge line segment of the specific area on the surface and a second edge line segment in the height direction of the three-dimensional object, The image feature of the area is a fourth edge line segment included in the second area, and the plurality of positions in the first area are the first edge line segment and the first edge line on the third edge line segment. Two edge line segments, and the plurality of positions in the second region are boundary candidates on the fourth edge line segment, and the determination unit includes the first region in the first region. Using each of a plurality of positions as the boundary on the third edge line segment and using each of the plurality of positions in the second region as the boundary on the fourth edge line segment, the third edge line segment And calculating the feature similarity between the fourth edge line segment and the fourth edge line segment. Placing an image analysis apparatus.
(Appendix 13)
Computer
An image feature of the first region is extracted from a first image taken at a first time by an imaging device installed on a moving body moving on the surface, and a second time different from the first time by the imaging device. Extracting the image features of the second region from the second image taken in
Based on the image characteristics of the first area, a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface is obtained,
Based on the amount of movement of the moving body between the first time and the second time, a second position that is a candidate for a boundary in the second region corresponding to the first position is obtained,
The image characteristics of the first area when the first position is used as the boundary in the first area, and the second area when the second position is used as the boundary in the second area. Determining the first position as the boundary in the first region based on a feature similarity with an image feature;
Identifying a region of the three-dimensional object in the first region based on the determined boundary;
An image analysis method characterized by the above.
(Appendix 14)
The computer obtains a plurality of positions including the first position as the boundary candidates in the first area, and determines a plurality of positions including the second position as the boundary candidates in the second area. The image characteristics of the first area when each of the plurality of positions in the first area is used as the boundary in the first area, and each of the plurality of positions in the second area are obtained as the first position. A feature evaluation value including a feature similarity with the image feature of the second area when used as the boundary in two areas is obtained, and the first position is used as the boundary in the first area. Based on the result of comparing the feature evaluation value and the feature evaluation value when a position other than the first position is used as the boundary in the first region, the first position is determined in the first region. The appendix is characterized in that the boundary is determined. The image analysis method according 13.
(Appendix 15)
The image feature of the first area is a third edge line segment including a first edge line segment of the specific area on the surface and a second edge line segment in the height direction of the three-dimensional object, The image feature of the area is a fourth edge line segment included in the second area, and the plurality of positions in the first area are the first edge line segment and the first edge line on the third edge line segment. A plurality of positions in the second area are candidates for boundaries on the fourth edge line segment, and the computer is configured to select the plurality of positions in the first area. Are used as the boundary on the third edge line segment, and each of the plurality of positions in the second region is used as the boundary on the fourth edge line segment, and the third edge line segment and The feature similarity with the fourth edge line segment is obtained. The image analysis method according 14.

101, 102, 301, 1041 Image 111, 112 Object 113, 211 Arrow 121, 122, 207, 401, 602, 603, 1903, 2511-2514 Point 131-134, 322, 323, 1821, 1822, 1831, 1832 End point 201 Road surface 202, 311 Pattern 203, 312, 2131 Three-dimensional object 204 Point of interest 205 Vehicle 206 Camera 212-214 Direction 321, 325, 326, 1801, 1802, 2101, 1022 Contour line 324, 502, 504 Dividing points 402, 403, 902 to 905, 1803 Partial contour lines 501, 503, 601 Region 701 Image analysis device 711 Extraction unit 712 Determination unit 713 Identification unit 901-1 to 901 -N Combination 1001 Imaging device 1011 Video input unit 1012 Force unit 1013 Storage unit 1021 Feature point extraction unit 1022 Contour line extraction unit 1031 Collation unit 1032 Generation unit 1033 Difference calculation unit 1034 Similarity calculation unit 1035 Division point determination unit 1042 Feature point image 1043 Outline line information 1044 Candidate list 1045 Division score 1046 Feature similarity 1047 Detection result 1301-1303 Pixel 1811-1813 Line 1823 Intersection 1901, 1902 Line segment 2111-2115, 2121-2125 Edge point 2501-2510, 2521-2528 Luminance value distribution 2701 CPU
2702 Memory 2703 Input device 2704 Output device 2705 Auxiliary storage device 2706 Medium drive device 2707 Network connection device 2708 Bus 2709 Portable recording medium

Claims (10)

An image feature of the first region is extracted from a first image taken at a first time by an imaging device installed on a moving body moving on the surface, and a second time different from the first time by the imaging device. Extracting the image features of the second region from the second image taken in
Based on the image characteristics of the first area, a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface is obtained,
Based on the amount of movement of the moving body between the first time and the second time, a second position that is a candidate for a boundary in the second region corresponding to the first position is obtained,
The image characteristics of the first area when the first position is used as the boundary in the first area, and the second area when the second position is used as the boundary in the second area. Determining the first position as the boundary in the first region based on a feature similarity with an image feature;
Identifying a region of the three-dimensional object in the first region based on the determined boundary;
An image analysis program that causes a computer to execute processing.   The computer obtains a plurality of positions including the first position as the boundary candidates in the first area, and determines a plurality of positions including the second position as the boundary candidates in the second area. The image characteristics of the first area when each of the plurality of positions in the first area is used as the boundary in the first area, and each of the plurality of positions in the second area are obtained as the first position. A feature evaluation value including a feature similarity with the image feature of the second area when used as the boundary in two areas is obtained, and the first position is used as the boundary in the first area. Based on the result of comparing the feature evaluation value and the feature evaluation value when a position other than the first position is used as the boundary in the first region, the first position is determined in the first region. The request is determined to be the boundary. Claim 1, wherein the image analysis program.   The image feature of the first area is a third edge line segment including a first edge line segment of the specific area on the surface and a second edge line segment in the height direction of the three-dimensional object, The image feature of the area is a fourth edge line segment included in the second area, and the plurality of positions in the first area are the first edge line segment and the first edge line on the third edge line segment. A plurality of positions in the second area are candidates for boundaries on the fourth edge line segment, and the computer is configured to select the plurality of positions in the first area. Are used as the boundary on the third edge line segment, and each of the plurality of positions in the second region is used as the boundary on the fourth edge line segment, and the third edge line segment and The feature similarity with the fourth edge line segment is obtained. Image analysis program of claim 2 wherein.   The computer corresponds to the third edge line segment based on the movement amount of the moving body and installation information of the imaging device from among a plurality of edge line segments extracted from the second image. A fourth edge line segment is identified, and using the condition that the plurality of positions in the first area are on the second edge line segment, the first edge line segment is determined from each of the plurality of positions in the first area. The image analysis program according to claim 3, wherein each of the plurality of positions in two regions is obtained.   The computer includes a pixel value of a first partial edge line segment between one end point of the third edge line segment and the boundary, and a first value between the one end point of the fourth edge line segment and the boundary. A first pixel value similarity indicating a similarity with a pixel value of a two partial edge line segment; a pixel value of a third partial edge line segment between the other end point of the third edge line segment and the boundary; Based on the second pixel value similarity indicating the similarity between the other end point of the fourth edge line segment and the pixel value of the fourth partial edge line segment between the boundary and the third edge line segment, 5. The image analysis program according to claim 3, wherein the feature similarity with the fourth edge line segment is obtained.   The computer includes a first inclination similarity indicating a similarity between the inclination of the first partial edge line segment and the inclination of the second partial edge line segment, the inclination of the third partial edge line segment, and the fourth part. 6. The feature similarity between the third edge line segment and the fourth edge line segment is obtained based on a second slope similarity indicating a similarity with the slope of the edge line segment. The image analysis program described.   The computer is arranged on the third edge line segment based on a first pixel value difference indicating a difference between the pixel value of the first partial edge line segment and the pixel value of the third partial edge line segment. A first evaluation value for the boundary of the second partial edge line segment, and based on a second pixel value difference degree indicating a difference degree between the pixel value of the second partial edge line segment and the pixel value of the fourth partial edge line segment, A second evaluation value for the boundary on a four-edge line segment is obtained, the feature similarity between the third edge line segment and the fourth edge line segment, the first evaluation value, and the second evaluation value; The image analysis program according to claim 5 or 6, wherein the feature evaluation value is obtained by using the function.   The computer calculates the first evaluation value based on a first inclination difference indicating a difference between the inclination of the first partial edge line segment and the inclination of the third partial edge line segment, and the second part 8. The image analysis program according to claim 7, wherein the second evaluation value is obtained based on a second inclination dissimilarity indicating a dissimilarity between an inclination of an edge line segment and the inclination of the fourth partial edge line segment. . An image feature of the first region is extracted from a first image taken at a first time by an imaging device installed on a moving body moving on the surface, and a second time different from the first time by the imaging device. An extraction unit for extracting image features of the second region from the second image taken in
Based on the image characteristics of the first area, a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface is obtained, and the first time And a second position that is a candidate for a boundary in the second region corresponding to the first position based on a movement amount of the moving body between the second time and the second time, and Feature similarity between the image feature of the first region when used as the boundary in one region and the image feature of the second region when the second position is used as the boundary in the second region A determination unit for determining the first position as the boundary in the first region based on:
Based on the boundary determined by the determining unit, a specifying unit that specifies a region of the three-dimensional object in the first region;
An image analysis apparatus comprising: Computer
An image feature of the first region is extracted from a first image taken at a first time by an imaging device installed on a moving body moving on the surface, and a second time different from the first time by the imaging device. Extracting the image features of the second region from the second image taken in
Based on the image characteristics of the first area, a first position that is a candidate for a boundary between the specific area on the surface included in the first area and the three-dimensional object existing on the surface is obtained,
Based on the amount of movement of the moving body between the first time and the second time, a second position that is a candidate for a boundary in the second region corresponding to the first position is obtained,
The image characteristics of the first area when the first position is used as the boundary in the first area, and the second area when the second position is used as the boundary in the second area. Determining the first position as the boundary in the first region based on a feature similarity with an image feature;
Identifying a region of the three-dimensional object in the first region based on the determined boundary;
An image analysis method characterized by the above.