JP5251460B2 - Approaching object detection device, approaching object detection program, approaching object detection method - Google Patents

Description

  The present invention relates to an approaching object detection device that detects an approaching object, an approaching object detection program, and an approaching object detection method.

  An in-vehicle blind spot camera image display device (blind corner) that displays on the in-vehicle display a vehicle left-right image acquired from a camera installed at the front end of the vehicle, etc., for the purpose of assisting the vehicle user in an intersection with poor visibility. Monitor).

  However, the conventional in-vehicle blind spot camera image display device has the following properties with respect to an object approaching from a distance.

(1) Since the image area size of the approaching object is small, the user can easily blend into the background and hardly notice the shape of the approaching vehicle, motorcycle, or the like.
(2) Since it is an approaching object, the amount of movement of the image area is small, so the user is less likely to notice the change.

  On the other hand, there has been a type of device that detects an approaching object by image processing from video data from a camera and highlights an image area of the approaching object. For example, using the optical flow method, using the relationship between the approaching object and the field of view of the camera, extracting the flow area that occurs in the center direction on the screen and identifying it as the approaching object area, realizing approaching object detection There is a device to do.

  In addition, the gyro sensor or vehicle sensor is used to calculate the horizontal flow component due to turning so that it can be detected even during turning, and this component is subtracted from the optical flow to perform flow correction processing. There is a device that realizes approaching object detection by determining whether or not it is in the upper center direction.

As a conventional technique, there is a nose view monitor device that can accurately detect an approaching object on the side of a nose portion of a vehicle when the vehicle is turning (see, for example, Patent Document 1).
JP 2005-276057 A

  However, depending on the stop position of the host vehicle, the flow toward the center of the screen does not necessarily occur in the approaching object due to the relationship between the position of the camera and the approaching object. For example, if the vehicle has entered the intersection at a position or angle where the end point of the approaching object looks constant with respect to the camera, or if a pedestrian, bicycle, or motorcycle approaches the wall, Since there was no flow toward the center, there were cases where detection was missed.

  In addition, if the turning speed is faster than the speed of the approaching vehicle during turning, subtracting the horizontal flow component due to virtual turning obtained from the gyro sensor or vehicle sensor will cause the residual correction flow to face outward. In some cases, it was a detection failure.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide an approaching object detection device, an approaching object detection program, and an approaching object detection method that improve the accuracy of approaching object detection.

  In order to solve the above-described problem, one aspect of the present invention is an approaching object detection device that detects an approaching object, and includes a captured first image and a second image captured before the first image. An image acquisition unit to acquire, and a search unit to search for a first point in the first image corresponding to a second point in the second image based on the first image and the second image acquired by the image acquisition unit; An optical flow calculation unit for calculating the optical flow from the second point to the first point searched by the search unit, and a vertical component of the optical flow from the second point to the first point calculated by the optical flow calculation unit. And a determination unit that determines whether or not the first point indicates an approaching object.

  One embodiment of the present invention is an approaching object detection program that causes a computer to detect an approaching object, and obtains a captured first image and a second image captured before the first image, Based on the acquired first image and second image, the first point in the first image corresponding to the second point in the second image is searched, and the optical from the searched second point to the first point is searched. Based on the calculated vertical component of the optical flow from the second point to the first point, the computer is caused to determine whether or not the first point indicates an approaching object.

  One embodiment of the present invention is an approaching object detection method for detecting an approaching object, which is obtained by acquiring a captured first image and a second image captured before the first image. Based on the first image and the second image, the first point in the first image corresponding to the second point in the second image is searched, and the optical flow from the searched second point to the first point is calculated. Then, based on the calculated vertical component of the optical flow from the second point to the first point, it is determined whether or not the first point indicates an approaching object.

  Moreover, what applied the component of this invention, or arbitrary combinations of a component to a method, an apparatus, a system, a recording medium, a data structure, etc. is also contained in this invention.

  According to the approaching object detection device, approaching object detection program, and approaching object detection method of the disclosure, it is possible to improve the accuracy of the approaching object detection.

  Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment 1)
In the present embodiment, an approaching object display device to which the approaching object detection device of the present invention is applied will be described. The approaching object display device of the present embodiment is mounted on a vehicle. Hereinafter, a vehicle equipped with the approaching object display device is referred to as a host vehicle.

  The approaching object display device according to the present embodiment calculates a vertical (vertical) component of the optical flow from continuously captured image data, and detects an approaching object based on the calculated vertical component.

  The configuration of the approaching object display device according to the first embodiment will be described below.

  FIG. 1 is a block diagram illustrating an example of the configuration of the approaching object display device according to the first embodiment. The approaching object display device includes a camera 11, an image processing unit 12 a, a display unit 14, a CPU, a storage unit 16, an angular velocity sensor 17 (first motion measurement unit, second motion measurement unit), and a vehicle speed sensor 18.

  The image processing unit 12a is realized by a CPU (Central Processing Unit), for example. This CPU implements an approaching object detection function by executing an approaching object detection program stored in the storage unit 16.

  Instead of the angular velocity sensor 17, an angular velocity sensor provided outside the approaching object display device may be used. In place of the vehicle speed sensor 18, a vehicle speed sensor provided outside the approaching object display device may be used. Here, the angular velocity sensor is used to acquire the angular velocity. However, in addition to the angular velocity sensor, a sensor that performs measurement related to the vertical motion of the camera 11 such as an acceleration sensor may be used.

  The configuration of the image processing unit 12a will be described below.

  FIG. 2 is a block diagram illustrating an example of the configuration of the image processing unit 12a. The image processing unit 12a includes a feature point extraction unit 21, an optical flow calculation unit 22, a feature point information recording unit 23, a pitching flow calculation unit 31 (motion component calculation unit), and a pitch correction unit 32 (optical flow correction unit). , An approaching object determination unit 33a (determination unit, region determination unit), a video acquisition unit 34 (image acquisition unit), a vehicle state acquisition unit 35 (measurement value acquisition unit), and a video synthesis unit 36 (image generation unit).

  The vehicle state acquisition unit 35 acquires the angular velocity measured by the angular velocity sensor 17 and the vehicle speed measured by the vehicle speed sensor 18.

  The operation of the approaching object display device of the present embodiment will be described below.

  The camera 11 has a prism lens adjusted so as to photograph the left and right direction of the host vehicle, and photographs at a predetermined frame time interval. By providing the camera 11 in the left-right direction of the host vehicle, it is possible to take an image of the traffic situation on the left and right roads that are the blind spot of the user at an intersection with poor visibility. The frame time interval is 1/30 seconds, for example. The camera 11 may have a fisheye lens that can photograph the left and right direction of the host vehicle.

  FIG. 3 is a flowchart illustrating an example of the operation of the approaching object display device according to the first embodiment. The video acquisition unit 34 acquires video data captured by the camera 11 at every lam time interval (S11). A plurality of cameras 11 may be provided. In this case, the video acquisition unit 34 synthesizes the video shot by each of the cameras 11 and generates video data. For example, the video from the camera 11 installed facing the left side of the host vehicle is arranged on the left half of the video data, and the video from the camera 11 installed facing the right side of the host vehicle is displayed on the right half of the video data. To place.

  Next, the feature point extraction unit 21 extracts feature points (attention points) in the video data (S12). For the feature point extraction, for example, a corner extraction process that reacts well to the uneven region of the vehicle outer shape is used. For the corner extraction process, a Harris operator or a KLT (Kanade-Lucas Track) operator is used.

  The feature point extraction unit 21 records information about the extracted feature points in the feature point information recording unit 23 as feature point information (S13). Here, the feature point information recording unit 23 is realized by the storage unit 16. FIG. 4 is a table showing an example of the configuration of the feature point information according to the first embodiment. This figure shows feature point information at time t and feature point information at time t + 1. The feature point information includes a feature point ID, peripheral image data (eg, 9 × 9 pixels) that is image data around the latest coordinate value of the feature point, and (x, y) of the corresponding point of the feature point for each frame. And a corresponding point list which is a list of coordinates and frame times.

  Next, the optical flow calculation unit 22 corresponds to the feature point in the current video data I (t + 1) based on the latest coordinate value of the feature point and the peripheral image data registered in the feature point information recording unit 23. Corresponding points that are points to be searched are searched within a predetermined range (S14). Next, the optical flow calculation unit 22 calculates an optical flow by, for example, a block matching method (S15). The optical flow calculation unit 22 adds the coordinate value of the corresponding point searched at I (t + 1) and the frame time t + 1 to the list in the feature point information, and the surrounding image data in the feature point information is the surrounding image at I (t + 1). The data is updated (S16).

  Here, the video data I (t) at time t corresponds to the second image. The video data I (t + 1) at time t + 1 corresponds to the first image.

  Next, the approaching object determination unit 33a selects a feature point satisfying a predetermined vertical component condition as the selected feature point for the latest feature point registered in the feature point information recording unit 23 (S17). Ends. Here, the vertical component condition is that the magnitude of the vertical component of the optical flow exceeds a predetermined vertical component threshold. The optical flow is calculated by subtraction of two coordinate values in the registered list. The first vertical component condition is expressed by the following equation using the vertical component of the optical flow and the vertical component threshold Th1.

| Y _{i, t + 1} −y _{i, t} |> Th1

  In addition, the optical flow calculation unit 22 calculates a long-time optical flow that is an optical flow from the past corresponding point to the latest corresponding point by the time interval n as the vertical component condition. When the approaching object determination unit 33a determines that the vertical component (motion amount) of the optical flow at the unit time interval (frame time interval) is small, the approaching object determination unit 33a selects the selected feature point using the second vertical component condition. The second vertical component condition is that the vertical component size of the long-time optical flow exceeds a predetermined vertical component threshold Th2. The second vertical component condition is expressed by the following equation using the vertical component of the long-time optical flow and the vertical component threshold Th2.

| Y _{i, t + 1} −y _{i, t−n + 1} |> Th2

  Here, the video data I (t−n + 1) at time t−n + 1 corresponds to the third image.

  Th1 and Th2 are, for example, one pixel.

  Thus, when the speed component at which the approaching object approaches the host vehicle is small, the detection accuracy of the selected feature point can be increased by calculating the optical flow for a long time.

  The image processing unit 12a of the present embodiment is a vertical component that is a vertical component of the background flow (background optical flow) that occurs when the vehicle is pitched (moves in the vertical direction) due to the brakes of the host vehicle or the unevenness of the road surface. The shaking flow is removed from the optical flow.

  The vehicle state acquisition unit 35 acquires the angular velocity measured by the angular velocity sensor 17. The pitching flow calculation unit 31 calculates the pitching flow based on the angular velocity obtained by the vehicle state acquisition unit 35. The pitch correction unit 32 subtracts the pitch flow from the vertical component of the optical flow to obtain a vertical residual flow. The approaching object determination unit 33a determines the vertical component condition for the vertical residual flow instead of the vertical component of the optical flow. That is, the first vertical component condition is that the magnitude of the vertical residual flow exceeds Th1.

  Similarly, the pitch correction unit 32 subtracts the pitch flow from the vertical component of the long-time optical flow to obtain a long-time vertical residual flow. The second vertical component condition is that the magnitude of the long-time vertical residual flow exceeds Th2.

  Here, details of operations of the pitching flow calculation unit 31 and the pitching correction unit 32 when the vehicle state acquisition unit 35 acquires the triaxial angular velocity from the angular velocity sensor 17 will be described.

  When the time interval (frame time interval) for evaluating the vertical component is Δt for a certain feature point, the pitching flow calculation unit 31 performs the calculation for a certain coordinate axis obtained from the angular velocity sensor from time t−Δt to time t. The displacement angle W is calculated by integrating the angular velocity w. W is expressed by the following equation.

W = Σ _t- Δ _t ^t w

  Next, the focal length of the camera 11 is f, and the pitch velocity flow calculation unit 31 performs angular velocity wx (pitch) (first measurement value), wy (roll), wz (yaw) (second measurement value) about three axes. ) Is obtained, and thereby displacement angles Wx (pitch), Wy (roll), and Wz (yaw) are obtained, the pitching flow vy at the point (x, y) can be calculated by the following equation.

vy =
(Y * y / f + f) * Wx-x * y * Wy / fx-Wz;

  Even when the camera 11 and the angular velocity sensor 17 are separated from each other, the pitching flow calculation unit 31 can calculate the pitching flow with high accuracy by using the triaxial angular velocity.

  When the vehicle state acquisition unit 35 acquires only the angular velocity in the vertical direction (pitch) from the angular velocity sensor 17, Wy = Wz = 0 may be used in the above vy calculation formula.

  When the camera 11 and the angular velocity sensor 17 are close and perform the same motion, the pitching flow calculation unit 31 can calculate the pitching flow using only the angular velocity in the vertical direction.

  The pitch correction unit 32 subtracts the pitch flow from the vertical component of the optical flow to calculate the vertical residual flow V. V is expressed by the following equation.

V = y _{i, t + 1} −y _{i, t−n + 1} −vy

  The approaching object determination unit 33a determines whether or not the vertical component condition represented by the following equation is satisfied.

  | V |> Th1

  Next, the approaching object determination unit 33a performs a clustering process on the coordinates of the selected selected feature point on the two-dimensional image, specifies the approaching object region, and sets it as the approaching object region. For example, the clustering process is a process in which feature points within a predetermined image distance are regarded as one cluster.

The video composition unit 36 synthesizes the approaching object region output from the approaching object determination unit 33a with the video data from the camera 11 to obtain composite video data (composite image). The display unit 14 displays the synthesized video data synthesized by the video synthesis unit 36.
(Embodiment 2)

  The approaching object display device according to the present embodiment calculates the vertical and horizontal components of the optical flow from continuously captured images, and detects the approaching object based on the calculated vertical and horizontal components.

  The configuration of the approaching object display device according to the second embodiment includes an image processing unit 12b instead of the image processing unit 12a as compared with the configuration of the approaching object display device according to the first embodiment.

  The configuration of the image processing unit 12b will be described below.

  FIG. 5 is a block diagram illustrating an example of the configuration of the image processing unit 12b. In this figure, the same reference numerals as those in FIG. 2 denote the same or corresponding parts as those in FIG. 2, and the description thereof will be omitted here. Compared with the image processing unit 12a, the image processing unit 12b includes an approaching object determination unit 33b instead of the approaching object determination unit 33a, and newly includes a turning flow calculation unit 41 (motion component calculation unit) and a turning correction unit 42. (Optical flow correction unit).

  For example, the angular velocity sensor 17 measures the angular velocity wz (second measured value) of turning (horizontal direction, yaw) of the host vehicle in which the camera 11 is installed, in addition to the angular velocity in the vertical direction. The turning flow calculation unit 41 calculates a turning flow that is a horizontal component of the background flow due to turning on the screen, based on the angular velocity acquired by the vehicle state acquisition unit 35. The turning correction unit 42 subtracts the calculated turning flow from the horizontal component of the optical flow to obtain a horizontal residual flow.

  The approaching object determination unit 33b selects a feature point that satisfies a predetermined vertical component condition or a predetermined horizontal component condition as a selected feature point. Here, the vertical component condition is that the magnitude of the vertical residual flow exceeds a predetermined vertical component threshold. The horizontal component condition is that the horizontal residual flow calculated by the turning correction unit 42 exceeds a predetermined horizontal component threshold value with the center direction as a positive direction.

  Note that the vertical component condition may be that the size of the vertical residual flow exceeds a predetermined vertical component threshold and that the vertical residual flow is directed toward the center.

  The approaching object determination unit 33b may further select the selected feature point using the vehicle speed measured by the vehicle speed sensor 18. In this case, the approaching object determination unit 33b first acquires the vehicle speed measured by the vehicle speed sensor 18. Next, the approaching object determination unit 33b determines whether or not the vehicle speed is below a predetermined vehicle speed threshold. When the vehicle speed falls below a predetermined vehicle speed threshold, a feature point that satisfies the horizontal component condition is selected as a selected feature point. When the vehicle speed is equal to or higher than the vehicle speed threshold, a feature point that satisfies the vertical component condition is selected as a selected feature point.

  Specific examples of the video data and the optical flow in each of the above-described embodiments will be described below.

  Here, with the prism lens provided in the camera 11, the left area, which is the left half area of the video data, is an image of the left side of the host vehicle, and the right area, which is the right half area of the video data, is This is an image of the right side of the vehicle. Hereinafter, a line dividing the left area and the right area in the video data is referred to as a center line.

  A comparative example in which an approaching object is detected by determining whether or not the horizontal component of the optical flow is in the center line direction will be described below.

  FIG. 6 is a plan view showing an example of the positional relationship between the host vehicle and the approaching object in the first state. This figure shows a host vehicle 62, a structure 63, camera fields of view 64a and 64b, approaching object positions 65a and 65b, and feature points 66a and 66b. The camera view 64a corresponds to the left half of the video data, and the camera view 64b corresponds to the right half of the video data. In this figure, an X axis and a Z axis are coordinate systems based on the host vehicle. The origin O indicates the imaging position of the camera 11, the X axis indicates the rear of the host vehicle, and the Z axis indicates the right side of the host vehicle. The feature point 66b is a corresponding point of the feature point 66a.

  The first state is a state in which the host vehicle 62 pauses before entering the intersection, and the approaching object in the camera view 64a moves from the approaching object position 65a to 65b and approaches the host vehicle 62. The approaching object position 65a indicates the position of the approaching object imaged at time t1, and the approaching object position 65a indicates the position of the approaching object imaged at time t1 + 1 (the time of the next frame after time t1).

  FIG. 7 is a conceptual diagram illustrating an example of an optical flow in the first state. This figure shows an optical flow horizontal component 71b of the feature point 66b, a swirl flow 72b, and a horizontal residual flow 73b that is the result of subtracting the swirl flow 72b from the horizontal component 71b of the optical flow. In the first state, the horizontal residual flow 73b points in the direction of the center line. Therefore, the feature point 66b can be detected as an approaching object by the horizontal component of the optical flow.

  FIG. 8 is a conceptual diagram showing an example of video data and an optical flow in the second state. The second state is a state in which the host vehicle 62 is temporarily stopped before entering the intersection, and the approaching object 61a in the camera field of view 64a and the approaching object 61b in the camera field of view 64b are approaching the host vehicle 62. In this figure, a black arrow represents an optical flow. In the second state, the horizontal component of the optical flow of each feature point of the approaching objects 61a and 61b indicates the direction of the center line. Accordingly, the approaching objects 61a and 61b can be detected as approaching objects by determining whether or not the horizontal component of the optical flow is in the center line direction.

  FIG. 9 is a plan view showing an example of the positional relationship between the host vehicle and the approaching object in the third state. This figure is similar to the first state described above, but shows approaching object positions 65c and 65d instead of approaching object positions 65a and 65b, and feature points 66c and 66d instead of feature points 66a and 66b.

  The third state is a state in which the host vehicle 62 pauses before entering the intersection, and the approaching object in the camera view 64a moves from the approaching object position 65c to 65d and approaches the host vehicle 62. The approaching object position 65c indicates the position of the approaching object imaged at time t2, and the approaching object position 65d indicates the position of the approaching object imaged at time t2 + 1 (the time of the next frame after time t1). In the third state, the horizontal component of the optical flow from the feature point 66c to the feature point 66d points in the direction opposite to the center line. Therefore, if only the determination of whether or not the horizontal component of the optical flow is in the center line direction is used, the feature point 66d is not detected as an approaching object.

  FIG. 10 is a conceptual diagram illustrating an example of an optical flow in the third state. This figure shows the horizontal component 71d of the optical flow at the feature point 66d, the swirling flow 72d, and the horizontal residual flow 73d that is the result of subtracting the swirling flow 72d from the horizontal component 71d of the optical flow. In the third state, the horizontal residual flow 73d points in the direction opposite to the center line. Therefore, if only the determination of whether or not the horizontal component of the optical flow is in the center line direction is used, the feature point 66d is not detected as an approaching object.

  FIG. 11 is a plan view showing an example of the positional relationship between the host vehicle and the approaching object in the fourth state. This figure is similar to the first state described above, but shows approaching object positions 65e and 65f instead of approaching object positions 65a and 65b, and feature points 66e and 66f instead of feature points 66a and 66b.

  The fourth state is a state in which the host vehicle 62 enters the intersection and turns right, and the approaching object in the camera view 64a moves from the approaching object position 65e to 65f and approaches the host vehicle 62. The approaching object position 65e indicates the position of the approaching object imaged at time t3, and the approaching object position 65f indicates the position of the approaching object imaged at time t3 + 1 (the time of the next frame after time t3). In the fourth state, there is almost no horizontal component of the optical flow from the feature point 66e to the feature point 66f. Therefore, if only the determination of whether or not the horizontal component of the optical flow is in the direction of the center line is used, the feature point 66f is not detected as an approaching object.

  FIG. 12 is a conceptual diagram illustrating an example of video data and an optical flow in the fifth state. The fifth state is a state in which the host vehicle 62 enters the intersection and turns right, and the approaching object 61a in the camera field of view 64a and the approaching object 61b in the camera field of view 64b are approaching the host vehicle 62. In this figure, a black arrow represents an optical flow. In the fifth state, the horizontal component of the optical flow of each feature point of the approaching objects 61a and 61b points to the left as the host vehicle turns to the right.

  FIG. 13 is a conceptual diagram illustrating an example of video data and a turning flow in the fifth state. In this figure, a black arrow represents a turning flow. In the fifth state, the turning flow of each feature point of the approaching objects 61a and 61b indicates the left direction due to the right turn of the host vehicle.

  FIG. 14 is a conceptual diagram illustrating an example of video data and a horizontal residual flow in the fifth state. In this figure, the black arrow represents the horizontal residual flow. In the fifth state, the horizontal residual flow of each feature point of the approaching objects 61a and 61b points to the left due to the horizontal component of the optical flow and the turning flow. Therefore, if only the determination of whether or not the horizontal component of the optical flow is in the center line direction is used, the approaching object 61b is detected as an approaching object, but the approaching object 61a is not detected as an approaching object.

  A specific example in the case of detecting an approaching object based on the vertical component of the optical flow as shown in the first and second embodiments will be described below.

  FIG. 15 is a conceptual diagram showing an example of video data and an optical flow in the sixth state. In the sixth state, the host vehicle 62 enters the intersection and turns right, the approaching object 61a in the camera view 64a and the approaching object 61b in the camera view 64b approach the own vehicle 62, and the camera 11 swings upward. It is in the state. In this figure, a black arrow represents an optical flow. In the sixth state, the horizontal component of the optical flow of each feature point of the approaching objects 61a and 61b indicates the left direction due to the right turn of the host vehicle.

  FIG. 16 is a conceptual diagram illustrating an example of vertical components of video data and an optical flow in the sixth state. In this figure, the black arrow represents the vertical component of the optical flow. In the sixth state, the vertical component of the optical flow other than the approaching objects 61a and 61b (background) indicates the downward direction due to the camera 11 shaking upward.

  FIG. 17 is a conceptual diagram illustrating an example of video data and pitching flow in the sixth state. In this figure, the black arrow represents the pitching flow. In the sixth state, each pitch flow indicates a downward direction due to the camera 11 swinging upward.

  FIG. 18 is a conceptual diagram illustrating an example of video data and a vertical residual flow in the sixth state. In this figure, the black arrow represents the vertical residual flow. In the sixth state, the vertical residual flow other than the approaching objects 61 a and 61 b (background) is almost eliminated by the processing of the pitch correction unit 32. Further, only the vertical residual flow of each feature point of the approaching objects 61a and 61b appears greatly. Therefore, the approaching objects 61a and 61b are detected as approaching objects by determining the magnitude of the vertical component of the optical flow as described in the first and second embodiments.

  According to each of the above-described embodiments, when focusing only on the horizontal component facing the center of the screen, even for an approaching object that is difficult to detect in principle due to the relationship between the camera and the approaching object, the approach is focused on the vertical component of the optical flow. Detection is possible by extracting an object.

  Further, according to each of the above-described embodiments, it is possible to detect an approaching object even when a pitching occurs in the host vehicle or the camera.

  Further, according to the second embodiment described above, it is possible to detect an approaching object even when the turning speed of the host vehicle is high.

  Further, according to the second embodiment described above, when the horizontal movement amount of the approaching object in the image is larger than the vertical movement amount, it is detected earlier if the selected feature point is extracted using the horizontal component condition. It becomes possible. According to the second embodiment, it is possible to improve the detection performance of the selected feature point by making an integrated determination using both the vertical component and the horizontal component of the optical flow.

  In addition, according to each of the above-described embodiments, by extracting and tracking feature points and selecting selected feature points from the feature points, it is not necessary to perform calculation processing on all points on the image, and calculation should be performed. Since the pixels can be effectively selected, it can be realized at low cost.

  Moreover, the approaching object detection device according to the present invention can be easily applied to a camera of a moving body, a fixed camera at an intersection, and the like, and can further improve the performance of the camera.

  The present invention can be applied to the following computer system. FIG. 19 is a diagram illustrating an example of a computer system to which the present invention is applied. A computer system 900 shown in this figure includes a main body 901 incorporating a CPU, a disk drive, and the like, a display 902 that displays an image according to an instruction from the main body 901, a keyboard 903 for inputting various information to the computer system 900, A mouse 904 for designating an arbitrary position on the display screen 902a of the display 902 and a communication device 905 for accessing an external database or the like and downloading a program or the like stored in another computer system are provided. The communication device 905 may be a network communication card, a modem, or the like.

  A program for executing the above-described steps in the computer system constituting the approaching object detection device as described above can be provided as an approaching object detection program. By storing this program in a recording medium readable by the computer system, the program can be executed by the computer system constituting the approaching object detection device. A program for executing the above steps is stored in a portable recording medium such as a disk 910 or downloaded from a recording medium 906 of another computer system by the communication device 905. Further, an approaching object detection program for causing the computer system 900 to have at least an approaching object detection function is input to the computer system 900 and compiled. This program causes the computer system 900 to operate as an approaching object detection system having an approaching object detection function. Further, this program may be stored in a computer-readable recording medium such as a disk 910, for example. Here, examples of the recording medium readable by the computer system 900 include an internal storage device such as a ROM and a RAM, a portable storage such as a disk 910, a flexible disk, a DVD disk, a magneto-optical disk, and an IC card. It includes a medium, a database holding a computer program, or other computer systems and the database, and various recording media accessible by a computer system connected via communication means such as a communication device 905.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The scope of the present invention is shown by the scope of claims, and is not restricted by the text of the specification. Moreover, all modifications, various improvements, substitutions and modifications belonging to the equivalent scope of the claims are all within the scope of the present invention.

The following additional notes are further disclosed with respect to the first and second embodiments.
(Appendix 1)
An approaching object detection device that detects an approaching object,
An image acquisition unit for acquiring a captured first image and a second image captured before the first image;
A search unit for searching for a first point in the first image corresponding to a second point in the second image based on the first image and the second image acquired by the image acquisition unit;
An optical flow calculation unit that calculates an optical flow from the second point to the first point searched by the search unit;
A determination unit that determines whether or not the first point indicates an approaching object based on a vertical component of the optical flow from the second point to the first point calculated by the optical flow calculation unit;
An approaching object detection device comprising:
(Appendix 2)
The determination unit determines whether the first point indicates an approaching object based on the magnitude of the vertical component of the optical flow to the first point calculated by the optical flow calculation unit.
The approaching object detection device according to appendix 1.
(Appendix 3)
The image acquisition unit further acquires a third image taken before the second image,
The search unit is further configured to, based on the third image, the first image, and the second image acquired by the image acquisition unit, in the first image corresponding to a third point in the third image. Search for the first point,
The optical flow calculation unit calculates an optical flow from the third point searched by the search unit to the first point;
When the magnitude of the vertical component of the optical flow from the second point to the first point calculated by the optical flow calculation unit is smaller than a predetermined vertical component threshold, the determination unit starts from the third point. Based on the vertical component of the optical flow to one point, it is determined whether or not the first point indicates an approaching object.
The approaching object detection device according to attachment 2.
(Appendix 4)
A measurement value acquisition unit that acquires a first measurement value related to the vertical movement of the approaching object detection device;
Based on the first measurement value acquired by the measurement value acquisition unit, a motion component calculation unit that calculates a first component that is a component of the optical flow due to vertical motion;
An optical flow correction unit that corrects the vertical component of the optical flow by subtracting the first component calculated by the motion component calculation unit from the vertical component of the optical flow;
Further comprising
The determination unit determines whether the first point indicates an approaching object based on a vertical component of the optical flow to the first point corrected by the optical flow correction unit.
The approaching object detection device according to appendix 1.
(Appendix 5)
The measurement value acquisition unit further acquires a second measurement value related to a horizontal movement of the approaching object detection device,
The motion component calculation unit further calculates a second component that is a component of the optical flow caused by a horizontal motion based on the second measurement value acquired by the measurement value acquisition unit,
The optical flow correction unit further corrects the horizontal component of the optical flow by subtracting the second component calculated by the motion component calculation unit from the horizontal component of the optical flow,
The determination unit determines whether the first point indicates an approaching object based on a vertical component and a horizontal component of the optical flow to the first point corrected by the optical flow correction unit.
The approaching object detection device according to appendix 4.
(Appendix 6)
The determination unit determines whether the first point indicates an approaching object based on the vertical component size of the optical flow to the first point calculated by the optical flow calculation unit and the direction and size of the horizontal component. To determine,
The approaching object detection device according to appendix 1.
(Appendix 7)
The measurement value acquisition unit further acquires the moving speed of the approaching object detection device,
The determination unit selects at least one of a vertical component and a horizontal component of the optical flow to the first point based on the moving speed acquired by the measurement value acquisition unit as a selection component. Based on whether the first point indicates an approaching object,
The approaching object detection device according to appendix 6.
(Appendix 8)
The determination unit selects, as the selection component, a vertical component of an optical flow to the first point when the moving speed is greater than a predetermined speed threshold.
The approaching object detection device according to appendix 7.
(Appendix 9)
A feature point extraction unit that extracts a feature point from the first image as the first point and extracts a feature point from the second image as the second point;
In addition,
The approaching object detection device according to appendix 1.
(Appendix 10)
The feature point extraction unit extracts a feature point by a corner extraction process;
The approaching object detection device according to appendix 9.
(Appendix 11)
An area determination unit that determines an area indicating the approaching object based on a plurality of first points determined to indicate the approaching object by the determination unit;
In addition,
The approaching object detection device according to appendix 1.
(Appendix 12)
The region determination unit determines a region indicating the approaching object by clustering processing on a plurality of first points determined to indicate the approaching object by the determination unit.
The approaching object detection device according to appendix 1.
(Appendix 13)
An image generation unit that generates a combined image obtained by combining the region determined by the region determination unit and the second image;
In addition,
The approaching object detection device according to attachment 12.
(Appendix 14)
A display unit for displaying the composite image generated by the image generation unit;
In addition,
The approaching object detection device according to attachment 13.
(Appendix 15)
A camera that captures the first image and the second image;
In addition,
The approaching object detection device according to appendix 1.
(Appendix 16)
A first motion measurement unit that measures the first measurement value related to the vertical motion of the approaching object detection device;
In addition,
The approaching object detection device according to appendix 1.
(Appendix 17)
A second motion measurement unit that measures the second measurement value related to the horizontal motion of the approaching object detection device;
In addition,
The approaching object detection device according to appendix 16.
(Appendix 18)
The first motion measurement unit is an angular velocity sensor.
The approaching object detection device according to appendix 16.
(Appendix 19)
An approaching object detection program for causing a computer to detect an approaching object,
Obtaining a first image taken and a second image taken before the first image;
Searching for a first point in the first image corresponding to a second point in the second image based on the acquired first image and the second image;
Calculating an optical flow from the searched second point to the first point;
Based on the calculated vertical component of the optical flow from the second point to the first point, it is determined whether or not the first point indicates an approaching object.
An approaching object detection program that causes a computer to execute this.
(Appendix 20)
An approaching object detection method for detecting an approaching object,
Obtaining a first image taken and a second image taken before the first image;
Searching for a first point in the first image corresponding to a second point in the second image based on the acquired first image and the second image;
Calculating an optical flow from the searched second point to the first point;
Based on the calculated vertical component of the optical flow from the second point to the first point, it is determined whether or not the first point indicates an approaching object.
An approaching object detection method that does this.

1 is a block diagram illustrating an example of a configuration of an approaching object display device according to a first embodiment. It is a block diagram which shows an example of a structure of the image process part 12a. 6 is a flowchart illustrating an example of an operation of the approaching object display device according to the first embodiment. 4 is a table illustrating an example of a configuration of feature point information according to the first embodiment. It is a block diagram which shows an example of a structure of the image process part 12b. It is a top view which shows an example of the positional relationship of the own vehicle and approaching object in a 1st state. It is a conceptual diagram which shows an example of the optical flow in a 1st state. It is a conceptual diagram which shows an example of the video data and optical flow in a 2nd state. It is a top view which shows an example of the positional relationship of the own vehicle and approaching object in a 3rd state. It is a conceptual diagram which shows an example of the optical flow in a 3rd state. It is a top view which shows an example of the positional relationship of the own vehicle and approaching object in a 4th state. It is a conceptual diagram which shows an example of the video data in the 5th state, and an optical flow. It is a conceptual diagram which shows an example of the video data in the 5th state, and a turning flow. It is a conceptual diagram which shows an example of the video data in a 5th state, and a horizontal residual flow. It is a conceptual diagram which shows an example of the video data and optical flow in a 6th state. It is a conceptual diagram which shows an example of the vertical component of the video data in the 6th state, and an optical flow. It is a conceptual diagram which shows an example of the video data in a 6th state, and a pitching flow. It is a conceptual diagram which shows an example of the video data and vertical residual flow in a 6th state. It is a figure which shows an example of the computer system to which this invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Camera, 12a, 12b Image processing part, 14 Display part, 16 Storage part, 17 Angular velocity sensor, 18 Vehicle speed sensor, 21 Feature point extraction part, 22 Optical flow calculation part, 23 Feature point information recording part, 31 Pitch flow calculation Part, 32 pitch correction part, 33a, 33b approaching object determination part, 34 video acquisition part, 35 vehicle state acquisition part, 36 video composition part, 41 turning flow calculation part, 42 turning correction part.

Claims (5)

An approaching object detection device that detects an approaching object,
An image acquisition unit that acquires a captured first image, a second image captured before the first image, and a third image captured before the second image ;
Based on the first image , the second image , and the third image acquired by the image acquisition unit, the second point in the second image corresponds to the third point in the third image. A search unit for searching for a first point in the corresponding first image;
An optical flow calculation unit that calculates an optical flow from the second point to the first point searched by the search unit and an optical flow from the third point to the first point ;
A speed acquisition unit for acquiring a moving speed of the approaching object detection device;
Based on the moving speed acquired by the speed acquisition unit , at least one of a vertical component and a horizontal component of the optical flow from the second point to the first point calculated by the optical flow calculation unit is selected. A determination unit that determines whether or not the first point indicates an approaching object based on the size of the selection component ;
Equipped with a,
When the magnitude of the vertical component of the optical flow from the second point to the first point calculated by the optical flow calculation unit is smaller than a predetermined vertical component threshold, the determination unit starts from the third point. based on the vertical component of the optical flow to a point, the first point is you determine whether indicating the approaching object approaching object detection device. A measurement value acquisition unit that acquires a first measurement value related to the vertical movement of the approaching object detection device;
Based on the first measurement value acquired by the measurement value acquisition unit, a motion component calculation unit that calculates a first component that is a component of the optical flow due to vertical motion;
An optical flow correction unit that corrects the vertical component of the optical flow by subtracting the first component calculated by the motion component calculation unit from the vertical component of the optical flow;
Further comprising
The determination unit determines whether the first point indicates an approaching object based on a vertical component of the optical flow to the first point corrected by the optical flow correction unit.
Claim 1 Symbol placement of the approaching object detection device. The measurement value acquisition unit further acquires a second measurement value related to a horizontal movement of the approaching object detection device,
The motion component calculation unit further calculates a second component that is a component of the optical flow caused by a horizontal motion based on the second measurement value acquired by the measurement value acquisition unit,
The optical flow correction unit further corrects the horizontal component of the optical flow by subtracting the second component calculated by the motion component calculation unit from the horizontal component of the optical flow,
The determination unit determines whether the first point indicates an approaching object based on a vertical component and a horizontal component of the optical flow to the first point corrected by the optical flow correction unit.
The approaching object detection device according to claim 2 . An approaching object detection program for causing a computer to detect an approaching object,
Obtaining a captured first image, a second image captured before the first image, and a third image captured before the second image ;
Based on the acquired first image , the second image , and the third image , the first corresponding to the second point in the second image and corresponding to the third point in the third image. Search for the first point in the image,
Calculating an optical flow from the second point to the first point, and an optical flow from the third point to the first point ,
Get their own movement speed,
Based on the obtained moving speed, at least one of a vertical component and a horizontal component of the optical flow from the second point to the first point calculated is selected as a selection component, and the size of the selection component To determine whether the first point indicates an approaching object ,
If the calculated vertical component of the optical flow from the second point to the first point is smaller than a predetermined vertical component threshold, the optical flow is calculated based on the vertical component of the optical flow from the third point to the first point. Te, the first point to determine whether indicating the approaching object,
An approaching object detection program that causes a computer to execute this. An approaching object detection method for detecting an approaching object,
Obtaining a captured first image, a second image captured before the first image, and a third image captured before the second image ;
Based on the acquired first image , the second image , and the third image , the first corresponding to the second point in the second image and corresponding to the third point in the third image. Search for the first point in the image,
Calculating an optical flow from the second point to the first point, and an optical flow from the third point to the first point ,
Get their own movement speed,
Based on the obtained moving speed, at least one of a vertical component and a horizontal component of the optical flow from the second point to the first point calculated is selected as a selection component, and the size of the selection component To determine whether the first point indicates an approaching object ,
If the calculated vertical component of the optical flow from the second point to the first point is smaller than a predetermined vertical component threshold, the optical flow is calculated based on the vertical component of the optical flow from the third point to the first point. Te, the first point to determine whether indicating the approaching object,
An approaching object detection method that does this.

Images