JP5773334B2 - Optical flow processing apparatus and display radius map generation apparatus - Google Patents

Description

  The present invention relates to a technical field for confirming a traveling environment by image processing, and more particularly to a technique for extracting an object of a traveling environment that is mounted on a moving body.

The moving body is, for example, a four-wheeled vehicle, a two-wheeled vehicle, an electric vehicle, or the like. The driver usually looks forward in front of the vehicle, except during an operation for parking. When changing lanes or turning left and right, check the back with various mirrors. Further, the moving body has a blind spot from the driver. The surroundings such as the front, side, and rear of the moving body are referred to as a traveling environment.
As a system that supports the confirmation of the driving environment by the driver, a method of imaging the driving environment with a camera installed in the vehicle that is a moving body, and displaying this image or a corrected / edited image on the display unit, image processing, etc. Thus, there is a method of issuing an alarm when a certain condition is satisfied.
As image processing useful for confirming the driving environment, there is a method of extracting the same object between two images having a time difference and generating a displacement vector between the objects as an optical flow.

In Patent Document 1, an optical flow group is divided into a plurality of small areas for the purpose of extracting a moving body in a running environment using an optical flow, and a small area included in a certain radial range is moved to a single movement. A method for recognizing a body (paragraphs 0018 to 0020, 0027 to 0029, FIG. 3) is disclosed.
In Patent Document 2, for the purpose of determining the approach of the following vehicle using the optical flow, the optical flow becomes larger as the relative speed is larger and the inter-vehicle distance is shorter, and therefore, based on the average value of the size of the optical flow. A method for determining the degree of approach (paragraphs 0010, 0020 to 0022, 0029 to 0030, FIG. 4) is disclosed.
In Patent Document 3, for the purpose of tracking and monitoring an intruder using an optical flow based on template matching, a window is set around the original template, and a window with a high optical flow matching rate is circulated around the window. A method of searching and translating a template in the direction of a window having a high matching rate (paragraphs 0009 and 0043 to 0047) is disclosed.
Patent Document 4 describes the relationship between two optical flows between three temporally continuous images for the purpose of extracting only the optical flow of a moving object even when a periodic structure exists in the background of the image. Thus, a method for determining a false optical flow caused by a stationary object is disclosed.
Patent Document 5 (same applicant and inventor) discloses a method of associating the coordinate value of an image captured by a camera using a wide-angle lens with the coordinate value of the world coordinate system by correction calculation on the image plane. ing.
Patent Document 6 (the same applicant) discloses a method of performing an optimal correction calculation for each region by dividing the image into three regions concentrically for correction calculation of an image having wide-angle distortion.

JP-A-8-83345 Japanese Patent Laid-Open No. 11-259634 JP 2002-163657 A JP-A-8-30792 JP 2010-218226 A JP 2011-054008 JP

In the above-mentioned patent document 1, since the grouping of optical flows is calculated in a two-dimensional image coordinate system (paragraphs 0026 to 0028), it depends on the distance between the vehicle and the moving body. The calculation accuracy is reduced. Further, in a state where the own vehicle and the moving body are close to each other and the moving body is imaged large, only the contour portion of the moving body is extracted, so that the optical flow becomes discrete and grouping becomes difficult.
In the above Patent Document 2, since the average value of the flow size is calculated by dividing the sum of the flows by the total number of flows for the grouping of the optical flows (paragraph 0029), When the total number of flows extracted according to the change in the inter-vehicle distance changes, the error included in the average value of the flow sizes increases.
In the above-mentioned Patent Document 3, for the optimization of template matching, a large number of windows must be generated to calculate the matching rate (paragraph 0043), and the window setting is also brute force (paragraph 0044). The calculation cost for detecting the body in real time becomes enormous.
In the above-mentioned Patent Document 4, since a method for removing a false optical flow generated from a stationary object periodically generated in the background is determined based on a ratio of lengths of optical flows (paragraph 0014), The distance between the vehicle and the stationary object is vulnerable to changes and cannot be removed stably.

[Problem 1] As described above, the conventional example has a disadvantage that the effectiveness of the optical flow cannot be satisfactorily determined if there is a change in the distance between the vehicle and the moving body.
[Problem 2] Further, in the above conventional example, there is an inconvenience that the calculation cost of the optical flow processing according to the change in the inter-vehicle distance between the own vehicle and the moving body becomes enormous and difficult to execute in real time.

  [Object of the Invention] An object of the present invention is to satisfactorily determine the effectiveness of the optical flow regardless of the distance between the vehicle and the moving body.

  [Focus Point] The inventor of the present invention performs an optical flow extraction process based on a two-dimensional image, and the state in which the size in the three-dimensional space appears in the two-dimensional image in the aspect of determining the flow property We focused on the point that it would be good to be able to consider. And it came to the idea that the above problem could be solved by performing image processing in consideration of the correspondence between the size in the three-dimensional space and the size in the image.

[Problem Solving Means 1] The first group of the present invention corresponding to the first embodiment is an optical flow processing apparatus, and a camera that continuously captures the periphery of the vehicle and an image input from the camera as a template. A flow extraction unit that extracts a flow between the divided images that are divided into divided images and that temporally moves back and forth, and a virtual block that has a predetermined position and size in a three-dimensional space of a range captured by the image A display block management unit that projects a display block onto a two-dimensional image coordinate system of the image, and a flow determination that determines the validity of the flow with reference to the size of the display block at the start point or end point of the flow It is configured to include a processing unit.
Thereby, the said subject 1 was solved.

[Problem Solving Means 2] A second group of the present invention corresponding to the second embodiment is a display radius map generation device, and has the same size predetermined in a three-dimensional space of a range captured by an image for flow extraction. A virtual block generation unit that generates a plurality of virtual blocks; a display block calculation unit that calculates a display block by projecting each of the plurality of virtual blocks onto a two-dimensional image coordinate system of the image; and the display An area calculation unit that calculates a block area as a display block area; a display block radius calculation unit that calculates a display block radius by calculating a radius when the display block area is a circle area; and the display block radius And a correlation generation unit that generates a display radius map by correlating the position of the image with the position of the image.
Thereby, the said subject 2 was solved.

  The present invention interprets the meaning of the terms described in each claim in consideration of the description of the present specification and the drawings, and certifies the invention according to each claim. There are the following advantageous effects in relation to

[Functional Effect 1 of the Invention] In the optical flow processing apparatus of the problem solving means 1, when the flow extraction unit extracts a flow between the divided images that move back and forth in time, the display block management unit in advance in a three-dimensional space. A virtual block having a predetermined position and size is projected as a display block on the two-dimensional image coordinate system of the image. The flow determination processing unit determines the validity of the flow with reference to the size of the display block at the start point or end point of the flow.
Therefore, the flow determination processing unit can determine the effectiveness of the flow within the range of the display block while referring to the size of the display block obtained by projecting the size in the three-dimensional space onto the two-dimensional image. . For this reason, the display-sized gap corresponding to the inter-vehicle distance can be eliminated by referring to the display block, and the effectiveness of the flow can be accurately determined.

[Operation Effect 2 of the Invention] In the display radius map generation device of the problem solving means 2, the display block calculation unit projects a virtual block of the same size in a three-dimensional space onto a two-dimensional image coordinate system. Then, the area calculation unit calculates the display block area of this display block. The display block radius calculation unit calculates a display block radius by calculating a radius when the display block area is an area of a circle, and the correlation generation unit calculates the display block radius and the position of the image. Generate a correlated display radius map.
Therefore, the display block calculation unit can calculate in advance display blocks having the same size in the three-dimensional space and different sizes in the image coordinate system.
Further, since the display block radius calculation unit calculates the display block radius based on the display block area, the three-dimensional information related to the two-dimensional position of the difference in the size of the display block is “radius”. It can be simplified to information of length.
Then, since the correlation generation unit generates a display radius map in which the display block radius is correlated with the position of the image, a map that can be easily referred to by the optical flow processing device may be generated in advance with a simple data structure. it can.

FIG. 1 is a block diagram illustrating a configuration example of an embodiment of the present invention. (Example 1) FIG. 2 is an explanatory diagram illustrating an example of a relationship between a flow and a display block. (Examples 1 and 2) FIG. 3 is an explanatory diagram illustrating an example of a relationship between a virtual block and a display block. (Examples 1 and 2) FIGS. 4A to 4C are explanatory diagrams illustrating an example in which a flow and a display block are calculated according to the characteristics of an approaching vehicle. (Examples 1 and 2) FIGS. 5A to 5B are explanatory diagrams showing an example of an image at a wide angle and an example of a display block at an image at this angle of view. (Examples 1 and 2) FIG. 6 is an explanatory diagram illustrating an example of the assumed ground height. (Examples 1 and 2) FIGS. 7A to 7B are explanatory diagrams illustrating an example of the relationship between the assumed ground height and the display block. (Examples 1 and 2) 8A and 8B are explanatory diagrams illustrating an example in which flows are grouped using display blocks. (Examples 1 and 2) FIG. 9 is a block diagram illustrating a configuration example of the second embodiment of the present invention. (Example 2) FIG. 10 is an explanatory diagram illustrating a calculation example of the display block area. (Examples 1 and 2) 11A and 11B are explanatory diagrams illustrating an example of the relationship among the world coordinate system, the camera coordinate system, and the image coordinate system. (Examples 1 and 2) It is a flowchart which shows an example of the information processing which calculates a display block and a display block radius. (Examples 1 and 2) It is explanatory drawing which shows an example of the virtual block vector etc. in the information processing shown in FIG. (Examples 1 and 2) 14A to 14C are explanatory diagrams illustrating an example of a situation corresponding to the procedure 4 to procedure 6 of the information processing illustrated in FIG. (Examples 1 and 2) FIGS. 15A to 15F are explanatory diagrams illustrating a relationship example between a wide-angle or corrected image and each display block. (Examples 1 and 2) FIG. 16 is an explanatory diagram illustrating an example of a correspondence relationship between a virtual block and a display block. (Examples 1 and 2) FIG. 17 is an explanatory diagram illustrating an example of a vector or the like in another solution for calculating a display block. (Example 2) FIGS. 18A to 18C are explanatory diagrams showing an example of another solution for rotating and shifting the virtual disk. FIGS. 19A to 19C are explanatory views showing an example in which the virtual disk is rotated in the procedure 5 of the information processing shown in FIG.

  Two embodiments are disclosed as modes for carrying out the invention. The first embodiment is an optical flow processing device, and the second embodiment is a display radius map generation device. Embodiments including Examples 1 and 2 are referred to as embodiments. The display radius map generated by the method of the second embodiment can be used in the optical flow processing apparatus of the first embodiment.

The present embodiment is a method of associating a fixed area as a base point of an arbitrary position and speed in real space with an image IM displayed on a driver, and in particular, a change in display size corresponding to the distance between vehicles. Smoothly correspond with technically rational methods.
That is, in this embodiment, the virtual block IB is associated with the display block BD and the display radius map 34 based on the size of the virtual block IB displayed on the screen, so that the position, speed, and range in the real space can be changed. It can be used as a discriminating material on a three-dimensional image IM.

Optical flow processing (correlation method) is information processing for two images IM that are temporally continuous, and compares the previous image IM captured last time with the current image IM captured next to the previous image IM. To extract the optical flow. In this specification, the optical flow is abbreviated as “flow Flw”.
In the flow extraction process, the previous image and the current image are divided into a number of rectangular areas (sections) using a predetermined template T, and a section of the current image having a high correlation with the section of the previous image is searched. This section comparison / search is called block matching. Then, a section of the current image that best matches the feature of the image of the section of the previous image is searched, and sections whose degree of feature correlation is equal to or greater than a certain level are connected by the flow Flw. This flow Flw is a vector, the magnitude (flow length) is proportional to the relative velocity, and the flow direction is related to the traveling direction.

As described above, the flow Flw is a movement vector that connects the section (rectangular area) of the template T arranged on the previous image IM before the movement to the section on the coordinates of the current image IM after the movement. The reference point of the partition before movement is the start point [U0, V0] of the flow Flw, and the reference point of the partition after movement is the end point [U1, V1] of the flow Flw. When the current image IM becomes the previous image IM, the section that was the flow end point [U1, V1] in the current image IM becomes the flow start point [U0, V0] as the previous image IM.
As feature quantities in block matching, there are evaluation values such as a degree of difference (SAD, Sum of Absolute Difference) and a normalized cross-corelation (NCC). The flow Flw is arranged between the matching sections from the previous image IM to the current image IM by searching for a matching position of the section having the smallest deviation of the evaluation value.

  In this flow extraction method, in a flat image having no characteristics such as a road surface of a road, coincident portions between sections are generated at an unspecified number of positions. The size and direction of the flow Flw generated from such a flat section image AIM is not fixed, and depending on how the road surface is rough and lighting conditions, the extracted flow Flw has the same properties as the flow Flw indicating the approaching vehicle MA May have. Such an erroneous detection flow Flw is a very large obstacle when determining the presence or absence of an approaching vehicle MA.

In order to distinguish such a road surface and the like from a moving body by information processing, the luminance value in the divided image AIM is subjected to dispersion processing (variation), difference, or differential processing such as a Sobel filter at the time of block matching. Then, it is preferable to calculate an evaluation value that is the processing result. When these evaluation values are used, it is possible to distinguish the segment image AIM that does not have the feature amount of the moving body and determine that it is not the detection target of the flow Flw. Thereby, the flow Flw that is not the approaching vehicle MA can be removed as noise.
By the way, when the distance between the own vehicle and the moving body becomes shorter, the size of the template T in the real space included in the section image AIM becomes smaller, and the surface of the moving body is enlarged. When the moving body is a four-wheeled vehicle, the section image AIM is enlarged to, for example, a state in which the plate surface of the body is closed up. Then, the section image AIM does not have a characteristic element, and the luminance value becomes a single image.

  Normally, the size of each section image AIM of the template T is the same for maintaining the accuracy of block matching and homogenizing the processing burden. For this reason, as the inter-vehicle distance becomes shorter, the correlation size in the real space corresponding to the section image AIM becomes smaller due to the perspective display. As a result, even in the divided image AIM in which the moving body having the feature is captured, the feature is lost and the flow Flw cannot be extracted because the inter-vehicle distance is shortened. In particular, in an example in which road surface determination processing such as differentiation processing functions, a section image AIM that does not have a feature amount is not processed.

That is, even in the section image AIM arranged in the imaging range of the approaching vehicle MA, a section in which the flow Flw is not successfully extracted may be generated depending on the inter-vehicle distance. As a result, the section image AIM that can be discriminated is only for the contour of each object of the approaching vehicle MA, the tire, the light bumper, and the like where the luminance change is large.
Under this circumstance, the flow Flw remaining after the approach is discretely arranged, and the interval between the adjacent flows Flw is widened, so that the processing related to the determination based on the set of adjacent flows Flw (flow group FlwG) becomes difficult. The accuracy of the process of continuously tracking the approaching vehicle MA will decrease.
As described above, when the inter-vehicle distance between the host vehicle MT and the approaching vehicle MA changes, first, the approaching vehicle MA as a whole that is approaching cannot be detected as the flow Flw in the same-sized section image AIM. Second, since the flow Flw generated from the same approaching vehicle MA is discrete and is not connected, the accuracy of the process of grouping and determining the flow Flw is deteriorated.
In this embodiment, the accuracy of detecting the approaching vehicle MA is improved by determining the effectiveness of the flow Flw with reference to the display block BD.

<1 Optical flow processor>
<1.1 Display block>
First, Example 1 of this embodiment is disclosed. The first embodiment is an optical flow processing apparatus 100 in which a three-dimensional virtual block IB and a two-dimensional display block BD are associated with each other in order to reduce detection errors due to changes in the inter-vehicle distance. As a result, the validity of the flow Flw detected in two dimensions can be determined by taking the three-dimensional information into consideration.

The optical flow processing apparatus 100 according to the first embodiment includes a camera 10, a flow extraction unit 12, a display block management unit 14, and a flow determination processing unit 16 as main elements.
In the example illustrated in FIG. 1, the display block management unit 14 includes a display block radius related process 18, a virtual block specifying process 20, an assumed ground height process 22, and a display radius map 34.

  The camera 10 continuously captures the periphery of the host vehicle MT. The captured image IM is input to the flow extraction unit 12. For this camera 10, an external parameter indicating the attitude of the camera 10 and an internal parameter indicating lens distortion or the like may be obtained in advance by calibration or the like. The lens of the camera 10 may be a normal lens or a wide angle. You may use the camera 10 for back assistance for parking assistance.

The flow extraction unit 12 partitions the image IM input from the camera 10 into the partition images AIM using the template T, and extracts the flow Flw between the partition images AIM that are temporally changed.
As shown in FIG. 2, the flow extraction unit 12 partitions each region of the image IM into a number of partitioned images AIM by superimposing the template T on the image IM. The compartments are preferably arranged with a gap, but can also be arranged without a gap. The flow extraction unit 12 then sets the Flw start point [U0, V0] and the end point [U1, V1] of the flow between the two partition images IM that are temporally adjacent to each other and the features of the partition image AIM match. Define. This “match” includes not only perfect match as described above, but also a case where the degree of match of feature amounts is high.
The start point [U0, V0] of the flow Flw is arranged in the temporally previous image IM, and the end point [U1, V1] of the flow Flw is arranged in the temporally subsequent (current) image IM. Since the features of the section image AIM match at the start point [U0, V0] and the end point [U1, V1] of the flow Flw, the flow Flw represents a state in which the same part such as the approaching vehicle MA is approaching.
In the example shown in FIG. 2, the template T is arranged on the entire surface of the image IM. However, the template T may be arranged only in a range where the approaching vehicle MA can travel in the image IM.
For the information processing itself of the flow Flw extraction, the same technique as in the conventional example can be adopted.

In the first embodiment, the display block management unit 14 uses a virtual block IB having a predetermined position and size in a three-dimensional space in a range captured by the image IM as a two-dimensional image coordinate system of the image IM. Project as UV display block BD. The virtual block IB has a three-dimensional shape, and the display block BD has a two-dimensional shape. A large number of virtual blocks IB generated in the three-dimensional space can be easily processed if they have the same size, but they may have different sizes when their positions can be managed individually.
The virtual block IB near the camera 10 is projected larger in the image coordinate system UV than the far virtual block IB. That is, according to the perspective method, the virtual block IB at the infinity point is extremely small, the far virtual block IB is small, and increases as the camera 10 is approached.
The display block BD is the size of the virtual block IB associated with the coordinate value (position) of the image coordinate system UV. Accordingly, it is possible to represent the degree to which the size of the object in the three-dimensional space changes in the perspective on the image IM. The display block BD may be calculated by performing a projection process or the like when necessary, or may be calculated in advance using the method of the second embodiment and the data of the display radius map 34 is recorded. You may make it leave.

The flow determination processing unit 16 determines the validity of the flow Flw with reference to the size of the display block BD at the start point [U0, V0] or the end point [U1, V1] of the flow Flw.
The flow determination processing unit 16 determines the effectiveness of the flow Flw using the size of the display block BD as a discrimination material. The display block BD represents the extent to which the size of the object in the three-dimensional space is changed by perspective on the image IM. For this reason, the flow determination processing unit 16 does not determine the effectiveness of the flow Flw by devising the image processing only for the image IM, but the size of the extracted flow Flw in the actual three-dimensional space. The validity of the flow Flw can be determined with reference to FIG. Therefore, in particular, it is possible to effectively extract the flow Flw of the object that moves in the image IM such as the approaching vehicle MA.

  The validity determination process by the flow determination processing unit 16 is indirectly performed by grouping the flow Flw using a display block in addition to the process of directly determining validity / invalidity of each flow Flw. Processing to determine the validity / invalidity of the flow Flw, processing to determine the validity of the continuity of the flow Flw, processing to determine the placement position of a new template T for that purpose, and dynamic processing of such a template T This includes processing for determining the effectiveness of the flow Flw by continuous processing such as placement.

  As shown in FIG. 3, in this embodiment, in the world coordinate system XYZ on the left side of the drawing, the flow Flw determines that the virtual block IB has moved from the start point [U0, V0] to the end point [U1, V1]. . In the example shown in FIG. 3, the virtual block IB is a sphere and has the same size (virtual block radius IBr). In the image coordinate system UV, the size of the sphere having the same size changes in the display on the image IM according to the perspective (the distance between the vehicle MT and the vehicle). That is, the distance from the camera 10 in the depth direction (z axis in FIG. 11A) becomes smaller, and the virtual block IB is imaged larger as it gets closer to the camera 10.

FIG. 4A shows an example of the approaching vehicle MA. In the present embodiment, as shown in FIG. 4B, the approaching vehicle MA is handled as a set of virtual blocks IB. In the example shown in FIG. 4B, it is assumed that a valid virtual block IB exists at the position where the flow Flw is detected. That is, the flow Flw is considered to have occurred based on the movement of “a certain block” in the virtual block group constituting the vehicle. Thus, in the present embodiment, the approaching vehicle MA is modeled as an aggregate of virtual planar blocks existing in the three-dimensional space.
The virtual block IB can be projected and converted onto the two-dimensional image IM. By converting the flow Flw extraction position to real space 3D coordinates (world coordinate system XYZ), and converting the virtual block IB set for the converted real space position back onto the plane of the 2D image IM by perspective transformation Corresponding to gaps between vehicle distance and object display size.
That is, as shown in FIG. 4C, when the validity determination process of the flow Flw is performed based on the virtual block IB, it is possible to deal with a display gap based on the inter-vehicle distance.

  Even if each virtual block IB has the same size in the real space, the display size changes on the two-dimensional image IM according to the distance from the camera 10 and the distortion of the image IM. The flow Flw handles the movement of a part of the virtual blocks IB in the entire virtual block IB in the three-dimensional space as capturing the change appearing in the two-dimensional image IM. Since the source of the flow Flw is a set of virtual blocks IB, there are other virtual blocks IB of similar nature around the virtual block IB at the start point [U0, V0] and the end point [U1, V1] of the flow Flw. It can be assumed that there exists a set by By processing based on the virtual block IB assumed in the real space and the display size on the image IM, the flow Flw indicating a part of the approaching vehicle MA is changed according to the distance between the vehicle and the own vehicle MT. Realizes processing that is less affected by the gap with the display size on the plane of the changing image IM.

1.1 Effect of Display Block As described above, when the flow extraction unit 12 extracts the flow Flw between the segment images AIM that are temporally changed, the display block management unit 14 is predetermined in the three-dimensional space. A virtual block IB having a position and a size is projected as a display block BD on the two-dimensional image coordinate system UV of the image IM. Then, the flow determination processing unit 16 refers to the size of the display block BD at the start point [U0, V0] or the end point [U1, V1] of the flow Flw to determine the validity of the flow Flw.
Therefore, the flow determination processing unit 16 can determine the effectiveness of the flow Flw within and outside the range of the display block BD while referring to the size of the object in the three-dimensional space. For this reason, the display-sized gap corresponding to the inter-vehicle distance can be eliminated by referring to the display block BD, and the effectiveness of the flow Flw can be accurately determined.
As described above, since the effectiveness of the flow Flw is determined by referring to the size of the object in the actual three-dimensional space, not the image processing only for the image IM, the own vehicle MT and the approaching vehicle MA, etc. Even if there is a change in the distance between the vehicle and the moving body, the detection accuracy of the flow Flw and its grouping can be stabilized. The movement of the same moving object can be continuously and stably extracted based on the flow Flw group.
With the flow Flw validity determination process using the display block BD, the display-size gap corresponding to the inter-vehicle distance can be eliminated, and the detection accuracy of the moving object can be improved.
In particular, in the detection of an approaching vehicle MA in which the inter-vehicle distance between the host vehicle MT and the approaching vehicle MA changes every moment, the number of flows Flw extracted from the same approaching vehicle MA increases with the approach due to the influence of perspective. In addition, even if the range is expanded, referencing the display block BD effectively handles the validity of the flow Flw, the identity of the approaching vehicle MA, the grouping of the flow Flw, the removal of the invalid flow Flw, etc. can do.

<1.2 Display block radius>
In the first embodiment, the data structure representing the display block can be a simpler mechanism. In this example, the display block management unit 14 includes a display block radius related process 18.
The display block radius related processing 18 approximates the display block BD as a circle and associates the approximate circle radius with the size of the display block BD as the display block radius BDr.

Referring to FIG. 3 again, when the display block BD is circularly approximated, the characteristics of the display block BD can be managed only by specifying the value of the display block radius BDr. In other words, the display block management unit 14 manages the display block radius BDr corresponding to the length of the virtual block radius IBr of the virtual block IB, so that the size of the real space becomes the coordinate value [U, V] of the image IM. It can be managed using the display block radius BDr.
The display block radius BDr may be calculated at a necessary time in real time, or may be calculated and mapped in advance by the method of the second embodiment. In the case of mapping, the display block management unit 14 stores the display radius map 34 in advance.
Also, the display block BD or the display block radius BDr is not made to correspond to the coordinate value [U, V] of the image IM, but to the coordinate value [X, Y, Z] of the world coordinate system XYZ representing the position in the real space. When making correspondence, it is preferable to make correspondence using a map (measurement map, XZtoUV map of Patent Document 5) in which the image coordinate system UV and the world coordinate system XYZ are made to correspond in advance.

  As shown in FIGS. 5A and 5B, the display block BD approximated by a circle specified by the display block radius BDr can satisfactorily represent a display gap corresponding to the perspective (distance between vehicles). For this reason, accurate processing can be executed at high speed using a simple data structure. Furthermore, even for the image IM using the wide-angle lens shown in FIG. 5A, the effect of the flow Flw can be simplified by eliminating the influence of the distortion.

1.2 Effect of Display Block Radius As described above, the display block radius related process 18 approximates the display block BD as a circle and relates to the size of the display block BD using the approximate circle radius as the display block radius BDr. Therefore, the information of the display block BD corresponding to the perspective in the three-dimensional space can be specified and expressed with simple data called the radius. The simple data structure of the coordinate value and the display block radius BDr allows the processing to be performed accurately and at high speed, and in particular, real-time processing can be realized at low cost.

<1.3 Virtual block and detection speed>
In the first embodiment, the flow Flw generated from a stationary object or the like can be effectively invalidated by referring to the display block BD. In this example, the display block management unit 14 includes a virtual block specifying process 20.
The virtual block specifying process 20 specifies the size of the virtual block IB in the three-dimensional space based on a predetermined detection speed for a vehicle approaching the host vehicle MT in the three-dimensional space. .

  Referring to FIG. 3 again, the size of each display block BD is determined according to the position and size of the virtual block IB. The size of the virtual block IB can be arbitrarily set as the length of the virtual block radius IBr when it is a sphere. This virtual block radius IBr may be determined according to the actual length in the world coordinate system XYZ, but in this example, by determining according to the detection speed, the invalid flow Flw determination process can be simplified. Realize.

  For example, the flow Flw start point [U0, V0] to the end point [U1, V1] detected by the movement of the approaching vehicle MA approaching between the video rates (1/30 [sec]) is used as a linear movement component in real space. Ineffective flow Flw can be eliminated by determining whether or not the threshold condition specified by the detection speed (set upper limit speed) is satisfied. That is, since the background of the road surface and the like is stationary and the speed is 0 [m / sec], the speed of the flow Flw generated from the road surface or the like is equal to or lower than the detection speed.

Since the virtual block IB is the source of the flow Flw generation, if the size of the virtual block IB is determined according to the detection speed, it becomes easy to eliminate the invalid flow Flw, and the size of the display block BD is effective. In order to correspond to a speed higher than the speed required for the flow Flw, in the flow Flw validity determination process with reference to the display block BD, similarly, it becomes easy to eliminate the flow Flw that is invalid when the speed is 0.
As described above, by setting a sphere (or N-gon) having a radius corresponding to the detection speed for the virtual block IB, an optimum size for determining the effectiveness of the flow Flw can be obtained.

1.3 Effect of Virtual Block and Detection Speed As described above, the virtual block specifying process 20 is performed based on the detection speed predetermined for a vehicle approaching the host vehicle MT in a three-dimensional space. In order to specify the size of the IB in the three-dimensional space, when there is a flow Flw that is less than the size in the three-dimensional space specified by the display block BD, the flow Flw is less than the detection speed. Therefore, it can be determined to be invalid. In this way, by determining the size of the virtual block BD based on the detection speed, it is possible to obtain an optimal size that can naturally determine the effectiveness of the virtual block IB and the display block BD flow Flw.

<1.4 Assumed ground height Yconst>
In the first embodiment, the two coordinate values [U, V] of the image IM can be practically associated with the three coordinate values [X, Y, Z] of the world coordinate system. In this example, the display block management unit 14 includes the assumed ground height processing 22.
The assumed ground height processing 22 assumes the height of the virtual block IB in the three-dimensional space to be the assumed ground height Yconst predetermined in the three-dimensional space, and displays the virtual block IB in the image coordinate system UV display block. Project to BD position.

Referring to FIG. 6, in the first embodiment, the camera 10 is installed behind the host vehicle MT and the traveling environment behind the host vehicle MT is photographed. By setting the camera 10 tilted downward, the range in which a moving body such as the approaching vehicle MA can be imaged is widened. Then, a plane that is a normal line with the optical axis (z axis) of the camera 10 and that is at the focal position of the camera 10 is an image plane IP (ideal plane in Patent Document 5).
As shown in FIG. 6, the assumed ground height Yconst is preferably a height near a bumper having a large amount of features for extracting the flow Flw. The first hypothetical ground height Yconst1 is a joint where the bumper and the vehicle body are connected, and the second hypothetical ground height Yconst2 is a height where a luminance difference is likely to appear at the boundary between the bumper lower part and the road surface GT.

As shown in FIG. 7A, when the assumed ground height Yconst is used, a cutting plane based on the assumed ground height Yconst can be defined in the three-dimensional space of the imaging range. Then, the flow Flw moves on the assumed ground height Yconst plane.
As shown in FIG. 7B, when the display block BD at the end point [U1, V1] of the flow Flw shown in FIG. 7A is referenced, the display block BD closer to the camera 10 appears greatly.
In this example, since the effectiveness of each flow Flw can be determined on the cutting plane based on the assumed ground height Yconst with reference to the size of the display block BD, each flow Flw is from the same approaching vehicle MA. Judgment of whether or not it occurs, whether each flow Flw is a continuous flow Flw by the same approaching vehicle MA, and calculating the vehicle speed of the approaching vehicle MA, the predicted position where each flow Flw appears next Calculations can be performed with high accuracy.
That is, by using the assumed ground height Yconst and the display block BD, it is possible to effectively perform the process of determining the effectiveness of each flow Flw at a low cost.

1.4 Effect of Assumed Ground Height Yconst As described above, the assumed ground height processing 22 assumes the height of the virtual block IB in the three-dimensional space as the assumed ground height Yconst predetermined in the three-dimensional space. Then, in order to project the virtual block IB onto the position of the display block BD in the image coordinate system UV, two coordinate values [U, V] of the image IM and three coordinate values [X of the world coordinate system XYZ [X , Y, Z] can be matched practically. Moreover, the continuity of the extracted Flw can be increased by assuming the height of the object to be a constant value.
And since the height of the virtual block IB is assumed to be the assumed ground height Yconst, the display block BD becomes data indicating the difference in size for the virtual block IB of the same height, and is linear from far to close. A conical display block BD can be obtained.

<1.5 Effectiveness judgment processing>
In the first embodiment, the display block management unit 14 does not generate the display block BD, but stores data about the display block BD created in advance as a map, which is referred to during the validity determination process of the flow Flw. You may make it do.

In this example, the display block management unit 14 may store the display radius map 34.
The display radius map 34 stores a virtual block IB having a predetermined position and the same size in a three-dimensional space in a range captured by the image IM as a display block BD in the two-dimensional image coordinate system UV of the image. Projected data. The data included in the display radius map 34 includes the coordinate values [U, V] of the image IM or the coordinate values [X, Y, Z] in the world coordinate system XYZ corresponding to the coordinate values, and the size of the display block BD. The data corresponds to the display block radius BDr. A map corresponding to the display block radius BDr is particularly referred to as a display radius map 34.
The flow determination processing unit 16 refers to the display radius map 34 when determining the validity of the flow Flw, so that the display block at the start point [U0, V0] or the end point [U1, V1] of the flow Flw. BD size data can be used.

The flow determination unit 16 can perform the same information processing both when referring to the data generated by the display block management unit 14 and when referring to the display radius map 34.
The effectiveness determination processing by the flow determination unit 16 varies depending on needs and required accuracy. Here, in addition to the above-described effectiveness determination processing, (1) grouping, (2) dynamic template, (3) Disclose the relationship with other maps.

・ 1.5 (1) Grouping One type of flow Flw effectiveness determination process is identification of approaching vehicle MA and determination of the degree of approach.
Generally, out of the detected flow group, a flow Flw that may be an approaching vehicle MA is extracted based on the flow position, flow length, and flow direction, and a set of flows Flw adjacent to this flow Flw is grouped, and this The approaching vehicle MA is identified assuming that the flow group FlwG is a flow group caused by the approaching vehicle MA.
Even if one approaching vehicle MA is closely monitored in parts (for example, tires, windows, lights, bodies, etc.), the amount and direction of movement of all parts is the same as that of the approaching vehicle MA. Since they are integral, they have the same moving component. Therefore, a set of flows Flw having the same properties (speed, direction, continuity) can be estimated as the approaching vehicle MA.

As described above, in the rear monitoring system or the like that extracts the flow Flw based on the image IM input from the camera 10, the presence / absence of the approaching vehicle MA and the degree of approach thereof are determined by grouping the detected flow Flw. Can do.
Further, it is possible to distinguish an area where there is a flow group FlwG having a strong similarity to the nature of the flow Flw and an area where the flow group FlwG does not exist by information processing. Evaluation based on the distribution of the flow Flw can be used in combination. On the other hand, it is difficult to determine the presence of the approaching vehicle MA by classifying the flow Flw that exists independently at any given time (between images), noise generated from other similar elements, and erroneous detection flows.

  The flow Flw grouping process is useful for determining the effectiveness of the flow Flw. However, the range of the inter-vehicle distance in which the flows Flw form an ideally aggregated distribution is short, and it is difficult to evaluate the adjacent flow group even if the inter-vehicle distance is too far or too close, and the desired grouping process cannot be performed.

  In the example illustrated in FIG. 8, the flow determination processing unit 16 converts the flow Flw extracted in FIG. 8A into the display block BD illustrated in FIG. 8B. In FIG. 8A, an arrow indicating the flow Flw is drawn large so that the flow Flw can be identified.

In the example shown in FIG. 8B, the flow determination processing unit 16 groups the flows Flw based on the overlapping of the display blocks BD. The extracted flow Flw is divided into a flow group FlwG (1), a flow group FlwG (2), and a flow group FlwG (3).
By grouping the flows Flw in this way, they can be grouped based on the size in the real space, not the size in the image IM. By this grouping, the presence / absence of the approaching vehicle MA can be determined more easily than the determination process of the approaching vehicle MA for the individual flow Flw.
Thus, by grouping the flows Flw based on the display block BD of the flow end point [U1, V1], the detection accuracy of the approaching vehicle MA can be improved.

・ 1.5 (2) Dynamic template Even if the approaching vehicle MA is found by the above-described grouping process, the approaching vehicle MA is zoomed in when the inter-vehicle distance is further reduced, and the approaching vehicle MA is compared with the section image AIM. Since the flow Flw is discretized due to the excessively large feature, and there is no flow Flw constituting the group, and there is only the single flow Flw, the approaching vehicle MA is lost.
In this case, regarding the single flow Flw, if the template T (section) can be dynamically added to the periphery, a group can be formed again with the flow Flw derived from the single flow Flw. Due to the dynamic addition of the template T, the continuous detection performance of the approaching vehicle MA can be secured even when the space between the vehicles is clogged.
When the dynamic template T is arranged, the template T can be arranged at a very effective position by using the display block BD.
In addition, when the flow Flw that is a candidate of the approaching vehicle MA is detected while reducing the amount of information processing by roughening the entire template T, the approaching vehicle MA is dynamically arranged around the flow Flw. In order to improve the detection accuracy, dynamic template placement with reference to the display block BD is also effective.

The addition of the dynamic template T is a process of arranging the dynamic template T on the assumption that another virtual block IB exists around the virtual block IB existing at the detection position of the normal flow Flw. . The dynamic template T is arranged based on the display block BD, that is, the arrangement based on the display size of the virtual block IB on the two-dimensional image plane projected from the detection position of the flow Flw. The approaching vehicle MA can be extracted effectively corresponding to the gap regarding the display size of the object.
In this example, the next image IM is added by dynamically adding a new template within the range of the display block BD located around the end point [U1, V1] of the detected flow Flw and the end point [U1, V1]. The flow Flw can be extracted successively from the image IM one after another.
Then, by confirming the behavior and continuous detectability of the flow Flw that is continuously detected, the presence of the approaching vehicle MA can be determined and information can be presented to the driver.

Let BDr [U, V] be the reference value of the radius map at the coordinate value [U, V] of the image IM. For the detection position [U, V] of the flow Flw, the line segment [U ± BDr [U, V], V] is treated as the intersection of the Yconst plane and the virtual block IB, and this line segment [U ± BDr [ Divide point coordinates [U [i], V [i]] by dividing U, V], V] into M equal parts.
[U [i], V [i]] = [(U [i] −2 × R [i]) + 2 × R [i] / M, V [i]] and the dividing point coordinates [U [i] , V [i]], a total of M sections (template T) are registered.

  Template matching can be optimized by the dynamic arrangement of the partitions using the display block BD and the display block radius BDr, and this processing can be performed at low cost. Then, by continuing the continuous optimization of the arrangement of the template T instead of the optimization between the two images IM, it is possible to reliably keep track of the characteristic portion of the moving object.

  Based on the display block radius BDr at the end of the detected approach flow Flw, in the method of dynamically arranging the section around this end, even when the detection flow Flw is detected from the position of the contour portion of the approaching vehicle MA, Regardless of the distance between the vehicles, the section can be arranged at a position indicating the main body of the approaching vehicle MA at least 1/2 or more.

・ 1.5 (3) Relationship with other maps By using the display block BD, the validity of the flow Flw can be determined with the two-dimensional coordinates of the image IM, but information related to the vehicle speed of the approaching vehicle MA, etc. Some items can be determined more accurately in the world coordinate system XYZ.
For this reason, it is convenient that the image coordinate system UV and the world coordinate system XYZ can be converted to each other. However, when the camera 10 needs to be corrected to be linear at a wide angle, complicated nonlinear equations and It is necessary to obtain a polynomial equation, and three information (position) in three dimensions cannot be obtained from two information in two dimensions.

Patent document 5 of the same applicant and the inventor includes two-dimensional coordinates of an image coordinate system UV based on a back-eye camera 10 using a wide-angle lens for parking assistance, and coordinate values of a world coordinate system XYZ in real space. A measurement map for smoothly acquiring the correlation is disclosed.
By setting the virtual block center IBc of the virtual block IB to be located on a certain arbitrary height Yconst plane, the display block center BDc [U, V] of the image coordinate system UV is obtained by the method disclosed in Patent Document 5. It can be converted to the virtual block center IBc [X, Yconst, Z] in the world coordinate system XYZ.
Further, if the disk circumference point ICei and the virtual block circumference point IBei disclosed in the second embodiment are associated with each other, the world coordinates regarding the circumference distribution and display size of the display block BD in the image IM, and the display block radius BDr. Data in system XYZ can be acquired.

In order to make the image coordinate system UV and the world coordinate system XYZ correspond to each other, according to the method described in Patent Document 5, when performing wide-angle correction processing with the image plane IP, when the distance from the center is a predetermined value or less, A distortion approximation is performed, and when a predetermined value is exceeded, a logarithmic approximation is calculated.
In the method described in Patent Document 6 of the same applicant, the calculation is performed by changing the correction method continuously and stepwise within a predetermined range instead of a predetermined value.
In relation to the validity determination processing of the flow Flw using the display block BD, either correction method can be adopted, and any measurement map creation method can be used.

-1.5 Effect of validity determination processing As described above, the display block BD or the display block BDr is useful for various flow validity determination processing, and includes group creation, dynamic template registration, and evaluation for each flow Flw detected continuously ( The efficiency of processing such as confirmation of continuity and change in moving components) can be increased and stabilized.

<2 Display radius map generator>
<2.1 Display radius map generation>
Next, Example 2 is disclosed. Example 2 is a display radius map generation device that attempts to take into consideration three-dimensional information in real time when processing an optical flow by projecting three-dimensional information in two dimensions with a simple data structure. It is.
When obtaining the size of the display block BD, a floating-point operation and a division process are required several times, so that a very high system calculation capability is required. Then, it is difficult to mount on an approaching vehicle detection system that requires real-time performance. Therefore, after offline calculation in advance, the size of the display block BD, for example, the display block radius BDr, may be registered in advance as an aperture map in the effective detection range of the flow Flw. By mapping in advance, the data of the display block BD can be acquired as needed in the form of a data map reference instead of numerical calculation, so the processing speed can be increased and there is no high calculation cost Implementation in an inexpensive system is possible.

  The display radius map generation apparatus 200 according to the second embodiment includes, as main elements, a virtual block generation unit 24, a display block calculation unit 26, an area calculation unit 28, a display block radius calculation unit 30, and a correlation generation unit 32. And a display radius map 34.

  The contents of each part (each process) will be described with reference to FIG. Since the information processing of each part can be calculated by a virtual disk IC, which will be described later, the number of the corresponding formula is specified. In the example shown in FIG. 9, the calculation result (display radius) is also obtained by information processing other than each formula. A map 34) can be obtained. Further, when the operations of the respective parts are grasped in order in time, it becomes a process constituting a series of methods. Information processing of each part and each corresponding process can be realized by a program executed by an ECU or the like.

First, the virtual block generation unit 24 generates a plurality of virtual blocks IB having the same size in a three-dimensional space in a range captured by the image IM (virtual block generation step, equation (2)).
The display block calculation unit 26 calculates the display block BD by projecting each of the plurality of virtual blocks IB onto the two-dimensional image coordinate system UV of the image IM (display block calculation step, equations (3), ( 4), (5), (6), (7); (10)).
The area calculation unit 28 calculates the area of the display block BD as the display block area Bda (area calculation step, equation (8)). As shown in FIG. 10, the display block radius BDr can be obtained from the N-gonal display block area BDa.
The display block radius calculation unit 30 calculates the display block radius BDr by calculating the radius when the display block area Bda is the area of a circle (display block radius calculation step, equation (9)).
The correlation generation unit 32 generates a display radius map 34 by correlating the display block radius BDr and the position of the image IM (correlation generation step).
Each process (process) of each of these units may include a calculation process according to a corresponding formula and the procedure shown in FIG.

The registered value in the display radius map 34 may be the diameter or the area itself, but by using the registered value in the display radius map 34 as the radius, it is convenient for the determination process of the approaching vehicle MA. If the camera 10 and the approaching vehicle MA are too close to each other, the radius will diverge, so an appropriate upper limit is clipped and the upper limit is clipped. In practice, registration to the display map 34 is limited to an appropriate upper limit. Good.
The virtual block IB and the display radius map 34 generate the dynamic registration of the template T, the continuity confirmation of the flow Flw, the reference for evaluating the change of the moving component, the flow group FlwG, and the continuous measurement result and history as Flw. This method can be used for a method shared by each other, a control parameter for noise determination, and the like.

・ Relationship with the measurement map The effective monitoring area is defined behind the vehicle using the coordinate values of the world coordinate system XYZ. The minimum value is Xmin and the maximum value is Xmax in the X direction corresponding to the left and right direction of the image IM, the minimum value is Zmin and the maximum value is Zmax in the Z direction corresponding to the vertical direction.
The effective monitoring area is [Xmin to Xmax] [Zmin to Zmax], and the measurement map is 8 bits. The coordinates [U, V] on the image coordinates UV can be obtained by referring to the X measurement map Xmap, the coordinate data Xmap [U, V], Zmap [U, V] on the Z measurement map Zmap. 8-bit converted value [Xpa, Yconst, Zpa] of coordinates [X, Y, Z]: −128 ≤ Xpa ≤ 127 0 ≤ Zpa ≤ 255 (Unit is map bit. Z: 255 [map bit] in real space Zmax [mm]).

Since the assumed ground height Yconst is a fixed value set when the measurement map is created, the possible values of Xpa, Zpa are 256 × 256 combinations.
Therefore, after creating the display radius map 34 (data table) of 256 × 256 size, fixing Yp = Yconst, and converting Xpa and Zpa to the original Xp and Zp, the virtual block center IBc of the virtual block IB By defining the position as [Xp, Yconst, Zp], information of the display block BD corresponding to all the flows Flw detected on the image IM can be registered in the display radius map 34.
When the information of the display block BD is set as the display block radius BDr, in this example, the display radius map 34 has a size of 256 × 256.

  By setting the registration to the display radius map 34 not to the display block area BDa but to the display block radius BDr (which may be a diameter), it is possible to reduce the registration value to the display radius map 34 and save resources. For example, when the radius is 20 pixels, 5 bits are required for registering the display radius map 34, and when the diameter is 40 pixels, 6 bits are required for registration. If the area is 1256 pixels, 11 bits are required for map registration.

In this way, the display radius map 34 having the X and Z positions of the measurement map on the coordinate axes (reference keys) and converting the display size of the corresponding display block BD into data can be created.
By introducing the display radius map 34 that defines the display size (display block radius BDr) of the virtual disk IC assumed in the real space, the determination of the approaching vehicle MA can be processed in real time. In other words, the virtual block IB is regarded as a disk, and the display size calculation on the image of the virtual block IB, which requires floating-point arithmetic, multiplication processing, etc., is calculated offline and mapped in advance, thereby determining the validity of the flow Flw The calculation cost can be reduced.
In particular, processing can be efficiently performed in correspondence with the wide-angle camera 10 so that the image IM includes distortion.

  Further, in order to obtain an inverse correlation between the three-dimensional coordinates on the measurement map and the coordinates on the image plane, it is preferable to map the X and Z positions and the U and V coordinates on the image IM (hereinafter referred to as XZtoUV map). By using this XZtoUV map, complicated numerical calculations during the flow Flw processing can be avoided, and efficient processing can be realized.

2.1 Effects of Generating Display Radius Map As described above, the area calculation unit 28 calculates the area of the display block BD as the display block area BDi, and the display block radius calculation unit 30 calculates the display block area BDi as a circle. Since the display block radius BDr is calculated by calculating the radius when the area is used, the extent to which the size of the three-dimensional shape in real space changes depending on the perspective can be expressed by simple length information called the radius. . Then, when the correlation generation unit 32 generates the display radius map 34 in which the display block radius BDr and the position of the image IM are correlated, it corresponds when the coordinate value [U, V] of the flow Flw of the image IM is specified. The information of the size reflecting the inter-vehicle distance in the three-dimensional space can be obtained. In the example in which the coordinate values [U, V] of the image coordinate system UV are associated in advance with the coordinate values [X, Y, Z] of the world coordinate system XYZ, the coordinate values [X , Y, Z], it is possible to immediately refer to the size of the virtual block BD at that position.
In addition, the difference in the size of the display block BD for the perspective corresponding to the vertical direction of the image IM is expressed as a radius, and even if the shape of the display block BD is distorted in the horizontal direction of the image IM, it is managed by the display block radius BDr. Thus, since the display block BD is handled in a circle, various validity determination processes using the display block BD can be stably executed at a low cost.
In this way, the display radius map BDr is calculated in advance, and the three-dimensional quality is improved by corresponding to the coordinate values [U, V] of the image IM or the coordinate values [X, Y, Z] of the world coordinate system XYZ. Therefore, the flow Flw validity determination process can be executed at low cost and with high accuracy.
Further, in this example, the display block radius BDr is calculated individually without setting the hardware state such as the posture of the camera 10, the presence / absence and degree of wide angle, the angle of view, and other lens distortion in the design. Thus, the perspective gap and the like can be resolved by software processing. That is, the method of calculating the display block radius BDr can greatly improve the design options for the camera 10 and the imaging range.

<2.2 Calculation using virtual disk>
In the second embodiment, more specific information processing is disclosed than associating the virtual block IB of the world coordinate system XYZ with the display block radius BDr of the image coordinate system UV through calculation in the camera coordinate system xyz. In this example, a virtual block IB is assumed to be a sphere, and a virtual disk IC is defined to project the virtual block IB, which is a sphere, two-dimensionally, and a display block BD is calculated by adding information processing to the virtual disk IC. To do.

  That is, in this example, the shape of the virtual block IB can be handled as a disk. For example, when the flow Flw indicating the approaching vehicle MA is detected at the position of the coordinate value [X, Y, Z] = [Xp, Yp, Zp] in the world coordinate system, the virtual disk IC constituting the approaching vehicle MA is detected. The size of the virtual disk IC displayed on the screen (the size of the display block BD) is set to the information processing from step 1 to step 9 below, assuming that any one of has moved around this [Xp, Yp, Zp] To obtain the projected area of the virtual disk IC on the image IM.

FIG. 11 shows a definition example of the coordinate system. In the example shown in FIGS. 11A and 11B, the x axis of the camera coordinate system xyz, the X axis of the world coordinate system XYZ, and the U axis of the image coordinate system UV are parallel, and the y axis of the camera coordinate system xyz. And the Y axis of the world coordinate system XYZ and the V axis of the image coordinate system UV are parallel. The Z axis of the world coordinate system XYZ is parallel to the road surface and faces the rear of the vehicle, and the z axis of the camera coordinate system xyz coincides with the optical axis of the camera 10.
In the following, the value of the mathematical data name (value or sign) with the suffix w indicates that it is a coordinate value, especially in the world coordinate system XYZ, and the suffix c is the camera coordinate system xyz Indicates that the coordinate value is. If w or c is not attached, it indicates that either can be calculated.

  In the example shown in FIG. 12, calculation processing by the virtual disk IC is from procedure 1 to procedure 9, procedures 3 to 5 are calculations in the camera coordinate system xyz, procedures 6 are calculations in the world coordinate system XYZ, and procedures 7 to 9 is a calculation in the image coordinate system UV.

First, the origin OC of the camera coordinate system xyz is expressed as follows. This camera coordinate origin OC is a lens focal position, and is located at a focal length f in the z direction of the camera coordinate system from the center of the image coordinate systems U and V.
World coordinate system [X, Y, Z] = [OrgXw, OrgYw, OrgZw]
Camera coordinate system [x, y, z] = [OrgXc, OrgYc, OrgZc] = [0,0,0]
The hypothetical ground height Yconst is adopted as the center position of the desired virtual block (sphere or disk) as follows.
World coordinate system [X, Y, Z] = [Xp, Yconst, Zp]

Procedure 1: Convert the camera coordinate origin OC to the world coordinate system XYZ.
Since the position of the camera coordinate origin OC is [x, y, z] = [0, 0, 0], it can be expressed by the following equation (1a) using the camera calibration data rnn, tn. World coordinates [OrgXw, OrgYw, OrgZw] can be obtained by the following equation (1b). Since the lens focal position is the center on the image plane IP, this coordinate conversion formula (1b) can be applied regardless of the presence or absence of lens distortion.
It should be noted that when an expression such as a, b, c, etc. is added to a numeral indicating an expression number, when referring to the numeral indicating the expression number, all expressions identified by the alphabet are included. For example, the expression (2) includes expressions (2a) and (2b).
The subscript -1 in equation (1b) indicates that there is an inverse matrix.

Procedure 2: Calculate virtual block vector IBv.
As illustrated in FIG. 13, the virtual block generation unit 24 generates a vector Vec from the camera coordinate origin world coordinate system OW [OrgXw, OrgYw, OrgZw] to the center IBo [Xp, Yp, Zp] of the virtual block IB. The block vector IBv [VecX, VecY, VecZ] is calculated in the world coordinate system. Here, the height Yp of the center IBo of the virtual block IB is the assumed ground height Yconst, and the virtual block generator 24 can calculate the virtual block vector IBv by the following equation (2).

Procedure 3: Calculate virtual block rotation angle IBei.
The virtual block generator 24 converts the virtual block vector IBv [VecX, VecY, VecZ] into the camera coordinate system xyz and rotates the virtual block vector IBv [VecX, VecY, VecZ] around the origin OC. The angle is calculated as a virtual block rotation angle IBei [0, φ, ψ].
That is, the virtual block generation unit 24 converts the virtual block vector IBv [VecXw, VecYw, VecZw] in the world coordinate system into the virtual block vector IBv [VecXc, VecYc, VecZc] in the camera coordinate system using the external parameters of the camera. To do.
Then, the virtual block generation unit 24 obtains a rotation angle [θ, φ, ψ] about each three-dimensional axis around the origin (lens focal point) in the camera coordinate system xyz. Since the rotation of the virtual disk IC is independent of the area change, it is assumed that θ = 0.0 (fixed). Then, since the following equation (3a) is established from the rotation transformation of the space, the virtual block rotation angle IBei [0, φ, ψ] can be obtained by the following equations (3b) and (3c).
Here, there are two solutions for the rotation angle φ around the Y-axis, φ and φ ± π, but they are front-back relations and do not affect the calculation results and need not be adjusted.

Procedure 4: Calculate the disk circumference point ICei.
As shown in FIG. 14A, the virtual block generator 24 generates a virtual disk IC having a radius r corresponding to the size of the virtual block IB around the origin OC. Then, the virtual block generation unit 24 calculates the disk circumference point ICei [Xei, Yei, Zei] by dividing the points on the circumference of the virtual disk IC into N. The symbol ei i is the number of the circumference point.
Here, it is assumed that the initial angle [θ0, φ0, ψ0] = [0, 0, 0] of the virtual disk IC. That is, the virtual disk IC is set on the yz plane of the camera coordinate system xyz. Dividing the point on the circumference of this virtual disk IC by 360 degrees / N, the division point number is ei, and the position of the division point ICei [Xei, Yei, Zei] on each circumference is given by the following equation (4) Ask for. The radius r of the virtual disk IC may be determined according to the size of the virtual block IB in the real space, or may be determined by the method disclosed in “1.3 Virtual Block and Detection Speed” described above. .

-Procedure 5: The disk circumference point ICei is rotationally converted.
As shown in FIG. 14B, the virtual block generation unit 24 calculates the plurality of disk circumference points ICei [Xei, Yei, Zei] in the camera coordinate system xyz by the equation (5), respectively. Rotation conversion is performed at the rotation angle IBei [0, φ, ψ].

  The calculation for obtaining the N-point N of the N-square that composes the virtual disk IC may be performed based on the world coordinate system (described later as another solution), but here, conversion calculation such as rotation is performed based on the camera coordinate system. Do. If the rotation is calculated around the origin of the camera coordinate system xyz, the amount of calculation and the discrimination process can be reduced without generating an unnecessary virtual image or the like.

Procedure 6: Calculate the virtual block circumference point IBei.
As shown in FIG. 14C, the virtual block generation unit 24 converts the plurality of the disk circumference points ICei [Xeiθ, Yeiφ, Zeiψ] that have been rotationally converted into the world coordinate system XYZ, 6) The virtual block circumferential point IBei [Xeiθp, Yeiφp, Zeiψp] is obtained in the world coordinate system XYZ by shifting the virtual block vector IBv [VecX, VecY, VecZ] by the calculation of 6).

  As described above, the virtual block generation unit 24 temporarily returns each vertex of the disk circumference point ICei [Xeiθ, Yeiφ, Zeiψ] to the world coordinate system using the inverse transformation of the external parameter, and then continues to the virtual disk IC. By adding the virtual block vector IBv [VecX, VecY, VecZ] to each world coordinate [Xeiθ, Yeiφ, Zeiψ], shift movement is performed in the real space represented by the world coordinate system, and the offset is adjusted.

Step 7: Display block circumference point BDei is calculated.
The display block calculation unit 26 performs perspective coordinate transformation on the virtual block circumference point IBei [Xeiθp, Yeiφp, Zeiψp] in the world coordinate system XYZ, thereby obtaining an image coordinate system of the image IM as shown in the following equation (7). Convert to display block circumference point BDei [Uei, Vei] in UV.
In other words, the display block calculation unit 26 projects all the circumferential points IBei [Xeiθp, Yeiφp, Zeiψp] of the virtual block IB onto the two-dimensional image coordinate system UV, so that the display block circumferential point BDei [Uei, Vei] Is calculated.

Procedure 8: Display block area Bda is calculated.
The area calculation unit 28 calculates an area of the display block BD surrounded by the plurality of display block circumference points BDei [Uei, Vei] as a display block area Bda.
That is, the display block area BBa is obtained for the region surrounded by the projection points [Ue1, Ve1] to [UeN, VeN] on the image IM. The display block area Bda can be obtained from the number of pixels drawn by actually drawing a region surrounded by [Ue1, Ve1] to [UeN, VeN] on the image IM. For example, as shown below, The virtual disk can be approximated by using Heron's formula and the like, assuming N-square.

First, similarly to [Ue1, Ve1] to [UeN, VeN], the projection coordinate transformation is performed on the center ICo [Xp Yconst, Zp] of the virtual disk IC to obtain the position [Ue0, Ve0] on the image plane.
Then, pay attention to a triangle surrounded by two adjacent points [Ue0, Ve0] and [Uei, Vei], [Uei + 1, Vei + 1].
The side [ai, bi, ci] connecting the three points is obtained by the following equation (8a).
This is repeatedly executed from [Ue1, Ve1] to [UeN, VeN], and the image IM of the virtual disk IC is obtained by the following formulas (8b) and (8c) from the total area of each divided triangle obtained by the Heron area formula. The display block area Bda above is obtained.

When [Ue0, Ve0] or [Ue1, Ve1] to [UeN, VeN] is converted outside the image IM, the corresponding side is clipped (cut) at the intersection with the frame of the image IM, and the screen Calculate the area enclosed by the corners and clip points.
When all the points are converted to the outside of the screen, or when no triangle is formed, the area is clipped with 0 and treated as invalid.

Step 9: Calculate the display block radius BDr.
Subsequently, the display block radius calculation unit 30 calculates a display block radius BDr by calculating a radius when the display block area BDi is set as a circle area.
That is, when the size of the virtual disk IC is indicated, it is possible to treat the display block radius BDr as the approximate radius by converting the display block radius BDr with the following expression (9) for the obtained display block area BDa, instead of using the display area. .
In particular, in the camera 10 having distortion, the display virtual disk becomes closer to a perfect circle because the point moved to the outside due to the influence of the distortion is eccentric to the center of the screen. Therefore, the efficiency of approximating the N-gon by a circle is good.

Step 10: Generate a display radius map.
Then, the correlation generation unit 32 generates the display radius map 34 by correlating the display block radius BDr and the position of the image IM (step S10).

15A is an original image using the wide-angle camera 10, and FIG. 15B is a corrected image obtained by correcting wide-angle distortion by the method described in Patent Document 5, both of which are images IM in this embodiment.
A hypothetical ground height Yconst was set, and a display block BD of a spherical virtual block IB arranged from a position corresponding to an infinite point according to the resolution of the image coordinate system to the vicinity of the host vehicle MT was obtained. FIGS. 15C and 15D show the display block BD superimposed on the image IM.
FIGS. 15E and 15F are diagrams in which only the display block BD of FIGS. 15C and 15D is extracted.
In this way, the display block can be obtained by a posteriori calculation regardless of the wide-angle level of the camera 10 and the correction method.

  Next, a modification when the display block BD is generated from the virtual block IB will be described with reference to FIGS. In procedure 8, the virtual disk IC is approximated by an N-gon when calculating the display block area BDa. This N-gon is originally a regular N-gon that approximates a perfect circle.

As shown in FIG. 16, in the display on the image IM, N-shaped polygons having different deformation and flatness are generated depending on the setting position (center position) of the virtual sphere (spherical virtual block IB) defined by the perspective at the time of projection, etc. To do. This degree of deformation is also affected by lens distortion and parallax of the camera 10.
Therefore, strictly speaking, it is ideal that the display block BD uses a deformed N-gon instead of a circle to determine the validity of the flow. However, it is very difficult to handle various N-gons individually according to the coordinate values (positions) of the image IM.
For this reason, the display block BD is preferably handled with an approximate circle. Even in the case of the display block BD that is an approximate circle, (1) it is possible to evaluate the correctness of the flow Flw within the range of the image IM [U, V] without conversion to the real space during the validity determination process of the flow Flw. Yes, (2) When evaluating with the image IM [U, V], it is easier to handle with the radius.

  Referring to FIG. 15C again, in the non-linear image IM with wide-angle distortion, the N-gon is close to a circle. Referring to FIG. 15D, the N-gon in the image IM in which the wide-angle distortion is corrected is an elliptical shape. These display blocks BD are circular regardless of image distortion or correction in the method of drawing a circle from the display block radius BDr.

Next, another solution such as the procedure 5 shown in FIG. 12 will be examined with reference to FIGS. The procedure number shown in FIG. 12 is set to n, and a is added to another solution of procedure n as procedure na. For example, another solution of the procedure 4 shown in FIG. 12 is a procedure 4a.
The creation of the display radius map 34 can be obtained based on the following disk model as another solution. Let [X, Y, Z] = [Xp, Yconst, Zp] radius be the center coordinate of the virtual block IB (sphere). The procedure up to procedure 3 is the same as that described above.
Procedure 1: Obtain the camera coordinate origin world system OW.
Procedure 2: As shown in FIG. 17, a distance L (vector length) connecting the camera coordinate origin world system OW and the center of the sphere is obtained by the following equation (10).

Further, the center position [Xp, Yconst, Zp] of the sphere is converted into the camera coordinate system [xp, yp, zp].
Procedure 3: Find the rotation angle [θ, φ, ψ] to [xp, yp, zp] in the camera coordinate system.
Here, θ: rotation angle in the camera coordinate system xyz around the x axis, φ: around the y axis, and ψ: around the z axis. Further, in order to obtain the rotation of the vector, θ = 0 is set as a constraint condition.

Procedure 4a: A model composed of a virtual disk IC and a vector of length L passing through the center of the disk is defined as shown in FIG. This virtual disk IC may be defined as an N square and given by N vertex coordinates.
Step 5a: As shown in FIGS. 18B and 18C, a virtual disk IC and a vector of length L passing through the center of the disk are represented by [θ, φ, ψ] = [θa, φa, ψa] camera coordinate system Rotate with

Procedure 6a: The virtual disk IC is returned to the world coordinate system XYZ by inverse transformation. In another solution, the procedure from step 7 is the same as the procedure shown in FIG.
However, in this alternative solution, there are two solutions for φ 1 at this time, φa and φ ± π, which must be separately determined. Therefore, as in the procedure 4b and subsequent steps, the method corresponding to the proposed current method shown in FIG.

Here, with reference to FIG. 19, a calculation example in the procedure shown in FIG. 12 will be described again as a procedure nb.
Procedure 4b: As shown in FIG. 19A, a model is defined in which a virtual disk is set on the yz plane. A virtual disk is defined by an N-gon and given by N vertex coordinates.
Procedure 5b: As shown in FIG. 19B, the virtual disk and a vector of length L passing through the center of the disk are rotated by [θ, φ, ψ] = [θa, φa, ψa] in the camera coordinate system.

Step 6b: Return the virtual disk to the world coordinate system by inverse transformation.
Step 7b: For the coordinate group returned to the world coordinate system, the offset of the camera coordinate origin and the virtual sphere center is taken into account, and the disk is shifted with [Vecx, Vecy, Vecz] to obtain the virtual disk N vertex in the world coordinate system .
In other words, in this operation, if there is no distance L from the camera coordinate origin OC at the time of setting (if handled independently), even if the solution generated for φ is either φa or φ ± π, FIGS. As shown in (2), the operation of the coordinate system is not affected because of the relationship between the front and back of the disk surface.

2.2 Effects of calculation by virtual disk As described above, the virtual block generation unit 24 calculates the virtual block vector IBv and the virtual block rotation angle IBb, and before and after this, the virtual block generation unit 24 sets the origin to the origin. A disk circular point ICei is calculated by generating a virtual disk IC having a radius corresponding to the size of the virtual block and dividing points on the circumference of the virtual disk IC into N parts. And if the virtual block IB itself is not calculated, but the rotational transformation is performed on the disk circumference point ICei of the virtual disk IC centered on the camera coordinate origin world system OW, a virtual image that is difficult to discriminate on calculation appears. It appears as the front and back of the virtual disk IC with the same position. Therefore, it is not necessary to individually determine whether the image is a virtual image. Further, the virtual block circumference point ICei of the world coordinate system XYZ is transformed into a display block circumference point BDei in the image coordinate system UV of the image IM by perspective coordinate transformation, and the plurality of the display block circumference points Since the area of the display block surrounded by BDei is calculated as the display block area, even if the shape of the display block circumference point BDei is not a perfect circle due to the relationship with lens distortion, angle of view, etc. The distortion can be accurately expressed in the area, and the display block BD can be corrected to a circle by obtaining the display block radius BDr from the area of the N-gon with distortion.
In this way, the calculation using a virtual disk reduces the amount of calculation for discriminating virtual images, etc., and the display block radius that accurately responds to changes in size (area) due to distortion even when there is wide-angle or other distortion BDr can be obtained.

DESCRIPTION OF SYMBOLS 10 Camera 12 Flow extraction part 14 Display block management part 16 Flow determination process part 18 Display block radius related process 20 Virtual block specific process 22 Assumed ground height process 24 Virtual block generation part 26 Display block calculation part 28 Area calculation part 30 Display block radius Calculation unit 32 Correlation generation unit 34 Display radius map
MT
MA approaching vehicle
Flw flow
FlwG flow group
UV image coordinate system
OC camera coordinate origin
OW Camera coordinate origin world system
IP image plane
xyz camera coordinate system
XYZ world coordinate system
IM image
IB virtual block
IBc virtual block center
IBv virtual block vector
IBb Virtual block rotation angle
IBr virtual block radius
IC virtual disk
ICei disk circumference
IBei virtual block circumference
BD display block
BDr Display block radius
BDa Display block area
BDei Display block circumference point
f Focal length

Claims (9)

A camera that continuously captures the area around the vehicle,
A flow extraction unit that divides an image input from the camera into division images with a template and extracts a flow between the division images that are temporally mixed;
A display block management unit that projects a virtual block having a predetermined position and size in a three-dimensional space of a range captured by the image as a display block on a two-dimensional image coordinate system of the image;
A flow determination processing unit that determines the validity of the flow with reference to the size of the display block at the start point or end point of the flow,
An optical flow processing device characterized by that. The display block management unit includes the display block radius related processing for approximating the display block to a circle and relating the approximated circle radius as a display block radius to the size of the display block.
The optical flow processing apparatus according to claim 1, wherein: The display block management unit specifies the virtual block size in the three-dimensional space based on a predetermined detection speed for a vehicle approaching the host vehicle in the three-dimensional space. With processing,
The optical flow processing apparatus according to claim 1 or 2, wherein The display block management unit assumes the height of the virtual block in the three-dimensional space to be an assumed ground height predetermined in the three-dimensional space, and displays the virtual block in the image coordinate system. With hypothetical ground processing that projects to the block position,
The optical flow processing apparatus according to claim 1, 2 or 3. A virtual block generation unit that generates a plurality of virtual blocks of the same size predetermined in a three-dimensional space in a range captured by an image for flow extraction;
A display block calculation unit for calculating a display block by projecting the plurality of virtual blocks onto a two-dimensional image coordinate system of the image,
An area calculation unit for calculating the area of the display block as a display block area;
A display block radius calculating unit that calculates a display block radius by calculating a radius when the display block area is an area of a circle;
A correlation generation unit that generates a display radius map by correlating the display block radius and the position of the image,
A display radius map generation device characterized by the above. The virtual block generator is
A vector from the camera coordinate origin world coordinate system converted from the origin that is the focal position in the camera coordinate system to the three-dimensional space of the world coordinate system to the center of the virtual block is calculated as a virtual block vector in the world coordinate system,
Converting the virtual block vector to the camera coordinate system, calculating a rotation angle around the origin of the virtual block vector as a virtual block rotation angle,
By generating a virtual disk with a radius corresponding to the size of the virtual block at the origin and dividing the point on the circumference of the virtual disk by N, the disk circumference point is calculated,
In the camera coordinate system, each of the plurality of disk circumferential points is rotationally transformed at the virtual block rotation angle, respectively.
The virtual block circumference point is calculated in the world coordinate system by shifting the rotation of the plurality of the disk circumference points to the world coordinate system and shifting the virtual block vector.
The display block calculation unit
By transforming the virtual block circumference point of the world coordinate system into a perspective coordinate transformation to the display block circumference point in the image coordinate system of the image,
The area calculation unit
Calculating an area of the display block surrounded by the plurality of display block circumference points as a display block area;
6. The display radius map generating apparatus according to claim 5, wherein the display radius map generating apparatus is a display radius map generating apparatus. A virtual block generating step for generating a plurality of virtual blocks having the same size in a three-dimensional space in a range captured by an image for flow extraction;
A display block calculating step of calculating a display block by projecting each of the plurality of virtual blocks onto a two-dimensional image coordinate system of the image;
Calculating an area of the display block as a display block area;
A display block radius calculating step of calculating a display block radius by calculating a radius when the display block area is an area of a circle;
A correlation generation step of generating a display radius map by correlating the display block radius and the position of the image,
A display radius map generation method characterized by that.   A display radius map generation program for executing the method according to claim 7 using an arithmetic unit. A camera that continuously captures the area around the vehicle,
A flow extraction unit that divides an image input from the camera into division images with a template and extracts a flow between the division images that are temporally mixed;
A display for storing a display radius map in which a virtual block having a predetermined position and the same size in a three-dimensional space of a range captured by the image is projected as a display block on the two-dimensional image coordinate system of the image The block manager,
A flow determination processing unit that determines the effectiveness of the flow by referring to the display radius map with respect to the size of the display block at the start point or the end point of the flow,
An optical flow processing device characterized by that.