JP6060612B2 - Moving surface situation recognition device, moving object, and program - Google Patents

Description

  The present invention relates to a moving surface situation recognition device, a moving body, and a program.

  In recent years, in order to improve the safety and convenience of automobiles, driving support systems using in-vehicle cameras have been put into practical use. In order to realize functions such as collision prevention in a driving support system, it is very important to recognize road surface conditions in a traveling space. Since a stereo camera can be used to obtain not only visual information but also distance information, recently, many studies on driving support systems using a stereo camera have been conducted.

  In Patent Document 1, for the purpose of accurately determining a road surface condition such as a road surface shape and an object on the road, the white line three-dimensional information obtained from the luminance image and the parallax is compared with a feature histogram template, The original shape is acquired. Further, the three-dimensional shape of the road surface is recognized by interpolating the three-dimensional shape of the white line. Then, a technique for generating a difference image indicating a difference between the converted image and the road surface image by converting the image on the assumption that the road surface three-dimensional shape is photographed, and detecting a road object based on the difference image is disclosed. Has been.

  However, the above conventional road surface condition recognition is based on the three-dimensional information by calculating the feature amount using both the luminance image photographed by the stereo camera and the histogram parallax obtained therefrom. It is. Therefore, the recognition algorithm is complicated, and there is a problem that implementation on a camera and real-time processing are difficult.

  In the technique described in Patent Document 1, a histogram obtained from a captured image of a stereo camera is compared with a reference feature amount. However, by acquiring the white line extraction and parallax of the luminance image, the road surface three-dimensional shape is acquired from the three-dimensional shape of the plurality of white lines, and the difference indicating the difference between the converted image of the road surface three-dimensional shape and the road surface image Since the image is generated and objects on the road are detected based on the difference image, the road surface condition recognition method is complicated, and it is difficult to implement the internal processing of the camera and to realize real-time performance. Has not been resolved.

  Therefore, the present invention has been made in view of the above-described problems of the prior art, and by using distance data and parallax images obtained using a stereo camera, a simple configuration and processing can be performed at high speed. It is an object of the present invention to provide a moving surface state recognition device, a moving body, and a program that can automatically recognize a road surface state.

In order to solve the above-described problem, the moving surface situation recognition apparatus according to the present invention described in claim 1 is configured to display the first and second images respectively captured by the first and second imaging means mounted on the moving body. stereo image acquisition means acquire a Sasei formed means vision for generating a parallax image including a parallax data corresponding to the first and second image based on the first and second image, based on the parallax data The first parallax map in which the highest point indicated by the parallax image and the parallax data is associated with the first parallax map that is associated with the lowest point indicated by the parallax image and the parallax data. A recognition process based on a height difference calculated from a height indicated by the first parallax map and a height indicated by the second parallax map; line It characterized in that it comprises a recognition processing means.

  According to the present invention, in addition to using parallax data for recognition processing, the feature quantity of the height difference of a road surface on which a moving body such as an automobile travels is used as known information, thereby speeding up and automating the system. Therefore, it is possible to obtain a moving surface state recognition device, a moving body, and a program that can automatically recognize a road surface state at a high speed with a simple configuration and processing using a stereo camera.

It is a schematic block diagram explaining the whole structure of the movement surface condition recognition apparatus which concerns on embodiment of this invention. It is a figure explaining the functional block of the movement surface condition recognition apparatus which concerns on embodiment of this invention. It is a figure explaining the functional block which produces | generates a parallax image and a brightness | luminance image using the stereo image acquired from the stereo camera of the moving surface condition recognition apparatus which concerns on embodiment of this invention. It is a figure explaining the structure and imaging relationship of the stereo camera of the moving surface condition recognition apparatus which concerns on embodiment of this invention. It is a figure explaining the (a) example of a left camera image obtained with the stereo camera of the movement surface condition recognition apparatus which concerns on embodiment of this invention, and the (b) example of a right camera image. It is a figure explaining the left-right luminance image and parallax image of the moving surface condition recognition apparatus which concern on embodiment of this invention. It is a figure explaining the relationship between the parallax image of the moving surface condition recognition apparatus which concerns on embodiment of this invention, and U-Map, U-Height-Map, and U-Low-Map. It is a figure explaining the relationship between the luminance image of the moving surface condition recognition apparatus which concerns on embodiment of this invention, and U-Map, U-Low-Map, and U-Height-Map. It is a figure explaining the road surface recognition process flow of the moving surface condition recognition apparatus which concerns on embodiment of this invention. It is a figure explaining the example of the road surface condition recognition of the moving surface condition recognition apparatus which concerns on embodiment of this invention.

  Next, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In addition, in each figure, the same code | symbol is attached | subjected to the part which is the same or it corresponds, The duplication description is simplified thru | or abbreviate | omitted suitably. The present invention relates to a stereo camera, and recognizes a road surface situation in real time with a simple configuration, that is, measures a distance using a stereo camera and recognizes a road surface situation based on distance information.

  Specifically, with respect to the road surface on which an automobile or the like moves, the difference amount of the height of the road surface that is the moving surface is arranged, and the moving surface height difference feature amount that is the specified data is preset in the memory, that is, stored. Keep it. Then, parallax data is acquired from the in-vehicle stereo camera, and an X direction parallax image histogram (X direction parallax map: U-Map) and an X direction parallax height map (U-Low-Map, U-Height-Map) Is calculated. Further, in the recognition process, the height difference is compared with the moving surface height difference feature amount on the U-Low-Map and the U-Height-Map based on the information of the line extracted from the U-Map, and the moving surface And objects are recognized. A feature is that the road surface range, road objects, and road objects are obtained based on the positional relationship with the luminance image.

  The characteristics of the present invention described above will be described in detail with reference to the drawings. First, the overall configuration of the moving surface situation recognition apparatus according to the embodiment of the present invention will be described. FIG. 1 is a schematic configuration diagram illustrating the overall configuration of a moving surface situation recognition apparatus according to an embodiment of the present invention.

  The moving surface state recognition apparatus according to the embodiment of the present invention is a stereo camera system 100 having a moving surface state recognition function as shown in FIG. From left camera 101 and right camera 102, which are two cameras mounted in parallel on the left and right sides in order to acquire a distance, and a stereo image that is an image of a subject on a moving plane acquired by two cameras 101, 102 An FPGA (Field Programmable Gate Array) 103 for calculating parallax data at high speed, an MPU (Micro Processing Unit) 106 for generating and recognizing a parallax map, and a captured image, parallax data, and a recognition result image are stored. Frame buffer 104, memory 107 for storing data specifying a road surface height difference feature quantity obtained from the camera for use in recognition processing, and an image display device for displaying the results of the road surface recognition processing and camera images 105.

  Next, functional blocks of the moving surface situation recognition apparatus according to the embodiment of the present invention will be described in detail. FIG. 2 is a diagram illustrating functional blocks of the moving surface state recognition device according to the embodiment of the present invention. 2 is mounted on a moving body that moves on the road surface, for example, an automobile or the like.

  The moving surface situation recognition apparatus 200 has a stereo image acquisition unit 201 that acquires stereo images from the two cameras 101 and 102 (FIG. 1), and a parallax that generates a parallax image by performing stereo matching processing from the acquired stereo images. The image generation unit 202, the road surface height difference feature amount storage unit 204 that stores the road surface height difference feature amount of the moving moving surface, and the frequency of the parallax data represented by the parallax data histogram on the parallax image, etc. A parallax map generation unit 203 that creates a parallax map, a recognition processing calculation unit 205 that performs a recognition process from the parallax map and the luminance image, and a recognition result image display unit that outputs and displays the recognition result image of the recognition processing calculation unit 205 206.

  Next, each component of the functional block shown in FIG. 2 will be described. The stereo camera image acquisition unit 201 attaches the left and right cameras 101 and 102 in parallel, and outputs images captured by the left camera 101 and the right camera 102, respectively. The parallax image generation unit 202 performs stereo matching processing for detecting partial images corresponding to each other between the acquired stereo camera images, and generates a parallax image composed of parallax between the partial images and including parallax data.

  Here, stereo matching processing performed based on the stereo camera image acquired by the stereo image acquisition unit 201 will be described with reference to FIGS. 3 to 5. FIG. 3 is a diagram illustrating functional blocks that generate a parallax image and a luminance image using a stereo image acquired from the stereo camera of the moving surface situation recognition apparatus 200 according to the embodiment of the present invention. FIG. 4 is a diagram for explaining the structure and imaging relationship of the stereo camera of the moving surface situation recognition apparatus according to the embodiment of the present invention. FIG. 5 is a diagram illustrating (a) an example of a left camera image and (b) an example of a right camera image obtained by a stereo camera of the moving surface situation recognition apparatus according to the embodiment of the present invention.

  The stereo matching process is executed by the luminance image input unit 301 and the parallax image calculation unit 302 in FIG. The stereo image acquired by the stereo image acquisition unit 201 is two stereo images acquired from the left and right camera lenses and sensors shown in FIG. As shown in FIG. 4, stereo images are input to the left lens L 401 and the right lens R 402, respectively. FIG. 5 shows an example of (a) a left camera image and (b) a right camera image of an input stereo image. The left lens L 401 and the right lens R 402 have different imaging positions of the same subject 403 in the left and right sensors.

  The luminance image input unit 301 receives a luminance image from the left image or the right image of the stereo image. The input image is stored in a memory area (not shown) of the stereo camera. In the parallax image calculation unit 302, the difference between the connection positions of the left and right images in the shooting target, that is, the parallax is calculated using the stereo image acquired by the stereo image acquisition unit 201. As shown in FIGS. 5A and 5B, a parallax image is calculated by performing block matching on the same portion of the stereo image formed by the left and right lenses 401 and 402.

  Specifically, the left and right images are divided into blocks, and when the difference between the blocks in the left and right images of the same object is the smallest, the parallax is obtained from the portion where the blocks match. That is, when the value Ir (x + Δ, y) of the block of the right image is found with respect to the value Il (x, y) of the block of the left image, the parallax data Δ that is the difference is obtained. For example, a 1280 × 960 pixel image is divided into blocks of 5 × 5 size, and a parallax value in each block is obtained. The optimum block size is adjusted and set by experiment.

  In the example shown in FIG. 4, the distance between the position of the point O on the subject 403 and the imaging center of the left lens 401 and the imaging center of the right lens 402, which are the imaging positions in the left and right images, is , Δ1 and Δ2, respectively. The parallax data that is the difference is Δ = Δ1 + Δ2. The parallax image has a parallax value at the pixel position.

  Next, examples of left and right luminance images and parallax images will be described with reference to FIG. FIG. 6 is a diagram illustrating the left-right luminance image and the parallax image of the moving surface situation recognition device according to the embodiment of the present invention. The parallax map generated in the embodiment of the present invention includes a U-Map that is a histogram of parallax data obtained by scanning in the X direction on a parallax image, a U-Height-Map that is a maximum height map of each parallax data, and a minimum. It is U-Low-Map which is a height map.

  That is, (a) the left image acquired by the left camera 101 (FIG. 1) and (b) the right image acquired by the right camera 102 are output as luminance images in the stereo image acquisition unit 201 (FIG. 3). Then, the stereo matching processing is performed in the parallax image generation unit 202, and (c) the parallax image is displayed in color.

  Next, the relationship between a parallax image and U-Map, U-Height-Map, and U-Low-Map will be described with reference to FIG. FIG. 7 is a diagram illustrating the relationship between the parallax image of the moving surface situation recognition device according to the embodiment of the present invention, and U-Map, U-Height-Map, and U-Low-Map.

  In FIG. 7, (a) in the 0-axis 701 on the X-axis in the parallax image, the parallax data of 2, 5, 3, and 3 are associated with the 0th to 3rd rows on the Y-axis. In (b) U-Map, since the number of appearances of parallax data, that is, the value indicating the appearance frequency becomes the value of U-Map, (a) In the 0 coordinate 701 on the X axis of the parallax image, 2 is 1 Since 3 times, 2 times and 5 times each appear, (b) 0-coordinate 702 of the X-axis of U-Map is 1 in the 2 column, 2 in the 3 column, 2 in the 5 column 1 is defined respectively.

  Next, in (c) U-Height-Map, since the highest number of lines of parallax data is the value of U-Height-Map, (a) 2 coordinates in the X coordinate 0 701 of the parallax image Is 0 row, 3 is 3 row, and 5 is 1 row, so that (c) 0-coordinate 703 of X-axis of U-Height-Map is 0, 3 column in 2 column 1 is set in the columns 3 and 5 respectively.

  Finally, in (d) U-Low-Map, since the lowest number of rows of parallax data is the value of U-Low-Map, (a) in the 0 coordinate 701 of the X axis of the parallax image, 2 is Lines 0 and 3 are 2, and line 5 is the lowest number of lines. Therefore, (d) 0-coordinate 704 of the X axis of U-Low-Map is displayed in columns 2 and 0 and 3. 1 is defined in columns 2 and 5 respectively. Hereinafter, (a) parallax data after the 0 coordinate of the X axis of the parallax image can be determined in the same manner.

  Next, an example of the relationship between a luminance image and U-Map, U-Height-Map, and U-Low-Map will be described with reference to FIG. FIG. 8 is a diagram for explaining the relationship between the luminance image of the moving surface situation recognition apparatus according to the embodiment of the present invention and U-Map, U-Low-Map, and U-Height-Map.

  In the road surface height difference feature amount storage unit 204 (FIG. 2), the road surface height difference feature amount to be detected is preset, that is, stored before processing. The road surface height difference feature quantity is used as a reference for discriminating a road shoulder from an object. Based on the parallax image, the recognition processing calculation unit 205 first generates three parallax maps, U-Map, U-Height-Map, and U-Low-Map. Then, with respect to the generated parallax map, a road shoulder, a road object existing on the road surface, and a road object existing outside the road surface are recognized.

  For example, in FIG. 8, (a) in the luminance image, the building 801 is recognized as a certain shadow 803 in (b) U-Map, but (c) is recognized as a road shoulder 803 in U-Low-Map. d) In the U-Height-Map, the building 801 is recognized as the outdoor object 801. In (a) the luminance image, the automobile 802 is recognized as a line segment 802 after (b) U-Map is subjected to noise approximation and line approximation for removing noise present in the parallax image. (D) In U-Height-Map, the automobile 802 is recognized as a road object 802.

  Next, a specific processing flow of the moving surface situation recognition apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 9 is a diagram illustrating a road surface recognition process flow of the moving surface state recognition device according to the embodiment of the present invention. First, U-Map, U-Height-Map, and U-Low-Map are generated in the processing of Step (hereinafter referred to as “S”) 901 based on the luminance image and the parallax image. In the processing of S902, noise removal and line segment approximation processing are performed based on the input U-Map, and target object extraction processing is performed.

  In the process of S903, the height difference is calculated for each line segment based on the positional relationship between the U-Low-Map and U-Height-Map with respect to the line segment extracted on the U-Map. Calculated. Based on the calculated height difference and the parallax map, the actual height deviation is obtained and compared with the road surface height difference feature amount stored in the road surface height difference feature amount storage unit 204 in advance. Is recognized.

  Specifically, when the actual height deviation is larger than the road surface height difference feature amount, it is recognized as an object. Further, when the actual height deviation is smaller than the road surface height difference feature amount, it is recognized as a road shoulder. In step S904, the road surface range is obtained based on the positional relationship of the road shoulder on the luminance image, and the recognized objects are classified into road objects and road objects based on the road surface range. Note that the recognition result image display unit 206 (FIG. 2) outputs various images from the camera and recognition result images (images in which road shoulders and objects are superimposed and displayed with lines, frames, etc.) as images for the user to confirm. It has a function.

  In addition, each operation | movement flow of the charging device in embodiment of this invention shown in FIG. 9 can also be made to perform the program on a computer. That is, a CPU (not shown) built in a control unit (not shown) of the moving surface situation recognition apparatus 100 stores a program stored in a storage unit (not shown) including a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. This is done by loading and executing each processing step of the program sequentially.

  The recognition result processing image in the embodiment of the present invention will be described with reference to FIG. FIG. 10 is a diagram illustrating an example of road surface situation recognition of the moving surface situation recognition apparatus according to the embodiment of the present invention. In FIG. 10, (a) in the luminance image, the building 801 is recognized as some shadow 803 in (b) U-Map, but (d) in U-Height-Map, it is recognized as an outdoor object 801 ( e) Recognized as a building 801 as a recognition result image.

  In (a) the luminance image, the automobile 802 is recognized as a road object 802 in (d) U-Height-Map, and (e) the automobile 802 is recognized as a recognition result image. Further, (a) in the luminance image, the road sign 804 is recognized as a road shoulder 805 in (c) U-Low-Map, and (e) is recognized as a road sign 804 as a recognition result image. As described above, (e) in the recognition result image, the road shoulder is used as a line, and the object on the road and the object outside the road are displayed in different colors so as to be superimposed on the luminance image and output.

  As described above, according to the present invention, in the moving surface recognition camera system including a stereo camera, the parallax information acquired by the stereo camera and the moving surface height difference feature amount stored in advance are obtained. To obtain a moving surface situation recognition device, a moving body, and a program capable of automatically recognizing road surface conditions such as road shoulder recognition and discrimination between roadside objects and roadside objects at high speed. It has become possible.

  The present invention has been described above by the preferred embodiments of the present invention. While the invention has been described with reference to specific embodiments thereof, various modifications and changes can be made to these embodiments without departing from the broader spirit and scope of the invention as defined in the claims. is there.

100 stereo camera system 101 left camera 102 right camera 103 FPGA
104 Frame buffer 105 Image display device 106 MPU
DESCRIPTION OF SYMBOLS 107 Memory 200 Moving surface condition recognition apparatus 201 Stereo image acquisition part 202 Parallax image generation part 203 Parallax map generation part 204 Road surface height difference feature-value memory | storage part 205 Recognition processing calculating part 206 Recognition result image display part 301 Luminance image input part 302 Parallax Image calculation unit 401 Left lens 402 Right lens 403 Subject 801 Building, roadside 802 Car, line segment 803 Shadow, shoulder 804 Road sign 805 Road shoulder

JP 2002-150302 A

Claims (7)

Each stereo image acquisition means acquire the first and second image captured by the first and second imaging means mounted on a mobile body,
And Visual Sasei forming means for generating a parallax image including a parallax data corresponding to the first and second image based on the first and second images,
Based on the parallax data, the first parallax map in which the parallax image and the point with the highest height indicated by the parallax data are associated, and the point with the lowest height indicated by the parallax image and the parallax data a parallax map generating means for generating a second parallax map is associated,
Recognition processing means for performing recognition processing based on a height difference calculated from the height indicated by the first parallax map and the height indicated by the second parallax map;
Moving surface situation recognition apparatus, which comprises a. A moving surface height difference feature amount storing means in which a moving surface height difference feature amount that is a height from the moving surface of the subject obtained from the both imaging means is stored;
Recognition processing calculation means for recognizing an object constituting the subject based on the parallax map and the moving surface height difference feature quantity;
The moving surface situation recognition apparatus according to claim 1, wherein: The object constituting the subject includes a moving surface outside object that exists outside the moving surface and a moving surface object that exists on the surface of the moving surface, and the recognition processing calculation means includes the moving surface. 3. The moving surface state recognition apparatus according to claim 1, wherein after the recognition, the object outside the moving surface and the object on the moving surface are distinguished and recognized. It said output means includes an image of the object, moving surface situation recognition apparatus according to any one of claims 1 to 3, characterized in that superimposed and displayed on the objects constituting the object. The disparity map generating means, the before generating the disparity map, moving surface situation recognition apparatus according to claim 1, any one of 4, characterized in that the removal of noise included in the parallax image. Each stereo image acquisition means acquire the first and second image captured by the first and second imaging means mounted on a mobile body,
Parallax generating means for generating a parallax image including parallax data corresponding to the first and second images based on the first and second images ;
Based on the parallax data, the first parallax map in which the parallax image and the point with the highest height indicated by the parallax data are associated, and the point with the lowest height indicated by the parallax image and the parallax data a parallax map generating means for generating a second parallax map is associated,
Recognition processing means for performing recognition processing based on a height difference calculated from the height indicated by the first parallax map and the height indicated by the second parallax map;
Mobile which comprises a. The first and second imaging means mounted on a mobile body and each processed that Tokusu preparative first and second images captured,
A process that generates a parallax image including a parallax data corresponding to the first and second image based on the first and second images,
Based on the parallax data, the first parallax map in which the parallax image and the point with the highest height indicated by the parallax data are associated, and the point with the lowest height indicated by the parallax image and the parallax data a process that generates a second parallax map is associated,
Processing for performing recognition processing based on a height difference calculated from the height indicated by the first parallax map and the height indicated by the second parallax map ;
The program characterized by including.