JP6855325B2 - Image processing equipment, stereo camera systems, moving objects, road surface shape detection methods and programs - Google Patents

Description

The present disclosure relates to an image processing device, a stereo camera system, a moving body, a road surface shape detecting method and a program.

In recent years, a stereo camera system has been provided on a moving body such as an automobile, and is used for detecting a three-dimensional object and measuring a distance. In a stereo camera system, a method has been proposed in which a distance distribution is obtained over the entire image from each of a plurality of images acquired from a plurality of cameras, and an obstacle is discriminated from the information of the distance distribution (for example, a patent). Reference 1).

Japanese Unexamined Patent Publication No. 5-265547

However, the conventional stereo camera system recognizes an object on the road on the premise that the road surface on which the moving body travels is horizontal. Therefore, if the actual road surface is inclined or undulated, the height of an object on the road may be erroneously determined. For example, when the right side of the road surface is high, a low-height object on the right side of the road surface may be determined to be a large high-height object. In addition, a high-height object on the left side of the road surface may be determined to be a small low-height object.

Therefore, an object of the present invention made by paying attention to these points is an image processing device, a stereo camera system, a moving body, and a road surface shape detection which improve the accuracy of object recognition by detecting shapes such as inclination and undulation of the road surface. To provide methods and programs.

The image processing apparatus according to the embodiment of the present disclosure includes an input unit and a processor. The input unit acquires a stereo image from the stereo camera. The processor generates a parallax image from the stereo image acquired by the input unit, calculates the parallax distribution from the parallax image, and calculates a plurality of parallax lines connecting points having the same parallax based on the parallax distribution. The processor detects the shape of the road surface based on a plurality of parallax lines.

A stereo camera system according to an embodiment of the present disclosure includes a stereo camera and a processor. A stereo camera captures a stereo image. The processor generates a parallax image from a stereo image captured by a stereo camera, calculates a parallax distribution from the parallax image, and calculates a plurality of parallax lines connecting points having equal parallax based on the parallax distribution. The processor detects the shape of the road surface based on a plurality of parallax lines.

The mobile according to one embodiment of the present disclosure includes a stereo camera system. The stereo camera system includes a stereo camera and a processor. A stereo camera captures a stereo image. The processor generates a parallax image from a stereo image captured by a stereo camera, calculates a parallax distribution from the parallax image, and calculates a plurality of parallax lines connecting points having equal parallax based on the parallax distribution. The processor detects the shape of the road surface based on a plurality of parallax lines.

In the road surface shape detection method according to the embodiment of the present disclosure, a parallax image is generated from a stereo image, a parallax distribution is calculated from the parallax image, and a plurality of parallax lines connecting points having equal parallax based on the parallax distribution. Is calculated, and the shape of the road surface is detected based on the plurality of isoparallax lines.

The program according to the embodiment of the present disclosure includes a procedure of generating a parallax image from a stereo image on a computer and calculating a parallax distribution from the parallax image, and a plurality of points connecting points having the same parallax based on the parallax distribution. The procedure for calculating the parallax line and the procedure for detecting the shape of the road surface based on the plurality of parallax lines are executed.

According to the image processing device, the stereo camera system, the moving body, the road surface shape detecting method and the program according to the embodiment of the present invention, the accuracy of object recognition can be improved.

FIG. 1 is a block diagram showing a schematic configuration of a stereo camera system according to one of a plurality of embodiments. FIG. 2 is a side view schematically showing a moving body equipped with the stereo camera system of FIG. FIG. 3 is a front view schematically showing a moving body equipped with the stereo camera system of FIG. FIG. 4 is a flowchart showing an example of processing performed by the processor of the image processing apparatus of FIG. FIG. 5 is a diagram showing an example of an image captured by either one of the stereo cameras of FIG. FIG. 6 is a diagram illustrating a first parallax image corresponding to the image of FIG. FIG. 7 is a diagram illustrating a second parallax image obtained by removing a three-dimensional object from the first parallax image of FIG. 6 of FIG. FIG. 8 is a diagram showing isoparallax lines obtained from the second parallax image of FIG. 7. FIG. 9 is a flowchart showing another example of processing performed by the control unit of the image processing apparatus of FIG. FIG. 10 is a diagram showing an example of the relationship between the parallax detected from the parallax image and the vertical position of the image. FIG. 11 is a diagram showing the relationship between the parallax obtained by complementing the image vertical position corresponding to the parallax missing in FIG. 10 and the image vertical position. FIG. 12 is a diagram showing the relationship between the image vertical position and the parallax obtained by estimating the image vertical position corresponding to the parallax smaller than the predetermined value by the Kalman filter in FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The figures used in the following description are schematic. The dimensional ratios on the drawings do not always match the actual ones.

As shown in FIG. 1, the stereo camera system 1 includes a stereo camera 10 and an image processing device 20. The stereo camera 10 and the image processing device 20 can communicate with each other via a wired or wireless network 30. The network 30 may include, for example, wireless, wired, CAN (Controller Area Network), or the like. Further, the stereo camera 10 and the image processing device 20 may be housed in the same housing and integrally configured. A device in which the stereo camera 10 and the image processing device 20 are housed in the same housing can be called a stereo camera device. The stereo camera 10 and the image processing device 20 may be provided in the moving body and can communicate with the ECU 40 (Electronic Control Unit) in the moving body.

A "stereo camera" is a plurality of cameras that have parallax with each other and cooperate with each other. Stereo cameras include at least two or more cameras. With a stereo camera, it is possible to collaborate with a plurality of cameras to capture an object from a plurality of directions. Stereo cameras include those capable of simultaneously capturing an object by coordinating a plurality of cameras. The “simultaneous” shooting is not limited to the exact same time. For example, (1) multiple cameras take pictures at the same time, (2) multiple cameras take pictures based on the same signal, and (3) multiple cameras take pictures at the same time in each internal clock. That is included in shooting "simultaneously" in the present disclosure. The time reference for imaging includes the start time of imaging, the end time of imaging, the transmission time of captured image data, and the time when the other device receives the image data. The stereo camera may be a device in which a plurality of cameras are included in one housing. A stereo camera may be a device including two or more cameras that are independent of each other and located apart from each other. Stereo cameras are not limited to multiple cameras that are independent of each other. In the present disclosure, for example, a camera having an optical mechanism that guides light incident on two distant places to one light receiving element can be adopted as a stereo camera. In the present disclosure, a plurality of images obtained by capturing the same subject from different viewpoints may be referred to as "stereo images".

The stereo camera 10 includes a first camera 11 and a second camera 12. The first camera 11 and the second camera 12 each include an optical system and an image sensor that define the optical axis OX. Although the first camera 11 and the second camera 12 have different optical axes OX, in the description of the present specification, the optical axes of both the camera 11 and the camera 12 are collectively represented by only a single reference numeral OX. The image sensor includes a CCD image sensor (Charge-Coupled Device Image Sensor) and a CMOS image sensor (Complementary MOS Image Sensor). The first camera 11 and the second camera 12 generate an image signal representing an image formed by the image sensor. Further, the first camera 11 and the second camera 12 may perform arbitrary processing such as distortion correction, brightness adjustment, contrast adjustment, and gamma correction on the captured image.

The optical axes OX of the first camera 11 and the second camera 12 are oriented so that they can image the same subject. The first camera 11 and the second camera 12 have different optical axes OX from each other. The optical axis OX and the position of the first camera 11 and the second camera 12 are determined so that the captured image includes at least the same subject. The optical axes OX of the first camera 11 and the second camera 12 are oriented so as to be parallel to each other. This parallelism is not limited to strict parallelism, but allows assembly deviation, mounting deviation, and these deviations over time. The optical axes OX of the first camera 11 and the second camera 12 are not limited to parallel, and may be in different directions from each other.

As shown in FIGS. 2 and 3, the stereo camera system 1 of FIG. 1 is mounted on the moving body 50. As shown in the side view of FIG. 2, the first camera 11 and the second camera 12 have the optical axis OX of each optical system of the first camera 11 and the second camera 12 so as to image the front of the moving body 50. It is arranged so as to be substantially parallel to the front of the moving body 50. In the present application, the front means the traveling direction of the moving body 50 when traveling straight, and the right and left mean the right and left with respect to the front. The height direction is perpendicular to the road surface and upward.

The "moving body" in the present disclosure may include, for example, a vehicle or an aircraft. Vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, living vehicles, fixed-wing aircraft traveling on runways, and the like. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, trolley buses and the like. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts, golf carts, and the like. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, lawnmowers and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, crane trucks, dump trucks, road rollers and the like. The vehicle may include a vehicle that travels manually. The classification of vehicles is not limited to the above examples. For example, an automobile may include an industrial vehicle that can travel on the road. The same vehicle may be included in multiple categories. Aircraft may include, for example, fixed-wing aircraft, rotary-wing aircraft, and the like.

The moving body 50 of the present disclosure travels on a road, a runway, or the like. The surface of the road or runway on which the moving body 50 travels is called the road surface.

The image pickup elements included in the first camera 11 and the second camera 12 are located in the same plane perpendicular to the respective optical axes OX. As shown in the front view of FIG. 3, the first camera 11 and the second camera 12 may be arranged so that a straight line (reference line) passing through a reference point such as the center of the image pickup element provided therein is horizontal. Further, the reference line of the image pickup device may be directed slightly above or below the horizontal direction.

The first camera 11 and the second camera 12 are located apart from each other in the direction intersecting the optical axis OX. In one of the plurality of embodiments, the first camera 11 and the second camera 12 are located along the width direction of the moving body 50. The first camera 11 is located on the left side of the second camera 12 when facing forward. The second camera 12 is located on the right side of the first camera 11 when facing forward. Due to the difference in position between the first camera 11 and the second camera 12, the positions of the subjects corresponding to each other in the two images captured by each camera are different. The first image output from the first camera 11 and the second image output from the second camera 12 are stereo images captured from different viewpoints. The first camera 11 and the second camera 12 take an image of a subject at a predetermined frame rate (for example, 30 fps).

In one of the plurality of embodiments, the optical axis OX of the first camera 11 and the second camera 12 is fixed toward the front of the moving body 50. In one of the plurality of embodiments, the first camera 11 and the second camera 12 can image the outside of the moving body 50 through the windshield of the moving body 50. In one of the plurality of embodiments, the first camera 11 and the second camera 12 may be fixed to any of the front bumper, fender grille, side fender, light module, and bonnet of the vehicle.

The image processing device 20 includes an input / output unit 21 and a control unit 22. The image processing device 20 can be arranged at an arbitrary position in the moving body 50. For example, the image processing device 20 can be arranged in the dashboard of the moving body 50.

The input / output unit 21 is an input interface of the image processing device 20 for inputting image data, and is an output interface for outputting information from the image processing device 20 to another device in the moving body 50. Therefore, the input / output unit 21 is both an input unit and an output unit. Other devices in the moving body include an ECU (Electronic Control Unit) 40. A physical connector and a wireless communication device can be used as the input / output unit 21. The physical connector includes an electric connector corresponding to transmission by an electric signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electrical connectors include IEC60603 compliant connector, USB standard compliant connector, RCA terminal compatible connector, EIAJ CP-1211A S terminal compatible connector, and EIAJ RC-5237 D terminal. The connector includes a connector corresponding to HDMI, a connector conforming to the HDMI (registered trademark) standard, and a connector corresponding to a coaxial cable including BNC. Optical connectors include various connectors according to IEC 61754. Wireless communication devices include wireless communication devices that comply with each standard including Bluetooth (registered trademark) and IEEE 802.11. The wireless communication device includes at least one antenna.

Image data of images captured by each of the first camera 11 and the second camera 12 is input to the input / output unit 21. The input / output unit 21 delivers the input image data to the control unit 22. The input / output unit 21 may correspond to the transmission method of the image pickup signal of the stereo camera 10. The input / output unit 21 may be connected to the output interface of the stereo camera 10 via the network 30. The image processing device 20 can output to the ECU 40 of another device in the moving body 50 via the input / output unit 21. The ECU 40 can appropriately use the information received from the image processing device 20.

The control unit 22 includes one or more processors. The control unit 22 or the processor may include one or more memories for storing programs for various processes and information during calculation. The memory includes a volatile memory and a non-volatile memory. The memory includes a memory independent of the processor and a built-in memory of the processor. Processors include general-purpose processors that load specific programs and perform specific functions, and dedicated processors that specialize in specific processing. Dedicated processors include application specific integrated circuits (ASICs). The processor includes a programmable logic device (PLD). The PLD includes an FPGA (Field-Programmable Gate Array). The control unit 22 may be either a SoC (System-on-a-Chip) in which one or a plurality of processors cooperate, or a SiP (System In a Package). The process executed by the control unit 22 can be rephrased as the process executed by the processor.

The image processing device 20 may be configured to read and implement a program recorded on a non-temporary computer-readable medium for the processing performed by the control unit 22 described below. Non-temporary computer-readable media include, but are not limited to, magnetic storage media, optical storage media, photomagnetic storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include optical discs such as CDs (Compact Discs), DVDs, and Blu-ray Discs (Blu-ray® Discs). The semiconductor storage medium includes a ROM (Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a flash memory.

The control unit 22 controls the entire image processing device 20 and each component, and processes a stereo image based on the image data acquired from the stereo camera 10. The processing performed by the control unit 22 will be described below with reference to FIG.

The control unit 22 acquires images captured by the first camera 11 and the second camera of the stereo camera 10 via the input / output unit 21 (step S101). The control unit 22 may perform preprocessing on the first image acquired from the first camera 11 and the second image acquired from the second camera 12 before detecting the parallax. The pre-processing may include processing such as image distortion correction, contrast adjustment between the first and second images, edge enhancement, software calibration of the displacement between the first camera 11 and the second camera 12. The control unit 22 stores the preprocessed first image and the second image in the memory.

FIG. 5 schematically shows an example of an image acquired from the first camera 11 or the second camera 12. FIG. 5 shows a moving body 50, which is a vehicle, traveling on the left lane of the road in the tunnel. The road curves to the right and the tunnel wall is visible in front. In addition, a vehicle traveling ahead is imaged in the image. The first image and the second image are images of the same subject except for the difference in parallax. 5 and 6 to 8 below are for illustration purposes only and are different from the images actually captured or processed.

After step S101, the control unit 22 calculates the parallax distribution of the first image acquired from the first camera 11 and the second image acquired from the second camera 12 (step S102). The control unit 22 divides one image (for example, the first image) of the first image and the second image into a large number of small areas. The small area is, for example, a rectangular area having 3 to 9 pixels in both vertical and horizontal directions. For example, the small area may be composed of 3 pixels in the vertical direction and 3 pixels in the horizontal direction. The control unit 22 matches each image of the plurality of divided small regions with the other image (for example, a second image) while shifting the image one pixel at a time in the horizontal direction. A method using a SAD (Sum of Absolute Difference) function is known for matching stereo images. This represents the sum of the absolute values of the differences in the brightness values in the small area. When the SAD function is minimized, both images are determined to be the most similar. The matching of stereo images is not limited to the method using the SAD function, and other methods may be adopted.

The control unit 22 calculates the parallax for each small region based on the difference in the position of the pixels in the lateral direction of the two regions matched between the first image and the second image. The magnitude of parallax can be expressed in units of the width of the pixel in the horizontal direction. The magnitude of parallax corresponds to the distance. If the parallax is large, it means that the distance is short, and if the parallax is small, it means that the distance is long. The stereo calculation unit generates a parallax image showing the calculated parallax distribution. Therefore, the parallax distribution can be rephrased as a parallax image. The pixels that represent the parallax that make up the parallax image are called parallax pixels. By performing the interpolation processing, a parallax image can be generated with the accuracy of the pixels of the original first image and the second image.

FIG. 6 schematically shows an example of a parallax image corresponding to the image of FIG. The parallax image of FIG. 5 is called a first parallax image. The control unit 22 can replace the difference in parallax with the difference in the brightness or color of the pixels and display it. In FIG. 6, the parallax is represented by different shading. In FIG. 6, the darker the shading, the smaller the parallax, and the lighter the shading, the larger the parallax. In FIG. 6, the equal shaded areas represent that they are located within a predetermined parallax range. In FIG. 6, the parallax distribution is represented by shading over the entire image. However, in an actual parallax image, it is not always possible to calculate the parallax over the entire image in this way. This is because it may be difficult to find a corresponding small area by matching in an area where there is little difference in color, brightness, etc. from the surrounding area such as a smooth road surface. Therefore, the parallax image data can be extracted for a portion where a white line is drawn, a portion having uneven unevenness on the road, a three-dimensional object on the road, and the like. The control unit 22 does not need to display the image as an image after calculating the parallax distribution. The parallax distribution may be retained as data inside the control unit 22 and necessary processing may be performed.

After step S102, the control unit 22 determines a three-dimensional object from the first parallax image and removes it (step S103). The control unit 22 identifies an object located at a distance corresponding to the parallax by detecting a region having a parallax equal to the height direction in the first parallax image.

For example, in the first parallax image of FIG. 6, a vehicle traveling in front can be detected as a region having substantially equal parallax in the height direction on the image. The control unit 22 can detect a vehicle, a person, a road sign, a utility pole, an obstacle on the road, or the like as a three-dimensional object from the first parallax image. The control unit 22 removes the parallax data of the three-dimensional object detected in this way from the first parallax image. FIG. 7 shows a second parallax image obtained by removing the parallax data of the vehicle as a three-dimensional object from the first parallax image. By doing so, the control unit 22 can remove most of the parallax due to the three-dimensional object other than the road surface from the road surface. In FIG. 7, the part of the parallax image corresponding to the deleted vehicle is shown by a broken line for explanation.

After step S103, the control unit 22 sets the largest parallax to be detected as the first parallax (k = 1) (step S104). The maximum parallax to be detected may be predetermined in the control unit 22 based on how close the distance is to be measured. Alternatively, the maximum parallax may be set by the user in the image processing apparatus 20. The control unit 22 extracts a plurality of pixels having the largest parallax from the second parallax image as a point cloud (step S105). The control unit 22 calculates an approximate line on the second parallax image with respect to the point group having the first parallax as an isoparallax line (S106). The parallax line may be a straight line, and the least squares method may be used for approximation, but the approximation is not limited to this. The parallax line may be curved. Further, in an actual parallax image, since the parallax data is obtained discretely in the image, a plurality of parallax lines may be calculated in different portions in the image.

After calculating the approximate line for the first parallax in step S106, the control unit 22 confirms that there is the next smallest parallax data (step S107). The control unit 22 sets the parallax to be detected as the parallax next to the largest parallax (step S108), extracts the point cloud of the parallax (step S105) and the parallax line for the second parallax (k = 2). Is calculated (step S106). Hereinafter, the control unit 22 repeats the processes of steps S105 and S106, sequentially targeting the next smallest parallax (step S108), until the minimum parallax of the detection target is reached (step S107). In one embodiment, the minimum parallax to be detected is the parallax of one pixel.

The control unit 22 calculates the parallax line to the minimum parallax, and when the parallax of the next calculation target disappears (step S107: N), the process proceeds to step S109. In step S109, the control unit 22 combines a plurality of parallax lines having the same parallax among the parallax lines calculated in steps S105 to S108 to obtain a single parallax line. The parallax line is a line connecting positions separated from the moving body 50 by a predetermined distance corresponding to the parallax. Road surface information including information on the inclination and undulations of the road surface located at the distance can be acquired from the inclination of the isoparallax line and the height on the screen (step S109).

FIG. 8 shows a second parallax image on which an isoparallax line is drawn, corresponding to the image of FIG. 7. In FIG. 8, only four isoparallax lines are typically drawn at the boundary of the shaded area. The control unit 22 can calculate as many parallax lines as there are detectable parallaxes. Further, since the three-dimensional object is removed from the first parallax image in step S103, the parallax line can be calculated without being affected by the three-dimensional object on the road surface. Therefore, the accuracy of the isoparallax line is improved.

Based on the isoparallax line, it is possible to obtain information on the road surface shape such as the inclination of the road ahead and the undulations of the road surface. The left-right inclination of the equidistant line indicates the left-right inclination of the road surface equidistant from the moving body 50. The interval between the parallax lines indicates the undulations of the road surface such as a slope. According to FIG. 8, it can be known that the road in front of the moving body 50 is tilted to the right due to the tilt of the isoparallax line.

As described above, according to the stereo camera system 1 of the present embodiment, the control unit 22 acquires the road surface shape in front of the moving body 50, so that the height of the three-dimensional object in front from the road surface can be detected with higher accuracy. It becomes possible to do. The three-dimensional object also includes the three-dimensional object removed in step S203. By detecting the road surface shape, it becomes possible to determine the height of the three-dimensional object from the road surface height again with respect to the first parallax image before removing the three-dimensional object. As a result, it is possible to reduce the omission of detection of an object having a high height from the road surface to be detected and the occurrence of erroneous detection of an object having a low height from the road surface which does not need to be detected. The accuracy of object recognition is improved.

The road shapes such as the slope and undulations of the road detected by the stereo camera system 1 are relative to the moving body 50. Therefore, in the stereo camera system 1, the accuracy of object recognition can be improved even when the moving body 50 on which the stereo camera system 1 is mounted is tilted and the road is horizontal.

The control unit 22 can transmit more accurate information on a three-dimensional object to the ECUs 40 of various devices of the mobile body 50 via the input / output unit 21 to improve the control accuracy of the mobile body 50. Can be done. Examples of the control of the moving body 50 include braking and steering operations automatically performed by the moving body 50. That is, it can be used for warning, automatic stop, automatic avoidance, etc. by detecting a three-dimensional object.

Further, according to the stereo camera system 1 of the present embodiment, the image processing device 20 can detect the relative inclination and undulation of the road ahead with respect to the moving body 50. Further, by using this information and the information on the traveling speed, the ECU 40 in the image processing device 20 or the moving body 50 can estimate how the posture of the moving body 50 changes after a specific time. Become. For example, the image processing device 20 can know the change in the inclination of the moving body 50 after 5 seconds from the inclination of the road surface 50 m ahead when the moving body 50 travels at a speed of 10 m / s. The image processing device 20 can recognize the actual moving direction or speed of the object in consideration of the movement of the object on the image due to the change in the posture of the moving body. The ECU 40 can utilize the road surface information from the image processing device 20 for improving safety, riding comfort, and the like. For example, the ECU 40 may reduce the traveling speed when the information that the road surface condition in front of the vehicle changes significantly during high-speed traveling is acquired.

In the above description, the control unit 22 of the image processing device 20 sequentially reduces the parallax by one pixel from the number of pixels of the largest parallax to be detected, and reduces the parallax pixel group by one pixel until the parallax becomes one pixel. Extracted. However, in the parallax image, the distance corresponding to the difference in parallax of one pixel in the region where the parallax is large is smaller than the distance corresponding to the difference in parallax of one pixel in the region where the parallax is small. Further, since the region having a large parallax is a short distance, image data can be obtained more densely than the region having a small parallax located at a long distance. Therefore, in a region where the parallax is large, the parallax distribution may be calculated by thinning out the parallax to be calculated in order to reduce the calculation amount of the control unit 22.

Therefore, in the stereo camera system according to the embodiment, the parallax is thinned out to calculate the parallax distribution. With respect to the thinned parallax, interpolation processing can be performed from the data of the front and rear parallax when obtaining the road surface shape.

Further, the region having the smallest parallax such as 1 or 2 corresponds to the region at the longest distance on the image. In this region, a difference of one pixel corresponds to a large difference in distance. Further, since the distance is long, the three-dimensional object on the image becomes small, and it is difficult to obtain continuous data. Therefore, long-distance data with small parallax can be unreliable.

Therefore, the control unit 22 applies a Kalman filter to the road surface position data from the short-distance side without performing matching processing for some pixels corresponding to the longest distance, and estimates the long-distance road surface shape. Can be done. The Kalman filter is usually applied to time-series data that changes over time, but in the present embodiment, it is used to estimate a long-distance road shape. That is, in the present embodiment, the time in a normal Kalman filter is calculated by replacing it with distance or parallax.

The flowchart of FIG. 9 shows the processing flow of the control unit 22 of the image processing apparatus 20 according to the embodiment, which complements the thinned parallax and applies the Kalman filter for the estimation of the longest distance. The parallax distribution calculation in step S202 of the flowchart of FIG. 9 is performed by thinning out the parallax corresponding to some pixels including the parallax of one pixel. Further, the processes of steps S204 to S208 are performed on the parallax for which the parallax distribution was calculated in step S202. In other respects, the processing of steps S201 to S209 is the same as that of steps S101 to S109 of the flowchart of FIG. 4, and thus the description thereof will be omitted.

FIG. 10 is a graph showing the vertical position of the image of the road surface with respect to the parallax in a certain direction as seen from the stereo camera 10 after step S209. The vertical position of the image represents the height on the image by the number of pixels from the top of the image as 0. Actually, the road surface information is acquired two-dimensionally in the front and left-right directions of the stereo camera 10, but in FIG. 10 and FIGS. 11 and 12 below, the road surface information is acquired one-dimensionally forward for the sake of explanation. It shall be. In this graph, the parallax corresponding to 1 pixel, 17 pixels, 19 pixels, 21 pixels and 23 pixels is not calculated.

In step S210, processing is performed in which the parallax data of 17 pixels, 19 pixels, 21 pixels, and 23 pixels of the thinned parallax is complemented from the parallax data before and after the parallax data. In FIG. 11, the vertical pixel positions of the road surface after the interpolation processing are represented by white circles. By complementing the thinned-out parallax data, the equi-parallax lines of these thinned-out parallax can be obtained. The amount of calculation related to interpolation is smaller than the case where the parallax distribution is calculated by the matching process in step S202.

In step S211, the road surface shape at a distance corresponding to the parallax of the farthest pixel is estimated by using a Kalman filter from the data on the short distance side where the parallax is large with respect to the parallax data. In FIG. 12, the vertical position of the image of the road surface at the farthest distance with a parallax of 1 pixel is estimated from the interpolated data of FIG. 11 by sequentially using a Kalman filter from the short distance side. By using the Kalman filter, it is possible to obtain road surface shape data with improved reliability for a long distance where the reliability of the data calculated by parallax is reduced.

Furthermore, the use of the Kalman filter makes it possible to smooth the data on the long-distance side and remove the apparently incorrect data by using the relatively reliable data on the short-distance side.

Although the above embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments and examples, and various modifications and modifications can be made without departing from the scope of claims. For example, it is possible to combine the plurality of constituent blocks described in the embodiments and examples into one, or to divide one constituent block into one.

1 Stereo camera system 10 Stereo camera 11 1st camera 12 2nd camera 20 Image processing device 21 Input / output unit 22 Control unit 30 Network 40 ECU
50 mobile

Claims (11)

The input section that acquires a stereo image and
A parallax image is generated from the stereo image acquired by the input unit, a parallax distribution is calculated from the parallax image, a plurality of parallax lines connecting points having the same parallax are calculated based on the parallax distribution, and the plurality of parallax lines are calculated. An image processing device including a processor that detects the shape of a road surface based on an isoparallax line. The image processing device according to claim 1, wherein the processor extracts a plurality of parallax pixels having parallax with each other from the parallax distribution, and calculates the parallax line by approximation based on the parallax pixels. The image processing device according to claim 1 or 2, wherein the processor determines a three-dimensional object from the parallax distribution, excludes the parallax distribution corresponding to the three-dimensional object, and then calculates the parallax line. The image processing apparatus according to claim 3, wherein the processor determines the three-dimensional object based on a region where parallax is continuous in the height direction. The image processing device according to any one of claims 1 to 4, wherein the processor detects the shape of a road surface by interpolating between the plurality of parallax lines. The image processing device according to any one of claims 1 to 5, wherein the processor estimates an parallax line corresponding to a parallax smaller than a predetermined number of pixels in the calculation of the parallax line by using a Kalman filter. The image processing device according to any one of claims 1 to 6, wherein the processor performs a stereo image recognition process based on a height from the road surface calculated from the relative inclination of the road surface. A stereo camera that captures stereo images and
A parallax image is generated from the stereo image captured by the stereo camera, a parallax distribution is calculated from the parallax image, an isoparallax line connecting points having equal parallax is calculated based on the parallax distribution, and the parallax line is calculated. A stereo camera system equipped with a processor that detects the shape of the road surface based on. A stereo camera that captures a stereo image, and a plurality of parallax images generated from the stereo image captured by the stereo camera, a parallax distribution is calculated from the parallax image, and points having the same parallax are connected based on the parallax distribution. A moving body including a stereo camera system including a processor that calculates the parallax line of the above and detects the shape of the road surface based on the plurality of parallax lines. A parallax image is generated from the stereo image, the parallax distribution is calculated from the parallax image, and the parallax distribution is calculated.
A plurality of parallax lines connecting points having the same parallax are calculated based on the parallax distribution.
A road surface shape detecting method for detecting the shape of a road surface based on the plurality of parallax lines. A procedure for generating a parallax image from a stereo image on a computer and calculating the parallax distribution from the parallax image, a procedure for calculating a plurality of parallax lines connecting points having the same parallax based on the parallax distribution, and the plurality of steps. A program that executes the procedure of detecting the shape of the road surface based on the isotropic parallax line.

Images