TWI640960B - Road image processing method and apparatus - Google Patents

Abstract

A road image processing method and a road image processing device, and the method includes the steps of: capturing a road image by using a first lens to obtain a first image, and capturing a road image using the second lens to obtain a second image; Obtaining a first depth map with the second image; analyzing first depth information corresponding to the first region in the first depth map to obtain first road information corresponding to the first region; and obtaining, according to the first road information, corresponding to the original region Second road information; and obtaining a second depth map according to the second road information and the first depth map.

Description

Road image processing method and road image processing device

The present invention relates to a road image processing technology, and more particularly to a road image processing method and a road image processing device that can automatically detect a front obstacle.

When the vehicle is driving on the road, it is necessary to maintain a safe distance from the vehicle in front and obstacles. In order to maintain a safe enough distance for the driver, a driving recorder can be used to detect that the driving vehicle maintains a sufficient safety distance from the vehicle or obstacle ahead to avoid an accident.

At present, various front-vehicle safety distance warning systems mostly use three methods to estimate the distance from the vehicle or obstacle in front. The first one uses traditional high-frequency millimeter-wave radar (30 GHz to 300 GHz) or laser radar technology to estimate the distance, and the distance estimation results prevent vehicle collisions, and the weather resistance and detection distance are excellent. . However, such systems are extremely expensive and do not have the function of road identification such as lane offset and road sign recognition. Therefore, the past traditional front vehicle safety distance warning system has relatively low visibility on general vehicles.

As the public's emphasis on driving safety is getting higher and higher, the second type of image-type vehicle safety distance warning system has gradually received attention. Compared with the first radar technology, the image capture technology has the ability to identify obstacles, lane lines and traffic signs, and has large integration functions and low price. However, the selected lens greatly affects the detection capability of the image-type front vehicle safety distance warning system, and cannot detect both short-distance and long-distance detection. Therefore, most of the current image-type vehicle safety distance warning systems use close-range detection.

Due to the low price of the image-based front safety distance warning system, a third type of dual-lens detection image-type front vehicle safety warning system has been developed in recent years, and obstacles are detected by the disparity values of the two lenses. distance. The focus of this technique is that the depth of the planar road needs to be pre-detected to compare with the depth of the obstacle. However, if there is already a vehicle or obstacle on the road surface before the detection, there will be an error in the depth detection of the flat road.

In view of this, the present invention provides a road image processing method and a road image processing device, which can effectively avoid the obstacles in the original image and improve the accuracy of detecting the road depth.

The invention provides a road image processing method, which comprises: capturing a road image by using a first lens to obtain a first image and capturing the road image by using a second lens to obtain a second image; according to the first image and the Obtaining, by the second image, a first depth map; analyzing first depth information corresponding to the first region in the first depth map to obtain first road information corresponding to the first region, where The total number of pixels is less than the total number of pixels in the original area in the first depth map; the second road information corresponding to the original area is obtained according to the first road information; and according to the second road information The first depth map obtains a second depth map, wherein the second depth map is used to represent depth information of at least one solid object in the road image.

The present invention further provides a road image processing apparatus including a first lens, a second lens, and a processor. The first lens is used to capture a road image to obtain a first image, and the second lens is used to capture the road image to obtain a second image. The processor is coupled to the first lens and the second lens, and is configured to receive the first image from the first lens and the second image from the second lens. The processor is further configured to obtain a first depth map according to the first image and the second image. The processor is further configured to analyze the first depth information corresponding to the first area in the first depth map to obtain first road information corresponding to the first area, where the pixels in the first area The total number is less than the total number of pixels in the original area in the first depth map. The processor is further configured to obtain second road information corresponding to the original area according to the first road information. The processor is further configured to obtain a second depth map according to the second road information and the first depth map, where the second depth map is used to represent depth information of at least one solid object in the road image. .

Based on the foregoing road image processing method and apparatus, the first road information of the first area in the first depth map may be detected to extend the second road information of the original area in the first depth map. Then, the second road information can be matched with the first depth map to obtain a second depth map indicating depth information of the three-dimensional object in the road image. By separately detecting the road information of the first area in the first depth map instead of completely detecting the road depth information of the entire first depth map, the obstacles in the original image can be effectively avoided and the accuracy of detecting the road depth can be improved.

The above described features and advantages of the invention will be apparent from the following description.

The components of the present invention will be described in detail in the following description in conjunction with the accompanying drawings. These examples are only a part of the invention and do not disclose all of the embodiments of the invention. Rather, these embodiments are merely examples of devices and methods within the scope of the patent application of the present invention.

1 is a block diagram of a road image processing apparatus according to an embodiment of the invention, but is for convenience of description and is not intended to limit the present invention. First, all components and configuration relationships of the electronic device will be described first in FIG. 1, and detailed functions will be disclosed below.

Referring to FIG. 1, the road image processing apparatus 10 is, for example, an electronic device having a video recording and/or photographing function, such as a driving recorder, a vehicle navigation device, a personal digital assistant, and a smart phone. Further, the road image processing device 10 may be provided on various types of vehicles, such as automobiles, bicycles, locomotives, and various types of mobile machines.

The road image processing apparatus 10 includes a first lens 110, a second lens 120, a processor 130, and a memory 140. The first lens 110 and the second lens 120 are respectively used to capture images. For example, the first lens 110 and the second lens 120 can simultaneously capture images toward the front of the traveling direction of the carrier, the side of the traveling direction, or the rear of the traveling direction. The first lens 110 and the second lens 120 are arranged side by side and the first lens 110 and the second lens 120 are separated by a predetermined distance. For example, this preset distance can be 6.5 cm or closer or more separate. In addition, the first lens 110 and the second lens 120 may be a Charge Coupled Device (CCD) lens, a Complementary Metal Oxide Semiconductor (CMOS) lens, or an infrared lens camera or camera. However, the invention is not limited thereto.

The memory 140 is coupled to the first lens 110, the second lens 120, and the processor 130. The memory 140 is used to store the code and other materials required for the operation of the road image processing apparatus 10. For example, the memory 140 can include an inline storage unit and/or an external storage unit. The embedded storage unit can be a random access memory (RAM), a read-only memory (ROM), a flash memory, a magnetic disk storage device (Magnetic disk storage). Device), etc., but the invention is not limited thereto. The external storage unit can be a Compact Flash (CF) memory card, a Secure Digital (SD) memory card, a Micro SD memory card, a Memory Stick (MS), etc., but The invention is not limited thereto.

The processor 130 is coupled to the first lens 110 , the second lens 120 , and the memory 140 . The processor 130 is used to control the overall operation of the road image processing apparatus 10. For example, the processor 130 can be a central processing unit (CPU), or other programmable microprocessor (Microprocessor), a digital signal processor (DSP), a programmable controller, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), or other hardware devices with computing power, but the invention is not limited thereto.

2 is a flow chart of a road image processing method in accordance with an embodiment of the present invention. FIG. 3 is a schematic diagram of a road image according to an exemplary embodiment of the invention. FIG. 4 is a schematic diagram of a first depth map according to an exemplary embodiment of the invention. FIG. 5 is a schematic diagram of a second depth map according to an exemplary embodiment of the invention. The road image processing method of FIG. 2 is applied to the road image processing apparatus 10 of FIG. Hereinafter, detailed steps of the road image processing method of the present invention will be described with reference to various elements in the road image processing apparatus 10, and will be further described with reference to FIGS. 3, 4, and 5.

Referring to FIG. 1 , FIG. 2 and FIG. 3 , firstly, in step S210 , the processor 130 captures the road image 30 by using the first lens 110 to obtain a first image, and captures the road image 30 by using the second lens 120 to obtain a first image. Two images. In an embodiment, the first image may also be referred to as a left eye image, and the second image may also be referred to as a right eye image. The first lens 110 and the second lens 120 capture the road image 30 according to the same parameters, for example, to obtain the first image and the second image, respectively. The parameters may include a reference focal length, an aperture, a shutter, a white balance, and the like. The road image 30 may have a three-dimensional object and a plane object, wherein the three-dimensional object is, for example, a vehicle, and the plane object is, for example, a road plane or a road marking, as shown in FIG. It should be noted that the road image 30 may also have other types of three-dimensional objects such as utility poles and flat objects such as a gutter cover that is flat on the road surface, and the present invention is not limited thereto. In addition, the obtained image data of the first image and the second image are stored in the memory 140.

In step S210, the processor 130 further obtains the first depth map 40 according to the first image and the second image. For example, the processor 130 may calculate a disparity corresponding to each pixel in the first image and the second image. Since the first lens 110 and the second lens 120 are arranged side by side and the first lens 110 and the second lens 120 are separated by a predetermined distance, if an object in front of the lens is closer to the first lens 110 and the second lens 120, , the object has a larger disparity value in the first image and the second image. On the other hand, if an object in front of the lens is farther away from the first lens 110 and the second lens 120, the object has a smaller disparity value in the first image and the second image. The processor 130 may convert the disparity value corresponding to each pixel (or pixel position) into a depth value and generate a first depth map 40 accordingly.

In an embodiment, the processor 130 may analyze a plurality of features in the first image and find pixels in the second image that also have the features. Then, the processor 130 calculates an offset of the feature in the first image and the second image and obtains a corresponding disparity value. Next, the processor 130 may obtain a depth value of all pixels in the image via a depth value formula: depth value=(shot pitch×reference focal length)/disparity value. The processor 130 may obtain the first depth map 40 based on the depth values of all pixels in the image. It should be noted that each pixel in the first depth map 40 corresponds to a depth value. In addition, the first depth map 40 corresponds to the original area of the entire road image 30, as shown in FIG. In an embodiment, the first depth map 40 can also be considered an original depth map. In the following embodiments, depth values are described in terms of depth information.

In an embodiment, the original region in the first depth map 40 may include the first region 410 and the second region 420, and the second region 420 does not overlap with the first region 410. For example, the second region 420 can encompass regions other than the first region 410 in the first depth map 40. It should be noted that in order to prevent the obstacle from affecting the accuracy of the road depth detection, the first area 410 may be set as a smaller area, and the first area 410 may be disposed to be close to or close to the lower edge of the first depth map 40. . In another embodiment, the first region 410 may also be located at other locations in the first depth map 40, and the size of the first region 410 (ie, the coverage of the first region 410) may also be larger or smaller, The invention is not limited. Further, the shape of the first region 410 may also be a rectangle or other shape. In an embodiment, the size of the first region 410 may be less than or equal to the size of the second region 420 (ie, the coverage of the second region 420). That is, the first region 410 may have fewer total pixels than the second region 420. In addition, the total number of pixels in the first region 410 is also less than the total number of pixels in the entire original region in the first depth map 40.

In step S220, the processor 130 analyzes the first depth information corresponding to the first region 410 in the first depth map 40 to obtain first road information corresponding to the first region 410. For example, the first region 410 can be considered to include a plurality of column regions 430(1), 430(2)... and 430(N), and the second region 420 can be considered to include a plurality of column regions 440(1), 440(2) ...and 440 (M). In an embodiment, M is greater than N. In another embodiment, M may also be equal to N. The first road information includes a plurality of first column depth information corresponding to the first area 410. Wherein, each of the first column depth information corresponds to one of the column regions 430(1), 430(2)... and 430(N). Each first column depth information can be used to describe road depth information on the corresponding road plane (or corresponding column area). In general, if there is no three-dimensional object in the first region 410, the pixels belonging to the same column region in the first region 410 should have the same depth value, and each column region 430(1), 430(2)... And 430(N) will increase the depth value from near to far (or from bottom to top).

Taking the column region 430(1) as an example, when the first column depth information of the column region 430(1) is to be calculated, the processor 130 may select the depth value of at least one pixel in the column region 430(1) or The average depth value of all pixels of this column area is taken as the first column depth information of column area 430(1). And so on, the first column depth information of column regions 430(1)~430(N) can be obtained. In an embodiment, the first column depth information of column regions 430(1)-430(N) may be considered as the first road information corresponding to the first region 410 as a whole. The first road information corresponding to the first area 410 can be used to describe the road depth of the first area 410.

In an embodiment, if the first region 410 is disposed in a region of the first depth map 40 that is closer to the first lens 110 and the second lens 120 (ie, lower) as shown in FIG. 4, the first lens 110 is The disparity value of the second lens 120 in the first region 410 when the road image 30 is captured may be greater than the disparity value of the first lens 110 and the second lens 120 in the second region 420 when the road image 30 is captured. In general, an area closer to the first lens 110 and the second lens 120 (eg, the first area 410 in FIG. 4) is less likely to contain a three-dimensional object in the road image 30 (eg, a vehicle or Other obstacles, if any, indicate that they are about to hit this three-dimensional object). Therefore, by analyzing the first depth information corresponding to the first area 410, there is a higher probability that more accurate road information (ie, road depth information) can be obtained.

In step S230, the processor 130 obtains second road information corresponding to the original area in the first depth map 40 according to the first road information. For example, the processor 130 may analyze the first column depth information of the column regions 430(1) to 430(N) to obtain statistical characteristics of the first column depth information, and obtain the statistical characteristics of the first column depth information according to the first column. Corresponding to the second column depth information of the second region 420.

In an embodiment, the processor 130 analyzes the first column depth information of the column regions 430(1)-430(N) to obtain a linear increase model of the first column depth information. For example, the increasing trend of the first column depth information of the column regions 430(1)~430(N) will conform to the linear increase model according to this. Based on the first column depth information and the linear increase model, the processor 130 may further derive the second column depth information for the column regions 440(1), 440(2)... and 440(M). For example, the processor 130 may bring the first column depth information of the column region 430(N) into the linear increase model, thereby calculating a column region 440 that conforms to the linear increase model and the value gradually increases (ie, the depth value gradually increases). The second column depth information of (1), 440(2)... and 440(M). The second column depth information of the column regions 440 (1), 440 (2), ... and 440 (M) may also be regarded as the second column depth information corresponding to the second region 420.

In an embodiment, the processor 130 may set the first column depth information of the column regions 430(1) to 430(N) and the second column of the column regions 440(1), 440(2)... and 440(M). The depth information is generally regarded as the second road information corresponding to the original area in the first depth map 40. In other words, the second road information includes the first column depth information of the column areas 430(1) to 430(N) and the second column depth information of the column areas 440(1), 440(2)... and 440(M). Thereby, this second road information can be used to describe the road depth of the entire original area in the first depth map 40.

In step S240, the processor 130 obtains the second depth map 50 according to the second road information and the first depth map 40, wherein the second depth map 50 (only) is used to represent the depth information of at least one solid object in the road image 30. . For example, the processor 130 removes the depth information of at least one pixel in the first depth map 40 according to the second road information. In particular, the depth information of the removed pixels is the depth information of the pixels corresponding to the non-three-dimensional object (ie, the planar object). It should be noted that the plane matter mentioned here refers to an object that is roughly flat on the road surface. In addition, the three-dimensional object does not include various types of planar objects such as road markings that are flat on the road surface in the road image 30.

In an embodiment, the processor 130 compares the second road information with the first depth map 40, and obtains at least one candidate pixel in the first depth map 40 according to the comparison result, wherein the candidate pixel corresponds to the road. At least one non-stereoscopic object in image 30. Next, the processor 130 removes the depth information of the candidate pixels from the first depth map 40 to obtain a second depth map 50. After removing the depth information corresponding to the pixel of the non-stereo object, only the depth information of the three-dimensional object in the original image 30 (or the first depth map 40) is left in the obtained second depth map 50, instead of having Non-stereoscopic pixel information.

In an embodiment, the processor 130 compares the second road information with the depth information of the pixels of each corresponding pixel position in the first depth map 40. If the difference between the second road information and the depth information in the first depth map 40 is less than a critical value at a pixel position, the processor 130 may treat the pixel of the pixel position as a candidate pixel. It is determined that this candidate pixel corresponds to a non-three-dimensional object of the road image 30 (for example, a road plane and a road marking). On the other hand, if the difference between the second road information and the depth information in the first depth map 40 is not less than the threshold value at a certain pixel position, the processor 130 does not regard the pixel of the pixel position as a candidate. The pixel can determine that the pixel corresponds to a three-dimensional object in the road image.

In an embodiment, the pixel position corresponding to the removed depth information in the second depth map 50 may be refilled with dummy information and the depth of the three-dimensional object retained in the second depth map 50. Information is generated. In addition, after obtaining the second depth map 50, the second depth map 50 can be further used for obstacle detection and the like during driving, and the invention is not limited.

In summary, the present invention can capture the road image by using the first lens and the second lens to obtain the first depth map, and set the first region in the first depth map. Next, the present invention can analyze the road information of the first area and estimate the road information of the entire plane road. Then, the second depth map without the depth information of the planar road is obtained by comparing the road information of the plane road with the depth information of the original area. From another point of view, the present invention can utilize the depth of the (close distance) region of interest to derive the depth of the planar object of the original region of the entire depth map (ie, the depth of the road), and by comparing the first depth map with the entire The depth information of the corresponding pixel position in the road depth of the original area is used to remove the depth information of the plane object. By separately detecting the road information of the first area in the first depth map instead of completely detecting the road depth information of the entire first depth map, the obstacles in the original image can be effectively avoided and the accuracy of detecting the road depth can be improved.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

10‧‧‧Road image processing device

110‧‧‧ first shot

120‧‧‧second lens

130‧‧‧Processor

140‧‧‧ memory

Steps for S210, S220, S230, S240‧‧‧ road image processing methods

30‧‧‧ Road imagery

40‧‧‧First depth map

410‧‧‧First area

420‧‧‧Second area

430 (1), 430 (2), 430 (N) ‧ ‧ the first region of the column area

440 (1), 440 (2), 440 (M) ‧ ‧ the second region of the column area

50‧‧‧Second depth map

FIG. 1 is a block diagram of a road image processing apparatus according to an exemplary embodiment of the invention. FIG. 2 is a flow chart of a road image processing apparatus according to an exemplary embodiment of the invention. FIG. 3 is a schematic diagram of a road image according to an exemplary embodiment of the invention. FIG. 4 is a schematic diagram of a first depth map according to an exemplary embodiment of the invention. FIG. 5 is a schematic diagram of a second depth map according to an exemplary embodiment of the invention.

Claims (14)

A road image processing method includes: capturing a road image by using a first lens to obtain a first image, and capturing the road image by using a second lens to obtain a second image; according to the first image and the first image Obtaining a first depth map in the second image; analyzing a first depth information corresponding to a first region in the first depth map to obtain a first road information corresponding to the first depth information of the first region, where The total number of pixels in the first region is less than the total number of pixels in an original region in the first depth map; obtaining a second road information corresponding to the original region according to the first road information; The second road map and the first depth map obtain a second depth map, wherein the second depth map is used to represent a depth information of at least one solid object in the road image. The road image processing method of claim 1, wherein the original area includes the first area and a second area, the first area and the second area do not overlap, and the first road information includes a plurality of first column depth information of the first area, and the second road information includes the plurality of first column depth information and a plurality of second column depth information corresponding to the second area. The road image processing method of claim 2, wherein the step of obtaining the second road information corresponding to the original area according to the first road information comprises: analyzing the plurality of first column depth information to obtain the a statistical characteristic of the plurality of first column depth information; and obtaining the plurality of second column depth information according to the statistical characteristic of the plurality of first column depth information. The road image processing method of claim 2, wherein the first lens and the second lens when the road image is captured, a disparity value in the first region is greater than the first lens and the The second lens is a disparity value in the second area when the road image is captured. The road image processing method of claim 1, wherein the obtaining the second depth map according to the second road information and the first depth map comprises: removing the first depth according to the second road information A depth information of at least one pixel in the figure. The road image processing method of claim 5, wherein the step of removing the depth information of the at least one pixel in the first depth map according to the second road information comprises: comparing the second road Information and the first depth map; obtaining at least one candidate pixel in the first depth map according to a comparison result, wherein the at least one candidate pixel corresponds to at least one non-stereoscopic object in the road image; The depth information of the at least one candidate pixel is removed from the first depth map to obtain the second depth map. The road image processing method according to claim 1, wherein the at least one three-dimensional object does not include a road marking line in the road image that is flat on the road surface. A road image processing device includes: a first lens that captures a road image to obtain a first image; and a second lens that captures the road image to obtain a second image, a processor coupled to the first The lens and the second lens are configured to: receive the first image from the first lens and receive the second image from the second lens; obtain a first depth map according to the first image and the second image; A first depth information corresponding to a first area in the first depth map is obtained to obtain a first road information corresponding to the first depth information of the first area, where a pixel in the first area The total number is less than a total number of pixels in an original area in the first depth map; obtaining a second road information corresponding to the original area according to the first road information; and according to the second road information and the first The depth map obtains a second depth map, wherein the second depth map is used to represent a depth information of at least one solid object in the road image. The road image processing device of claim 8, wherein the original area includes the first area and a second area, the first area and the second area do not overlap, and the first road information includes a plurality of first column depth information of the first area, and the second road information includes the plurality of first column depth information and a plurality of second column depth information corresponding to the second area. The road image processing apparatus of claim 9, wherein the obtaining the second road information corresponding to the original area according to the first road information comprises: analyzing the plurality of first column depth information to obtain the a statistical characteristic of the plurality of first column depth information; and obtaining the plurality of second column depth information according to the statistical characteristic of the plurality of first column depth information. The road image processing device of claim 9, wherein the first lens and the second lens have a disparity value in the first region greater than the first lens and the second lens when the road image is captured The second lens is a disparity value in the second area when the road image is captured. The road image processing apparatus of claim 8, wherein the obtaining the second depth map according to the second road information and the first depth map comprises: removing the first depth according to the second road information A depth information of at least one pixel in the figure. The road image processing apparatus of claim 12, wherein the removing the depth information of the at least one pixel in the first depth map according to the second road information comprises: comparing the second road Information and the first depth map; obtaining at least one candidate pixel in the first depth map according to a comparison result, wherein the at least one candidate pixel corresponds to at least one non-stereoscopic object in the road image; The depth information of the at least one candidate pixel is removed from the first depth map to obtain the second depth map. The road image processing device of claim 8, wherein the at least one three-dimensional object does not include a road marking line in the road image that is flat on the road surface.