CN103686126A - Stereo image processing device and method thereof - Google Patents

Stereo image processing device and method thereof Download PDF

Info

Publication number
CN103686126A
CN103686126A CN201310380608.7A CN201310380608A CN103686126A CN 103686126 A CN103686126 A CN 103686126A CN 201310380608 A CN201310380608 A CN 201310380608A CN 103686126 A CN103686126 A CN 103686126A
Authority
CN
China
Prior art keywords
image
camera
region
stereo
extraction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201310380608.7A
Other languages
Chinese (zh)
Other versions
CN103686126B (en
Inventor
金周英
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hyundai Mobis Co Ltd
Original Assignee
Hyundai Mobis Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hyundai Mobis Co Ltd filed Critical Hyundai Mobis Co Ltd
Publication of CN103686126A publication Critical patent/CN103686126A/en
Application granted granted Critical
Publication of CN103686126B publication Critical patent/CN103686126B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/246—Calibration of cameras

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The present invention relates to a stereo image processing method which effectively performs stereo matching of stereo images and a device thereof. The stereo image processing device provided by the invention is characterized in that the stereo image processing device is used for processing stereo images of vehicles and comprises the components of: stereo cameras which comprises a first camera and a second camera for acquiring stereo images; and a stereo matching part which performs stereo matching by means of the stereo image; wherein the stereo matching part performs correction between a first image of the first camera and a second image of the second camera by means of a maximal inclination between the first camera and the second camera, which is set according to the vehicle state.

Description

Stereoscopic image processing device and method
Technical field
The present invention relates to stereoscopic image processing method and device.Specifically, effectively implement stereoscopic image processing method and the device of stereo-picture Stereo matching.
Background technology
The method of the distance to locality comprising the image calculation image that utilization is obtained from a plurality of cameras has become known technology.Stereo matching (stereo matching) is for comparing typical method, by the arbitrary patterns the image obtaining from some cameras, from the image of other camera of mating with the same place three dimensions, find out, utilize the corresponding relation of its pattern and trigonometry to learn the position on three dimensions.Described pattern may be the various factorss such as pixel, window block or characteristic point (edge, track) in image.
Fig. 1 is that explanation utilizes Stereo matching method to calculate the diagram of an example of distance.
Utilization be take in two images of two cameras shootings that any baseline (b) distance is phase spacing and is obtained the poor (displacement to the distance between picture, disparity), learn the focal length (f) of camera lens, geometry structure that can application drawing 1, as the distance of following mathematical expression 1 between obtaining from camera to object.
[mathematical expression 1]
In mathematical expression 1, r is the distance between object and camera, and b is two parallax ranges between camera, and f is the focal length of camera lens, and d is by poor (disparity) of the distance between the same image under printing, i.e. d in two images l-d r.
As mentioned above, while implementing Stereo matching, its prerequisite is that the pattern of special object in two images is placed in (hypothesis only has the gap (disparity) of horizontal direction) on same line from vertical direction.Then from explore point corresponding in two images to the same circuit of horizontal direction.For this reason, core (epipolar) line in two images should be consistent, and described operation is called correction (rectification).Generally to utilize rotation/calibration model (Matrix) correcting image after gradient between characteristic point (feature) the judgement left and right image of exploring in image or distortion degree.
But conventional method is to exist to take an image and change as benchmark makes another image rotation match point, and poor (disparity) of actual range problem of also changing.And the characteristic point in exploring image and confirm to there will be in the process of gradient the problem of delay (delay).Also inapplicable for the vehicle image Stereo matching that described delay is especially processed in time for needs.
Summary of the invention
Technical task
The object of the present invention is to provide a kind of mistake occurring in the process of image rectification (rectification) for the Stereo matching of three-dimensional camera 110 images to minimize, especially be effectively applied to the stereoscopic image processing method of three-dimensional camera applicable on vehicle, to solve described problem.
The object of the present invention is to provide a kind of stereoscopic image processing device of implementing above-mentioned stereoscopic image processing method.
Technical scheme
The invention provides a kind of stereoscopic image processing device, it is characterized in that, the device of the stereo-picture of vehicle is processed in operation, comprising: three-dimensional camera, comprises the first camera and the second camera that obtain stereo-picture; Stereo matching portion, utilizes described stereo-picture to implement Stereo matching; Described Stereo matching portion utilizes described the first camera of setting according to the state of described vehicle and the maximum inclination between described second camera, implements the correction between the first image of described the first camera and the second image of described second camera.
As an embodiment, described Stereo matching portion, comprising: pixel extraction portion, utilizes described maximum inclination to extract respectively the first extraction region and second and extract region for proofreading and correct from described the first image and described the second image; Feature Points Matching portion, extracts region and described second from described first and extracts extracted region characteristic of correspondence point; Gradient calculating portion, utilizes described characteristic point, calculates described first and extracts region and the described second gradient of extracting between region; Image converter section, utilizes described gradient to proofread and correct described the first image and described the second image, implements final Stereo matching.
Described the first extraction region is from X row of longitudinal extraction from described the first image.
Described the second extraction region is to consider maximum inclination described in described the second image, extracts a plurality of row that comprise described X row.
The invention provides a kind of stereoscopic image processing method, as the method for processing the stereo-picture of vehicle, this implementation step comprises: (a) from the three-dimensional camera of vehicle, obtain the first image and the second image; (b) the pixel extraction portion in Stereo matching portion utilizes according to described first camera of the state setting of described vehicle and the maximum inclination between second camera, by set extracted region in described the first image and the second image, is that region is extracted in the first extraction region and second; (c) in described Stereo matching portion, calculate and described first, extract region and described second and extract region characteristic of correspondence point, calculate the gradient between described the first image and described the second image; (d) in described Stereo matching portion, utilize described gradient to proofread and correct described the first image and described the second image, implement final Stereo matching.
Beneficial effect
According to the present invention, its beneficial effect is, when the image obtaining for the three-dimensional camera from vehicle is implemented Stereo matching, consider the particular surroundings of so-called vehicle and non-image Zone Full, only the trimming process between stereo-picture is implemented in the subregion of image, thereby reduce the processing time, and simplify system.
Accompanying drawing explanation
Fig. 1 is that explanation utilizes Stereo matching method to calculate the exemplary plot of distance;
Fig. 2 is the block diagram of the stereoscopic image processing device of the preferred embodiment of the present invention;
Fig. 3 is the processing exemplary plot of stereo-picture of the stereoscopic image processing device of the preferred embodiment of the present invention;
Fig. 4 is the precedence diagram of the stereoscopic image processing method of the preferred embodiment of the present invention.
Description of reference numerals
100: stereoscopic image processing device; 110: three-dimensional camera;
120: Stereo matching portion; 122: pixel extraction portion;
124: Feature Points Matching portion; 126: gradient calculating portion;
128: image converter section; 130: storage part;
140: electronic-controlled installation.
Embodiment
With reference to the accompanying drawings, invention preferred embodiment is described in detail.The same member of reality in description and accompanying drawing is to represent with same symbol, no longer repeat specification.And explanation is when of the present invention, if aim of the present invention is thickened to illustrating of relevant known function or structure, omits this explanation.Embodiment based in the present invention, those of ordinary skills, not making the every other embodiment obtaining under creative work prerequisite, belong to the scope of protection of the invention.
Fig. 2 is the block diagram of the stereoscopic image processing device of the preferred embodiment of the present invention.Fig. 3 is the processing exemplary plot of stereo-picture of the stereoscopic image processing device of the preferred embodiment of the present invention.
The stereoscopic image processing device 100 of the preferred embodiment of the present invention comprises three-dimensional camera 110, Stereo matching portion 120 and the storage part 130 possessing on vehicle.Described stereoscopic image processing device can be connected in the electronic-controlled installation 140(ECU that utilizes stereo-picture to implement the various controls of vehicle).
Three-dimensional camera 110 is assemblied on vehicle, is generally assemblied in vehicle front.Described three-dimensional camera 110 comprises the first camera 110a and second camera 110b.Described the first camera 110a and second camera 110b are that the method for obtaining the image in same place is installed.But the three-dimensional camera 110 in the embodiment of the present invention is not only vehicle front, also can be installed on side or rear.
In an embodiment, Stereo matching portion comprises pixel extraction portion 122, Feature Points Matching portion 124, gradient calculating portion 126 and image converter section 128.
Pixel extraction portion 122 implements the first image 150a obtain from the first camera 110a and the second image 150b of obtaining from second camera 110b extracts the function of the piece of intending implementing coupling.According to Fig. 2, pixel extraction portion 122 is the running environment of considering vehicle, sets contingent maximum inclination A between the first camera 110a and second camera 110b.For example, when the first camera 110a and second camera 110b are installed on the place ahead of vehicle, the maximum of the differential seat angle that may mutually occur between the first camera 110a that the impact because applying on vehicle or vibration are occurred and second camera 110b is set as maximum inclination A.Described maximum inclination A, estimates that the car travelling on the road surface spreading is to set littlely, estimates that the offroad racing travelling on the road surface not spreading is to set greatlyr.Described maximum inclination A can set when vehicle outbound, but also can set in real time according to the actual travel condition of vehicle.
Pixel extraction portion 122 is extracted as the first extraction region by the row of X pixel of vertical direction from the first image 150a.If the first image 150a and the second image 150b are comprised of the pixel of M * N respectively, first extract the pixel that region has M * 1.
In addition, the second image 150b considers maximum inclination A, can calculate the several Y that intend to the pixel of vertical direction consideration.Described Y be can utilize as
Figure DEST_PATH_IMAGE004
numerical expression calculate.Pixel extraction portion 122 considers described Y, from the second image 150b, extract with ± the corresponding region of Y is the second extraction region.Second extracts the region of Y up and down that region comprises X the row of the second image 150b, has the pixel of M * (1+2Y).
In embodiments of the invention, described X can be arranged in the first image 150a and the second image 150b from being longitudinally positioned at central part.But described X is not only longitudinal middle body in the first image 150a and the second image 150b, can also be obliquely installed up and down.
The first information of extracting region and the second extraction region of extracting from pixel extraction portion 122 is to be stored into storage part 130 temporarily.
Feature Points Matching portion 124 utilizes described the first extraction region and second to extract region to implement the symmetric points that Stereo matching is calculated coupling.For example, Feature Points Matching portion 124 utilizes SSD (Sum of Squared Differences, poor quadratic sum), SAD (Sum of Absolute Differences, poor absolute value and) or the Stereo Matching Algorithm such as NCC (Normalized Cross Correlation, normalized crosscorrelation) calculate the first extraction region and second and extract the symmetric points in region.But embodiment of the present invention neutral body matching algorithm is to be not limited to foregoing, for Stereo matching also can be used other algorithm.
The present invention extracts the first pixel and described second of extracting region being comprised of a line the pixel comparison calculating symmetric points in region, compares and can accelerate its arithmetic speed with the past.
Gradient calculating portion 126 utilizes the characteristic point of calculating from described Feature Points Matching portion 124 to calculate the first gradient of extracting between the image that extracts region in region and second.
Image converter section 128 is to utilize the gradient of calculating in described gradient calculating portion 126, with some images in the first image 150a or the second image 150b, be as the criterion, after making another image rotation, implement the Stereo matching between the first image 150a and the second image 150b.
Storage part 130 is to implement the information of described the first image 150a and the second image 150b and described first to extract the interim function storing of the information of extracting region in region and second.
The electronic-controlled installation 140 of vehicle is the result of utilizing at described image converter section 128 final Stereo matchings, implement as FCW (Forward Collision Warning, frontal collisions warning), the function such as LDWS (Lane Departure Warning System, lane-departure warning system).
Fig. 4 is the precedence diagram of the stereoscopic image processing method of the preferred embodiment of the present invention.
The three-dimensional camera 110 of vehicle is to utilize the first camera 110a and second camera 110b to obtain the first image 150a and the second image 150b(step S200).
The pixel extraction portion 122 of Stereo matching portion 120 is that the maximum inclination A that considers vehicle extracts respectively for calculating the first extraction region and the second extraction region (step S210) of gradient between the first image 150a and the second image 150b from the first image 150a and the second image 150b.
Feature Points Matching portion 124 extracts region and second to described first to extract region and compare and calculate characteristic point (step S220).
Gradient calculating portion 126 utilizes the characteristic point of calculating from described Feature Points Matching portion 124 to calculate the first extraction region and the second gradient (step S230) of extracting between region.
Image converter section 128 is to utilize described gradient, with some being as the criterion in the first image 150a or the second image 150b, makes another image rotation (step S240).
Image converter section 128 be rotation be that the first image 150a and the second image 150b that image rectification completes implements the Stereo matching (step S250) between the first image 150a and the second image 150b.
Final Stereo matching result in image converter section 128 is transmitted in the electronic-controlled installation 140 of vehicle, and electronic-controlled installation 140 is to utilize Stereo matching result to implement the control of vehicle.
Above embodiment only, in order to technical scheme of the present invention to be described, is not intended to limit; Although the present invention is had been described in detail with reference to previous embodiment, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in aforementioned each embodiment, or part technical characterictic is wherein equal to replacement; And these modifications or replacement do not make the essence of appropriate technical solution depart from the scope of technical scheme described in various embodiments of the present invention.Protection scope of the present invention should make an explanation according to following claim scope, and all technical schemes in its equal scope should all belong to claim scope of the present invention.

Claims (7)

1. a stereoscopic image processing device, is characterized in that,
Device as processing the stereo-picture of vehicle, comprising:
Three-dimensional camera, comprises the first camera and the second camera that obtain stereo-picture; And
Stereo matching portion, utilizes described stereo-picture to implement Stereo matching;
Described Stereo matching portion utilizes described the first camera of setting according to the state of described vehicle and the maximum inclination between described second camera, implements the correction between the first image of described the first camera and the second image of described second camera.
2. stereoscopic image processing device according to claim 1, is characterized in that,
Described Stereo matching portion comprises:
Pixel extraction portion, utilizes described maximum inclination for proofreading and correct, to extract respectively the first extraction region and the second extraction region from described the first image and described the second image;
Feature Points Matching portion, extracts region and described second from described first and extracts extracted region characteristic of correspondence point;
Gradient calculating portion, utilizes described characteristic point, calculates described first and extracts region and the described second gradient of extracting between region; And
Image converter section, utilizes described gradient to proofread and correct described the first image and described the second image, implements final Stereo matching.
3. stereoscopic image processing device according to claim 2, is characterized in that,
Described the first extraction region is from X row of longitudinal extraction from described the first image.
4. according to the stereoscopic image processing device described in claim 2 or 3, it is characterized in that,
Described the second extraction region is to consider maximum inclination described in described the second image, extracts a plurality of row that comprise described X row.
5. a stereoscopic image processing method, as the stereoscopic image processing method of vehicle, this implementation step comprises:
(a) from the three-dimensional camera of vehicle, obtain the first image and the second image;
(b) the pixel extraction portion in Stereo matching portion utilizes according to described first camera of the state setting of described vehicle and the maximum inclination between second camera, by set extracted region in described the first image and the second image, is that region is extracted in the first extraction region and second;
(c) in described Stereo matching portion, calculate and described first, extract region and described second and extract region characteristic of correspondence point, calculate the gradient between described the first image and described the second image;
(d) in described Stereo matching portion, utilize described gradient to proofread and correct described the first image and described the second image, implement final Stereo matching.
6. stereoscopic image processing method according to claim 5, is characterized in that,
Described the first extraction region is to X row of longitudinal extraction from described the first image.
7. according to the stereoscopic image processing method described in claim 5 or 6, it is characterized in that,
Described the second extraction region is to consider that from described the second image described maximum inclination extraction comprises a plurality of row of described X row.
CN201310380608.7A 2012-09-06 2013-08-28 Stereo image processing device and method thereof Active CN103686126B (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020120098990A KR101907852B1 (en) 2012-09-06 2012-09-06 Method and Apparatus for Processing Stereo Images
KR10-2012-0098990 2012-09-06

Publications (2)

Publication Number Publication Date
CN103686126A true CN103686126A (en) 2014-03-26
CN103686126B CN103686126B (en) 2017-05-24

Family

ID=50322229

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310380608.7A Active CN103686126B (en) 2012-09-06 2013-08-28 Stereo image processing device and method thereof

Country Status (2)

Country Link
KR (1) KR101907852B1 (en)
CN (1) CN103686126B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10063831B2 (en) 2015-02-09 2018-08-28 Samsung Electronics Co., Ltd. Method and apparatus for stereo image rectification
KR102250289B1 (en) * 2015-04-02 2021-05-11 현대자동차주식회사 Apparatus and Method for Compensating of Image Data Obtained by Stereo Camera
KR101766149B1 (en) 2016-07-05 2017-08-07 현대자동차주식회사 Apparatus and Method for Stereo Matching which needs small Computation

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101354796A (en) * 2008-09-05 2009-01-28 浙江大学 Omnidirectional stereo vision three-dimensional rebuilding method based on Taylor series model
US20120026295A1 (en) * 2009-03-31 2012-02-02 Hirofumi Nishimura Stereo image processor and stereo image processing method
CN102404595A (en) * 2011-08-16 2012-04-04 上海交通大学 Epipolar line correction method capable of providing 3D program shooting guidance

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3765862B2 (en) * 1996-02-15 2006-04-12 本田技研工業株式会社 Vehicle environment recognition device
JP2008286648A (en) * 2007-05-17 2008-11-27 Sumitomo Electric Ind Ltd Distance measuring device, distance measuring system, and distance measuring method
JP4702569B2 (en) 2008-09-30 2011-06-15 マツダ株式会社 Image processing apparatus for vehicle

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101354796A (en) * 2008-09-05 2009-01-28 浙江大学 Omnidirectional stereo vision three-dimensional rebuilding method based on Taylor series model
US20120026295A1 (en) * 2009-03-31 2012-02-02 Hirofumi Nishimura Stereo image processor and stereo image processing method
CN102404595A (en) * 2011-08-16 2012-04-04 上海交通大学 Epipolar line correction method capable of providing 3D program shooting guidance

Also Published As

Publication number Publication date
KR20140032283A (en) 2014-03-14
CN103686126B (en) 2017-05-24
KR101907852B1 (en) 2018-10-15

Similar Documents

Publication Publication Date Title
US9729858B2 (en) Stereo auto-calibration from structure-from-motion
CN106295560B (en) Lane keeping method based on vehicle-mounted binocular camera and segmented PID control
CN103231708B (en) A kind of intelligent vehicle barrier-avoiding method based on binocular vision
JP7485612B2 (en) Multispectral systems for providing pre-crash alerts
EP3082066B1 (en) Road surface gradient detection device
JP4956452B2 (en) Vehicle environment recognition device
JP5011049B2 (en) Image processing system
US9912933B2 (en) Road surface detection device and road surface detection system
CN106428209A (en) Steering assistant
CN107792070B (en) Driving assistance device for vehicle
US9898669B2 (en) Traveling road surface detection device and traveling road surface detection method
JP2009176090A (en) Environment recognition system
US20090184845A1 (en) Vehicle Detecting System
CN113196007B (en) Camera system applied to vehicle
CN104835136A (en) System and method for generating a depth map through iterative interpolation and warping
CN105243655A (en) System and method for estimating vehicle dynamics using feature points in images from multiple cameras
CN112435287A (en) Distance estimation apparatus, operation method thereof, and host vehicle apparatus
CN102136136A (en) Luminosity insensitivity stereo matching method based on self-adapting Census conversion
CN110949254A (en) Vehicle running environment detection device and running control system
CN103686126A (en) Stereo image processing device and method thereof
US20170057508A1 (en) Vehicle travel control device
CN110807347A (en) Obstacle detection method and device and terminal
CN108010005B (en) Method and device for adjusting image brightness and vehicle
CN105389790B (en) Apparatus, method and computer readable medium for correcting interpolation coefficients for stereo matching
JP4696925B2 (en) Image processing device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant