US20140184814A1 - Calibration reference pattern for vehicle camera and setting method thereof, and image conversion method and device - Google Patents
Calibration reference pattern for vehicle camera and setting method thereof, and image conversion method and device Download PDFInfo
- Publication number
- US20140184814A1 US20140184814A1 US14/065,852 US201314065852A US2014184814A1 US 20140184814 A1 US20140184814 A1 US 20140184814A1 US 201314065852 A US201314065852 A US 201314065852A US 2014184814 A1 US2014184814 A1 US 2014184814A1
- Authority
- US
- United States
- Prior art keywords
- calibration reference
- pattern
- characteristic patterns
- characteristic
- patterns
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 27
- 239000003086 colorant Substances 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 3
- 238000003708 edge detection Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 description 11
- 230000009466 transformation Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 241000212749 Zesius chrysomallus Species 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/40—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the details of the power supply or the coupling to vehicle components
- B60R2300/402—Image calibration
Definitions
- the disclosure relates in general to automatic calibration for a vehicle camera, and more particularly to an automatic calibration reference pattern for a vehicle camera and a setting method thereof, and an image conversion method and an image conversion device.
- Driving safety is crucial for drivers and passengers. Many techniques are available for assisting driving safety. For example, when reversing a vehicle, images behind the vehicle can be captured by a rear camera. Thus, in addition to observing with the naked eye, a driver may also determine whether obstacles or pedestrians are behind the vehicle though images captured by a rear safety assist system.
- Safety assist systems e.g., all-around-view driver assistance systems, forward collision avoidance systems, rear safety assist systems, and side blind-spot detection systems
- vehicle cameras Before being shipped out of the factory, the vehicle cameras installed in vehicles need to be calibrated first.
- a vehicle camera is manually fine-tuned by a calibration staff, such that characteristic patterns in captured images satisfy predetermined conditions and then image capture is adjusted to correct position/direction of the camera through the human-interface operation.
- a system then calculates camera extrinsic parameters and camera intrinsic parameters for coordinate conversion.
- the disclosure is directed to a calibration reference pattern for a vehicle camera, a method for setting a calibration reference pattern for a vehicle camera, and an image conversion and device applying the same.
- a calibration reference pattern for automatically calibrating a vehicle camera comprises: a first calibration reference sub-pattern, comprising a first characteristic pattern group, at least one straight line and a plurality of first grid lines, wherein the first characteristic pattern group has at least two identical first characteristic patterns; and a second calibration reference sub-pattern, comprising a second characteristic pattern group and a second straight line, wherein the second characteristic pattern group has at least two identical second characteristic patterns.
- the first characteristic patterns of the first characteristic pattern group are equidistant from a border of the first calibration reference sub-pattern.
- the second characteristic patterns of the second characteristic pattern group are equidistant from a border of the second calibration reference sub-pattern.
- a method for setting a calibration reference pattern comprises: placing two marking lines to align with two sides a vehicle and crossing the marking lines to position a central extension line of the vehicle respectively; aligning the first straight line of the first calibration reference sub-pattern with the central extension line of the vehicle; and aligning two ends of the second straight lines of the second calibration reference sub-pattern to the first grid lines of the first calibration reference sub-pattern, respectively.
- an image conversion method converts a captured input image in a first view angle coordinate system to an output image in a second view angle coordinate system.
- An image as the foregoing calibration reference pattern is captured.
- the image conversion method comprises: identifying the first and second characteristic patterns of the calibration reference pattern; obtaining coordinates of the first and second characteristic patterns in the first view angle coordinate system; and obtaining the first and second characteristic patterns in the second view angle coordinate system, to obtain a coordinate conversion relationship between the first view angle coordinate system and the second view angle coordinate system; and converting the input image to the output image according to the coordinate conversion relationship.
- an image conversion device for converting a captured input image in a first view angle coordinate system to an output image in a second view angle coordinate system, comprises an image capturing unit, a characteristic retrieval unit and a coordinate conversion unit.
- the image capturing unit captures an image as the foregoing calibration reference pattern.
- the characteristic retrieval unit identifies the first and second characteristic patterns of the calibration reference pattern.
- the coordinate conversion unit obtains coordinates of the first and second characteristic patterns in the first view angle coordinate system and obtains the first and second characteristic patterns at the second view angle coordinate system, to obtain a coordinate conversion relationship between the first view angle coordinate system and the second view angle coordinate system; and converts the input image to the output image according to the coordinate conversion relationship.
- FIG. 1 is a block diagram of a vehicle camera calibration device according to one embodiment.
- FIG. 2 is an example of a calibration reference pattern according to one embodiment.
- FIGS. 3A to 3E show setting of a calibration reference pattern according to one embodiment.
- FIGS. 4A and 4B are a calibration reference pattern and placement of the calibration reference pattern according to one embodiment.
- FIG. 5 is a flowchart of a vehicle camera calibration process according to one embodiment.
- FIG. 6 shows details of steps 530 and 540 in FIG. 5 .
- FIG. 7 shows conversion from an image coordinate system to a bird view coordinate system according to one embodiment.
- a length and a width of a calibration reference pattern as well as distances between characteristic points of the calibration reference pattern are known. Considering a longest bird view distance required by a user, position relationships of the characteristic points under different view angle coordinate systems are estimated to obtain a conversion relationship between different view angle coordinate systems.
- the calibration reference pattern, image contour characteristic recognition and optional color classification are used to achieve automatic calibration for a vehicle camera.
- FIG. 1 shows a block diagram of a vehicle camera calibration device 100 according to one embodiment.
- the vehicle camera calibration device 100 includes an image input unit 105 , an image processing unit 106 and a memory unit 107 .
- an image capturing unit 102 is a vehicle camera which is carried by a vehicle, and has an image capturing horizontal view angle of for example but not limited to 40 to 190 degrees, an image capturing frequency of for example but not limited to 15 to 30 frames/second, and an output frequency of for example but not limited to 15 to 30 frames/second.
- the image input unit 105 receives image data transmitted from the image capturing unit 102 .
- the image data is from the image capturing unit 102 when the image capturing unit 102 captures the calibration reference pattern 103 .
- the image processing unit 106 performs image characteristic identification, color classification and calculation of a view angle coordinate conversion relationship according to image of the calibration reference pattern.
- the view angle coordinate conversion relationship describes relationship between an image coordinate system and a bird view coordinate system.
- the image processing unit 106 includes a characteristic retrieval unit 106 A, a color classification unit 106 B and a coordinate conversion unit 106 C.
- the characteristic retrieval unit 106 A performs image characteristic identification on the calibration reference pattern 103 , with associated details to be described shortly.
- the color classification unit 1068 performs color classification on the calibration reference pattern 103 , with associated details also to be described shortly.
- the coordinate conversion unit 106 C calculates the relationship (e.g., a single transformation matrix H) between different view angle coordinate systems.
- the color classification unit 1066 may be optional. The reason is as follows.
- the calibration reference pattern may include a plurality of characteristic patterns which may be patterns (e.g., circles, squares and triangles), lines or texts or any combination thereof. For characteristic patterns having different shapes/lines but the same color, the color classification unit 1066 may be ignored.
- the memory unit 107 temporarily stores the view angle coordinate conversion relationship calculated by the image processing unit 106 , temporary image data and system operation values.
- the memory unit 107 is for example but not limited by a memory.
- the units 105 , 106 , 106 A, 1066 , 106 C are implemented by hardware or software or firmware.
- a digital signal processing-control unit e.g., a DSP or CPU
- an image outputting unit may be included.
- the digital signal processing-control unit performs functions such as peripheral circuit initialization, image characteristic and color identification, color classification, coordinate conversion and bird view conversion (e.g., converting a real-time captured image in front/behind/at the left/at the right of the vehicle to a real-time bird view image).
- the image outputting unit outputs images processed by the digital signal processing-control unit to display to a user.
- FIG. 2 shows an example of the calibration reference pattern 103 according to one embodiment.
- the calibration reference pattern 103 includes three calibration reference sub-patterns 23 , 24 and 25 , which are rectangular, for example.
- the first calibration reference sub-pattern 23 is placed at an extension line along a center of the vehicle, with details for determining a position for placing the first calibration reference sub-pattern 23 to be described shortly.
- the second calibration reference sub-pattern 24 and the third calibration reference sub-pattern 25 having substantially the same or similar structure, are placed on the calibration reference sub-pattern 23 and perpendicularly cross the calibration reference sub-pattern 23 .
- the first calibration reference sub-pattern 23 includes a first characteristic pattern group, first straight lines and first grid lines.
- the first characteristic pattern group has at least two identical first characteristic patterns, e.g., two circular sub-patterns 32 and 33 .
- the first straight lines include horizontal lines 30 and 31
- the first grid lines include grid lines 26 , 27 , 28 and 29 .
- the circular sub-patterns 32 and 33 are blue.
- the horizontal line 30 crosses and protrudes at the grid lines 26 and 27
- the horizontal line 31 crosses and protrudes at the grid lines 28 and 29 .
- the calibration reference sub-pattern 23 has a width W1 and a total length W3.
- the grid lines 26 and 27 are spaced by the distance W1
- the grid lines 27 and 28 are spaced by the distance W1
- the grid lines 28 and 29 are spaced by the distance W1
- the grid line 29 and a border of the calibration reference sub-pattern 23 are spaced by the distance W1
- the first characteristic patterns are equidistant from the border of the first characteristic pattern 23 .
- characteristic points e.g., centers, which are utilized as examples in the following descriptions
- characteristic points e.g., centers, which are utilized as examples in the following descriptions
- Extension lines of the horizontal lines 30 and 31 pass through the centers of the circular sub-patterns 32 and 33 , respectively.
- the grid lines 26 , 27 , 28 and 29 are equidistant from one another, and a distance between one of the grid lines and another border of the first calibration reference sub-pattern 23 (e.g., the distance W1 between the grid line 29 and a right border of the first calibration reference sub-pattern 23 ) equals the distances between the grid lines.
- the second calibration sub-pattern 24 includes a second characteristic pattern group and at least one second straight line.
- the second characteristic pattern group has at least two identical characteristic patterns, e.g., sub-patterns 34 , 35 , 36 , 37 , 38 and 39 .
- the straight line has a horizontal line 40 .
- the sub-patterns 34 , 35 , 36 , 37 , 38 and 39 are circular.
- the second calibration reference sub-pattern 24 has a width W1 and a total length W4. A distance between centers of two neighboring circular sub-patterns 35 , 36 , 37 , 38 and 39 is W1.
- the horizontal line 40 is located at the center of the second calibration reference sub-pattern 24 .
- the second characteristic patterns are equidistant to a border of the second calibration reference sub-pattern 24 .
- the centers of circular sub-patterns 35 , 36 , 37 , 38 and 39 are equidistant to a border (having a length W4) of the second calibration reference sub-pattern 24 .
- the horizontal line 40 is located on a perpendicular bisector by connecting the centers of the sub-patterns 36 and 37 , or the centers of the sub-patterns 35 and 38 or the centers of the sub-patterns 34 and 39 .
- the horizontal line 40 (i.e. the second straight line) is between the second characteristic patterns ( 34 ⁇ 39 ).
- the third calibration reference sub-pattern 25 includes a third characteristic pattern group and at least a third straight line.
- the third characteristic pattern group has at least two identical third characteristic patterns, e.g., sub-patterns 41 , 42 , 43 , 44 , 45 and 46 .
- the third straight line includes a horizontal line 47 .
- the sub-patterns 41 , 42 , 43 , 44 , 45 and 46 are circular.
- the third calibration reference sub-pattern 25 has a width W1 and a total length W4. A distance between centers of two neighboring circular sub-patterns 41 , 42 , 43 , 44 , 45 and 46 is W1.
- the horizontal line 47 is located at the center of the calibration reference sub-pattern 25 .
- the distances from the centers of the circular sub-patterns 41 , 42 , 43 , 44 , 45 and 46 to a border of the third calibration reference sub-pattern 25 are known.
- the third characteristic patterns are equidistant from a border of the third calibration reference sub-pattern 25 .
- the centers of circular sub-patterns 41 , 42 , 43 , 44 , 45 and 46 are equidistant from a border (having a length W4) of the third calibration reference sub-pattern 25 .
- the horizontal line 47 is located on a perpendicular bisector formed by connecting the centers of the sub-patterns 43 and 44 , or the centers of the sub-patterns 42 and 45 or the centers of the sub-patterns 41 and 46 .
- the calibration reference sub-patterns 24 and 25 are substantially the same.
- the calibration reference sub-patterns placed perpendicularly on the calibration reference sub-pattern 23 may be different.
- the calibration reference sub-patterns 24 and 25 may be substantially the same so that the manufacturing process is simple.
- the circular sub-patterns 34 , 35 , 36 , 37 , 38 and 39 have different colors.
- the circular sub-patterns 35 , 38 , 42 , and 45 are red
- the circular sub-patterns 34 , 36 , 37 , 39 , 41 , 43 , 44 and 46 are black.
- the vehicle camera calibration device 100 may determine whether the identified circular sub-patterns are in the calibration reference sub-patterns 24 or 25 , and determine the orders of the identified circular sub-patterns in the calibration reference sub-patterns 24 or 25 .
- the calibration reference pattern 103 is utilized in cooperation with two marking lines 48 and 49 .
- each of the marking lines 48 and 49 is marked with two positioning points (e.g., red spots) X and Y at two predetermined positions during installation of the calibration reference pattern.
- the marking lines 48 and 49 have the same length.
- the marking lines 48 and 49 may have different lengths.
- FIGS. 3A to 3E show setting of the calibration reference pattern according to one embodiment.
- the image capturing unit 102 captures images behind a vehicle is illustrated, but the application is not limited by this.
- the marking lines 48 and 49 are placed to align with two sides of a vehicle 1000 , respectively.
- the marking lines 48 and 49 are crossed, such that the two positioning points X of the marking lines 48 and 49 intersect each other, with the intersection denoted as an intersection C.
- the marking lines 48 and 49 are crossed, such that the two positioning points Y of the marking lines 48 and 49 intersect each other, with the intersection denoted as an intersection D.
- the two horizontal lines 30 and 31 of the calibration reference sub-pattern 23 are aligned with the intersections C and D, respectively.
- two ends of the central line 40 of the calibration reference sub-pattern 24 are aligned to the grid lines 26 and 27 of the calibration reference sub-pattern 23
- two ends of the central line 47 of the calibration reference sub-pattern 25 are aligned to the grid lines 28 and 29 to complete setting of the calibration reference pattern.
- the image capturing unit 102 captures images behind the vehicle is described for illustration purposes.
- the image capturing unit 102 may capture images in front/at the right/at the left of the vehicle, and the marking lines 48 and 49 and the calibration reference sub-patterns 23 to 25 are placed in front/at the right/at the left of the vehicle. Associated details may be referred from the above descriptions, and shall be omitted herein.
- the horizontal lines 30 and 31 of the calibration reference sub-pattern 23 may be transparent or light-transmissive, so that the two horizontal lines 30 and 31 of the calibration reference sub-pattern 23 may be easily aligned with the intersections C and D. Further, as previously described, the horizontal line 30 crosses and protrudes at the grid lines 26 and 27 , and the horizontal line 31 crosses and protrudes at the grid lines 28 and 29 . Such design is helpful for the two ends of the central line 40 of the calibration reference sub-pattern 24 to be aligned to the grid lines 26 and 27 of the calibration reference sub-pattern 23 , and the two ends of the central line 47 of the calibration reference sub-pattern 25 to be aligned to the grid lines 28 and 29 of the calibration reference sub-pattern 23 .
- FIGS. 4A and 4B show a calibration reference pattern according to one embodiment and the placement thereof.
- a first calibration reference pattern 400 includes a first calibration reference sub-pattern 410 and a second calibration reference sub-pattern 420 .
- the first calibration reference sub-pattern 410 and the second calibration reference sub-pattern 420 are rectangular.
- the first calibration reference sub-pattern 410 is placed at an extension line along a center of a vehicle. Details about how to place the first calibration reference sub-pattern 410 may be referred from FIGS. 3A to 3C .
- the second calibration reference sub-pattern 420 is aligned the first calibration reference sub-pattern 410 and perpendicularly crosses the first calibration reference sub-pattern 410 .
- the first calibration reference sub-pattern 410 includes a first characteristic pattern group, at least one first straight line and a plurality of first grid lines.
- the first characteristic pattern group has at least two identical first characteristic patterns, e.g., two circular sub-patterns 411 and 412 .
- the first straight line includes a horizontal line 413
- the first grid lines include grid lines 414 and 415 .
- the circular sub-patterns 411 and 412 are blue.
- the horizontal line 413 crosses and protrudes at the grid lines 414 and 415 .
- a width and a length of the first calibration reference sub-pattern 410 , a distance between the grid lines 414 and 415 , a distance from the grid line 414 to a border of the first calibration reference sub-pattern 410 , and a distance from the grid line 415 to another border of the first calibration reference sub-pattern 415 are known. Further, distances from centers of the circular sub-patterns 411 and 412 to a border of the first calibration reference sub-pattern 410 are known.
- the first characteristic patterns are equidistant from a border of the first calibration reference sub-pattern 410 . For example, centers of the circular sub-patterns 411 and 412 are equidistant from a border of the first calibration reference sub-pattern 410 .
- An extension line from the horizontal line 413 passes through the centers of the circular sub-patterns 411 and 412 .
- the grid lines 414 and 415 are equidistant from each other, and a distance from one of the grid lines 414 and 415 to another border of the first calibration reference sub-pattern 410 (e.g., a distance from the grid line 414 to an upper border of the first calibration reference sub-pattern 410 ) equals the distance between the grid lines 414 and 415 .
- the circular sub-patterns 411 and 412 are identical in shape and color.
- the horizontal line 413 is transparent or light-transmissive for facilitating the alignment of the calibration reference sub-patterns 420 and 410 .
- the second calibration reference sub-pattern 420 includes a second characteristic pattern group and a straight line.
- the second characteristic pattern group has at least two identical second characteristic patterns, e.g., two sub-patterns 421 and 422 .
- the second straight line is a horizontal line 423 .
- the sub-patterns 421 and 422 are circular.
- a width and a length of the calibration reference sub-pattern 420 and a distance between centers of the two circular sub-patterns 421 and 422 are known.
- the horizontal line 423 is located at the center of the calibration reference sub-pattern 420 .
- the second characteristic patterns are equidistant from a border of the second calibration reference sub-pattern 420 .
- the centers of the sub-patterns 421 and 422 are equidistant from a border of the second calibration reference sub-pattern 420 .
- the horizontal line 423 is located on a perpendicular bisector formed by connecting the centers of the sub-patterns 421 and 422 .
- a distance between the grid lines 414 and 415 of the first calibration reference sub-pattern 410 is equal to the width of the second calibration reference sub-pattern 420 .
- the sub-patterns 421 and 422 and are identical in color and shape.
- the calibration reference sub-pattern 410 is perpendicular to the vehicle and the calibration reference sub-pattern 420 is perpendicular to the calibration reference sub-pattern 410 .
- the calibration reference pattern includes a plurality of characteristic patterns (e.g., the circular sub-patterns), and the length and width of the calibration reference pattern, the distances between the characteristic patterns, and the distances between the characteristic patterns and the border are known.
- the length of the marking lines is also known.
- the characteristic patterns may be patterns (e.g., circles, squares and triangles), lines or texts or any combination thereof.
- the characteristic points of squares, triangles and lines may be geometric centers or corners, and the characteristic points of texts may be corners.
- the characteristic patterns may be in a same shape or different shapes. Alternatively, for characteristic patterns having different shapes, the characteristic patterns may have the same color or different colors.
- FIG. 5 shows a flowchart of a vehicle camera calibration process according to one embodiment.
- the calibration reference pattern is placed. Details for setting/placing the calibration reference pattern can be referred from associated descriptions above, and shall be omitted herein.
- step 520 an image of the calibration reference pattern which is placed is captured, e.g., by the image capturing unit 520 .
- step 530 automatic calibration for the vehicle camera is performed, with details thereof to be described shortly.
- characteristic patterns of the calibration reference pattern are identified through image identification and color classification to determine whether the video camera is placed in a correct position/direction. If not, the vehicle camera is automatically adjusted and step 520 is repeated until the video camera is placed in a correct position/direction.
- steps 510 to 530 are sufficient for calibration of the vehicle camera.
- images which are real-time captured may be converted to bird view images, for fast automated view angle conversion.
- step 540 after the automatic calibration, for the characteristic patterns of the calibration reference pattern which are identified, coordinates of the characteristic patterns of the calibration reference pattern on an image coordinate system are found to obtain a conversion relationship (e.g., a homography matrix) between the image coordinate system and a bird view coordinate system. Further, the conversion relationship between the image coordinate system and the bird view coordinate system may be stored for coordinate system conversion and as a reference for distance estimation. That is, bird view images are obtained by converting the captured images behind the vehicle, and the bird view images are displayed to a user/driver for assisting driving safety. Details of step 540 are described below. For example, step 540 is performed by the coordinate conversion unit 106 C in FIG. 1 .
- a conversion relationship e.g., a homography matrix
- FIG. 6 shows details of steps 530 and 540 in FIG. 5 according to one embodiment.
- step 530 includes steps 610 and 620
- step 540 includes steps 630 and 640 .
- contour recognition and optional color classification are performed on the characteristic patterns.
- Contour recognition is performed on the characteristic patterns of the calibration reference pattern to identify contour characteristics of the characteristic patterns to thereby obtain the characteristic patterns (e.g., center positions of circular sub-patterns).
- the image identification is performed by the characteristic retrieval unit 106 A in FIG. 1 .
- the contour recognition may be performed by such as a Hough transform algorithm, an edge detection algorithm, a morphology algorithm or an active contour method, or any combination thereof.
- Color classification e.g., performed by the color classification unit 106 B in FIG. 1 identifies colors of characteristic patterns having different colors. Thus, it determines that the identified characteristic patterns belong to which of the calibration reference patterns, for positioning. Further, an arrangement order of the identified characteristic patterns can be confirmed and the center positions thereof are recorded. If the characteristic patterns have the same color, color classification may be optional.
- step 620 it is determined whether a straight line connecting two center positions of the identified blue circular sub-patterns 32 and 33 is perpendicular to a rear edge of the vehicle, to determine whether the vehicle camera is installed at a correct position/angle or not.
- the straight line may be formed by connecting two or more coordinates of the first characteristic patterns ( 32 , 33 ) and/or the second characteristic patterns ( 34 ⁇ 39 ) and coordinates of the first characteristic patterns ( 32 , 33 ) and/or the second characteristic patterns ( 34 ⁇ 39 ) are the geometric centers, the corner points or the end points of the first characteristic patterns and the second characteristic patterns.
- the vehicle camera calibration device 100 After identification by the vehicle camera calibration device 100 , if the horizontal lines 30 and 31 are not perpendicular to the rear edge of the vehicle, it means that a lens of the image capturing unit 102 is deviated to the left or to the right.
- the vehicle camera calibration device 100 automatically sends out an instruction to adjust lens of the image capturing unit 102 to again capture images, as in step 520 , until the horizontal lines 30 and 31 are substantially perpendicular to the rear edge of the vehicle. Therefore, it is demonstrated that in the embodiment, instead of manual adjustment, the lens of the image capturing unit 102 may be automatically adjusted.
- the actual size of the calibration reference pattern, the sizes of the characteristic patterns, and the distances from the characteristic patterns to the calibration reference pattern are known.
- step 630 coordinates of the characteristic patterns in a bird view coordinate system are obtained.
- step 640 a coordinate conversion relationship between different view angle coordinate systems is obtained, with details thereof described below.
- FIG. 7 shows conversion between different view angle coordinate systems according to one embodiment.
- the upper part is an image coordinate system (corresponding to the captured image)
- the lower part is a bird view coordinate system (corresponding to the bird view image converted from the captured image).
- a farthest desirable view range e.g., a farthest visible range in the bird view image
- a farthest desirable view range e.g., a farthest visible range in the bird view image
- the farthest view range is defined as Wd (in meters), and a resolution of the bird view image is Rw ⁇ Rh (the same as a resolution of the captured image).
- the center coordinates of the circular sub-pattern 33 are defined as (X1, Y1); and in the bird view coordinate system, the coordinates of the converted blue circle 33 are (X1′, Y1′).
- the center coordinates of the circular sub-pattern 32 are defined as (X2, Y2); and in the bird view coordinate system, the coordinates of the converted blue circle 32 are (X2′, Y2′).
- the bird view coordinates of all of the characteristic patterns in the calibration reference patterns under the bird view coordinate system can be deduced accordingly.
- x, y and z represent pixel coordinates in the image coordinate system; x new , y new and z new represent pixel coordinates (estimated according to the equation (1)) in the bird view coordinate system.
- equation (1) can be rewritten as:
- equation (3) can be defined as:
- a 28 ⁇ 9 ( a x1 T ,a y1 T , . . . ,a x14 T ,a y14 T ) T (6).
- the transformation matrix H between the coordinate system and the bird view coordinate system is estimated based on 14 characteristic points. That is to say, to achieve distortion correction for an image captured at edges of a lens, it needs calibration characteristic points on image edges.
- the 14 characteristic points are described in the above example. In practice, four characteristic points are sufficient for obtaining the transformation matrix H between the image coordinate system and the bird view coordinate system.
- equation (6) can be rewritten as:
- a 8 ⁇ 9 ( a T x1 ,a T y1 , . . . ,a T x4 ,a T y4 ) T (7)
- equation (6) can be rewritten as:
- a 2n ⁇ 9 ( a T x1 ,a T y1 , . . . ,a T xn ,a T yn ) T (8)
- n represents the number of characteristic points, and is a positive integer.
- Equation (5) is solved through an optimization algorithm to obtain a single transformation matrix H having minimum errors.
- the optimization algorithm may be a linear least squares algorithm, a singular value decomposition algorithm, a random sample consensus (RANSAC) algorithm, a least median of squares (LMEDS) algorithm or any combination thereof.
- coordinates of non-characteristic patterns in the image coordinate system can also be converted to coordinates of non-characteristic patterns in the bird view coordinate system according to the single transformation matrix and equation (1).
- the single transformation matrix may be stored (e.g., in the memory unit 107 ) for use of a vehicle safety system.
- a vehicle safety system is capable of real-time converting a captured image to a bird view image for displaying to a user and for distance detection or distance estimation.
- the automatic calibration for vehicle cameras of other vehicles may be performed without again installing the calibration reference pattern.
- the automatic calibration for vehicle cameras of other vehicles may be performed by utilizing the calibration reference pattern previously placed, to offer a fast and simple vehicle camera calibration process for vehicle manufacturers.
- the view angle coordinate conversion relationship obtained from the automatic calibration for a vehicle camera in the embodiments may be applied for bird view projection conversion and distance estimation.
- the view angle coordinate conversion relationship may be provided to a vehicle image safety assist system for obstacle detection and distance estimation.
- the bird view image converted may then be outputted to a vehicle display device and presented to a user to assist in driving safety.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Closed-Circuit Television Systems (AREA)
- Mechanical Engineering (AREA)
- Studio Devices (AREA)
Abstract
Description
- This application claims the benefit of Taiwan application Serial No. 101151114, filed Dec. 28, 2012, the disclosure of which is incorporated by reference herein in its entirety.
- The disclosure relates in general to automatic calibration for a vehicle camera, and more particularly to an automatic calibration reference pattern for a vehicle camera and a setting method thereof, and an image conversion method and an image conversion device.
- Driving safety is crucial for drivers and passengers. Many techniques are available for assisting driving safety. For example, when reversing a vehicle, images behind the vehicle can be captured by a rear camera. Thus, in addition to observing with the naked eye, a driver may also determine whether obstacles or pedestrians are behind the vehicle though images captured by a rear safety assist system.
- Safety assist systems (e.g., all-around-view driver assistance systems, forward collision avoidance systems, rear safety assist systems, and side blind-spot detection systems) are usually equipped with a vehicle camera for capturing images outside a vehicle. Before being shipped out of the factory, the vehicle cameras installed in vehicles need to be calibrated first.
- In a conventional human-interface calibration, a vehicle camera is manually fine-tuned by a calibration staff, such that characteristic patterns in captured images satisfy predetermined conditions and then image capture is adjusted to correct position/direction of the camera through the human-interface operation. A system then calculates camera extrinsic parameters and camera intrinsic parameters for coordinate conversion.
- However, due to human factors such as different observations and different carefulness of different calibration staff, image errors are resulted in the current calibration procedure. Further, the calibration procedure requires paying special attention on setting of calibration reference patterns and solving extrinsic parameters may be a tremendous computation load on the system.
- The disclosure is directed to a calibration reference pattern for a vehicle camera, a method for setting a calibration reference pattern for a vehicle camera, and an image conversion and device applying the same.
- According to one embodiment, a calibration reference pattern for automatically calibrating a vehicle camera is provided. The calibration reference pattern comprises: a first calibration reference sub-pattern, comprising a first characteristic pattern group, at least one straight line and a plurality of first grid lines, wherein the first characteristic pattern group has at least two identical first characteristic patterns; and a second calibration reference sub-pattern, comprising a second characteristic pattern group and a second straight line, wherein the second characteristic pattern group has at least two identical second characteristic patterns. The first characteristic patterns of the first characteristic pattern group are equidistant from a border of the first calibration reference sub-pattern. The second characteristic patterns of the second characteristic pattern group are equidistant from a border of the second calibration reference sub-pattern.
- According to another embodiment, a method for setting a calibration reference pattern is provided. The calibration reference pattern is as described above. The method comprises: placing two marking lines to align with two sides a vehicle and crossing the marking lines to position a central extension line of the vehicle respectively; aligning the first straight line of the first calibration reference sub-pattern with the central extension line of the vehicle; and aligning two ends of the second straight lines of the second calibration reference sub-pattern to the first grid lines of the first calibration reference sub-pattern, respectively.
- According to another embodiment, an image conversion method is provided. The image conversion method converts a captured input image in a first view angle coordinate system to an output image in a second view angle coordinate system. An image as the foregoing calibration reference pattern is captured. The image conversion method comprises: identifying the first and second characteristic patterns of the calibration reference pattern; obtaining coordinates of the first and second characteristic patterns in the first view angle coordinate system; and obtaining the first and second characteristic patterns in the second view angle coordinate system, to obtain a coordinate conversion relationship between the first view angle coordinate system and the second view angle coordinate system; and converting the input image to the output image according to the coordinate conversion relationship.
- According to yet another embodiment, an image conversion device is provided. The image conversion device, for converting a captured input image in a first view angle coordinate system to an output image in a second view angle coordinate system, comprises an image capturing unit, a characteristic retrieval unit and a coordinate conversion unit. The image capturing unit captures an image as the foregoing calibration reference pattern. The characteristic retrieval unit identifies the first and second characteristic patterns of the calibration reference pattern. The coordinate conversion unit obtains coordinates of the first and second characteristic patterns in the first view angle coordinate system and obtains the first and second characteristic patterns at the second view angle coordinate system, to obtain a coordinate conversion relationship between the first view angle coordinate system and the second view angle coordinate system; and converts the input image to the output image according to the coordinate conversion relationship.
-
FIG. 1 is a block diagram of a vehicle camera calibration device according to one embodiment. -
FIG. 2 is an example of a calibration reference pattern according to one embodiment. -
FIGS. 3A to 3E show setting of a calibration reference pattern according to one embodiment. -
FIGS. 4A and 4B are a calibration reference pattern and placement of the calibration reference pattern according to one embodiment. -
FIG. 5 is a flowchart of a vehicle camera calibration process according to one embodiment. -
FIG. 6 shows details ofsteps FIG. 5 . -
FIG. 7 shows conversion from an image coordinate system to a bird view coordinate system according to one embodiment. - In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. One or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are schematically shown in order to simplify the drawing.
- In the embodiments, a length and a width of a calibration reference pattern as well as distances between characteristic points of the calibration reference pattern are known. Considering a longest bird view distance required by a user, position relationships of the characteristic points under different view angle coordinate systems are estimated to obtain a conversion relationship between different view angle coordinate systems. In the embodiments, the calibration reference pattern, image contour characteristic recognition and optional color classification are used to achieve automatic calibration for a vehicle camera.
-
FIG. 1 shows a block diagram of a vehiclecamera calibration device 100 according to one embodiment. As shown inFIG. 1 , the vehiclecamera calibration device 100 includes animage input unit 105, animage processing unit 106 and amemory unit 107. - For example, an
image capturing unit 102 is a vehicle camera which is carried by a vehicle, and has an image capturing horizontal view angle of for example but not limited to 40 to 190 degrees, an image capturing frequency of for example but not limited to 15 to 30 frames/second, and an output frequency of for example but not limited to 15 to 30 frames/second. - Details of a
calibration reference pattern 103 are described shortly. - The
image input unit 105 receives image data transmitted from theimage capturing unit 102. For example, the image data is from theimage capturing unit 102 when theimage capturing unit 102 captures thecalibration reference pattern 103. - The
image processing unit 106 performs image characteristic identification, color classification and calculation of a view angle coordinate conversion relationship according to image of the calibration reference pattern. The view angle coordinate conversion relationship describes relationship between an image coordinate system and a bird view coordinate system. - The
image processing unit 106 includes acharacteristic retrieval unit 106A, acolor classification unit 106B and acoordinate conversion unit 106C. Thecharacteristic retrieval unit 106A performs image characteristic identification on thecalibration reference pattern 103, with associated details to be described shortly. The color classification unit 1068 performs color classification on thecalibration reference pattern 103, with associated details also to be described shortly. Thecoordinate conversion unit 106C calculates the relationship (e.g., a single transformation matrix H) between different view angle coordinate systems. Further, the color classification unit 1066 may be optional. The reason is as follows. The calibration reference pattern may include a plurality of characteristic patterns which may be patterns (e.g., circles, squares and triangles), lines or texts or any combination thereof. For characteristic patterns having different shapes/lines but the same color, the color classification unit 1066 may be ignored. - The
memory unit 107 temporarily stores the view angle coordinate conversion relationship calculated by theimage processing unit 106, temporary image data and system operation values. Thememory unit 107 is for example but not limited by a memory. For example, theunits - In an alternative embodiment, other optional elements such as a digital signal processing-control unit (e.g., a DSP or CPU) or an image outputting unit may be included. The digital signal processing-control unit performs functions such as peripheral circuit initialization, image characteristic and color identification, color classification, coordinate conversion and bird view conversion (e.g., converting a real-time captured image in front/behind/at the left/at the right of the vehicle to a real-time bird view image). The image outputting unit outputs images processed by the digital signal processing-control unit to display to a user.
-
FIG. 2 shows an example of thecalibration reference pattern 103 according to one embodiment. As shown inFIG. 2 , thecalibration reference pattern 103 includes three calibration reference sub-patterns 23, 24 and 25, which are rectangular, for example. The firstcalibration reference sub-pattern 23 is placed at an extension line along a center of the vehicle, with details for determining a position for placing the firstcalibration reference sub-pattern 23 to be described shortly. The secondcalibration reference sub-pattern 24 and the thirdcalibration reference sub-pattern 25, having substantially the same or similar structure, are placed on thecalibration reference sub-pattern 23 and perpendicularly cross thecalibration reference sub-pattern 23. - The first
calibration reference sub-pattern 23 includes a first characteristic pattern group, first straight lines and first grid lines. The first characteristic pattern group has at least two identical first characteristic patterns, e.g., twocircular sub-patterns horizontal lines grid lines calibration reference sub-pattern 24 and the third calibration reference sub-pattern 25 (to be perpendicularly placed on the first calibration reference sub-pattern 23), thehorizontal line 30 crosses and protrudes at thegrid lines horizontal line 31 crosses and protrudes at thegrid lines calibration reference sub-pattern 23 has a width W1 and a total length W3. The grid lines 26 and 27 are spaced by the distance W1, thegrid lines grid lines grid line 29 and a border of thecalibration reference sub-pattern 23 are spaced by the distance W1, and thegrid line 26 and one other border of thecalibration reference sub-pattern 23 are spaced by a distance W2. That is, W3=4*W1+W2. In other words, distances between centers of the circular sub-patterns 32 and 33 and the borders of thecalibration reference sub-pattern 23 are known. The first characteristic patterns are equidistant from the border of the firstcharacteristic pattern 23. For example, characteristic points (e.g., centers, which are utilized as examples in the following descriptions) of the circular sub-patterns 32 and 33 are equidistant from a horizontal border of the firstcalibration reference sub-pattern 23. Extension lines of thehorizontal lines grid line 29 and a right border of the first calibration reference sub-pattern 23) equals the distances between the grid lines. - The
second calibration sub-pattern 24 includes a second characteristic pattern group and at least one second straight line. The second characteristic pattern group has at least two identical characteristic patterns, e.g., sub-patterns 34, 35, 36, 37, 38 and 39. For example, the straight line has ahorizontal line 40. In the embodiment, for example, the sub-patterns 34, 35, 36, 37, 38 and 39 are circular. The secondcalibration reference sub-pattern 24 has a width W1 and a total length W4. A distance between centers of two neighboringcircular sub-patterns horizontal line 40 is located at the center of the secondcalibration reference sub-pattern 24. Distances from the centers of the circular sub-patterns 35, 36, 37, 38 and 39 to a border of the secondcalibration reference sub-pattern 24 are known. The second characteristic patterns are equidistant to a border of the secondcalibration reference sub-pattern 24. For example, the centers of circular sub-patterns 35, 36, 37, 38 and 39 are equidistant to a border (having a length W4) of the secondcalibration reference sub-pattern 24. Thehorizontal line 40 is located on a perpendicular bisector by connecting the centers of the sub-patterns 36 and 37, or the centers of the sub-patterns 35 and 38 or the centers of the sub-patterns 34 and 39. Distances between the first calibration reference sub-pattern and thegrid lines calibration reference sub-pattern 24. The horizontal line 40 (i.e. the second straight line) is between the second characteristic patterns (34˜39). - Similarly, the third
calibration reference sub-pattern 25 includes a third characteristic pattern group and at least a third straight line. The third characteristic pattern group has at least two identical third characteristic patterns, e.g., sub-patterns 41, 42, 43, 44, 45 and 46. For example, the third straight line includes ahorizontal line 47. In the embodiment, for example, the sub-patterns 41, 42, 43, 44, 45 and 46 are circular. The thirdcalibration reference sub-pattern 25 has a width W1 and a total length W4. A distance between centers of two neighboringcircular sub-patterns horizontal line 47 is located at the center of thecalibration reference sub-pattern 25. The distances from the centers of the circular sub-patterns 41, 42, 43, 44, 45 and 46 to a border of the thirdcalibration reference sub-pattern 25 are known. The third characteristic patterns are equidistant from a border of the thirdcalibration reference sub-pattern 25. For example, the centers of circular sub-patterns 41, 42, 43, 44, 45 and 46 are equidistant from a border (having a length W4) of the thirdcalibration reference sub-pattern 25. Thehorizontal line 47 is located on a perpendicular bisector formed by connecting the centers of the sub-patterns 43 and 44, or the centers of the sub-patterns 42 and 45 or the centers of the sub-patterns 41 and 46. Distances between the firstcalibration reference sub-pattern 23 and thegrid lines calibration reference sub-pattern 25. InFIG. 2 , thecalibration reference sub-patterns calibration reference sub-pattern 23 may be different. However, thecalibration reference sub-patterns - To meet identification requirements of the vehicle
camera calibration device 100, in the embodiment, the circular sub-patterns 34, 35, 36, 37, 38 and 39 have different colors. For example, the circular sub-patterns 35, 38, 42, and 45 are red, and the circular sub-patterns 34, 36, 37, 39, 41, 43, 44 and 46 are black. As such, when identifying the circular sub-patterns, the vehiclecamera calibration device 100 may determine whether the identified circular sub-patterns are in thecalibration reference sub-patterns calibration reference sub-patterns - When calibrating the vehicle camera, the
calibration reference pattern 103 is utilized in cooperation with two markinglines lines lines lines lines -
FIGS. 3A to 3E show setting of the calibration reference pattern according to one embodiment. In the embodiment, an example that theimage capturing unit 102 captures images behind a vehicle is illustrated, but the application is not limited by this. - As shown in
FIG. 3A , the markinglines vehicle 1000, respectively. As shown inFIG. 3B , the markinglines lines FIG. 3C , the markinglines lines - As shown in
FIG. 3D , the twohorizontal lines calibration reference sub-pattern 23 are aligned with the intersections C and D, respectively. - As shown in
FIG. 3E , two ends of thecentral line 40 of thecalibration reference sub-pattern 24 are aligned to thegrid lines calibration reference sub-pattern 23, and two ends of thecentral line 47 of thecalibration reference sub-pattern 25 are aligned to thegrid lines - In the embodiment, the example that the
image capturing unit 102 captures images behind the vehicle is described for illustration purposes. In an alternative embodiment, theimage capturing unit 102 may capture images in front/at the right/at the left of the vehicle, and the markinglines - To facilitate the alignment operation, the
horizontal lines calibration reference sub-pattern 23 may be transparent or light-transmissive, so that the twohorizontal lines calibration reference sub-pattern 23 may be easily aligned with the intersections C and D. Further, as previously described, thehorizontal line 30 crosses and protrudes at thegrid lines horizontal line 31 crosses and protrudes at thegrid lines central line 40 of thecalibration reference sub-pattern 24 to be aligned to thegrid lines calibration reference sub-pattern 23, and the two ends of thecentral line 47 of thecalibration reference sub-pattern 25 to be aligned to thegrid lines calibration reference sub-pattern 23. -
FIGS. 4A and 4B show a calibration reference pattern according to one embodiment and the placement thereof. As shown inFIG. 4A , a first calibration reference pattern 400 includes a firstcalibration reference sub-pattern 410 and a secondcalibration reference sub-pattern 420. For example, the firstcalibration reference sub-pattern 410 and the secondcalibration reference sub-pattern 420 are rectangular. The firstcalibration reference sub-pattern 410 is placed at an extension line along a center of a vehicle. Details about how to place the firstcalibration reference sub-pattern 410 may be referred fromFIGS. 3A to 3C . The secondcalibration reference sub-pattern 420 is aligned the firstcalibration reference sub-pattern 410 and perpendicularly crosses the firstcalibration reference sub-pattern 410. - The first
calibration reference sub-pattern 410 includes a first characteristic pattern group, at least one first straight line and a plurality of first grid lines. The first characteristic pattern group has at least two identical first characteristic patterns, e.g., twocircular sub-patterns 411 and 412. For example, the first straight line includes ahorizontal line 413, and the first grid lines includegrid lines 414 and 415. In the embodiment, for example, thecircular sub-patterns 411 and 412 are blue. To facilitate the alignment of thecalibration reference sub-pattern 420, thehorizontal line 413 crosses and protrudes at thegrid lines 414 and 415. A width and a length of the firstcalibration reference sub-pattern 410, a distance between thegrid lines 414 and 415, a distance from thegrid line 414 to a border of the firstcalibration reference sub-pattern 410, and a distance from the grid line 415 to another border of the first calibration reference sub-pattern 415 are known. Further, distances from centers of thecircular sub-patterns 411 and 412 to a border of the firstcalibration reference sub-pattern 410 are known. The first characteristic patterns are equidistant from a border of the firstcalibration reference sub-pattern 410. For example, centers of thecircular sub-patterns 411 and 412 are equidistant from a border of the firstcalibration reference sub-pattern 410. An extension line from thehorizontal line 413 passes through the centers of thecircular sub-patterns 411 and 412. The grid lines 414 and 415 are equidistant from each other, and a distance from one of thegrid lines 414 and 415 to another border of the first calibration reference sub-pattern 410 (e.g., a distance from thegrid line 414 to an upper border of the first calibration reference sub-pattern 410) equals the distance between thegrid lines 414 and 415. Thecircular sub-patterns 411 and 412 are identical in shape and color. For example, thehorizontal line 413 is transparent or light-transmissive for facilitating the alignment of thecalibration reference sub-patterns - The second
calibration reference sub-pattern 420 includes a second characteristic pattern group and a straight line. The second characteristic pattern group has at least two identical second characteristic patterns, e.g., twosub-patterns horizontal line 423. In the embodiment, for example, thesub-patterns calibration reference sub-pattern 420 and a distance between centers of the twocircular sub-patterns horizontal line 423 is located at the center of thecalibration reference sub-pattern 420. Distances from the centers of thecircular sub-patterns calibration reference sub-pattern 420 are known. The second characteristic patterns are equidistant from a border of the secondcalibration reference sub-pattern 420. For example, the centers of thesub-patterns calibration reference sub-pattern 420. Thehorizontal line 423 is located on a perpendicular bisector formed by connecting the centers of thesub-patterns grid lines 414 and 415 of the firstcalibration reference sub-pattern 410 is equal to the width of the secondcalibration reference sub-pattern 420. Thesub-patterns - As shown in
FIG. 4B , in placement, thecalibration reference sub-pattern 410 is perpendicular to the vehicle and thecalibration reference sub-pattern 420 is perpendicular to thecalibration reference sub-pattern 410. - Referring to
FIGS. 2 and 4A , in the embodiments, the calibration reference pattern includes a plurality of characteristic patterns (e.g., the circular sub-patterns), and the length and width of the calibration reference pattern, the distances between the characteristic patterns, and the distances between the characteristic patterns and the border are known. The length of the marking lines is also known. Further, the characteristic patterns may be patterns (e.g., circles, squares and triangles), lines or texts or any combination thereof. For example, the characteristic points of squares, triangles and lines may be geometric centers or corners, and the characteristic points of texts may be corners. Further, for characteristic patterns having different colors, the characteristic patterns may be in a same shape or different shapes. Alternatively, for characteristic patterns having different shapes, the characteristic patterns may have the same color or different colors. -
FIG. 5 shows a flowchart of a vehicle camera calibration process according to one embodiment. As shown inFIG. 5 , instep 510, the calibration reference pattern is placed. Details for setting/placing the calibration reference pattern can be referred from associated descriptions above, and shall be omitted herein. - In
step 520, an image of the calibration reference pattern which is placed is captured, e.g., by theimage capturing unit 520. - In
step 530, automatic calibration for the vehicle camera is performed, with details thereof to be described shortly. Instep 530, characteristic patterns of the calibration reference pattern are identified through image identification and color classification to determine whether the video camera is placed in a correct position/direction. If not, the vehicle camera is automatically adjusted and step 520 is repeated until the video camera is placed in a correct position/direction. In principle, steps 510 to 530 are sufficient for calibration of the vehicle camera. In the embodiment, images which are real-time captured may be converted to bird view images, for fast automated view angle conversion. - In
step 540, after the automatic calibration, for the characteristic patterns of the calibration reference pattern which are identified, coordinates of the characteristic patterns of the calibration reference pattern on an image coordinate system are found to obtain a conversion relationship (e.g., a homography matrix) between the image coordinate system and a bird view coordinate system. Further, the conversion relationship between the image coordinate system and the bird view coordinate system may be stored for coordinate system conversion and as a reference for distance estimation. That is, bird view images are obtained by converting the captured images behind the vehicle, and the bird view images are displayed to a user/driver for assisting driving safety. Details ofstep 540 are described below. For example,step 540 is performed by the coordinateconversion unit 106C inFIG. 1 . -
FIG. 6 shows details ofsteps FIG. 5 according to one embodiment. As shown inFIG. 6 ,step 530 includessteps steps - In
step 610, contour recognition and optional color classification are performed on the characteristic patterns. Contour recognition is performed on the characteristic patterns of the calibration reference pattern to identify contour characteristics of the characteristic patterns to thereby obtain the characteristic patterns (e.g., center positions of circular sub-patterns). For example, the image identification is performed by thecharacteristic retrieval unit 106A inFIG. 1 . For example but not limited by, the contour recognition may be performed by such as a Hough transform algorithm, an edge detection algorithm, a morphology algorithm or an active contour method, or any combination thereof. Color classification (e.g., performed by thecolor classification unit 106B inFIG. 1 ) identifies colors of characteristic patterns having different colors. Thus, it determines that the identified characteristic patterns belong to which of the calibration reference patterns, for positioning. Further, an arrangement order of the identified characteristic patterns can be confirmed and the center positions thereof are recorded. If the characteristic patterns have the same color, color classification may be optional. - In
step 620, it is determined whether a straight line connecting two center positions of the identified bluecircular sub-patterns - In calibration for a vehicle camera according to one embodiment, after identification by the vehicle
camera calibration device 100, if thehorizontal lines image capturing unit 102 is deviated to the left or to the right. The vehiclecamera calibration device 100 automatically sends out an instruction to adjust lens of theimage capturing unit 102 to again capture images, as instep 520, until thehorizontal lines image capturing unit 102 may be automatically adjusted. - In the embodiment, the actual size of the calibration reference pattern, the sizes of the characteristic patterns, and the distances from the characteristic patterns to the calibration reference pattern are known. In
step 630, coordinates of the characteristic patterns in a bird view coordinate system are obtained. - In
step 640, a coordinate conversion relationship between different view angle coordinate systems is obtained, with details thereof described below. -
FIG. 7 shows conversion between different view angle coordinate systems according to one embodiment. InFIG. 7 , the upper part is an image coordinate system (corresponding to the captured image), and the lower part is a bird view coordinate system (corresponding to the bird view image converted from the captured image). In the embodiment, a farthest desirable view range (e.g., a farthest visible range in the bird view image) of the bird view coordinate system is user defined. - For example, the farthest view range is defined as Wd (in meters), and a resolution of the bird view image is Rw×Rh (the same as a resolution of the captured image).
- The sizes (e.g., W1 to W4 in
FIG. 2 ) in the calibration reference pattern are known. Thus, inFIG. 7 , Wd_c1=W3−(W1/2), and Wd_c2=W2+3*(W1/2). - In the identification result, the center coordinates of the
circular sub-pattern 33 are defined as (X1, Y1); and in the bird view coordinate system, the coordinates of the convertedblue circle 33 are (X1′, Y1′). -
X1′=Rw/2 -
Y1′=Rh*[1−(Wd — c1/Wd) - Similarly, in the identification result, the center coordinates of the
circular sub-pattern 32 are defined as (X2, Y2); and in the bird view coordinate system, the coordinates of the convertedblue circle 32 are (X2′, Y2′). -
X2′=Rw/2 -
Y2′=Rh*[1−(Wd — c1/Wd) - As such, the bird view coordinates of all of the characteristic patterns in the calibration reference patterns under the bird view coordinate system can be deduced accordingly.
- Through a homography algorithm, the transformation matrix H between the image coordinate system and the bird view coordinate system is defined as the equation below:
-
- In equation (1), x, y and z represent pixel coordinates in the image coordinate system; xnew, ynew and znew represent pixel coordinates (estimated according to the equation (1)) in the bird view coordinate system.
- From a perspective of a non-homogenous coordinate system
-
- As the image is in a two-dimensional coordinate system, it is assumed that z=1, and equation (1) can be rewritten as:
-
x′ new(H 31 x+H 32 y+H 33)=H 11 x+H 12 y+H 13 -
y′ new(H 31 x+H 32 y+H 33)=H 21 x+H 22 y+H 23 (3) - To solve coefficients H11 to H33 in the transformation matrix H, equation (3) can be defined as:
-
a x T=0,a y T h=0 (4), - where
-
- h=(H11, H12, H13, H21, H22, H23, H31, H32, H33)T,
- ax=(−x,−y,−1,0,0,0,x′newx,x′newy,x′new)T,
- ay=(0,0,0,−x,−y,−1,y′newx,y′newy,y′new)T.
- By substituting the coordinates of the 14 circular sub-patterns in the image coordinate system and the coordinates of the 14 circular sub-patterns in the bird view coordinate system into equation (4), it is obtained that:
-
Ah=0 (5), -
where -
A 28×9=(a x1 T ,a y1 T , . . . ,a x14 T ,a y14 T)T (6). - Due to lens distortion, in the above example, the transformation matrix H between the coordinate system and the bird view coordinate system is estimated based on 14 characteristic points. That is to say, to achieve distortion correction for an image captured at edges of a lens, it needs calibration characteristic points on image edges. For clear illustrations, the 14 characteristic points are described in the above example. In practice, four characteristic points are sufficient for obtaining the transformation matrix H between the image coordinate system and the bird view coordinate system. When four characteristic points are utilized, equation (6) can be rewritten as:
-
A 8×9=(a T x1 ,a T y1 , . . . ,a T x4 ,a T y4)T (7) - For an n number of characteristic points utilized, equation (6) can be rewritten as:
-
A 2n×9=(a T x1 ,a T y1 , . . . ,a T xn ,a T yn)T (8) - In equation (8), n represents the number of characteristic points, and is a positive integer.
- Equation (5) is solved through an optimization algorithm to obtain a single transformation matrix H having minimum errors. For example but not limited by, the optimization algorithm may be a linear least squares algorithm, a singular value decomposition algorithm, a random sample consensus (RANSAC) algorithm, a least median of squares (LMEDS) algorithm or any combination thereof.
- After obtaining the optimized single transformation matrix, coordinates of non-characteristic patterns in the image coordinate system can also be converted to coordinates of non-characteristic patterns in the bird view coordinate system according to the single transformation matrix and equation (1). Further, the single transformation matrix may be stored (e.g., in the memory unit 107) for use of a vehicle safety system. By applying the single transformation matrix, a vehicle safety system is capable of real-time converting a captured image to a bird view image for displaying to a user and for distance detection or distance estimation.
- In the embodiment, if the calibration reference pattern is placed according to requirements, after calibrating the vehicle camera of a first vehicle, the automatic calibration for vehicle cameras of other vehicles (that are parked at the same parking position as the first vehicle) may be performed without again installing the calibration reference pattern. Thus, the automatic calibration for vehicle cameras of other vehicles may be performed by utilizing the calibration reference pattern previously placed, to offer a fast and simple vehicle camera calibration process for vehicle manufacturers.
- The view angle coordinate conversion relationship obtained from the automatic calibration for a vehicle camera in the embodiments may be applied for bird view projection conversion and distance estimation. Thus, the view angle coordinate conversion relationship may be provided to a vehicle image safety assist system for obstacle detection and distance estimation. The bird view image converted may then be outputted to a vehicle display device and presented to a user to assist in driving safety.
- It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.
Claims (18)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
TW101151114A | 2012-12-28 | ||
TW101151114 | 2012-12-28 | ||
TW101151114A TWI517670B (en) | 2012-12-28 | 2012-12-28 | Automatic calibration for vehicle camera and image conversion method and device applying the same |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140184814A1 true US20140184814A1 (en) | 2014-07-03 |
US9319667B2 US9319667B2 (en) | 2016-04-19 |
Family
ID=51016775
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/065,852 Active 2034-07-26 US9319667B2 (en) | 2012-12-28 | 2013-10-29 | Image conversion method and device using calibration reference pattern |
Country Status (3)
Country | Link |
---|---|
US (1) | US9319667B2 (en) |
CN (1) | CN103916572B (en) |
TW (1) | TWI517670B (en) |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6211157B1 (en) * | 2016-09-01 | 2017-10-11 | 三菱電機株式会社 | Calibration apparatus and calibration method |
US20180052222A1 (en) * | 2015-05-05 | 2018-02-22 | Bayerische Motoren Werke Aktiengesellschaft | Diagnostic Method for a Vision Sensor of a Vehicle and Vehicle Having a Vision Sensor |
US9918010B2 (en) | 2015-06-30 | 2018-03-13 | Industrial Technology Research Institute | Method for adjusting vehicle panorama system |
CN108876707A (en) * | 2018-05-25 | 2018-11-23 | 北京市商汤科技开发有限公司 | Birds-eye view generates and neural network training method, device, storage medium, equipment |
US10261515B2 (en) * | 2017-01-24 | 2019-04-16 | Wipro Limited | System and method for controlling navigation of a vehicle |
CN109767473A (en) * | 2018-12-30 | 2019-05-17 | 惠州华阳通用电子有限公司 | A kind of panorama parking apparatus scaling method and device |
US10466027B2 (en) | 2017-06-21 | 2019-11-05 | Fujitsu Ten Corp. Of America | System and method for marker placement |
US10482626B2 (en) * | 2018-01-08 | 2019-11-19 | Mediatek Inc. | Around view monitoring systems for vehicle and calibration methods for calibrating image capture devices of an around view monitoring system using the same |
DE102019117198A1 (en) * | 2019-06-26 | 2020-12-31 | Zf Active Safety Gmbh | Method for determining a distortion of an image to be recorded by a camera of a driver assistance system and method for compensating for an optical distortion in a driver assistance system |
US11115643B2 (en) * | 2016-09-16 | 2021-09-07 | Xion Gmbh | Alignment system |
US20220046185A1 (en) * | 2020-08-05 | 2022-02-10 | Weltrend Semiconductor Inc. | Camera Image Conversion Method Capable of Reducing Processing Time |
US11270462B2 (en) * | 2018-05-16 | 2022-03-08 | Motherson Innovations Company Limited | Calibration devices and methods |
US11420643B2 (en) * | 2017-02-27 | 2022-08-23 | Robert Bosch Gmbh | Device and method for calibrating vehicle assistance systems |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106464843B (en) * | 2014-09-05 | 2019-05-14 | 堺显示器制品株式会社 | Video generation device, image generating method and computer readable storage medium |
US10013761B2 (en) * | 2015-03-24 | 2018-07-03 | Intel Corporation | Automatic orientation estimation of camera system relative to vehicle |
EP3173975A1 (en) * | 2015-11-30 | 2017-05-31 | Delphi Technologies, Inc. | Method for identification of candidate points as possible characteristic points of a calibration pattern within an image of the calibration pattern |
EP3174007A1 (en) | 2015-11-30 | 2017-05-31 | Delphi Technologies, Inc. | Method for calibrating the orientation of a camera mounted to a vehicle |
EP3173979A1 (en) | 2015-11-30 | 2017-05-31 | Delphi Technologies, Inc. | Method for identification of characteristic points of a calibration pattern within a set of candidate points in an image of the calibration pattern |
KR101736996B1 (en) * | 2015-12-11 | 2017-05-17 | 현대자동차주식회사 | Vehicle head unit, and method of operating the same |
TWI645372B (en) * | 2017-08-28 | 2018-12-21 | 華利納企業股份有限公司 | Image calibration system and image calibration method |
CN109520495B (en) | 2017-09-18 | 2022-05-13 | 财团法人工业技术研究院 | Navigation positioning device and navigation positioning method using same |
EP3534333A1 (en) | 2018-02-28 | 2019-09-04 | Aptiv Technologies Limited | Method for calibrating the position and orientation of a camera relative to a calibration pattern |
EP3534334B1 (en) | 2018-02-28 | 2022-04-13 | Aptiv Technologies Limited | Method for identification of characteristic points of a calibration pattern within a set of candidate points derived from an image of the calibration pattern |
JP6696102B1 (en) * | 2019-05-20 | 2020-05-20 | 東芝エレベータ株式会社 | Image processing device and marker |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080231710A1 (en) * | 2007-01-31 | 2008-09-25 | Sanyo Electric Co., Ltd. | Method and apparatus for camera calibration, and vehicle |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3632563B2 (en) * | 1999-10-19 | 2005-03-23 | 株式会社豊田自動織機 | Image positional relationship correction device, steering assist device including the image positional relationship correction device, and image positional relationship correction method |
JP2004200819A (en) | 2002-12-17 | 2004-07-15 | Clarion Co Ltd | Method of calibrating mounted camera and calibrator |
TWI243235B (en) * | 2004-03-10 | 2005-11-11 | Contrel Technology Co Ltd | Method for positioning sheets by means of dual positioning marks in cooperation with a camera device |
CN100430693C (en) * | 2004-03-17 | 2008-11-05 | 东捷科技股份有限公司 | Sheet material positioning method through double location mark in combination with image taking by photographic arrangement |
JP4744823B2 (en) * | 2004-08-05 | 2011-08-10 | 株式会社東芝 | Perimeter monitoring apparatus and overhead image display method |
JP4681856B2 (en) * | 2004-11-24 | 2011-05-11 | アイシン精機株式会社 | Camera calibration method and camera calibration apparatus |
JP4820221B2 (en) | 2006-06-29 | 2011-11-24 | 日立オートモティブシステムズ株式会社 | Car camera calibration device and program |
JP2008187564A (en) | 2007-01-31 | 2008-08-14 | Sanyo Electric Co Ltd | Camera calibration apparatus and method, and vehicle |
JP5124147B2 (en) | 2007-02-01 | 2013-01-23 | 三洋電機株式会社 | Camera calibration apparatus and method, and vehicle |
CN101242546A (en) * | 2007-02-06 | 2008-08-13 | 鸿富锦精密工业(深圳)有限公司 | Image correction system and method |
TWI397668B (en) * | 2007-02-12 | 2013-06-01 | Hon Hai Prec Ind Co Ltd | System and method for correcting an image |
TW200927537A (en) | 2007-12-28 | 2009-07-01 | Altek Corp | Automobile backup radar system that displays bird's-eye view image of automobile |
JP4874280B2 (en) * | 2008-03-19 | 2012-02-15 | 三洋電機株式会社 | Image processing apparatus and method, driving support system, and vehicle |
JP5339124B2 (en) | 2008-09-30 | 2013-11-13 | アイシン精機株式会社 | Car camera calibration system |
TWM387029U (en) | 2010-04-08 | 2010-08-21 | H P B Optoelectronics Co Ltd | Driving recorder with a photography angle calibration window feature |
TWI411969B (en) | 2010-12-24 | 2013-10-11 | Ind Tech Res Inst | Method and system for matching texture feature points in images |
-
2012
- 2012-12-28 TW TW101151114A patent/TWI517670B/en active
-
2013
- 2013-05-08 CN CN201310165649.4A patent/CN103916572B/en active Active
- 2013-10-29 US US14/065,852 patent/US9319667B2/en active Active
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080231710A1 (en) * | 2007-01-31 | 2008-09-25 | Sanyo Electric Co., Ltd. | Method and apparatus for camera calibration, and vehicle |
Cited By (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180052222A1 (en) * | 2015-05-05 | 2018-02-22 | Bayerische Motoren Werke Aktiengesellschaft | Diagnostic Method for a Vision Sensor of a Vehicle and Vehicle Having a Vision Sensor |
US10578710B2 (en) * | 2015-05-05 | 2020-03-03 | Bayerische Motoren Werke Aktiengesellschaft | Diagnostic method for a vision sensor of a vehicle and vehicle having a vision sensor |
US9918010B2 (en) | 2015-06-30 | 2018-03-13 | Industrial Technology Research Institute | Method for adjusting vehicle panorama system |
JP2018036189A (en) * | 2016-09-01 | 2018-03-08 | 三菱電機株式会社 | Calibration device and calibration method |
JP6211157B1 (en) * | 2016-09-01 | 2017-10-11 | 三菱電機株式会社 | Calibration apparatus and calibration method |
DE102017207696B4 (en) | 2016-09-01 | 2024-01-11 | Mitsubishi Electric Corporation | Calibration device and calibration method |
US11115643B2 (en) * | 2016-09-16 | 2021-09-07 | Xion Gmbh | Alignment system |
US10261515B2 (en) * | 2017-01-24 | 2019-04-16 | Wipro Limited | System and method for controlling navigation of a vehicle |
US11420643B2 (en) * | 2017-02-27 | 2022-08-23 | Robert Bosch Gmbh | Device and method for calibrating vehicle assistance systems |
US10466027B2 (en) | 2017-06-21 | 2019-11-05 | Fujitsu Ten Corp. Of America | System and method for marker placement |
US10482626B2 (en) * | 2018-01-08 | 2019-11-19 | Mediatek Inc. | Around view monitoring systems for vehicle and calibration methods for calibrating image capture devices of an around view monitoring system using the same |
US11270462B2 (en) * | 2018-05-16 | 2022-03-08 | Motherson Innovations Company Limited | Calibration devices and methods |
CN108876707A (en) * | 2018-05-25 | 2018-11-23 | 北京市商汤科技开发有限公司 | Birds-eye view generates and neural network training method, device, storage medium, equipment |
CN109767473A (en) * | 2018-12-30 | 2019-05-17 | 惠州华阳通用电子有限公司 | A kind of panorama parking apparatus scaling method and device |
DE102019117198A1 (en) * | 2019-06-26 | 2020-12-31 | Zf Active Safety Gmbh | Method for determining a distortion of an image to be recorded by a camera of a driver assistance system and method for compensating for an optical distortion in a driver assistance system |
US20220046185A1 (en) * | 2020-08-05 | 2022-02-10 | Weltrend Semiconductor Inc. | Camera Image Conversion Method Capable of Reducing Processing Time |
US11483493B2 (en) * | 2020-08-05 | 2022-10-25 | Weltrend Semiconductor Inc. | Camera image conversion method capable of reducing processing time |
Also Published As
Publication number | Publication date |
---|---|
TW201427389A (en) | 2014-07-01 |
CN103916572B (en) | 2017-07-07 |
CN103916572A (en) | 2014-07-09 |
US9319667B2 (en) | 2016-04-19 |
TWI517670B (en) | 2016-01-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9319667B2 (en) | Image conversion method and device using calibration reference pattern | |
CN110264520B (en) | Vehicle-mounted sensor and vehicle pose relation calibration method, device, equipment and medium | |
CN108475437B (en) | 360 degree look-around system for vehicle with corner-mounted camera, calibration system and method | |
US10192309B2 (en) | Camera calibration device | |
EP1954063A2 (en) | Apparatus and method for camera calibration, and vehicle | |
US6618076B1 (en) | Method and apparatus for calibrating projector-camera system | |
US20080309763A1 (en) | Driving Support System And Vehicle | |
JP4695167B2 (en) | Method and apparatus for correcting distortion and enhancing an image in a vehicle rear view system | |
KR100869570B1 (en) | Camera calibrating method and camera calibrating device | |
US20120314073A1 (en) | Apparatus and Method for Detecting Posture of Camera Mounted on Vehicle | |
CN111508027B (en) | Method and device for calibrating external parameters of camera | |
US20130002861A1 (en) | Camera distance measurement device | |
KR101583663B1 (en) | Method for generating calibration indicator of camera for vehicle | |
US11880993B2 (en) | Image processing device, driving assistance system, image processing method, and program | |
WO2022257569A1 (en) | Calibration of image acquisition apparatus | |
EP2471689B1 (en) | Driving support display device | |
CN112330755B (en) | Calibration evaluation method and device of all-round system, storage medium and terminal | |
KR101926258B1 (en) | Method of automatic calibration of AVM system | |
KR101816068B1 (en) | Detection System for Vehicle Surroundings and Detection Method for Vehicle Surroundings Using thereof | |
CN110884428A (en) | Vehicle-mounted panoramic driving auxiliary device and method based on catadioptric panoramic camera | |
CN116671100A (en) | Method and system for calibrating a camera device of a vehicle vision system using a programmable calibration plate device | |
EP3709265A1 (en) | Road surface area detection device | |
TWI645372B (en) | Image calibration system and image calibration method | |
CN113223089B (en) | Calibration device, calibration field and calibration method | |
CN116409243A (en) | Truck panoramic pedestrian positioning and early warning method based on 360-degree fisheye camera |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INDUSTRIAL TECHNOLOGY RESEARCH INSTITUTE, TAIWAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIN, YU-CHEN;LIU, WEI-CHENG;LEE, SHAO-YUAN;REEL/FRAME:031501/0650 Effective date: 20131014 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |