JP2008177742A - Camera parameter acquisition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide "a camera parameter acquisition device" capable of improving precision of camera parameters to be acquired. <P>SOLUTION: First data indicative of design values / reference values of internal parameters of a camera to be corrected and second data indicative of design values / reference values of a relative posture for a correction pattern of the camera are prepared, and a control means generates and displays the correction pattern on a display device, corrects the position of an expected correction control point, generated to have a nearly expected distribution on an image, using the first and second data so that the difference between the expected correction control point and a correction control point extracted from a result of imaging of the correction pattern on the display device by the camera is within a permissible range, and regenerates and displays a correction pattern corresponding to the result of the correction on the display device. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for acquiring camera parameters used for distortion correction, projective transformation, and the like of an image captured by a camera using a lens having a wide angle of view, and in particular, generates a top view image used for parking assistance. In particular, the present invention relates to a camera parameter acquisition device useful for correcting lens aberration of a captured image of a (digital) camera that is the basis thereof.

  Camera parameters are parameters that indicate camera characteristics such as camera lens focal length, lens distortion characteristics, optical axis coordinates of the lens on the image sensor, and are related to internal camera factors. Also called “internal parameters”. “TOPVIEW” having the same name as the top view is a registered trademark of Alpine Corporation.

  Digital cameras are used in various fields because they can reproduce a captured image immediately and can be easily used as an image input device such as a personal computer (PC). For example, when used for in-vehicle use, if the vehicle surroundings are photographed and the captured image is displayed on the screen of a display device such as an LCD monitor, the convenience of the driver (parking) Support). In this case, image data acquired by an in-vehicle camera arranged around the vehicle is transferred to a computer such as an in-vehicle ECU (electronic control unit), and image processing such as distortion correction and projective transformation is appropriately performed in the computer. Then, the image is supplied to the display device and the vehicle surrounding image is displayed on the screen. In other words, the computer acquires images obtained by photographing each direction around the vehicle (front, rear, left, right) with each in-vehicle camera, and converts the viewpoint of each acquired image into a viewpoint from above the vehicle. An image (overhead image) is generated and combined, and further, a top view image is generated by combining images of the host vehicle and supplied to the display device.

  As described above, when generating the top view image, it is necessary to perform image processing such as high-precision distortion correction and projective transformation on the image around the vehicle acquired by the in-vehicle camera. It is necessary to accurately acquire the internal parameters (camera parameters).

  In the conventional method of acquiring camera parameters, for example, a checker pattern (checkered pattern) drawn in a predetermined arrangement on a plane structure such as a solid surface such as a plane such as a sheet or a combination of three planes in a direction orthogonal to each other. ) Or the like (hereinafter also referred to as “calibration pattern”) is captured by a camera, and the captured image is processed to extract a reference feature point (hereinafter also referred to as “calibration control point”). An internal parameter (camera parameter) of the camera is calculated from the extracted calibration control point.

  In this case, in order to acquire camera parameters with high accuracy, it is necessary to accurately detect as many calibration control points as possible from the captured calibration pattern. For example, in the example of the image captured by the fisheye lens shown in FIG. 1, a checker pattern is imaged as a calibration pattern. In this case, the center point or corner point of each rectangular pattern (black and white graphic pattern) is set to the pattern. It is necessary to accurately detect the feature point.

As a technique related to such a conventional technique, for example, as described in Patent Document 1, an internal parameter necessary for correcting lens aberration of a digital camera or the like in a calibration device can be easily measured. There is. The calibration apparatus includes an extraction unit that extracts a calibration chart having a first mark and a second mark from a calibration image photographed in at least two directions with a camera that performs calibration, An approximate mark position measuring unit for obtaining an approximate position of the second mark by projective transformation using the extracted first mark, and a precise mark position for obtaining an imaged position of the second mark position in the vicinity of the approximate position of the second mark A measurement unit; a position of a second mark in the calibration chart; and an arithmetic processing unit that calculates a lens calibration element from the position of the second mark in the calibration image corresponding to the second mark. Yes.
JP 2003-307466 A

  As described above, in the conventional technique, a calibration pattern (such as a checker pattern illustrated in FIG. 1) drawn in a predetermined arrangement on a surface structure such as a sheet is imaged with a camera, and the pattern is extracted from the captured image. A feature point (calibration control point) is extracted to calculate an internal parameter (camera parameter) of the camera. In this case, there is no problem if an image of the calibration pattern captured by the camera can be acquired without distortion, but a fisheye lens or a wide-angle lens with a wide angle of view is used in the camera or the like used when generating the top view image as described above. Therefore, there were the following problems.

  That is, when a calibration pattern as described above is imaged with a camera using a fisheye lens or the like, image distortion such as reduction or deformation increases on the captured image, and a feature point (calibration control) to be detected according to the amount of distortion. There was an inconvenience that the error of the coordinates of point) was enlarged. For example, in the case of a picked-up image of a checker pattern as shown in FIG. 1, the individual rectangular pattern is deformed relatively small in the peripheral portion of the lens, and the number of pixels (of the image sensor) assigned to the image of the portion is also Since the number is relatively small, it is difficult to accurately acquire the coordinates of the calibration control point in the portion.

  In other words, many calibration control points that should be detected from the captured calibration pattern image cannot be detected accurately, and as a result, there is a problem in that the acquisition accuracy of the camera parameters decreases.

  The present invention has been created in view of the problems in the related art, and an object of the present invention is to provide a camera parameter acquisition apparatus that can improve the accuracy of camera parameters to be acquired.

  In order to solve the above-described problems of the related art, in the present invention, the design value or reference value (one or more values set in advance) of the internal parameter (camera parameter) of the camera to be calibrated and the imaging target of the camera Assuming that the design value or reference value (one or more values set in advance) representing the “relative posture” with respect to the calibration pattern is known information, the calibration control point on the captured image can be determined using this information. A calibration pattern that is distributed almost as expected is obtained. The calibration control points that are distributed on the image as expected (for example, uniformly distributed) are also referred to as “expected calibration control points” in the following description.

  The “relative posture” of the camera represents the relationship between the relative position and inclination of the camera and the reference coordinates of the calibration pattern (for example, the coordinates of the center of the calibration pattern) as illustrated in FIG. It is expressed by coordinate information of six axes of translation (Tx, Ty, Tz) and rotation (Rx, Ry, Rz). Such information representing the relative posture is also referred to as “camera external parameter” because it relates to a factor outside the camera.

  The camera parameter acquisition device according to the present invention includes a camera to be calibrated, at least first data indicating a design value or a reference value of an internal parameter of the camera, and a relative posture with respect to a calibration pattern to be imaged by the camera. Storage means storing second data designating design values or reference values, a display device capable of variably displaying the calibration pattern, a calibration pattern generated and displayed on the display device, and the first And the second data, the position of the expected calibration control points generated so as to be distributed almost as expected on the image, and the result of imaging the calibration pattern displayed on the display device with the camera The position of the expected calibration control point is corrected so that the difference from the position of the extracted calibration control point is within the allowable range, and the calibration pattern according to the result of the correction Regenerate characterized by comprising a control means for displaying on said display device.

  According to the camera parameter acquisition device of the present invention, the calibration pattern variably generated by the control means is displayed on the display device, and the position of the calibration control point extracted from the result of imaging the displayed calibration pattern with the camera, The position of the expected calibration control point is corrected so that the difference from the position of the generated expected calibration control point is within the allowable range, and the calibration pattern corresponding to the correction result is regenerated and displayed on the display device. As a result, the accuracy of the internal parameters (camera parameters) of the camera finally obtained can be improved.

  According to another aspect of the present invention, the camera to be calibrated, the first data indicating the design value or the reference value of the internal parameter of the camera, and the relative posture with respect to the calibration pattern to be imaged by the camera. Using the storage means storing the second data designating the design value or the reference value, the surface structure on which the calibration pattern is drawn, the first data and the second data, the image is almost Generate expected calibration control points that are distributed as expected, backproject the expected calibration control points on the generated image onto the calibration pattern imaged by the camera, and position the control points on the calibration pattern There is provided a camera parameter acquisition device characterized by comprising control means for calculating.

  Other structural features of the camera parameter acquisition apparatus according to the present invention and specific processing modes based on the characteristics will be described in detail with reference to embodiments of the invention described later.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the accompanying drawings.

  FIG. 3 schematically shows a part of the configuration of a camera parameter acquisition apparatus according to an embodiment of the present invention. In the present embodiment, it is assumed that the camera parameter acquisition device according to the present invention is realized by a personal computer (PC).

  The camera parameter acquisition device 20 according to the present embodiment is adapted to variably display a camera 1 as a camera parameter acquisition target (calibration target) and a calibration pattern as an imaging target of the camera 1 as illustrated. A display device 2, a hard disk drive (HDD) 3 as a storage medium, an operation unit 4 as a user interface, a speaker 5, and a control unit 10 constituted by a microcomputer or the like are provided. The control unit 10 and the speaker 5 are built in the PC main body, and the camera 1, the display device 2, the HDD 3, the operation unit 4, and the speaker 5 are operatively connected to each functional block in the control unit 10.

  The display device 2 includes three flat display units (for example, an LCD monitor) as shown in the figure, and each display unit constitutes a three-dimensional screen in the XYZ coordinate system. On the screens 2a, 2b, and 2c of each display unit, a calibration pattern dynamically (variably) generated by the control unit 10 is displayed as will be described later. Further, the display device 2 is used as a function related to the present invention to display a message for instructing “calibration failure” to be described later.

  The camera 1 is attached to the top of a structure (not shown) such as a stay (post) having a predetermined height, and the orientation of the camera lens (fisheye lens in the present embodiment) depends on the display device 2. The calibration patterns displayed on the display screens 2a, 2b, and 2c are adjusted so that they can be obtained equally (within approximately 1/3 of the imaging range).

  The HDD 3 stores in advance the design value or reference value data of the internal parameter (camera parameter) of the camera 1 to be calibrated and the design value or reference value of the relative orientation (external parameter) with respect to the calibration pattern to be imaged by the camera 1. Are stored. Further, the HDD 3 stores “expected calibration control point pattern” data, which will be described later, and determines the positional relationship (degree of proximity) between the calibration control point calculated from the calibration pattern and the expected calibration control point. Information indicating the “allowable range” used as the reference value (reference value) and information indicating the “calibration count” for performing camera calibration processing (feedback processing described later) are stored. The number of times of calibration is reset to “0” in an initial state when processing related to camera parameter acquisition described later is performed. Further, the “allowable range” is set, for example, within 20% of the distance between the calibration control points when the calibration control points are uniformly arranged. Note that the control points on the calibration pattern are expressed as, for example, the coordinates of the intersections of circle centers and grid lines.

  The operation unit 4 is a unit that includes various operation keys for inputting user instructions. In this embodiment, the operation unit 4 is a keyboard or a mouse connected to the PC body. As a function related to the present invention, the operation unit 4 gives an activation signal for causing the control unit 10 to execute camera calibration processing related to camera parameter acquisition described later. In the present embodiment, the speaker 5 can be used when outputting a voice message indicating “Calibration failure”, which will be described later. However, since the message is also displayed on the display device 2, the speaker 5 is not necessarily used. There is no need.

  The control unit 10 includes, as functional blocks related to the present invention, a CPU 11 as a main control unit that comprehensively controls camera calibration processing related to camera parameter acquisition, which will be described later, a camera image acquisition unit 12, and a calibration control point detection unit. 13, a calibration control point comparison unit 14, a back projection calibration pattern generation unit 15, a display control unit 16, a camera parameter calculation unit 17, and a message output unit 18. Each of the functional blocks 12 to 18 executes a predetermined process in cooperation with the CPU 11 as will be described later. Specific processing contents will be described in detail later.

  In the camera parameter acquisition apparatus 20 of the present embodiment configured as described above, the HDD 3 corresponds to “storage means”, and the control units 10 (11 to 18) correspond to “control means”.

  Hereinafter, camera calibration processing related to camera parameter acquisition performed in the camera parameter acquisition device 20 of the present embodiment will be described with reference to FIG. 4 showing an example thereof. In addition, a supplementary explanation will be given with reference to FIG.

  In the first step S1, in response to an activation signal input by the CPU 11 based on a user instruction via the operation unit 4, the design value / reference value of the camera parameter of the camera 1 to be subjected to this processing is retrieved from the HDD 3 in the HDD 3. Each data (information) of the design value / reference value of the value and the relative posture is acquired. The acquired information is temporarily stored in a memory (not shown) in the CPU 11.

  In the next step S2, when an arbitrary calibration pattern is first imaged by the back projection calibration pattern generation unit 15 and the display control unit 16 based on the control from the CPU 11, the calibration control point of the calibration pattern is set on the captured image. A virtual image (expected calibration control point pattern) that is uniformly distributed is generated, and then the camera parameters (design value / reference value) of the camera 1 once stored in a memory (not shown) in the CPU 11 are stored. Using the information and relative posture (design value / reference value) information, the expected calibration control point on the generated image is back projected onto the calibration pattern and the coordinates of the control point (that is, the expected calibration control point) The position on the calibration pattern is calculated, and the calculation result is displayed on each display screen 2a, 2b, 2c of the display device 2.

  FIG. 5A shows an example of the expected calibration control point pattern (virtual image) generated in this step. In the figure, each point indicated by EP represents an “expected calibration control point”. ing. FIG. 5B shows a display example of the backprojection calibration pattern generated in this step. The generated “expected calibration control point pattern” data is temporarily stored in the HDD 3 via the CPU 11.

  In the next step S3, the calibration patterns displayed on the display screens 2a, 2b, and 2c of the display device 2 are imaged by the camera 1 via the camera image acquisition unit 12 based on the control from the CPU 11.

  In the next step S4, the calibration control point detection unit 13 detects the calibration control point on the calibration pattern from the captured image acquired from the camera 1 via the camera image acquisition unit 12 based on the control from the CPU 11, and coordinates thereof. (Calibration control point position) is calculated.

  In the next step S5, the calibration control point comparison unit 14 compares the position of the calibration control point detected (calculated) by the calibration control point detection unit 13 with the position of the expected calibration control point based on the control from the CPU 11, It is determined whether the difference is within the allowable range (YES) or not (NO). If the determination result is yes, the process proceeds to step S6, and if the determination result is no, the process proceeds to step S7. For example, as shown in FIG. 5C as an example, the calculated position of the calibration control point CP is compared with the position of the expected calibration control point EP, and the difference between them (the error indicated by R1 in the figure) is within an allowable range. To determine if it is within. In the illustrated example, the case where the error R1 is out of the allowable range and “calibration processing” is required is shown.

  In the next step S6 (when the difference between the calculated calibration control point and the expected calibration control point is within the allowable range), the camera parameter calculation unit 17 calculates the calibration control point detection unit 13 based on the control from the CPU 11. The internal parameter (camera parameter) of the camera 1 is calculated using the information of the calibration control point. Then, this processing (camera calibration processing) is “finished”.

  On the other hand, in step S7 (when the difference between the calculated calibration control point and the expected calibration control point is out of the allowable range), the number of calibrations set in the HDD 3 (initial value “0”) based on the control from the CPU 11. )) Is counted up (+1).

  In the next step S8, the CPU 11 determines whether the number of calibrations set in the HDD 3 has reached a predetermined value (YES) or not (NO). If the determination result is YES, the process proceeds to step S9, and if the determination result is NO, the process proceeds to step S10. In this embodiment, this “predetermined value” is set to “3 times”, for example.

  In step S9, based on the control from the CPU 11, the display controller 16 instructs the screen of the display device 2 (at least one of the display screens 2a, 2b, 2c) to fail calibration based on the control from the CPU 11. A message to that effect is displayed. That is, camera calibration based on feedback processing described later (correction of backprojection pattern to bring the expected calibration control point closer to the position of each calibration control point on the calibration pattern) was performed a predetermined number of times (three times). Information (message) reporting that the control point could not be converged within the allowable range is output. As described above, the message may be output from the speaker 5 through the message output unit 18 as described above. In this case, audio output is performed together with display via the screen or instead of the screen display.

  Further, after outputting a calibration failure message, the count value of the number of times of calibration (in this case, “3 times”) set in the HDD 3 based on the control from the CPU 11 is reset to “0”. Then, this processing (camera calibration processing) is “finished”.

  On the other hand, in step S10 (when the difference between the calibration control point and the expected calibration control point has not yet converged within the allowable range and the number of calibrations is less than 3), the CPU 11 coordinates the calibration control point ( Using the (position) information, an internal parameter (camera parameter) and a relative posture of the camera 1 are calculated, and the position of the expected calibration control point is recalculated based on the calculation result. And it returns to step S2 and repeats said process.

  In this embodiment, starting from the process of step S2 (back projection of the expected calibration control point onto the calibration pattern), the process of step S3 (imaging the displayed calibration pattern), the process of step S4 (from the captured image to the calibration pattern). If the difference between the calibration control point and the expected calibration control point is out of the allowable range after the above calculation of the calibration control point position), the process of step S10 (recalculation of the position of the expected calibration control point) A series of processing until the process returns to step S2 is called “feedback processing (camera calibration based on)”.

  FIG. 5D shows the difference between the position of the expected calibration control point EP1 and the position of the calibration control point CP after performing camera calibration based on such feedback processing several times (in this case, less than 3 times) (FIG. 5D). The error indicated by R2 is improved over the state before the calibration (error R1 in FIG. 5C) (that is, improved in a direction to converge the expected calibration control point within the allowable range). Schematically).

  As described above, according to the camera parameter acquisition device 20 according to the present embodiment, the calibration pattern (back projection calibration pattern) dynamically (variably) generated by the control unit 10 is displayed on the stereoscopic screen 2a of the display device 2. , 2b, 2c, and the above-described feedback processing is performed from the result of imaging the displayed calibration pattern with the camera 1, so that the camera parameter calculation unit 17 finally acquires the camera 1 The accuracy of internal parameters (camera parameters) can be increased. In other words, the back projection calibration pattern to be displayed on the display device 2 is corrected by performing feedback processing so that the parameter acquisition error based on the imaging result of the calibration pattern is minimized (see FIG. 5).

  Thus, the control unit 10 dynamically generates a calibration pattern (back projection calibration pattern), displays the generated pattern on the display device 2, and displays the result of imaging the displayed pattern with the camera 1. By feeding back to the second side and regenerating the pattern, it is possible to improve the accuracy of the acquired camera parameters.

  Further, in the present embodiment, since the calibration pattern can be dynamically generated, it is possible to suitably cope with a camera having various parameter characteristics.

  In the above-described embodiment, the case where the calibration pattern is variably displayed using the display device 2 has been described as an example. However, the gist of the present invention (known values “camera parameter design value / reference value” and “camera” As will be clear from the fact that the calibration control points are distributed almost as expected on the captured image using the design value / reference value of the relative posture (external parameter) of It is not always necessary to display the calibration pattern that is the imaging target of the camera on the screen of the display device. In short, it is sufficient that the calibration pattern is arranged at a predetermined position (within the range) that can be imaged by the camera to be calibrated. For example, a plane structure such as a plane such as a sheet or a three-dimensional plane combining these planes. It is sufficient that a pattern having a predetermined shape (such as a checker pattern illustrated in FIG. 1) is drawn in a predetermined arrangement on the body by printing or the like.

  Basically, the embodiment in this case, that is, the embodiment in which a camera pattern is acquired by capturing a calibration pattern drawn on a surface structure such as a sheet without using a display device, and a camera parameter is obtained. Since this is the same as in the case of the above-described embodiment, detailed description thereof is omitted. The processing mode of the part related to the present invention will be described as follows. That is, in this embodiment, the control unit 10 refers to the design value / reference value data of the camera parameter and the design value / reference value data of the relative attitude of the camera (external parameter), and is almost expected on the image. Expected calibration control points that are distributed in the same manner (for example, uniformly distributed), and the expected calibration control points are back-projected onto the calibration pattern imaged by the camera 1 and the control points on the calibration pattern. The position is calculated.

  In the above-described embodiment, the case where the calibration pattern is displayed on the display device 2 including the three display units (three-dimensional screens 2a, 2b, and 2c in the XYZ coordinate system) has been described as an example. Obviously, the display target of the calibration pattern is not limited to the three-dimensional screen. Although not particularly illustrated, the calibration pattern can be dynamically displayed on one flat display unit (planar screen of the XY coordinate system).

  In addition to the above-described three-dimensional screen or three-dimensional surface, a polyhedron, a spherical surface, or the like can be appropriately used as a three-dimensional structure for displaying the calibration pattern on the screen (or drawing on the surface structure). Further, in the example shown in FIG. 5A, the case where the calibration control points (expected calibration control points EP) are uniformly distributed on the image has been described as an example. However, as is clear from the gist of the present invention, the calibration is performed. The control points do not necessarily have a uniform distribution mode, and can be appropriately set to a non-uniform distribution mode according to the purpose of use of the camera to be calibrated.

  In the above-described embodiment, a hard disk drive (HDD) is used as a storage medium for storing camera parameter design value / reference value data, camera relative orientation (external parameter) design value / reference value data, and the like. However, a DRAM, a flash memory, or the like can be used as appropriate as another storage medium.

It is a figure for demonstrating the conventional problem which concerns on camera parameter acquisition. It is explanatory drawing of the relative attitude | position (external parameter) of a camera. It is a block diagram which shows a part of structure of the camera parameter acquisition apparatus which concerns on one Embodiment of this invention typically. It is a flowchart which shows an example of the camera calibration process which concerns on the camera parameter acquisition performed in the apparatus of FIG. FIG. 5 is a diagram for supplementarily explaining the processing flow of FIG. 4.

Explanation of symbols

1 ... Camera (to be calibrated)
2 ... display device,
2a, 2b, 2c ... the screen of each display unit,
3. HDD (storage means),
4 ... operation part,
5 ... Speaker,
10 (11-18) ... control unit (control means),
20 ... Camera parameter acquisition device,
CP: Detected (calculated) calibration control point,
EP, EP1 ... Expected calibration control point.

Claims (6)

A camera to be calibrated,
At least first data indicating a design value or reference value of an internal parameter of the camera and second data indicating a design value or reference value of a relative posture with respect to a calibration pattern to be imaged by the camera are stored. Storage means;
A display device capable of variably displaying the calibration pattern;
A calibration pattern is generated and displayed on the display device, and using the first data and the second data, the position of the expected calibration control point generated so as to be distributed almost as expected on the image, The position of the expected calibration control point is corrected so that the difference from the position of the calibration control point extracted from the result of imaging the calibration pattern displayed on the display device with the camera is within an allowable range, and the result of the correction And a control unit that regenerates a calibration pattern according to the display and displays the calibration pattern on the display device. The control means includes
Means for generating a back-projection calibration pattern on the calibration pattern of the generated expected calibration control point using the first data and the second data;
Means for performing display control on the display device of the generated backprojection calibration pattern;
Means for obtaining an image of a calibration pattern imaged by the camera;
Means for detecting position information of the calibration control point from the acquired calibration pattern image;
Means for comparing the position of the detected calibration control point with the position of the expected calibration control point;
Means for calculating camera parameters of the camera using position information of the calibration control point when the difference between the calibration control point and the expected calibration control point is within the allowable range based on the comparison result; The camera parameter acquisition apparatus according to claim 1, wherein:   3. The camera parameter acquisition apparatus according to claim 2, wherein the control unit further includes a unit that notifies the fact that the expected calibration control point cannot be converged within the allowable range. .   The camera parameter acquisition apparatus according to claim 1, wherein the display device includes three flat display units combined in a direction orthogonal to each other, or one flat display unit. A camera to be calibrated,
Storage means storing first data designating a design value or reference value of an internal parameter of the camera and second data designating a design value or reference value of a relative posture with respect to a calibration pattern to be imaged by the camera When,
A surface structure on which the calibration pattern is drawn;
Using the first data and the second data, expected calibration control points that are distributed as expected on the image are generated, and the expected calibration control points on the generated image are captured by the camera. A camera parameter acquisition apparatus comprising: control means for back projecting onto the calibration pattern and calculating a position of the control point on the calibration pattern.   6. The planar structure is a planar structure in which a pattern of a predetermined shape is drawn in a predetermined arrangement, or a three-dimensional planar structure in which three planar structures are combined in directions orthogonal to each other. The camera parameter acquisition device described in 1.

Images