WO2016146105A1

WO2016146105A1 - Method and device for calibrating a camera

Info

Publication number: WO2016146105A1
Application number: PCT/DE2016/100112
Authority: WO
Inventors: Harald Hoppe
Original assignee: Hochschule Offenburg
Priority date: 2015-03-16
Filing date: 2016-03-11
Publication date: 2016-09-22
Also published as: DE102015103785A1

Abstract

The invention relates to a method for calibrating a camera (110) using a screen (120), said screen (120) having a number of image points (122) and said camera (110) using a plurality of pixels (112) to depict the image. The method comprises the following steps: (a) depicting at least one image value (BW) in at least one image point (122) of the screen (120), on the basis of an image value assignment; (b) capturing the at least one image value (BW) using a pixel (112a) of the camera (110); and (c) determining the position of the at least one image point (122) on said screen (120), on the basis of the at least one captured image value (BW) and the image value assignment. The method also comprises the step of displacing the screen (120) or the camera (110) in one direction by a certain amount such that the at least one image point (122) is at a different distance from the pixel (112a) of the camera (110) than it was prior to the displacement, and repeating at least steps (b) and (c) for the displaced screen (120i). The camera (110) has a variable focus during displacement of the screen (120) relative to the camera (110), and the method also comprises the step of storing an assignment relating to the pixel (112) and the positions of the at least one image point (122) for different displaced screen positions (120i).

Description

Method and device for calibrating a camera

The present invention relates to a method and apparatus for calibrating a camera, and more particularly to non-model based calibration of cameras using a monitor.

background

Digital cameras typically use a CCD sensor or a CMOS sensor to capture an image through a plurality of pixels. For many applications, it is not sufficient to capture only a digital image. Rather, information is desirable, such as from which directions the individual pixels of the camera receive light. To achieve this, a calibration of the camera is used, which, for example, associates an optical path with a pixel. This makes it possible to determine the position of a detected pixel for each pixel in a certain image area (for example perpendicular to the viewing direction). Thus, the calibration determines those points which a particular camera pixel is viewing or which point of the scene under consideration onto which camera pixels are imaged. The use of cameras for the purpose of industrial or non-industrial (eg medical) image processing usually depends largely on how accurately the cameras used can be calibrated.

The current state of the art for calibrating cameras is based on the basic assumption that the camera to be calibrated can be described sufficiently accurately by a hole camera model. In this model, there is a unique projection center, through which all the lines that are assigned to the individual camera pixels run. In addition, these models usually require that the real scene is largely undistorted on the image plane of the camera. At best, another model is used to describe radial and tangential lens distortion effects.

The overall model for describing the imaging properties of a camera is based on six extrinsic parameters (three rotational and three translatory degrees of freedom) used to describe the transformation from the world coordinate system to the camera coordinate system and five intrinsic parameters used to describe the perspective projection onto a projection center. The radial lens distortion is usually described with two to three parameters, the tangential with two parameters. The total model thus includes 16 parameters to be calibrated, of which, however, only 12 are usually used. To determine these parameters, for example, patterns representing points with known 3-dimensional (3-D) world coordinates are held in front of the camera and the associated 2-dimensional (2-D) image coordinates representing these points are determined. By means of an optimization, the unknown parameters of the model are then calculated from these point correspondences (3D <=> 2D). Frequently, three to ten images of a flat calibration pattern with approximately 50 to 250 uniquely definable feature points are used.

Such models are described, for example, in R.Y. Tsai: "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses" and Z. Zhang: "A Flexible New Technique for Camera Calibration" To calculate models, few (usually less than 1000) point correspondences between known world points and their pixel position are used in the camera image.

FIG. 8 shows a calibration pattern which is used to determine the point correspondences and represents a plurality of black dots on a white surface. The calibration pattern is captured by the camera, with the camera pixels detecting either a white screen background or a black dot imaged thereon. In order to be able to further determine which of the black dots was detected, a first calibration point 501, a second calibration point 502 and a third calibration point 503 are shown in a central area. The calibration points 501 to 503 define a local coordinate system with respect to which the coordinates of the centers of all circles are determined, for example by simply counting the black points to the calibration points. After the point correspondences have been determined, the corresponding models are taken to determine the parameters.

A similar calibration method is described in DE 10 2010 031 215, where a three-dimensional object is used for calibration, based on a model for an intensity distribution of the used CCD chip and the calibration determines the model coefficients. Another conventional method is described in DE 197 27 281 C1, where a three-dimensional test structure is generated by means of a hologram and is used for calibration, assuming idealized camera optics.

Although the aforementioned conventional calibration methods describe the imaging properties of a camera by relatively few parameters (for example ten to sixteen), they have the following disadvantages: (a) The parameters used to model the radial and tangential lens distortion are usually not sufficient to describe true imaging properties of the lens system. In particular, it often happens in the peripheral areas of an image that deviations from the idealized-assumed lens system occur.

(b) Highly distorting lenses or lens systems (such as fisheye lenses) or cameras with optical filters placed in front of them (such as reflecting or refracting surfaces) can not be adequately described with the usual parameters or with other models. A model description with parameters is also cumbersome because a separate model has to be developed for each complete system of camera, lenses and other optical elements.

(c) The number of dot correspondences used to optimize the parameters of the underlying model is usually in a range of 100 to 1000. Compared to the number of pixels of the camera to be calibrated, ranging from 1 to 5 million or even more pixels can be, this is often too little.

(d) Taking pictures from multiple, randomly selected positions of a calibration pattern does not guarantee even distribution of dot correspondences over the entire image of the camera. In particular, there are usually few point correspondences in the edge regions of the image.

It is an object of the present invention to provide a method and apparatus for calibrating a camera that overcomes the aforementioned disadvantages and achieves greater accuracy and flexibility of camera calibration. New and unconventional solutions should be made possible.

Summary

The above object is achieved by the method of calibration according to claim 1 or an apparatus according to claim 15. The dependent claims give advantageous embodiments of the subject matter of claim 1 again.

The present invention relates to a method of calibrating a camera using a screen, wherein the screen has a set of pixels and the camera uses a plurality of pixels to display the image. The method comprises the following steps: (a) displaying at least one image value in at least one pixel of the screen based on an image value assignment; (b) detecting the at least one image value by a pixel of the camera; and (c) determining the position of the at least a pixel on the screen based on the at least one captured image value and the image value assignment. The method further comprises shifting the screen or the camera in one direction by an amount such that the at least one pixel is at a different distance from the pixel of the camera than before the shifting, and repeating at least the aforementioned steps for the shifted screen. The camera includes a variable focus when moving the screen relative to the camera, and the method further comprises storing an association with the pixel and the positions of the at least one pixel for different shifted screen positions.

Thus, according to the present invention, no model is used to calibrate the camera, but the pixels of the camera are individually calibrated. It is understood that the position determination requires that the position of the detected image value on the screen is known. This is done by the image value assignment, which determines which image value is displayed at which position on the screen. However, the pixels themselves may be constructed of screen pixels, i. A pixel can consist of one or more screen pixels.

The subject invention solves the above-mentioned technical problem by calibrating each individual camera pixel independently of all other pixels (not model-based) with the aid of a screen. In particular, it is thus possible to use a commercially available screen (for example monitor, television or another display), preferably with the highest possible resolution. The only requirement is that he is able to generate a corresponding calibration pattern. The present invention thus includes a new, monitor-based and parameter-free method for calibrating cameras.

In contrast to the conventional method, which uses a specific model for calibration, whose parameters are to be determined, it is determined according to the invention for each pixel of the camera, from which world point or from which world points it receives light signals (i.e., where it sees). For example, the result of the calibration may be written to a table (e.g., a lookup table) by storing for each camera pixel a set of points mapped to that pixel. In the simplest case, these are in each case straight lines for each pixel, provided the camera focus is not changed during the calibration.

The steps (a) to (c) defined for a particular pixel of the camera may be performed in parallel for all pixels of the camera or for a discrete set of pixels of the camera. After the calibration has been performed by a discrete set of pixels, For example, an interpolation can be used to unambiguously assign the intervening pixels as well. However, the interpolation can also be used to perform subpixel-exact mapping. The more image correspondence pixel / pixel of the camera are determined, the more accurate the result, but at the same time also increases the computational effort required to calibrate all pixels.

The present invention also relates to a method of calibrating a unit of a first camera and a second camera using a screen, wherein the screen has a set of pixels and the first camera uses a plurality of pixels to display an image and the second Camera uses a plurality of pixels for displaying an image and the first and second camera are in a predetermined relation to each other. The calibration comprises the steps of: (a) displaying at least one image value in at least one pixel of the screen based on an image value assignment; (b) detecting the at least one image value by a pixel of the first camera and by a pixel of the second camera; and (c) determining the position of the at least one pixel on the screen based on the at least one captured image value (BW) and the image value assignment. It is understood that this method is not limited to two cameras. Rather, so that a variety of cameras can be calibrated in parallel.

The present invention also relates to a method for determining a shift direction of a screen relative to a camera. The method includes (A) calibrating the camera using a screen, the screen having a set of pixels, the camera using a plurality of pixels to display an image, and calibrating comprising the steps of: (a) displaying at least one image value in at least one pixel of the screen based on an image value assignment; (b) detecting the at least one image value by a pixel of the camera; and (c) determining the position of the at least one pixel on the screen based on the at least one captured image value and the image value assignment; (j) shifting the screen or camera along the direction of displacement by an amount such that the at least one pixel is at a different distance from the pixel of the camera than prior to translation, and (g) repeating at least steps (b) and (c) for the moved screen (120i). The method further comprises: (B) rotating the translation direction or the screen so that the camera is slidable relative to the screen along another direction of translation; (C) recovering the calibration according to steps (A); and (D) determining the direction of displacement from a comparison of calibrations (A) and (C). The present invention also relates to a method for determining refractive properties of a protective layer (eg windshield or coating) of a screen using an uncalibrated camera, wherein the camera uses a plurality of pixels to display an image and the screen has a set of pixels , The method comprises the following steps: (A) displaying at least one image value in at least one pixel of the screen; (B) detecting the at least one image value by a pixel of the camera; (C) rotating the camera or screen about an axis of rotation by an angle; (D) detecting the at least one image value by the pixel of the camera; and (E) determining the refractive properties from a spatial displacement of the detected image value in step (B) in comparison to the detected image value in step (D) and of the angle. Preferably, the axis of rotation is in the screen plane or in the vicinity thereof. To increase the measurement accuracy, it is also possible that not only is rotated by a known angle, but that a variety of rotated shots are performed. Preferably, a pixel of the camera is used, the viewing line is always in rotation in a plane that is perpendicular to the monitor plane.

In order to calibrate the camera not only for a single pixel point, in further embodiments, several pixels are displayed on the screen, which allow a clear assignment. This can be done, for example, by the image value being a color value or a grayscale value of a digital image captured by the camera. Thus, for example, if the screen has a very high resolution, it is possible for each pixel of the camera to receive a different image value (eg, a different color value or gray scale value), thus accurately determining the direction where the corresponding pixel of the camera will look , The screen defines a unique (world) coordinate system that assigns each pixel a unique 3D coordinate. Thus, further exemplary embodiments relate to a method wherein the at least one image value assigns a color value or a grayscale value of a digital image and the image value assignment respectively adjacent pixels of the screen different image values from a plurality of image values such that in at least a part of the screen a unique assignment of Image values allows a location based on the image values.

Since the number of displayed gray level values and color values is limited or the resolution sensitivity of the camera used is limited, it may be advantageous that not all points are calibrated simultaneously. For example, modern cameras have several Megapixel resolutions. With such a resolution, it is often difficult to display on each screen one pixel with a different image value, so as to ensure a clear assignment. Therefore, the calibration can be performed as a successive process in which screen portions (for example, horizontal positions and vertical positions) are successively determined. Likewise, it is not necessary that, for example, the entire horizontal extent of a screen is calibrated in one step. Instead, individual sections or periods can be determined, within which an unambiguous assignment takes place, so that, with knowledge of the position of the period on the screen, an unambiguous assignment of the pixel points to the screen points is made possible.

Therefore, in further exemplary embodiments, the steps (a) to (c) are carried out independently of each other for a horizontal position determination and for a vertical position determination, wherein the neighboring pixels are correspondingly horizontally adjacent pixels for the horizontal position determination or vertically adjacent pixels for the vertical position determination. In addition, optionally, step (a) may comprise displaying on the screen at least one period of a periodic pattern of image values, the image value being an image value from the periodic pattern, and step (b) further comprising the steps of: (b1) changing the Image value to a changed image value that is part of the periodic pattern of image values; (b2) detecting the changed image value by the camera; and (b3) determining a difference of the acquired image values and the detected changed image value for the at least one pixel. The step (c) may then include determining the position within a period in a period direction of the periodic pattern for the at least one pixel based on the determined difference.

As mentioned earlier, it may be useful to first calibrate horizontal and then vertical pixels of the camera (i.e., create the point correspondences for those pixels). Therefore, further embodiments include a method comprising the steps of: (d) rotating the periodic pattern of image values on the screen by a predetermined angle; and (e) repeating steps (b1) through (b3) for the rotated periodic pattern.

If a periodic pattern is used whose period length extends over the entire screen (for example, if there are enough image values to calibrate all the pixels of the camera in the horizontal direction), such a periodic pattern can be used. However, it is often the case that, as a result of the high number of pixels and the desired sensitivity to the distinctness of image values of neighboring pixels, it is advantageous if several periods are displayed on the screen. The period length can be selected, for example, such that a clear detection of the image values (or the differences between the image values) by the corresponding pixel becomes possible. If several such periods are used, in further steps, a determination of the period in which the particular pixel looks can take place. After determination of the period a clear point correspondence is reached.

The determination of the period can be done as follows. Starting from a midpoint, edge point or any other fixed point on the screen, each period may be characterized by a period offset, the period offset indicating, for example, how many periods the period under consideration is shifted from the fixed point. The period offset is determined according to further embodiments by representing a plurality of periods of the periodic pattern in step (a) and the method further comprising the steps of: (f) displaying a fixed image pattern having a predetermined position on the screen, each Period of the plurality of periods has a period offset relative to the fixed image pattern; (g) acquiring the fixed image pattern by at least one particular pixel of the camera; and (h) assigning the predetermined position to the at least one particular pixel; and (i) determining the period offset of the period for which the position was determined in step (c) based on the predetermined position and the position of the at least one particular pixel determined in step (c). This can be done, for example, by counting the periods that lie between the fixed image pattern and the period (for which the position was determined in step (c)). The predetermined position can be defined for example via a center of the image pattern.

According to further embodiments, the periodic pattern is detected only by a part of the camera and can optionally be represented by a harmonic function. Optionally, the method may then further include shifting the periodic pattern to another part of the camera, with the part and the other part of the camera partially overlapping; and customizing. In the adjustment, combining calibration for the part of the camera with calibration for the other part of the camera can be made so that a common calibration for the union of the part and the other part of the camera is obtained.

However, the distance of the screen and the relative position to the camera need not be measured explicitly. Instead, other embodiments allow determination of the distance of the screen and its location relative to the camera. This can be done, for example, by further executing the following steps in a method: moving the screen or the camera by an amount, so that the at least one pixel of the camera has a different distance to the screen than before the move; and repeating at least steps (b) and (c), ie, detecting and determining the shifted screen. Optionally, part or all of the above steps may also be repeated for the shifted position.

This determines at least 2 points through which a line can be laid, which then provides the calibration for the camera pixel. Optionally, however, the screen may be shifted multiple times, such that for each pixel, a plurality of positions are determined, each relating to different distances of the screen. The number can for example be adapted to the desired calibration quality.

According to further embodiments, the camera either has a fixed focus which remains fixed during the detection steps or a variable focus which is changed by the camera when the screen is moved and / or rotated. With a constant focus, each camera pixel can be assigned a straight line that contains all the dots that are mapped to a specific pixel. When using an autofocus function, each pixel can be assigned a set of points that are mapped to a specific pixel, which points can be as close as desired. But even in this case, each pixel can be assigned a direction with a certain focus.

If the relative position of the individual screen positions among each other is known, it can be achieved by repeating the said steps that it is possible to uniquely determine for each pixel of the camera where each individual pixel of the camera is facing. However, it is often not easy or errory to determine the exact relative position of the individual screen positions among each other (e.g., by measurement). Such a determination is achieved according to further embodiments by the following steps: rotating the screen by a predetermined angle; Repeating at least steps (a) to (d) for the rotated screen position; and determining the direction of the shift of the screen based on a comparison between the result obtained in the unrotated position and the result obtained in the rotated screen position. Optionally, part or all of the above steps may also be repeated for the rotated position.

Thus, in the calibration method according to the present invention, no static (printed) calibration pattern as used in the conventional methods is used. Rather, the determination of the point correspondences (3D world points - 2D pixel points), for example, by a sequence of dynamic images on a monitor. As stated, however, the dynamic sequence of points on the monitor is not mandatory. An unambiguous assignment of pixels and pixel points is already possible if the monitor can display sufficiently different image values and if the camera is able to clearly capture each individually displayed image value. However, such a unique assignment, without the use of a periodic pattern, places high demands on both the monitor and the camera. Therefore, advantageous embodiments of the present invention are based in particular on a dynamic sequence of a periodic pattern, wherein the period can be selected so that only a small number of image values by the camera must be clearly detectable and a successive determination of the position of the image value, which a pixel of the camera is detected can take place.

A windshield of the screen / display may possibly cause a refraction of light, which can lead to a distortion of the underlying pixel positions, which should be taken into account and excluded in the end result. Therefore, in other embodiments, the screen includes a windshield, and the method further includes correcting for distortions caused by the windshield.

Embodiments also include a storage medium having a computer program stored thereon and configured to cause a controller to execute a previously described method when running on a processor (processing unit). The storage medium may be a machine-readable medium including mechanisms for storing or transmitting data in a form readable by a machine (e.g., a computer).

Further embodiments also relate to a use of a display for calibrating a camera.

Also, embodiments include apparatus for calibrating a camera that uses a plurality of pixels to display the image using a screen, the screen having a set of pixels. The apparatus includes the following features: an output module configured to drive the screen so that the screen presents at least one image value in at least one pixel based on an image value assignment; an input module configured to input at least one image value captured by the camera; and a processing module configured to determine the position of the at least one pixel based on the at least one acquired image value and the image value assignment. The device may, for example, be a control module with a processor running the computer program.

Brief description of the figures

The invention will be explained in more detail with reference to the embodiments illustrated in Figures 1 to 7. Showing:

1 shows a flow chart for a method for calibrating a camera;

FIG. 2 shows a schematic representation of the assignment of screen dots to camera pixels; FIG.

3 shows an exemplary change of image values of a screen point with time;

4 shows an exemplary distribution of image values on adjacent screen points and their temporal change;

5 shows a periodic calibration pattern;

6 shows a pattern for determining a phase offset;

FIG. 7 shows a change in the screen position for determining an optical path; FIG.

Fig. 8 is a conventional calibration pattern; and

9A, B further embodiments of the present invention.

Detailed description of the figures

Fig. 1 shows a method of calibrating a camera using a screen, wherein the screen has a set of pixels and the camera uses a plurality of pixels to display the image. The method comprises the steps of: displaying S1 10 at least one image value in at least one pixel of the screen based on an image value assignment; Detecting S120 of the at least one image value by a pixel of the camera; and determining S130 the position of the at least one pixel on the screen based on the at least one captured image value and the image value assignment.

FIG. 2 shows a representation with further details for the calibration of the camera 110 using the screen 120 according to an embodiment of the present invention. The camera 110 has an image plane 1 13, in which a plurality of camera pixels 112 are arranged. For example, the camera pixels 1 12 can be arrayed on the image plane 113 may be arranged, wherein a first set of camera pixels 1 12 may be arranged along the horizontal line and a second amount along the vertical line.

The coordinate system can be defined by the (first) screen position in the present invention. The orientation of the camera towards the screen generally does not matter. The coordinate system can also be chosen differently. For example, the camera 110 may be spaced from the screen 120 along the z-direction, wherein the screen 120 and / or the image plane 1 13 within the camera 110 may extend in the x.y plane. For example, the x-direction may represent the horizontal direction and the y-direction the vertical direction of the recorded scene. However, in other embodiments, the coordinate system may still be chosen differently.

The camera 110 may further comprise an optical system 114, through which light signals 1 18 run from the screen 120, in order subsequently to be projected onto the image plane 1 13 of the camera 110. The screen 120 may be divided into a plurality of pixels 122, which may be defined, for example, such that each camera pixel 112 receives light signals from a pixel 122 of the screen 120. If the screen also has pixels, each pixel may include one or more screen pixels. The screen can be, for example, a monitor or display of a computer, a television or another projection device on which, for example, moving images can be displayed. The number of pixels 122 on the screen need not be correlated with the number of camera pixels 112. For example, the number of pixels 122 on the screen 120 may be less than the resolution of the camera 110 (number of camera pixels 112). However, it is also possible that a correspondingly high-resolution screen 120 is used, which has the same number of (or more) pixels 122 as the number of camera pixels 112. For a high quality of the calibration, it is advantageous if the screen 120 is as high-resolution as possible ,

In the calibration, a direction is determined from which a given camera pixel 112 receives a light signal or, in other words, in which direction the corresponding camera pixel 112 is looking. If, for example, the light signal along an optical path 1 18 is projected from a first pixel 122a of the screen 120 onto a first camera pixel 1 12a (after passing the optic 1 14), a determination of the location takes place during the calibration of the first camera pixel 11a of the first pixel 122a. Thus, an association of the camera pixel 112a with the position of the first pixel 122a is obtained. This assignment is not only for a first camera pixel 1 12a, but can be done for each camera pixel 112. The calibration thus determines Also, a second camera pixel 112b receives a light signal from a second pixel 122b at a particular position on the screen 120.

According to the present invention, it can be determined for each individual camera pixel 1 12 to which pixel 122 this one pixel is looking. It should be noted, however, that the camera pixel 112 itself has no further information regarding adjacent camera pixels or the orientation of the screen 120. In order to be able to determine the position of the pixel 122 independently of all other information, the corresponding image value represented by the pixel 122 is determined. If, for example, all the pixels 122 of the screen 120 represent a different image value (for example, an own gray value or color value) and the camera 110 is designed to determine all displayed image values, a calibration of all the camera pixels 1 12 can already take place by a single acquisition of the screen 120 ,

However, this can be problematic. On the one hand, it requires a screen 120 that is able to display as many image values as the camera has 110 pixels 112. Moreover, for such a unique association, it is also necessary for the camera to be able to uniquely identify all displayed image values. Often, however, it is the case that due to real conditions a clear identification of the displayed image value is not readily possible or is associated with an error. For example, a detected gray level value may change as the screen is moved closer to or farther from the camera (the brightness decreases with distance), or other external influences may result in unambiguous identification of the displayed gray level value on one Pixel 122 is not immediately possible.

These problems can be avoided by the embodiments as shown in FIGS. 3 to 6. In this case, no static image is displayed by the screen 120, but each pixel 122 changes over time the image value BW shown. The change may, for example, be periodic, and upon passage of the period, it may be determined which image value was initially displayed on the respective pixel 122 (assuming the period is known).

In addition, it should be noted that often the relative position of the camera 110 and the screen 120 are not known (including their relative orientation to each other) and that from the knowledge of a single particular pixel 122 the optical path 118 is unique given that all optical paths in a point of optics (or lens) 1 14 cut. However, even if all paths intersect in one point, one generally does not know this intersection and can not calculate straight lines. NEN. In such a case, the calibration could also be done with fewer steps. Since these conditions are often not met, the calibration will take place in several steps, which will be described in more detail below (see FIG. 7).

3 shows an exemplary embodiment in which different image values BW1 to BW6 are traversed at a specific pixel 122a over time t. For example, a first image value BW1 can be displayed on the specific pixel 122a for a first time t1. At a second time t2 there is a change to a second image value BW2, at a third time t3 a change takes place to a third image value BW3, etc., up to a sixth time t6, where a change to a sixth image value BW6 occurs. After the sixth time t6, the image values BW can be traversed in reverse order (for example decreasing again), until the eleventh time t11 again changes to the first image value BW1.

These image values BW can increase uniformly and relate, for example, to different gray levels or to color values. It is only important that the image values BW are different, it being advantageous if the changes between the image values BW are strong enough so that the camera 110 can detect the change. From the acquisition of all image values BW within the one period from the first time t1 to the eleventh time t11, it is possible to determine which image value BW was displayed at the starting time on the corresponding pixel.

4 shows further details of such a determination of the image value BW at the corresponding image points 122 of the screen 120. The pixels 122 of the screen 120 are shown horizontally. In this example, there are 80 pixels. These 80 pixels may, for example, refer to the horizontal pixels of the screen 120 or to the vertical pixels or in any other direction. These 80 pixels are subdivided by way of example into 8 periods p, with each period showing 10 image values.

In FIG. 4, the time is shown vertically, so that the 80 pixels at a first time t1 show the image 1 and at a tenth time t10 show the image 10. Each period p shows 10 image values, starting in image 1 with the first image value BW1 (white), the second image value BW2 up to the sixth image value BW6 (black) and then falling from the fifth image value BW5 until the first image value BW1 is reached again. The pattern is similar to the pattern shown in FIG. However, it may be different in other embodiments.

For example, when a particular pixel 122a is extracted, this one pixel 122a shows the fifth image value BW5 at the first time t1. At time t2 the changes Image value at the pixel 122a from the image value BW5 to the sixth image value BW6 (black), at time t3 to the fifth image value BW5. At the seventh time t7, the first image value BW1 (white) is displayed, and at the tenth time t10, the fourth image value BW4 is displayed. Thus, along the time direction (vertically downward), for each of the pixels 122, the period having the period width 10 is traversed once. Based on the knowledge of this particular period, it is possible to determine which image value was displayed at the start time t1 on the corresponding pixel.

Since the pattern of image values at the initial time t1 (period length) and the number of periods p on the screen 120 (for example, in the horizontal direction) is known, it is possible to determine the position of each pixel 122 subpixel accurately within a period. It should be noted that the first image value BW1 and the sixth image value BW6 occur exactly once in each period shown. Therefore, by determining the maximum (e.g., BW6) and the minimum (BW1) in each period, it can be determined which image value was displayed at the initial time t1. But if you could only determine the minimum and maximum, you could make the assignment only pixel accurate. In fact, a (co-) sine curve can be applied through all recorded image values and their displacement can be determined with subpixel precision. Advantageously, not only the minimum and maximum, but all image values are included in the evaluation.

For example, the six image values shown may refer to gray levels, for example, an 8-bit grayscale pattern being representable (i.e., having a size of 256 image values from 0 (black color) to 255 (white color)).

Fig. 5 shows an embodiment of the aforementioned periodic pattern. Grayscale values were again used for illustration. However, (other) color values can also be used. In the embodiment shown, the periodicity runs along the vertical direction, in which example more than eight periods are shown. In other embodiments, any number of periods may be represented. Advantageously, the number of periods can be chosen such that the difference between the image values within a period to neighboring pixels is so large that it can be clearly detected by the corresponding camera. For example, the difference between adjacent image values should be so great that, as they pass through the individual images, as can be seen in FIG. 4, the jumps in the image values BW1, BW2,... Can be detected unambiguously by the camera. Therefore, the number of periods can be selected depending on the specific camera (eg, their resolution) or the screen and the external conditions. As previously mentioned, it is not yet possible to determine in which period (eg in which of the eight periods of FIG. 4) the particular pixel 122a is located. This can be done, for example, by detecting at least one further pixel for which the exact horizontal and vertical period offset on the screen is known. The absolute period position of a period is known, if this one point is known. Using the known period pattern, the period offsets of all other periods relative to the known period can be determined from the absolute period position of a specific period. Thus, the (absolute) positions of all screen points are known.

This can be done, for example, by a pattern (calibration pattern), as can be seen in FIG. In this case, a pattern is shown on the screen 120 (in front of a background 220), which represents a specific image value at a pixel 124 (calibration pixel). Optionally, instead of a pixel, a predetermined number of pixels (for example adjacent pixels) can be used for this purpose. In the illustrated embodiment of FIG. 6, the calibration pixel 124 is shown in the fourth vertical period and the sixth horizontal period.

If this pattern is now detected by the camera 110, only one pixel or set of pixels will capture the calibration pixel 124. If the calibration pixel comprises several pixels, a center 125 can be determined, which can then be assigned to only one camera pixel. Thus, when this is accomplished, it is clear that the one camera pixel that has captured the calibration pixel 124 (or its midpoint 125) is "looking" (or its midpoint 125) towards the exemplary fourth vertical period and sixth horizontal period, and thus After the absolute phase (ie, not just within a period) of the pixel 124 having the one calibration image has been accurately captured on the screen 120, the phase of the pixel 124 is known With the well-known periodic pattern, the position of any other period relative to the one known period is also known, so it is also known for all other pixels in which period they "looked" at the previous determination.

If this is not the case, no clear direction for the pixel can be determined from the previous pixel pixel assignment. The unambiguous determination of a straight line is only possible if at least two points are determined through which the straight line passes. It should also be noted that all the light rays emanating from the screen 120 and projected on the projection plane 1 13 in the camera 110 are not necessarily cut in one point (in any real camera this is not the case). Therefore, the straight line can not be clearly fixed by the determination of a single point. Rather, it is necessary to identify at least one second point which is located at a different distance from the camera 110.

This can be done by a method as shown in FIG. The first (or any other) position of the screen 120 defines a global (calibration) coordinate system. For example, such that the screen 120 lies in the x.y plane and the z-direction represents the surface normal.

In this embodiment, the camera 110 receives a light signal 118 from a particular pixel 122a on the screen 120. The camera 110 detects the displayed gray level value (or any other image value) from the sweep of the periodic pattern (see FIGS. 4 and 5). , The period offset can be determined by using the calibration pattern as shown in FIG. 6.

Now, the screen 120 is shifted (and / or rotated) along a direction z _s to a new position 120i and the procedure described above is repeated (see FIGS. 4 to 6). The translation direction z _s is not necessarily parallel to the z-axis, and the z-axis may be defined as the direction parallel to the surface normal on the original screen 120 (before translation). As a result, a screen point 122i is again determined for the new screen position 120i, which is projected on the same pixel 1 12a in the camera 110. By determining the x and y coordinates of the new screen point 122i, it is possible to draw a straight line through the screen point 122 and the screen point 122i and thus uniquely define the direction or optical path 118.

Optionally, it is also possible to repeat these steps for further shifted screen positions so that for each pixel 1 12 a plurality of positions in the 3D space can be determined. The obtained positions can now be interpolated or approximated by a straight line so as to obtain the calibration.

If the shift was not parallel to the z-axis (eg, surface normal of the screen), a relative angle of the screen to the z-axis (ie, the z _s direction) may be determined as follows. First, the line 118 is determined as described above by measurements for at least two distances from the camera. The accuracy can be further increased by, as I said, additional distances of the screen 120i (i = 1, 2,...) Are taken by the camera 110 so as to achieve the most accurate determination of the straight line 118. After the line 118 for the thus oriented image screen 120 is determined, the screen 120 is rotated by a predetermined angle and again shifted relative to the camera 1 10 along the rotated axis. The measurement is repeated for such a rotated coordinate system. Since a given camera pixel 112 is still looking in the same direction and the camera 10 itself has not been changed, it is possible by comparing the first arrangement to the second, rotated arrangement, the relative position and orientation of the screen 120i to the coordinate system defined by the first screen position 120. Thereafter, all pixels 122 seen from a camera pixel, originating from different screen positions, may be displayed in the same coordinate system defined by the first (or any other) screen position and the associated line 118 calculated.

The presented method can also be summarized as follows.

FIG. 5 shows a snapshot of a monitor image for calibrating the camera 110. The camera 110 is positioned, for example, in such a way that the monitor 120 is visible through the camera 110. On the monitor 120, for example, a horizontal periodic pattern (for example in the form of a sine function) may be displayed as a calibration pattern (see FIG. 7). This pattern is advanced in equidistant steps (for example, four to as many as desired, preferably eight to ten) by a total of one period. Each pixel of the camera 110, which is currently viewing the monitor 120, now sees a gray value oscillation shifted by a certain phase due to the shift. This phase shift (between 0 and 2 ττ) can be calculated back from the recorded gray values. Since the monitor 120 displays several gray scale periods, it is finally only necessary to determine the phase position. This can be done by means of the display of another image (the calibration image, as shown in FIG. 6). This calibration image comprises a mark of a known phase position, which may for example be the center of the screen 120 or may also comprise any other point of the screen - it should only be known exactly. Then it is possible to easily determine adjacent phases. The steps shown can also be repeated vertically, so that a calibration with respect to the vertical direction is also possible.

From the horizontal and vertical absolute phase position, it is now possible to determine subpixel exactly the monitor pixel 122, which was seen by a specific camera pixel 1 12. The monitor screen position can then also be converted to mm and specified in a world coordinate system W. For example, the origin of the world coordinate system W may be at the center of the monitor 120, with the x-axis after to the right and the y-axis points downwards, and the z-axis is perpendicular thereto, ie a surface normal of the monitor plane is (in the Fig. 7 to the rear). This refers to a first position of the monitor 120 in which each camera pixel (n, m) is assigned a first world coordinate xow (n, m).

Furthermore, it is possible to select additional monitor positions whose positions are known in the world coordinate system W, or can be back-calculated from further data. One of these options can be outlined as follows:

The monitor 120 (or equivalent to the camera 1 10) is moved on a linear guide rail by a certain amount 5 _z s, for example, in an unknown direction. The x and y values associated with the monitor pixel 122 are calculated as before, but the z coordinate is measured along the unknown displacement direction. This defines a new sheared co-ordinate system S with non-orthogonal z-axis (see Fig. 7), since the displacement direction is generally non-parallel to the z-axis. The origin, the x and y directions of S coincide with the world coordinate system W. For z = 0, the two coordinate systems agree, with the following relation: xow (n, m) = xos (nm).

Subsequently, additional 3D points can be recorded, which are mapped to a specific pixel. One thus obtains s (n, m) with 0 <i <N-1, where N is the number of monitor positions used. In the case of a fixed camera focus, a regression line can now be calculated for each camera pixel (n, m) by the positions Xis (n.m). In the case of autofocus, the values Xis (n.m) can simply be stored. The camera 110 is now calibrated in the sheared coordinate system S.

The missing transformation between the sheared coordinate system S and the world coordinate system W, which can be used to transform the calibration into a rectangular coordinate system, can be determined by recording further monitor positions. These other monitor positions are not parallel to the calibration positions. At best, the monitor 120 is simply rotated by a certain angle about its vertical axis of symmetry and again shifted on the linear guide rail. The unknown direction zs can now be determined from this data.

It is not mandatory to calibrate all camera pixels 1 12 simultaneously. With highly distorting optics, the monitor 120 may not fill the entire camera image. Then only portions of the camera 1 10 are always calibrated, with the individual sections overlap. By means of a so-called "stitching", the tailor-made Joining the sub-calibrations, then the calibration can be achieved again in a common coordinate system.

In contrast to the conventional methods, therefore, no static (printed) calibration pattern is used to determine the imaging properties of the camera 110, but a commercially available monitor (monitor, television) with the highest possible resolution, which makes it possible to generate arbitrary calibration patterns.

Furthermore, the calibration is no longer based on a model whose parameters are to be determined. It is simply determined for each pixel of the camera 1 10 (about 1 to 5 million), which world points it sees. The result of the calibration is thus a look-up table in which, for each camera pixel 112, a set of points which are mapped onto this pixel are stored. In the simplest case, these are in each case straight lines, for example if the camera focus was not changed during the calibration.

The disclosed method offers the following advantages in particular:

(i) The number of dot correspondences used to determine the imaging characteristics of the overall system of camera, lenses, and optical elements is significantly greater in this process. For example, it can be a multiple of the number of camera pixels, that is, several millions, compared to 100 to 1000 in conventional methods based on a model. The time to determine these correspondences is of comparable magnitude.

(ii) Each pixel of the camera is calibrated individually. This guarantees maximum precision right into the edge areas of the camera image and thus significantly better results when using the calibration in image processing.

(iii) It is also possible to calibrate optically complicated systems comprising camera, lenses and optical properties (for example mirrors). This creates new, unconventional solutions.

9A shows a further embodiment in which the method described above is not used to calibrate a single camera but is used to calibrate a unit of at least two cameras 110, 120, wherein the plurality of cameras 110, 120 are in fixed relation to one another stands. The fixed relation, for example, defines the relative position and orientation of the first camera 110 and the second camera 120, and is achieved by attaching the cameras 110, 120 to the guide rail 150. For example, the distance and / or relative tilt between the first camera 110 to the second camera 120 may be predetermined. Shown are only two cameras 110, 120. However, it should be understood that this method is applicable to any number of cameras. With the method, all cameras can be automatically calibrated in the same coordinate system, as they all look at the same positions of the monitor 120 used. For example, the cameras 1 10, 210 simultaneously capture the image value of a monitor pixel 122, the first camera 110 with the camera pixel 122, and the second camera 210 with the camera pixel 212. One of the monitor positions, in a preferred embodiment, the first monitor position, defines that for all cameras 1 10, 120 same calibration coordinate system. In a preferred embodiment, the origin of the calibration coordinate system lies in the center of the image plane of the first monitor position, the x-axis points horizontally to the right, the y-axis points vertically downward, and the z-axis is perpendicular to the monitor plane and points into it (ie along the guide rail 150).

If each camera were calibrated in its own calibration coordinate system (and not as a unit), the relative transformations between the individual calibration or camera coordinate system would have to be determined after the actual calibration. According to this embodiment, this step is omitted, which is an advantage. In addition, higher accuracy of the overall system is achieved, which can be used, for example, for triangulating points in space (e.g., in stereoscopic or 3D cameras).

In a preferred embodiment of the invention, the direction of the linear guide rail 150 in the - preferably defined by the first monitor position Kalibrierkoordinatensystem - is not known (see, for example, Fig.7 or Fig. 9). While the z-axis of the Kalibrierkoordinatensystem is perpendicular to the image plane, the monitor 120 and the camera 110 moves by means of linear guide rail 150 is not exactly in the z-direction of the Kalibrierkoordinatensystems, but in an initially unknown direction. If the monitor pixels of the different monitor positions are always assigned the same x and y coordinates, the camera 110 or the cameras 1 10, 120 are first calibrated in a sheared calibration coordinate system whose z-axis is parallel to the direction of the linear guide rail 150.

Embodiments make it possible to determine the direction of the linear guide rail 150 in the calibration coordinate system in order to transfer the line of sight of the individual camera pixels 122, 212 from the sheared to the Cartesian calibration coordinate system. For this purpose, the monitor is rotated by an unknown angle after recording the actual calibration positions and again in this new angular position with the aid of the linear guide rail 150 brought in at least one, preferably in a plurality of positions and in turn displayed and recorded the described calibration pattern. The position of the monitor pixels seen by the individual camera pixels 112, 212 that appear distorted in the sheared calibration coordinate system can be unequivocally closed by means of suitable algorithms on the direction of the linear guide rail 150 and thus the shear of the calibration coordinate system can be corrected.

9B shows an exemplary embodiment for determining refractive properties of a screen 120. In this case, an uncalibrated camera 1 10 captures at least one pixel of the screen 120. Subsequently, the camera 110 is rotated about an axis of rotation (at an angle a), which is preferably in a vicinity or at the image plane of the screen 120. Thereafter, the camera 1 10 again detects the pixel, which has moved due to the refraction spatially on the screen surface. Finally, the refractive properties are determined from the observed shift of the pixel. The refractive properties include, for example, a refractive index and / or a thickness of the refractive layer.

This exemplary embodiment therefore takes into account the fact that a protective layer (front pane or coating) can still be located above the radiation-generating elements of the monitor 120 whose refractive properties ensure that the underlying monitor pixels are slightly shifted from the camera 110 to be calibrated Be seen. This effect increases as the angle a increases as the viewing beam of a camera pixel views the monitor 120. This shift can be corrected if the refractive properties of the protective layer - such as its thickness and refractive index - are known.

The embodiment shown in FIG. 9B precisely measures the refractive properties of the protective layer of the monitor used with the aid of the uncalibrated camera. In the exemplary configuration for determining the refractive characteristics of a monitor, the uncalibrated camera 220 is mounted on a turntable 140, the axis of rotation of which is preferably near or at the image plane of the monitor 120 to be measured. The uncalibrated camera 110 subsequently views the monitor under known conditions a, with the monitor 120 showing the known horizontal and vertical patterns. From the amount of monitor pixel positions that a single camera pixel has identified at different angles in this way, one can clearly conclude the refractive properties of the protective layer. For example, up to 10 or 40 or more angles can be measured, the invention not being limited to the specific number of angles. According to further embodiments, it is not necessary that each camera pixel 112 is assigned a unique line of sight, ie that the set of all points imaged on the same camera pixel 112 lies exactly on a straight line. This assumption is only correct if the focus of the camera 1 10 is fixed and does not change between calibration and use. If the focus of the camera 1 10 changed or this is operated with automatic readjustment of the focus, this assumption is no longer true.

Therefore, in the method according to the invention, it is not necessary for the set of all dots imaged on the same camera pixel 12 to lie on a straight line. Rather, they can form a curved, one-dimensional curve in three-dimensional space. According to further exemplary embodiments, therefore, regression lines from two or more points imaged on the same camera pixel 12 are not determined, but for each camera pixel 112 a plurality of world points are stored in the calibration coordinate system which are imaged onto a specific camera pixel 12. The curved, one-dimensional curve of each camera pixel can be scanned arbitrarily granular and stored according to the number of monitor positions used.

It is understood that all of the above-described method steps or features of the devices can also be implemented in these other methods / devices. To avoid repetition, these other limitations should not be described again here.

The features of the invention disclosed in the description, the claims and the figures may be essential for the realization of the invention either individually or in any combination.

LIST OF REFERENCE NUMBERS

110, 210 cameras

112a, b, 212 camera pixels

113 image plane of the camera

114 optics

118 optical path

120 screen

120i screen in a different position

122a, b pixels

122i pixels on screen in a different position

124 fixed image pattern, calibration pixel

125 center of the calibration pixel

140 turntable

150 guide rail

220 background

501..503 calibration points of a conventional pattern

BW image values

Claims

claims

A method of calibrating a camera (110) using a screen (120), wherein the screen (120) comprises a set of pixels (122) and the camera (110) uses a plurality of pixels (112) to display an image, with the following steps

(a) displaying at least one image value (BW) in at least one pixel (122) of the screen (120) based on an image value assignment;

(b) detecting the at least one image value (BW) by a pixel (1 12a) of the camera (110); and

(c) determining the position of the at least one pixel (122) on the screen (120) based on the at least one captured image value (BW) and the image value assignment;

(c1) shifting the screen (120) or the camera (1 10) in one direction by an amount such that the at least one pixel (122) is at a different distance from the pixel (112a) of the camera (1 10) than before Move, and

(c2) repeating at least steps (b) and (c) for the shifted screen (120i),

wherein the camera (110) has a variable focus when moving the screen (120) relative to the camera (110), and the method further storing an association with the pixel (112) and the positions of the at least one pixel (122) includes various shifted screen positions (120i).

Method for calibrating a unit from a first camera (1 10) and a second camera (210) using a screen (120), wherein the screen (120) has a set of pixels (122) and the first camera (1 10) uses a plurality of pixels (112) to display an image, and the second camera (210) uses a plurality of pixels (212) to display an image and the first and second cameras (110, 210) are in a predetermined relation to each other following steps:

(a) displaying at least one image value (BW) in at least one pixel (122) of the screen (120) based on an image value assignment; (b) capturing the at least one image value (BW) by a pixel (1 12) of the first camera (110) and by a pixel (212) of the second camera; and

(c) determining the position of the at least one pixel (122) on the screen (120) based on the at least one captured image value (BW) and the image value assignment.

Method for determining a direction of displacement of a screen (120) relative to a camera (110), comprising the following steps:

(A) calibrating the camera using a screen (120), wherein the screen (120) has a set of pixels (122), and the camera (110) uses a plurality of pixels (112) to display an image through the following Steps:

(c1) shifting the screen (120) or the camera (1 10) along the direction of displacement by an amount such that the at least one pixel (122) is at a different distance from the pixel (1 12a) of the camera (1 10) than before moving, and

(c2) repeating at least steps (b) and (c) for the shifted screen (120i); and

(B) rotating the direction of displacement or the screen (120) so that the camera (120) is slidable relative to the screen (120) along another direction of translation;

(C) repeating the calibration according to the steps (A); and

(D) Determining the direction of displacement from a comparison of the calibrations (A) and (C). A method of determining refractive characteristics of a screen (120) using an uncalibrated camera (110), wherein the camera (110) uses a plurality of pixels (1 12) to display an image and the screen (120) comprises a set of pixels (122 ), comprising the following steps:

(A) displaying at least one image value (BW) in at least one pixel (122) of the screen (120);

(B) detecting the at least one image value (BW) by a pixel (1 12) of the camera (110);

(C) rotating the camera (1 10) or the screen (120) about an axis of rotation about at least one angle (a);

(D) detecting the at least one image value (BW) by the pixel (112) of the camera (110); and

(E) determining the refractive properties from a spatial displacement of the detected image value in step (B) compared to the detected image value in step (D) and the at least one angle (a).

Method according to one of claims 1 to 4, wherein the at least one image value (BW) is a grayscale value or a color value of a digital image and the image value assignment respectively adjacent pixels (122) of the screen (120) different image values (BW) from a plurality of image values ( BW) such that in at least one part of the screen (120) a clear assignment of image values (BW) makes it possible to determine the location based on the assigned image values (BW).

Method according to claim 5, wherein the steps (a) to (c) are carried out independently of each other for a horizontal position determination and for a vertical position determination.

Method according to one of claims 1 to 6, wherein the step (a) comprises representing at least one period of a periodic pattern of image values (BW) on the screen (120), and wherein the image value (BW) is an image value (BW) from the periodic pattern and step (b) further comprises the steps of: (b1) changing the image value (BW5) to a changed image value (BW6) that is part of the periodic pattern of image values (BW);

(b2) detecting the changed image value (BW6) by the camera; and

(b3) determining a difference between the captured image value (BW5) and the acquired changed image value (BW6) for the at least one pixel (122),

wherein step (c) comprises determining the position within a period and thus a phase position in a period direction of the periodic pattern for the at least one pixel (122) based on the determined difference.

8. The method of claim 7, further comprising the steps of:

(d) rotating the periodic pattern of image values (BW) on the screen (120) by a predetermined angle; and

(e) repeating steps (b1) to (b3) for the rotated periodic pattern.

9. The method of claim 7 or 8, wherein in step (a) a plurality of periods of the periodic pattern are displayed and the method further comprises the following steps:

(f) displaying a fixed image pattern (124) having a predetermined position on the screen (120), each period of the plurality of periods having a period offset relative to the fixed image pattern;

(g) capturing the fixed image pattern (124) by at least one particular pixel (1 12a) of the camera (110);

(h) assigning the predetermined position to the at least one particular pixel (1 12a); and

(i) determining the period offset of the period for which the position is determined in step (c) based on the predetermined position and the position of the at least one particular pixel (1 12a) determined in step (c).

Method according to one of claims 7 to 9, wherein the periodic pattern is detected only on a part of the camera (110) and the method further comprises a displacement of the periodic pattern in another part of the camera (1 10), wherein the Partially overlap part and the other part of the camera (110); and adjusting a calibration for the part of the camera (110) to a calibration for the other part of the camera (110), so that a common calibration for a combination of the part and the other part of the camera (1 10) is obtained.

A method according to any one of the preceding claims, further comprising the steps of:

Rotating the screen (120) a predetermined angle;

Repeating steps (c1) through (c2) for the rotated screen position; and

Determining the direction of displacement of the screen (120) from the step (c1) based on a comparison between a result obtained in the unrotated position and a result obtained in the rotated screen position.

12. The method of claim 3, further comprising storing an association in a table, wherein the association for each pixel of the plurality of pixels includes a plurality of position data of the captured pixels in assigns to different distances.

13. The method of claim 3, wherein the screen comprises a windshield, and the method further comprises correcting for distortions caused by the windshield.

14. Use of a display (120) for calibrating at least one camera (1 10) with variable focus.

Apparatus for controlling calibration of a camera (110) using a plurality of pixels (112) to display the image using a display (120), the display (120) comprising a set of pixels (122) , with the following features:

an output module configured to drive the screen (120) such that the screen (120) displays at least one image value (BW) in at least one pixel (122) based on an image value assignment; an input module configured to input at least one image value (BW) acquired by the camera (110);

a processing module configured to determine the position of the at least one pixel (122) based on the at least one captured image value (BW) and the image value assignment; and

a displacement module configured to displace the screen (120) or the camera (110) in one direction by an amount such that the at least one pixel (122) is at a different distance from the pixel (112a) of the camera (110) has as before moving,

wherein the camera (110) has a variable focus when moving the screen (120) relative to the camera (110), and that the device is further configured to be associated with the pixel (1 12) and the positions of the at least one pixel (122) for various shifted screen positions (120i).

A computer program product having a computer program stored thereon configured to cause the apparatus of claim 15 to perform a method as claimed in any one of claims 1 to 13 when running on a processor.