DE102021200197A1 - Procedure for calibrating a camera - Google Patents

Abstract

Method for calibrating a moving camera (10, 20), in which model parameters of an intrinsic calibration model are determined on the basis of image correspondences, the image correspondences describing an image of the same objects at different recording times of the moving camera (10, 20) and being stored continuously and Finally, the model parameters are determined by minimizing epipolar errors.

Description

The invention relates to a method for calibrating a camera and an arrangement for carrying out the method.

State of the art

Cameras are photographic equipment that can record static or moving images on photographic film or electronically on a storage medium or transmit them via an interface.

Calibration is a measurement process that is used to determine a deviation of a measuring device from another device, a reference device. This deviation is then taken into account when the measuring device is subsequently used to correct the values read. As part of the calibration of a camera, its imaging behavior is thus determined in comparison to the imaging behavior of a reference camera.

In order to be able to use a camera as a measuring instrument, the process of image creation, in this case the projection of the three-dimensional environment onto the image, must be known. Just like the position of the camera in the environment, the imaging properties of the camera itself are also important. The latter is typically referred to as intrinsic calibration. This is to be distinguished from the so-called extrinsic calibration, which describes the position of the camera in the environment or in relation to a specific reference. Note that photometric calibration is not considered here.

In contrast, the estimation of the intrinsic calibration is considered here in particular. This describes the geometric relationship between a point on the image sensor or imager and a line of sight in the world. To estimate the parameters of an intrinsic calibration model, specially manufactured calibration bodies are typically used, the geometry of which is known. This process is typically performed after camera manufacture and/or installation, e.g. B. in the vehicle performed. Since this process is complex and the calibration can also change over the lifetime of a camera, it is desirable to be able to determine the model parameters during operation and without special calibration bodies.

The intrinsic calibration parameters, which describe the geometric properties of the image sensor and lens system, are typically determined using specially manufactured and measured calibration bodies. These methods are referred to as offline calibration because the calibration does not happen at runtime. Offline methods are opposed to online methods in which an attempt is made to estimate the calibration parameters during runtime, i.e. during normal operation.

Methods that do not require special calibration bodies are among the online methods. Offline methods typically achieve higher accuracy and are therefore to be preferred. However, there are several reasons to additionally or exclusively carry out an online calibration. These include (a) cost reasons, since calibration after manufacture and after installation leads to increased costs and (b) a change in the calibration over time, e.g. B. by temperature influences, which would require a renewed offline calibration.

It should be noted that methods for online calibration are often limited to the special case of a binocular (stereo) system, which is much easier to solve and should not be specifically considered here. In the case of a single camera, i. H. monocular, there have been few procedures to date. The known methods usually use certain known and frequently occurring geometric objects, such as line markings, or attempt to form temporal chains of image correspondences, which are then used in a large balancing problem.

Disclosure of Invention

Against this background, a method with the features of claim 1 and an arrangement according to claim 10 are proposed. Embodiments emerge from the dependent claims and from the description.

The method presented is used to calibrate a moving camera, in which model parameters of an intrinsic calibration model are determined and/or corrected on the basis of image correspondences, with the image correspondences describing an image of the same objects at different recording times of the moving camera and being continuously stored and finally the determination the model parameter is done by minimizing epipolar errors.

An efficient compression of image correspondences and epipolar geometry is proposed, which enables camera calibration. Epipolar geometry is a mathematical model from geometry that represents the geometric relationships between different camera images.

A method is thus proposed that makes it possible to estimate model parameters on the basis of image correspondences. It is important how large amounts of image correspondence can be stored or compressed efficiently and in terms of calibration. The way it is saved also makes it possible to determine in an efficient way whether there are enough correspondences available for a successful calibration.

In contrast to known methods, the method proposed here also works if there is only correspondence between consecutive images. It is therefore not necessary to build up chains, especially across several images. With the help of the correspondences, the parameters of the intrinsic calibration model can be determined or corrected. In order for this method to work well in practice, it is useful for the camera to perform various movements. A pure straight movement z. B. may not be sufficient. For this reason, it is expedient to store image correspondence over a longer period of time, in which the vehicle performs different movements, and then carry out a calibration based on the stored data. For this, the movement of the camera should also be known or estimated from the image correspondences. However, an estimate from the image correspondences will be subject to errors, since it is based on an incorrect calibration.

In principle, calibration and movement can be estimated simultaneously. However, the number of parameters to be determined is usually so large that the problem cannot be solved in an acceptable amount of time or with an acceptable use of computing resources. For this reason, the error-prone estimation of the movement is typically used in the presented method. It must be taken into account that errors in the calibration and motion estimation can be gradually reduced by performing the calibration method presented here multiple times.

A fundamental problem of the continuous storage of the measurements, namely image correspondences and motion estimation, is that the storage requirement continues to grow, typically linearly, and this large amount of data also has to be processed during the collision. One way to solve this problem is to only save new measurements when the vehicle or camera performs a new movement that has not been performed before. The disadvantage here is that it must be decided from the outset whether a movement is sufficiently different from the movements already carried out and that multiple measurements of the same movement do not lead to greater accuracy.

Therefore, it is suggested to continuously summarize similar measurements and to average them for the calibration. The similarity of the measurements is determined via the positions of the corresponding pixels and the epipolar geometry or the movement carried out. The storage is then realized via a hash map. A key is generated from the image positions and the movement. In the hash map, this refers to a value vector in which the image positions and parameters of the movement are added or accumulated and finally averaged before calibration. If a measurement consisting of the coordinates of the image correspondences and the parameters of the motion estimation is made, the same key is typically generated and the new measurement can be added to the existing value vector. An additional counter can then be used to determine the mean value of the measurements at any time.

A hash map represents a way of storing the data. A hash value is generated from the key on which the correspondence and movement information is based, which describes the storage location in a table. The generation of the hash value from the key is typically designed in such a way that the entire table is filled evenly, even if the keys are very similar. Hash collisions that occur when the same hash value was generated for different keys can be resolved using lists, for example. Each table entry therefore contains a list and each list object contains a value vector in which measurements can be accumulated and the number of accumulated values can be recorded. Since the generated hash typically does not allow any conclusions to be drawn about the key, the value vector also contains the key itself. The advantage of a hash map is typically the short and almost constant access time. Alternatively, search trees can also be used instead of lists. Likewise, a search tree can also replace the entire hash map. Typically, however, hash maps are more efficient, especially when the order of magnitude of the number of objects is known approximately.

Efficient storage of image correspondences and motion is achieved in this way by combining similar measurements. The summarization takes place in the sense of the calibration. The information relevant for the calibration is largely retained in the combination or compression.

The proposed method only requires new storage space as soon as new movements or observations that deviate from those previously made are made. As a result, memory requirements and computing time can be reduced. It can also be efficiently determined whether there are enough measurements for the calibration.

Efficient storage of measurements, taking into account the quantities required for calibration, is important here. A picture correspondence between two pictures can be described by four parameters and the movement by five parameters. The parameters describe the rotation and translation, whereby the length of the translation is not important here and can therefore be described by only two instead of three parameters. A total of nine parameters should therefore be stored. The five parameters of the movement would only have to be stored once for a pair of images. From the nine continuous parameters, four particularly discrete new parameters are determined here, which are sufficiently discriminative to distinguish cases that are important for the calibration and to combine similar measurements with the same four parameters. This means that the four discrete parameters serve as keys. Measurements with the same key are accumulated and later averaged. The storage can therefore be implemented efficiently as a hash map.

For any number of similar measurements, e.g. B. moving straight ahead at the same speed and a similar distance from the camera to the environment, no more memory is required than for a single such measurement. If, on the other hand, new movements are performed or a different scene structure is observed, the hash map grows because new keys are created. The diversity required for the calibration is thus retained. However, the influence of over-represented measurements, which are often disadvantageous for the calibration, can be reduced in this way.

An analysis of the hash map also makes it possible to determine in an efficient manner whether there are enough and sufficiently diverse measurements for a calibration. Details on this and how the key is calculated are explained below.

Further advantages and refinements of the invention result from the description and the accompanying drawings.

It goes without saying that the features mentioned above and those still to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention.

character list

• 1 shows a schematic representation of a projection of a world point in the two images of a camera at two different points in time and the resulting epipolar plane and epipolar lines in the camera images.
• 2 shows a superimposition of the point projections and the epipolar lines in one image.
• 3 shows an example of a calculation of keys.
• 4 shows a possible sequence of the presented method in a flowchart.

Embodiments of the invention

The invention is shown schematically on the basis of embodiments in the drawings and is described in detail below with reference to the drawings.

In the following, the special case of image correspondences between two images and a variant of the calculation of the key from four values are shown. Derivations or variants of this are presented in the next section.

1 shows a projection of a world point X into the image of a camera at two different points in time, which is considered and represented here as two cameras. The projection centers and the world point span a plane that describes epipolar lines in the respective camera images.

The representation shows a first camera 10 and a second camera 20, the second camera 20 corresponding to the first camera 10 at a different point in time. The representation also shows a world point 30. The camera 10, 20 is therefore a moving camera 10, 20.

The illustration thus clarifies the projection of the world point 30 into the image of a camera 10 or 20 at two different points in time, which is viewed here as two cameras 10, 20. The projection of the projection center 12 or 22 of the respective other camera 10 or 20 into the image designates the epipoles. The projection centers 12 , 22 span a triangle 40 with the world point 30 . The intersection of the area spanned by triangle 40 with the image area 14 or 24 describes the epipolar lines 16 or 26.

2 shows the first epipolar line 16 and the second epipolar line 26, the epipoles e 18 and e' 28 and image correspondences u 19 and v 29. The representation illustrates the superimposition of the epipoles e 18 and e' 28, the epipolar lines 16, 18 and the image correspondences u 19 and v 29 of both images.

The image correspondences u 19 and v 29 lie on the epipolar lines 16 and 26. Depending on the optics, epipolar lines do not necessarily result in the image, but arbitrary curves. However, the geometric connection is retained. For further explanation, it is restricted to the case of lines. However, all methods can also be used in the general case with any optics.

If the movement of the camera 10 or 20 is known, the epipolar line 26 in the second camera 20 can be determined solely from the image of the world point 30 in the first camera 10 . The image of the world point 30 in the second camera 20 must then lie on this line 26 , but its position on the line 26 depends on the distance from the world point 30 . The general result is that the images of the world point 30 must lie on corresponding epipolar lines 16, 26.

If the imaging behavior of the camera is incorrectly modeled, i. H. If the camera is decalibrated, it may not be possible to find a plane, so that the projection centers and image points or corresponding lines of sight lie on one plane. This circumstance can be used to correct or determine the calibration of the camera. It should be noted that only the error perpendicular to the epipolar line can be determined since the position along the epipolar line depends on the unknown distance of the world point 30 to the respective camera 10 or 20. Each point correspondence therefore introduces only a one-dimensional constraint equation. In order to calibrate the camera 10 or 20, many of these observations should therefore be used. Since the movement also determines the direction in which epipolar lines run and thus the direction in which errors can be measured and corrected, it is also necessary for various movements to be carried out.

It is useful to collect correspondences and determine the movement of the camera over a period of time until there is enough information to perform a calibration. Since the motion determination itself is affected by errors in the calibration, the calibration should be repeated several times, with the hash map being emptied each time and new measurements being accumulated. The procedure for efficiently storing correspondence and movement is discussed below.

Correspondences can be determined from any pair of consecutive images or across multiple images. Since similar correspondences are often observed, it makes sense to combine them in order to be more robust against errors in determining the correspondence or moving objects. The relationship described above only applies to the movement of a camera in a static environment. A naïve approach to grouping similar correspondences is to do so directly based on the four image coordinates and the five parameters of motion. One difficulty here is that there are too many combinations of different parameters. A different approach is therefore proposed.

It was recognized that, in addition to the position of the correspondences in the image, the position of the epipoles is particularly important for the calibration. It is suggested that binning, e.g. B. by a square grid to carry out the correspondence coordinates. The exact positions of the image correspondences u and v are discretized. Instead of the exact image position, the number of the grid cell (bin) is now used, so that there are two numbers for each image correspondence. In addition, the quantized direction to the epipole is given in the respective image. These four values result in a key that can be used to identify similar correspondence. Each key in the hash map refers to a vector of values that results from the image coordinates of the correspondence from the five parameters of the movement as well as a counter and the key itself. If two pairs of image correspondences have the same key, their parameters are added up in a value vector that serves as an accumulator. Of course, only the coordinates of the correspondences and the movement parameters are added up. The counter can be used to track how many correspondences have been added up. Finally, for the calibration, the values in the accumulator are divided by the value in the counter to determine the mean value.

Similar movements and similar scene structures should lead to correspondences with the same conclusions. The hash value vector pairs can e.g. B. store it using a hash map. The advantage of this table is that only data that has been observed is stored. Will e.g. B. the same observations are always made during a long straight drive, no new keys are created, no more memory is required takes. By adding up within the value vector, however, all

observations and generally allow a more accurate or more robust estimation of the parameters. An example of such key generation is in 3 to see.

3 shows an example for the calculation of keys 44 by dividing (binning) the image into square, consecutively numbered grid cells (bins) and quantization/mapping of the direction of the image correspondence to the respective epipole. For this measurement, the key would be (22, 24, 10, 9) 44 since the image correspondences are in grid cells 22 and 24 (from reference numeral 44) and the epipoles are in directions 10 and 9, respectively. The non-quantized position of the image correspondence coordinates and the motion parameters are added up in the value vector. The number of accumulations is tracked via a counter 50, here N.

4 shows a possible flow chart of the procedure. Image correspondences (step 72) and the camera movement (step 74) are determined for a new image pair 70 . For each correspondence, the key is then determined (step 76) and either a new entry in the hash map is created (step 78) and the values of the measurement, image positions u and v, and the five parameters of the estimated motion, in the value vector along with the The key is entered and the counter is set to one if a similar measurement has not yet been made, or added up in the corresponding value vector if there is already an entry in the hash map. In addition, the counter of the value vector of the corresponding entry is increased. After all image correspondences have been entered, an analysis of the hash map is made (step 80).

By analyzing the keys 44 ( 3 ) with regard to the directions to the epipoles and occupancy of the bins, it is determined whether there are sufficiently different measurements for a calibration. The decisive factor here is a high diversity of directions to the epipoles. If enough information is available, the calibration can be determined by simultaneously minimizing all distances to the epipolar lines (step 82). For this purpose, the average values are formed. In addition, the quality is estimated, for example based on the size of the residual error of the calibration (step 84). If the distances to the epipolar lines are still high after the calibration, the quality is low and further runs should be carried out. To do this, the hash map is emptied (step 86) and the entire process is repeated. If the measurements were not yet sufficiently diverse or there were too few measurements, then further pairs of images are considered.

This algorithm is characterized by the way it decides whether measurements, i. H. Motion and image correspondence coordinates are similar and are summarized and stored in them by accumulation through a hash table.

In the previous section, a variant was presented that carries out a binning of the image correspondences and then uses quantized directions to the epipoles. A further refinement consists in also specifying the positions of the epipoles via binning. All positions can also be relative to a coordinate, e.g. B. one of the epipoles are determined. The binning can also be done geometrically by other grid shapes, e.g. B. hexagonal or radius and angle.

Furthermore, the method can also be applied to more than two images. However, the keys then become very long and it is less likely that the same key will be generated.

If the range of movement is restricted to such an extent that only certain movements can occur, other keys can of course be generated on this basis. The procedure for generating the keys should generally be designed in such a way that all relevant measurements can be recorded with consistent accuracy, in particular taking into account the runtime for calculating the keys.

Claims (10)

Method for calibrating a moving camera (10, 20), in which model parameters of an intrinsic calibration model are determined on the basis of image correspondences (19, 29), the image correspondences (19, 29) being an image of the same objects at different recording times of the moving camera ( 10, 20) and stored continuously, and finally the model parameters are determined by minimizing epipolar errors. procedure after claim 1 , where the model parameters are corrected. procedure after claim 1 or 2 , in which the epipolar geometry is determined from the image correspondences (19, 29). Procedure according to one of Claims 1 until 3 , in which the motion underlying the epipolar geometry is described by five parameters. procedure after claim 3 or 4 , in which similar measurements, consisting of the coordinates of the image correspondences (19, 29) and the determined epipolar geometry, are summarized and represented by a mean value. Procedure according to one of claims 3 until 5 , in which the similarity of measurements is determined by discretizing the coordinates of the image correspondences (19, 29) and the respective direction of the epipoles (18, 28). procedure after claim 3 or 4 , in which the similarity of measurements is determined by discretizing the coordinates of the image correspondences (19, 29) and the respective direction of the epipoles (18, 28). procedure after claim 6 or 7 , in which the discretized values are used as keys (44) and floating point values of the measurements are accumulated in a hash map or a search tree and counted in order to be able to determine an average value at any given point in time. Procedure according to one of Claims 1 until 8th , in which the decision to carry out the calibration is made on the basis of an analysis of the diversity of the measurements. Arrangement for calibrating a moving camera for carrying out a method according to one of Claims 1 until 9 is set up.

Images