WO2018211057A1

WO2018211057A1 - Marker-based camera tracker

Info

Publication number: WO2018211057A1
Application number: PCT/EP2018/063048
Authority: WO
Inventors: Markus Michaelis
Original assignee: Signum Bildtechnik GmbH
Priority date: 2017-05-19
Filing date: 2018-05-18
Publication date: 2018-11-22
Also published as: DE102017208526A1

Abstract

The invention relates to a method for determining camera parameters based on image data recorded by means of a camera. The method comprises the following steps: recording a plurality of images of a scene, wherein the images contain a plurality of markings arranged in the scene; detecting the positions of the markings in the space; recording further images of the scene, wherein the further images contain at least some of the markings arranged in the scene; associating the spatial positions of the markings in the further images with their respective image positions; determining the camera parameters when recording the further images based on the associated space/image positions of the markings in the further images.

Description

Marker based camera tracker

The present document relates to a marker-based "through the

Camera tracker, the camera parameters based on shots of

Marks detected by the taking lens of the camera, and

appropriate method for determining camera parameters.

background

Virtual studios (studios with a green background) need to be calibrated to display backgrounds using moving cameras, so that the virtual background is always visually displayed correctly, especially with respect to real objects in the studio, those of the cameras

be recorded. Today's tracking systems are complex in preparation and calibration, usually very expensive, the user does not have the necessary knowledge about a calibration, additional devices are required (cumbersome structure). Despite many advantages of virtual studios any virtual environment can be used even in smallest studios) this often leads to the denial of the use of virtual studios.

Consequently, there is a need for virtual studio calibration and tracking systems that provide simplified calibration for determining

Can perform acquisition parameters without additional equipment during production. The recording parameters may be

Properties of recording cameras (also called production cameras) act, such as their position and orientation in space or their focal length.

Summary of aspects of the invention

The proposed procedure is based on a measurement of markings mounted in the studio by means of a camera and the determination of the spatial positions of the markings. These marks are then from captured by a camera during the recording (production) of image or video recordings and used to determine the recording parameters. For example, the spatial position of the camera during production and its

Zoom parameters such as focal length can be determined continuously during production without additional equipment (such as other

Cameras) for determining the position of the production camera needed. Such an approach is also called "through the lens."

Production camera). These recording parameters can then be used to calibrate the virtual studio, so that virtual contents can be correctly faded into a recorded scene, even with regard to real objects present in the studio.

According to a first aspect, a method for determining

Camera parameters based on recorded by a camera

Image data revealed. The method includes capturing a plurality of images of a scene, wherein the images include a plurality of markers disposed in the scene, and determining the positions of the markers in the space. In this way, all markers in the scene can be measured. The scene may be a television studio or other suitable studio for producing image or video content. The determined positions of the markers can then be stored for later use. Thus, the preparatory measures are completed and the production of image or video content can begin.

Subsequently, in the production, further images of the scene are taken, wherein the further images contain at least some of the markers arranged in the scene, and an assignment of the spatial positions of the scene

Markings in the other pictures to their respective picture positions. The other images can be the produced image or video content. For their processing, for example in a virtual studio, the camera parameters may be useful in taking the additional images. These are determined on the basis of the assigned spatial / image positions of the markings in the further images. The camera parameters thus determined may include at least one of camera position, camera orientation, and camera zoom. In the exemplary embodiments, the positions of the markers are determined in the rough by means of a "structure from motion" method The method can have a further step of using the determined camera parameters for inserting virtual content into the further images n the background of the virtual studio can be optically filled depending on the camera parameters. At least some of the markers may be designed to allow unique identification of the particular mark. For example, the markers may include a visual coding of their identification number and / or a reference position or direction. This allows at least some of the markers in the other pictures to be unique

be identified so as to their previously determined spatial positions

to access and assign to the respective image positions. Preferably, the markers are arranged three-dimensionally in the space of the scene and their respective three-dimensional positions in space are determined.

In order to be able to reliably recognize a large number of markings in the images, the markings can have uniquely identifiable master markings and slave markings whose identifier is unique only in relation to a master mark. Furthermore, the markers can be colored in such a way that they can be easily removed from the other images by means of their color coding. This is particularly advantageous when using a green box.

In embodiments, the images of the scene are taken with a calibrated surveying camera. This can be previously subjected to an intrinsic calibration to their camera parameters, such as

For example, focal length, optical center and distortion based on

To determine recordings of a calibration object with known dimensions. The further images can be taken with a production camera (e.g., a broadcast camera).

In order to record as many markings as possible in an image, a plurality of recorded individual images, each of which contains only a part of the scene and the markings arranged therein, can be stitched by means of so-called stitching

composed of composite images. Based on

composite images can then be determined the positions of the markers in the room. According to a second aspect, an apparatus for carrying out the method described above is proposed.

Furthermore, a system for determining camera parameters is provided which has at least one camera for recording images of a scene, a plurality of markings arranged in the scene, and an evaluation unit. The evaluation unit is connected to the at least one camera in order to obtain the recorded images and to determine the positions of the markings in the space on the basis of the obtained images. Further pictures of the scene are then taken by the at least one camera and the Transfer evaluation unit. The evaluation unit is further configured to associate the spatial positions of the markers in the further images with their respective image positions; and determining the camera parameters when capturing the further images based on the assigned spatial / image positions of the markers in the further images. Basically, the

Evaluation unit perform the steps of the method described above and all aspects described there can be transferred to the proposed system. In embodiments, 2 cameras can be used: an intrinsically calibrated surveying camera for recording the mark for determining their spatial positions and a

Production camera for taking picture or video content.

Brief description of the figures

The invention will now be described by way of example with reference to the accompanying drawings.

FIG. 1 shows an example of a marking.

FIG. 2 shows a calibration object.

FIG. 3 shows the geometric relationships of a projective camera.

Detailed description

In one embodiment, the described system for dynamically determining acquisition parameters of a production camera is based on a measurement of markings mounted in the studio by means of a

Surveying camera and the determination of the spatial positions of the markers by photogrammetric method. For this purpose, two or more individual images of the studio with the attached markings using a photogram metric surveying camera created and

evaluated. To determine the 3D positions of the markers, the so-called "structure from motion" method can be used, which is a stereo measurement with unknown extrinsic

Camera calibration, as described in detail below. The proposed procedure does not require a calibrated stereo rig for the

Surveying, but allows to work with freehand shots. The markers are designed in such a way that on the one hand they are easy to find or recognize in a recorded image and on the other hand enable an exact determination of a position. For this purpose, a round basic shape is appropriate, for example a circle with 2 filled,

opposite sectors that meet at the center of the marker. In general, 3 marks define the coordinate system.

The markings are preferably visually coded by means of markings, so that their identity can be determined by evaluating recorded images of the markings. For example, in addition to the round basic shape of the marker additional structures may be provided for coding. These additional structures can be filled circles (blobs), which are arranged at certain positions in a circle around the possibly also round basic shape. For example, a binary coding of the marker ID by means of these additional structures is possible. In other words

the presence of such an additional structure at a particular position represents a binary one and the absence of a binary zero. Using 4 possible structures on a marker so 16 different

Labels (IDs) are attached.

In addition to their basic form and the ID structures for coding the marker ID, the markers may also have an additional reference structure (Blob). This indicates the zero direction for decoding and ID recognition. On the other hand, the additional structure may be smaller than the ID structures for coding. If this additional structure is not recognized in an image, there is a risk that ID structure detection will be unreliable and the ID may be misrecognized. In this case, the marker should be discarded.

It is advantageous if the visual coding of the marker IDs can be recognized over a wide zoom range, so that the markings can be unambiguously identified during production under a wide variety of recording conditions. Therefore, as many markers as possible should be placed in the studio in different locations and possibly also in different sizes. So it can be ensured that among the

different recording conditions always enough marks are captured in the picture.

In order to facilitate the identification of the markers, even if they are only small in a captured image and their detail structures are difficult to recognize, a "master-slave" concept can be used, whereby an easily recognizable master marker ( for example, with larger dimensions) and multiple slave markers used in known spatial arrangements for master marking are, for example, annular in a known direction of rotation to the master mark. After the master tag has been identified by its visual ID, the slave tags can be identified, even if their visual IDs are too small in the captured image for safe evaluation.

The "master-slave" concept can also be used to simplify the necessary visual identifiers of the markers for unique identification with a large number of markers, since too many unique ones

Signs may require too small structures on the markers.

Such small structures are not under all recording conditions

(for example, wide zoom) to clearly recognize and distinguish. In such cases, slave markers whose identifiers are unique only in relation to a master marker can be used. In order to obtain a sufficiently large number of markers in an image even under unfavorable conditions for the surveying camera, individual images can also be stitched together using so-called stitching. In this way, a 3D survey for scenes that can not be captured with a single photogrammetric view (e.g., due to the size of the studio) can also be performed. By stitching wide scenes that do not fit a survey, extended images can be created with a sufficient number of markers. The accuracy in assembling the frames can be improved by overlapping markers contained in multiple frames. Stitching can also be stabilized by means of additional distance measurements between markers (preferably extreme markers on outer positions) or via fish-eye images.

These markers with known 3D position are then used by the

Production camera during the recording (production) of image or video in hold recorded and used to determine the recording parameters. For example, the rough position and orientation of the production camera and its zoom parameters such as focal length can be determined continuously during production without the need for additional devices during production

Position determination needed. These recording parameters can then be used to calibrate the virtual studio, so that virtual contents can be displayed correctly in a recorded scene. This is particularly advantageous if the production camera (or the

Production cameras) is moved and / or pivoted during the recording or whose zoom is changed. When using the proposed calibration or tracking system for virtual studios for generating image or video content, so-called green box, the markings are preferably executed green-in-green, so that they are in the display of virtual content together with the green Background image can be removed. The markers can be made from a picture over their

Color coding will be removed.

However, the proposed calibration and tracking system or the corresponding method is not limited to virtual studios and can also be used elsewhere, for example wherever recording parameters of a camera are to be determined dynamically. Applications are for example in the recording of sporting events or vehicle crash tests. The procedure for through-the-lens calibration of recordings can be used for broadcast video cameras and all other popular cameras (for virutal reality and augmented reality productions).

Special advantages of the presented approach are:

Very easy to implement (no dedicated hardware required)

Markers can be applied almost arbitrarily on a green wall

Position of the markers is almost arbitrary

- No great knowledge of calibration procedures

Can be used in all virtual studios

In one embodiment, a tracking system includes a production video camera (e.g., a broadcast camera)

Surveying camera (can be the broadcast camera, but a special camera is advantageous in handling and for accuracy), markers that have a clearly recognizable from the image marking, and a

Processing unit on. The processing unit receives the images taken by the surveying camera with the markers located in the scene to determine their 3D positions. For this purpose, at least 2 images of the markers of different

Camera positions required so that all markers are visible on all shots. To assist in locating the markers, the real distance between at least 2 markers can be measured and provided to the method. Subsequently, the processing unit performs a structure-from-motion reconstruction of the 3D positions of the markers (corresponds Stereo with unknown camera positions) using the 2 or more shots and the measured distance. Furthermore, the parameters of an intrinsic calibration of the surveying camera are required. This can be done before the recording of markers in the studio and independently. In general, the surveying camera is calibrated once and the intrinsic calibration parameters determined and

stored. These are only properties of the survey camera, are entered into the processing unit before the measurement of the studio and can be used as often as desired.

For intrinsic calibration of the survey camera, a calibration object of known dimensions is used. From this, one or more images of the calibration object are made with the surveying camera for intrinsic calibration of the surveying camera. In this way you can

Camera parameters, such as focal length, optical center, possibly distortions can be determined, which later for stereo measurement with unknown

extrinsic camera calibration using the so-called "structure from motion" - method for the determination of the 3D positions of the markers are used.

FIG. 1 shows an example of a marker with a coded identifier. The basic shape of the marker is circular with two opposite solid sectors meeting at the center of the circle. Detecting rounds

Structures in a picture is particularly easy. Due to the special design of the circle, the center can be determined very precisely. To the

Basic form of the marker around are more visual structures (Blops), here filled circles, arranged. The upper center circle identifies the orientation of the marker and marks its zero position or direction. In the example shown, this will be the zero-degree orientation of the marker

characterized. The further circles can be used to encode the ID of the marker, for example by means of a binary coding. Here, 4 circles are provided at predetermined positions of 45, 135, 225 and 315 degrees with respect to the zero-degree direction, so that the binary value 1111 is coded. By providing a circle at a position, the corresponding binary value is set to one. Of course, more coding points can be used to uniquely identify a larger number of markers. However, then the recognition accuracy for the coded marker IDs is reduced. In order to reduce misrecognition for a larger number of markers, the master-slave principle described above can be used. Of course, other designs of the markers are conceivable. In an application for a virtual studio with green box, the markers may be green-in-green, so that they can easily be removed from the shapes together with the green background.

Once the 3D positions of the markers have been determined in the studio, they can be used during the production of image or video content to calibrate the position, orientation and zoom of the production camera. For this purpose, the image or video data of the production camera with the recorded markers and other picture elements of a

Image processing unit supplied. This determines the marker positions in the images, determines their identifier and assigns the markers in the image via their I Ds their respective 3D position in space. The image processing unit may be separate from the processing unit used to measure the SD positions of the markers. Alternatively, both can

Processing units by an appropriately established

Computing device can be implemented. Due to the known 3D position of the markers and their respective

Image positions can be the recording geometry and thus the

Recording parameters such as position, orientation and zoom of the

Production Camera to be determined. The markers can then be removed from the pictures due to their color coding. The knowledge of the recording parameters obtained in this way (the lens of the production camera) makes it possible to carry out a wide variety of image evaluations and measurements of the content produced, for example when evaluating images of a vehicle crash test. Because the

Recording parameters in real time while recording the image or

Video data are detected by the production camera, these are for dynamic shooting conditions, such as moving or tilted zoom camera, of particular importance. They can be used in a virtual studio with a green box to visualize virtual content with correct orientation to the recording situation. The following are details about intrinsic calibration and about

Reconstruction of marker positions as described in

Embodiments can be used. It should be noted that not all details are required for the execution of the invention and embodiments are suitable without these details for carrying out the invention.

The intrinsic calibration assigns each pixel of a camera one

Direction vector, wherein the 3D coordinate system for the direction vector is anchored in the focal point of the camera. The intrinsic calibration is one Feature of the camera regardless of its position or orientation in the world. The intrinsic calibration is based on at least one image of an object with multiple calibration points and exactly known dimensions. FIG. 2 shows a CNC-produced calibration object with dark calibration points (filled circles) whose positions on the object are exactly known.

To detect the calibration points, the dark areas of the points are detected via an adaptive threshold method and in a binary mask

saved. Adaptive threshold methods do not work with a fixed threshold, but find areas that are darker in their local environment than the environment. Thus, differences in illumination across the object can be compensated. All contiguous areas in the resulting binary mask are then examined for circularity. This can also distortions by the projection of the circles in ellipses

be taken into account. All circles are then filtered again. Only circles of a certain size and only circle groups whose distances between the circles have a certain relationship to the circle diameters are accepted. In this way you will find exactly the calibration points on the calibration object.

The calibration points are then numbered according to their position on the object by rows and columns and assigned a 3D position based on the known object dimensions. The associated world coordinate system is anchored here on the calibration object.

For the intrinsic calibration the model of a projective camera is used. In a projective camera, a world point X is imaged onto the pixel (u, v) on the image sensor chip of the camera, which is located on a beam between the world point and the (virtual) focal point of the camera. The intrinsic camera parameters are determined by the focal point. This is described by the focal length f (distance from the image plane) and the optical center cu, cv (penetration point of the optical axis, which is perpendicular to the image plane). FIG. 3 illustrates the geometric relationships of a projective camera.

At the focal point the 3D camera coordinate system is anchored. in the

Calibration object is anchored to the world coordinate system. Between them, a rotation matrix R and a translation vector T, x = R * X + T, where x stands here for a vector (x, y, z) in camera coordinates, transforms X for a vector in world coordinates.

From Figure 3 takes the geometric relationship From Figure 3 takes the geometric relationship

which expresses that (in units of pixels rather than millimeters) the vector (u, v, f) for one pixel (u, v) and the focal length f extended by an unknown λ times equals the world point transformed into the camera system , u, v here and in the following designate pixel coordinates relative to the optical center, i. u = ub - cu, v = vb - cv, where ub, vb are the usual pixel image coordinates.

To get rid of the unknown factor λ becomes vectorial

Cross roduct of both sides taken and we receive:

Or written out in the coordinates:

v * (r31 * X + r32 * Y + r33 * Z + T3) - f * (r21 * X + r22 * Y + r23 * Z + T2) = 0 (3) f * (rll * X + rl2 * Y + Π3 * Ζ + T1) - u * (r31 * X + r32 * Y + r33 * Z + T3) = 0 (4) u * (r21 * X + r22 * Y + r23 * Z + T2) - v * (rll * X + rl2 * Y + r13 * Z + T1) = 0 (5) where rll etc. the components of the matrix R, T1, 2.3 are the components of the translation vector T, f is the focal length and u, v is a pixel , and Χ, Υ, Ζ are a world point to one of the calibration points on the calibration object. These formulas are also known under the name "Direct Linear Transformation." For a sufficient number of N calibration points, one obtains N equations (5), from which rll, rl2, r13, r21, r22, r23, T1, T2 can be determined The orthogonality conditions of the rotation matrix are given the rest of them

Elements. Equations (3) and (4), again for N calibration points, can then be used to calculate T3 and f. In consequence, the parameters of the surveying camera, such as focal length, optical center, possibly distortions, can be determined.

be determined.

This calibration is determined by linear equations and therefore not yet optimal. To increase the accuracy, a non-linear optimization such as a Gauss-Newton method can be used. At the Gauss Newton method iteratively minimizes an error function of the following type (see eg Wikipedia):

Here, the xi, l..n are the parameters l ... n of the calibration (ie f, cx, cy, R, T); i passes over the calibration points. The function f in (6) are the

Mapping equations of the projective camera from world points

Image coordinates and the yi correspond to the measured pixels (u, v, ie 2 terms per pixel).

With this optimization, 2 distortion parameters of the camera can be optionally allowed for a higher accuracy, which image coordinates to distorted image coordinates:

u '= u (1 + Kir ² + K2I- ⁴ )

v '= v (1 + K ₂ r ² + K ₂ r ⁴ ) (7)

After determining the parameters of the survey camera, the marker positions can be determined using the "structure from motion" method, in which case we have N calibration points with now unknown world coordinates, which are no longer located on a precision calibration object, but rather from In the formulas (3) to (5) u, v (for N points) and f are known, the rest unknown, including the N world points (X, Y, Z) i.

Determining the world points for the markers seems very difficult at first glance. The geometry of camera images, however, contains very strong structures that can also be used here. It is shown geometrically that when shooting a scene from two different positions from and with projective cameras, corresponding pixels (u, v) 1 and (uv) 2 in the first and second images are linked by the following linear relationship:

where F, the so-called fundamental matrix, is a 3x3 matrix. For a sufficient number of corresponding pixels, however, these systems of equations can be solved. Mathematical methods can be found under the keyword "Homography Estimation".

Equation (1) indicates for projective cameras how world points and pixels are related. Applying this consistently from one pixel in shot 1 to the world point and from there into shot 2, one can show that for matrix F, which links these pixels, the following applies:

Here, R, T are the sought-after extrinsic calibration parameters from (1) and K the known matrix of the intrinsic calibration (T is the transpose):

The right side in (9) can thus be determined by point correspondences between the 2 images. It now turns out that there are methods that can be used to factorize KTFK (also called essential matrix) into the R and T matrix on the right. Such a method is e.g. in Richard Hartley, Andrew Zisserman, Multiple View Geometry in Computer Vision,

Cambridge University Press, chapter 8.6.2 "Extraction of Cameras from the Essential Matrix".

If the two camera positions are known, the 3D positions of the world points (markers) can also be calculated by a stereo reconstruction. From (1) this results directly in two lines of sight as straight lines in the room, which can be cut to determine the desired 3D point.

With the marker positions now known, the position R, T and the focal length of the camera for this recording can be calculated for any recording with a production camera. For this one uses again the formulas (3) to (5). It is basically the same task as the

intrinsic calibration. Only R, T were an unimportant byproduct of the computation while these are now the sought sizes (along with the focal length).

Virtual environments and objects are in their own virtual

Coordinate system generated as a 3D world. In order to map this 3D world onto a 2D image, this virtual world is transformed by a virtual camera Virtual environments and objects are in their own virtual

Coordinate system generated as a 3D world. To map this 3D world to a 2D image, this virtual world is captured by a virtual camera with a specific position, orientation, and opening angle (zoom) called rendering.

In order for the view of the objects in the rendered image to match the real camera, the two coordinate systems must be connected. The virtual objects thus receive a position and orientation in the real world, and the position of the virtual camera for rendering then corresponds to the position of the real camera.

This requires the definition of a common coordinate system, which is done by marking selected markers as origin, X-axis and Y-axis. The marker of the Y-axis indicates only the plane of the Y-axis, the actual axis is calculated automatically perpendicular to the X-axis, so that one does not necessarily rely on an exactly orthogonal arrangement of the marker. The Z axis is then dictated by the origin and location of that plane.

In addition, you need the position, orientation, zoom of the real camera in the real coordinate system and, by linking the coordinate systems, so that in the virtual coordinate system. These parameters are provided by the described step of extrinsic calibration from the SD-measured markers.

The above methods can be set up with a suitable

Computing device, in particular a digital image processing device, are executed, wherein the calculation of the respectively sought parameters by means of software programming takes place. Alternatively, special hardware circuits or a mixture of both can be used for this purpose. On the production camera itself, no changes are required compared to conventional models. The requirements for the surveying camera are not particularly high, so that conventional models can also be used here.

It should be noted that the description and figures merely set forth the principles of the proposed device. Based on the present

Revelation, it is possible for the skilled person to create various variants of the described embodiments. These variations, although not expressly described, are also disclosed by this reference and are covered by the claims.

Claims

claims

1. A method for determining camera parameters based on recorded by a camera image data, with

Taking a plurality of images of a scene, the images including a plurality of markers arranged in the scene;

Determining the positions of the marks in the space;

Taking further pictures of the scene, wherein the further pictures contain at least some of the markers arranged in the scene;

Assigning the spatial positions of the markers in the further images to their respective image positions; and

Determining the camera parameters when recording the other images based on the assigned space / image positions of the markers in the other images.

2. The method of claim 1, wherein at least some of the markers allow a unique identification of the respective mark.

3. The method of claim 2, wherein at least some of the markers in the further images are identified so as to determine their previously determined spatial positions and assign them to the respective image positions.

4. The method according to any one of the preceding claims, wherein the determination of the positions of the markers in space by means of a "structure from motion" method.

5. A method according to any one of the preceding claims, wherein the determined camera parameters are at least one of camera position,

Camera orientation and camera zoom included.

A method according to any one of the preceding claims, wherein the images of the scene are taken with a previously calibrated surveying camera and the other pictures will be taken with a production camera.

7. The method of claim 6, wherein the surveying camera is subjected to an intrinsic calibration to determine their camera parameters based on the images of a calibration object with known dimensions.

8. The method according to any one of the preceding claims, wherein the markers have uniquely identifiable master markers and slave markers whose identifier is unique only in spatial relation to a master mark.

9. The method according to any one of the preceding claims, wherein the markings include a coding of their identification number and / or a reference position or direction.

10. The method according to any one of the preceding claims, wherein the markers are arranged three-dimensionally in the space of the scene and their respective three-dimensional positions are determined in space.

A method according to any of the preceding claims, wherein a plurality of captured still images, each containing only a portion of the scene and the markers disposed therein, are assembled into composite images to determine the positions of the markers in space from the composite images.

12. The method according to any one of the preceding claims, wherein the markers are colored in such a way that they can be removed from the further images by means of their color coding.

The method of any one of the preceding claims, further comprising the step of using the determined camera parameters to insert virtual content into the further images.

14. Apparatus for carrying out the method according to one of the preceding claims.

15. System for determining camera parameters, with

at least one camera for taking pictures of a scene, a plurality of markers arranged in the scene, and an evaluation unit which is connected to the at least one camera in order to obtain the recorded images and to determine the positions of the markers in the space on the basis of the obtained images, whereby further images of the scene are taken by the at least one camera and transferred to the evaluation unit Evaluation unit is designed for