BR112014027853B1 - MARKERLESS SYSTEM AND METHOD FOR REAL-TIME MIXING OR COMPOSITION OF COMPUTER-GENERATED 3D OBJECTS AND A VIDEO SIGNAL FROM A VIDEO CAMERA - Google Patents

Abstract

SYSTEM AND METHOD FOR REAL-TIME MIXING OR COMPOSITION OF COMPUTER-GENERATED 3D OBJECTS AND A VIDEO SIGNAL FROM A FILMING CAMERA, for treating the invention of a method of real-time mixing or compositing of computer-generated 3D objects and a video signal from a film camera, where the body of the film camera can be moved in 3D and sensors on or attached to the film camera provide real-time positioning data defining the 3D position and 3D orientation of the film camera. filming, or enabling the 3D position of the filming camera to be calculated.

Description

Technical Field

[001] The present invention relates to a real-time mixing or compositing system of computer-generated 3D objects and a video signal from a film camera, such as a video camera, for generating a video in real-time with augmented reality for broadcast TV, cinema or video games.

Description of Prior Art

[002] Over the past 20 years, there has been considerable research and commercial activity in this field; Reference may be made to video insertion or augmented reality systems from companies such as Sportvision, Inc., which has developed television viewing enhancements for sporting events such as American football, adding a virtual first line of bass that viewers can be observed superimposed on the field. Other companies that have developed systems in this area include PVI, Inc. A common feature of the known systems is that they rely primarily on analyzing the characteristics of the camera's video signal to determine which part of the real-world scene the camera is pointing at. ; the system needs to create in advance a 3D map of the scene, in such a way that once it knows where it is pointing in the scene, it is able to add or compose computer generated objects (first virtual line from the bottom) in the video signal, in such a way that the position of the object and its orientation appear to be a natural part of the scene. Therefore, a disadvantage of relying exclusively on optical flow lies in the fact that systems of this type can be unreliable.

[003] Other systems rely on purely marker-based approaches (Lightcraft Technologies, for example). These require an operator to place real physical markers (with dimensions of 1 m x 1 m) on the scene, to be detected by the system. They are highly inefficient, as they require hours or days of work to set up a scenario, which is very unlikely to be done in film production. They also have many physical limitations, as the markers always need to remain in the system's field of vision. Scientific references can also be made to the documents cited in Annex 1.

Brief Description of the Invention

[004] The present invention consists, in one aspect, of a system for mixing or compositing computer-generated 3D objects in real time and a video signal from a film camera, such as a video camera, for generating a real-time augmented reality video for broadcast TV, cinema or video games, in which: (a) the film camera body can be moved in 3D and sensors on or connected directly or indirectly to the film camera provide data real-time positioning data that define the 3D position and 3D orientation of the filming camera, or that allow the 3D position and 3D orientation of the filming camera to be calculated, and (b) real-time positioning data are then automatically used by the system to create, recall, render or modify computer-generated 3D objects and (c) the resulting computer-generated 3D objects are then mixed or composited with the video from the filming camera to provide a reality video enhanced for broadcast TV, cinema or video games.

[005] Optional features - some or all of which may be combined with each other, include the following: • computer-generated 3D objects are mixed or composited in real time with the real-time video from the filming camera. • The filming camera's real-time zoom, focus and iris settings are measured and used, along with real-time positioning data, such that 3D objects are correctly rendered in a desired location and orientation in a scene 3D. • Sensors include an accelerometer and a gyroscope (6DOF sensor). • The sensors include a 3-axis accelerometer measuring translational acceleration in 3D, a 3-axis gyroscope measuring angular velocity in 3D, and a magnetometer measuring absolute height in 3D and, consequently, constituting a 9DOF sensor. • Sensors include a 3D range sensor, such as a structured light or time-of-flight camera. • 3D range sensor captures the depth of each pixel in a camera's video output. • Edge depth is refined by back-projecting depths from the 3D range sensor onto the high-resolution video from the movie camera. • The sensors are formed into a unit that can be firmly attached to the film camera. o The unit includes one or two proof cameras. o The unit includes a 3D range sensor that captures the depth of each pixel in a video output. o The unit includes a 6DOF or 9DOF sensor. o The unit uses switchable brackets to allow it to be attached to different sizes and models of film cameras and booms. o The unit is extendable to include other types of sensors. o The unit can form an inspection device that can be used to inspect a complex scene and transmit data defining the 3D scene being inspected wirelessly to a computer that then tracks or recovers tracking of the scene. o The movie camera includes conventional encoders to read the camera's zoom, focus, and iris. • The system includes a single proof camera (in the monoscopic case) or two proof cameras (in the stereoscopic case), equipped with lenses that may consist of 180-degree fisheye lenses. • The test camera(s) is displaced from the filming camera and this displacement is obtained using a calibration card including a first and second group of circles, each circle being in a location known relative to one or more of the other circles, and each circle being recognized using a blob image algorithm. • The film camera lens is calibrated for optical distortion using a calibration card including a number of circles, each circle being in a known location relative to one or more of the other circles, and each circle being recognized with the use of a blob image algorithm. • The Filming Camera consists of one of the following: crane camera; Steadicam camera; handheld camera; cart mounted camera, tripod mounted camera, smart phone, augmented reality glasses. • The 3D position and orientation of the filming camera is determined with reference to a 3D map of the real world generated, in part using real-time 3D positioning data from sensors, and also from an optical flow in which the( The proof camera(s) inspects a scene and software running on a processor detects natural markers in the scene ("feature points") that have not been manually or artificially added to this scene. • The system uses one or two high-speed test cameras (at least 100 fps) to allow the system to be fully initialized without a separate step of just inspecting the scene to be tracked (called "instant inspection"), but instead, the inspection is performed continuously as the camera is being used to capture video. • The stereoscopic system allows the software to process the images and, even if the camera system is not moved as a whole, it can be used to generate an instantaneous 3D point cloud (for example, associating a large number of points from the scene to its position in 3D space using knowledge about the separation between the two cameras and epipolar geometry). • The depth of each pixel of the 3D point cloud is obtained using corresponding 2D texture fragments obtained from each test stereo camera and an epipolar line search algorithm. • The system runs a fusion algorithm that combines optical flow data from the probe camera system with real-time positioning data from hardware sensors. • The fusion algorithm relies on an Extended Kalman Filter prediction/correction technique to integrate the outputs of, and to recalibrate, all sensors, which may include an accelerometer, a gyroscope, a magnetometer, a 3D range sensor, to determine the position and orientation of the camera. • The EKF fusion algorithm uses confidence level data, associated with the output of each sensor, to determine how to merge data from each sensor. • The keyframes generated by the proof camera system are part of the visual tracking process, and consist of real-time images computed at 4 different resolution levels from the proof camera video signal. • The system comprises (a) a content generation computer that provides 3D animations of computer-generated virtual figures, objects and locations, and (b) a rendering computer (which may or may not be separate from the content generation computer ), in which the real-time positioning data that defines the 3D position of the filming camera is used by either or both the content-generating computer and the rendering computer to cause 3D objects to be computer-generated in real time and can be input and mixed in real time with the video signal from the film camera to form a natural part of the scene shown in that video signal. • Computer-generated 3D objects consist of animations that can move anywhere within the scene and can change their shape and appearance in a manner determined by the content-generating computer. • Computer generated 3D objects consist of animated figures of people or creatures that move (e.g. run, dance, walk, fight, fly, jump,...) realistically when mixed into the scene. • Camera positioning or tracking data is also made available for use in post-production to facilitate post-production CGI. • The 3D range sensor is used to increase the accuracy of the depth measurement associated with a reconstructed 3D point, or to reject this reconstructed 3D point. • The 3D range sensor is used for real-time depth keying to enable dynamic occlusion and suppress the potential use of a green stage. • The system uses a small camera recording object, such as a card of known size and covered with a known pattern, placed in the scene in such a way that a corner of the detected pattern is treated as the origin of the 3D point cloud (and , therefore, of the world). • The camera registration object comprises at least two spheres of known size arranged in a true vertical and recognized using a blob image recognition algorithm. • The system comprises a magnetometer to indicate magnetic North, an accelerometer to indicate the direction of gravity (and thus provide the true vertical), a gyroscope to indicate whether the system is tilted up/down, or whether it has been panned left or right, or whether it has been rotated around the optical axis, and a 3-axis accelerometer to allow 3D translation from an initial position to be deduced. • The software tries to generate a cloud of points evenly distributed on the 3D map to significantly reduce tracking losses and to increase tracking accuracy (a greater parallax is generated, and thus the estimated camera position is more accurate). • The camera tracking system can be connected wirelessly to the camera system and can therefore be quickly moved around the array when generating the 3D point cloud - without the need for cable tracks over the array, unlike systems from previous test cameras. • The camera tracking system combines real-time inspections of a 3D scene (in the monoscopic case) or instant inspections (in the stereoscopic case) with tracking of the filming camera as the director/camera operator tracks, produces panning movements and tilts the camera tracking system attached to the film camera. • The system fully automates all aspects of film camera tracking, including rotation, translation, focus, iris, focal length; and automates the sizing, positioning and orientation of computer-generated 3D content to be mixed with the video. • The system allows continuous inspections of a scene in real time, to generate a more complete point cloud defining the scene. • The system couples rotation invariant descriptors, for example, using ORB, to characteristic points detected in the scene to facilitate tracking recovery. • The system uses a constant velocity model, associated with the information provided by the sensors, to predict the next position of the filming camera using the correctly computed or previously confirmed position. It uses this prediction to reproject the 3D point cloud into the current frame, to enable a point matching algorithm to match the points identified in the real-time video signal from the test camera system and the points projected in the new cloud, of 3D points. • The system uses a camera tracking Levenberg-Marquardt minimization scheme to minimize the error between the points identified in the real-time video feed from the test camera system and the points projected in the new 3D point cloud. • The user can use the 3D point cloud generated by the camera tracking system to define 3D masks, such as Garbage Match 3D masks. • 3D objects comprise static objects, dynamic animations, virtual worlds, virtual people, virtual buildings, virtual sets, virtual sets, and all data from an animation database. • The film camera and test camera are calibrated for frame acquisition delay using a modulated light source, such as comparing light intensity curves associated with a flashing LED. Other Concepts - each of them can be combined with any of the characteristics described above, or with any other concept defined below.

[006] A method of mixing or compositing real-time computer-generated 3D objects and a video signal from a film camera, in which the body of the film camera can be moved in 3D and sensors on or connected to the camera provide real-time positioning data that defines the 3D position and 3D orientation of the camera, or that allows the 3D position to be calculated.

[007] A method of mixing or compositing in real time, computer-generated 3D objects and a video signal from a film camera, such as a video camera, to generate augmented reality video for broadcast TV, cinema or video games, in which: (a) the film camera body can be moved in 3D and sensors on or connected directly or indirectly to the film camera provide real-time positioning data that defines the 3D position and 3D orientation of the filming camera, or that allow the 3D position and 3D orientation of the filming camera to be calculated, and (b) that real-time positioning data is then automatically used by the system to create, recall, render or modify objects computer generated 3D and (c) the resulting computer generated 3D objects are then mixed or composited with video from the film camera to provide an augmented reality video for broadcast TV, cinema or video games.

[008] Methods or systems such as those defined above and used to enable a director (or a director of photography) to frame on a camera the pre-production of computer-generated 3D materials, typically pre-visualization materials or visual effects in real-time footage over the live action footage or video footage recorded by the camera, enabling the director to explore possible camera angles and movements in real time, with the computer-generated 3D materials being automatically mixed into the video observed by the director.

[009] Methods or systems such as those defined above, in which real-time positioning data is recorded and stamped with time codes to provide motion coincidence data for post-production processes.

[010] Methods or systems such as those defined above, used to enable virtual objects or images to be inserted into a video transmission stream.

[011] Methods or systems as defined above, used to enable one or more of the following actions: • Real-time tracking of studio cameras • Real-time tracking of a Steadicam • Real-time tracking of crane-mounted cameras • Tracking real-time from cart-mounted cameras • Real-time tracking from a Steadicam • Real-time tracking from Offsite Broadcast (OB) • Using real-time data (e.g. tracking data) for 2D post-production • Usage of real-time data (e.g., tracking data) for post-conversion to stereoscopic 3D content • Using real-time data (e.g., tracking data) for native stereoscopic 3D content • Inserting 3D graphics • Insertion of 3D graphics for studio or on-site placement • 3D graphics insertion for OB • 3D graphics insertion into other sponsored images • 3D graphics insertion specific to user locations • User specific 3D graphics insertion • Specific 3D graphics insertion at times • Insertion of 3D graphics to fill in green crowd scenes • Insertion of 3D graphics to replace teaching aid screens, • Insertion of graphics in museums and 3D centers with educational interpretation content in cultural, historical and natural - Measurement of the absolute or relative size of objects in a scene.

[012] Methods or systems as defined above, in which the filming camera consists of one of the following: - All cameras with a standardized safety strap - Cameras requiring tactical fiber optic connection - Cameras requiring RF/wireless connection

[013] Methods or systems as defined above, implemented in one of the following markets: - Film/TV (non-live) - Commercial (non-live) - Live commercials - Broadcast (non-sports) - Broadcast (OB) - Studio-Based Sports - OB-Based Sports - Live TV Product Placement - Internet Use (Non-Live) - Live Internet - Territory-Based Live Internet - Live Internet Product Placement - Museum Content /heritages - Commercials about museums/heritages - Architecture - Games

[014] Methods or systems as defined above, used to allow augmented reality images to be displayed by any viewing device, including a smart phone, augmented reality glasses, the appearance of the augmented reality images being automatically changed depending on the real-time positioning data.

[015] A film, film, TV show or video game, in which real-time computer-generated 3D objects are mixed with a video signal from a camera, and the camera body can be moved in 3D and sensors on or attached to the camera provide real-time positioning data that defines the 3D position of the camera, or that allows the 3D position to be calculated.

[016] A scene inspection and tracking device adapted to be coupled to a conventional camera, in which the camera body can be moved in 3D and hardware sensors in the scene inspection and tracking device provide real-time positioning data that define the 3D position of the camera, or that allow the 3D position to be calculated.

[017] A handheld or portable camera including sensors on or attached to the camera that provide real-time positioning data and that define the 3D position and 3D orientation of the camera relative to a frame of reference, or that allow, in full or as part of a system that analyzes other data, such as optical flow data, so that 3D position can be calculated.

[018] A filming camera, including or connected to a stereoscopic proof camera system, the proof camera system generating wide-angle (e.g., 180-degree) stereoscopic images, allowing the software to process the images, and, without the camera system as a whole being tracked/moved, it generates an instantaneous 3D point cloud.

[019] A tracking system to be coupled to a film camera, with the test camera system generating stereoscopic images, allowing the software to process the images and, without the camera system as a whole being moved, generate a instant 3D point cloud and provide real-time tracking (position, orientation, zoom, focus and iris) of the filming camera.

[020] Augmented reality glasses including sensors in or attached to the glasses to provide real-time positioning data, which define the 3D position and 3D orientation of the glasses relative to a frame of reference, or which allow, in whole or as part of a system that analyzes other data, such as optical flow data, so that 3D position can be calculated

[021] The system currently implemented is called Ncam. Ncam's various applications include the following: Visual Effects in Film, TV and Commercials

[022] Visual effects in pre-production and production: the NCAM system aims to help bridge the gap between previews (pre-production) and VFX (post-production) during the recording process (production). The NCAM system offers a solution for those who want to frame in-camera VFX while recording in real time. Using previously created content, often from the preview process, the NCAM system is able to render these preview materials, typically 3D models and animations, onto the live action footage board in real time.

[023] Using carefully crafted and approved previews as a whole can save enormous amounts of time. Filmmakers can frame pre-visualizations or VFX, either by shooting on green screens and composing a virtual background, or by superimposing a creature or object in the foreground. Film producers then have the ability to take back control of pre-visualizations and VFX by exploring possible camera angles and movements in the moment and in real time, without the usual restrictions.

[024] In turn, the data is recorded and time-stamped, giving the department's VFX camera motion coincidence data (the 6 DOF track), while also providing an editorial with a 'temp' of the final VFX .

[025] By capturing the VFX 'temp' in camera and pre-designing the VFX in the process, it is possible to eliminate a lot of VFX guess work in the post-production process. We can reduce the VFX design process, thus avoiding waste. Virtual Advertising - Live Broadcast

[026] NCAM technology lends itself well to real-time virtual advertising with broadcast space. Digital product placement, in which a product is inserted after recording during post-production, as opposed to an actual physical product recorded on camera, is becoming more popular. However, with the ability to increase live digital product placement, we are able to open up several possibilities. Examples might include brand logos on sports fields or canned drinks on soap operas. This is the environment in which the ability to render photorealistic CG images, in real time, based on the lighting of shooting environments, becomes critical.

Holographic Games

[027] The combination of NCAM technology with holographic screens and glasses (such as Google Glass) can provide a fully immersive gaming experience, mixing the real world with virtual worlds and characters. The possibilities are practically limitless. Museums &Heritages

[028] NCAM technology offers a wide range of potential applications for the museum and heritage sectors.

[029] As an exploratory tool, it could be useful in reconstructing heritage sites such as ruins, showing what the site might have looked like in its glory days.

[030] Inside museums, NCAM could be used as an educational tool, perhaps to demonstrate the size of a dinosaur and its movements, its internal organs or its skeletal structure. Another example could be exploring the operation of an integral combustion engine, effectively visualizing an animated exploded diagram, but in a completely immersive 3D space.

Virtual Camera - VCS

[031] NCAM technology lends itself perfectly to this application. The virtual camera essentially consists of an LCD screen that displays a full 3D CG environment. As the operator moves the LCD screen, it behaves as if it were a physical camera in terms of movement. The LCD camera is tracked in XYZ translations and rotations and shows the entire CG environment from a CG unit, in real time. There are currently several VCS solutions (virtual camera systems) available, but they all have limitations in terms of technology. They tend to be time-consuming to install, limited in space and the environment in which they will operate, and are expensive. An NCAM-based VCS would likely be in the form factor of a tablet, with processing calculated locally in combination with an LCD screen. CG computations could be performed from a remote PC or possibly locally depending on the requirements. The idea is to enable open access through usability and price points to various departments within a film, who previously did not have the ability to access these tools. This feature is very useful for designers, architects, VFX, gaming companies, and CG and animation companies, etc.

[032] This VCS also forms the backbone of the potential solution for museums and heritage sites.

Virtual Recognition System - VSS

[033] Imagine filming a movie and it being shot predominantly on a blue screen/green screen. How could I, as a film producer or director, determine the best angles and shots for filming? After all, I may have a small amount of physical space and a few actors, but I have no idea, beyond my imagination, which camera angles would be best, let alone how to explain to the entire crew where we should position the cameras and how to make a valid choice of lenses. These days we would transport a film camera and all its accessories to a filming location, set up the Ncam and take a look, only to realize that another position would be best. If we only had a portable, lightweight system to make these decisions quickly and efficiently, this is Ncam VSS.

[034] VSS is like Ncam today, but built into a small form factor camera - think DSLR. It essentially consists of a digital display with the benefits of Ncam. We are combining all the elements of Ncam as it is today, into one filming suite, but on a handheld device. The virtual production of TV episodes would also benefit greatly from this tool, as museum/heritage websites as well as architects could showcase their new creations to potential investors.

Brief Description of the Drawings

[035] Below is a brief description of the drawings according to the present invention. - Figure 1: Hardware description for configuration 2.3. Note that the version drawn here is the one with the tracking workstation and rendering workstation combined. - Figure 2: Global tracking algorithm for obtaining the position and rotation of a filming camera in real time. - Figure 3: The self-leveling 3D Registration Object created for our system. This object is used in the real scene to automatically compute the origin, orientation and size of the real world in our software. A planar version is shown in the right image of Figure 5. - Figure 4: The Pyramid Image used for the detection of natural markers. This figure also shows some 2D texture fragments associated with the markers. - Figure 5: Calibration cards developed and used in Ncam. Left: Calibration card used to calibrate all lenses. Medium: calibration card used to align the computation between a film camera and a test camera. The card on the left can also be used for cases in which the focal length of the film camera is short enough (experimentally decided on site). Right: planar version of the registration object. Easier to handle and can be placed anywhere on the ground surface. Circles provide an orthonormal basis by providing world orientation, scale, and origin (0,0,0) when adding virtual objects to the real scene. The center of the upper left circle consists of the origin of the world. - Figure 6: The Camera Bar to be mounted on a film camera with the different sizes of supports (for 15 mm and 19 mm rods). - Figure 7: The wireless inspection device. When an inspection is required (e.g. in the monoscopic case, (see section 2.1, or in very complex assemblies (see end of section 2.1.3))), the entire tracking process is then achieved using this device inspection. When the inspection is complete, all tracking data (point clouds, keypoints, etc.) is transmitted wirelessly to the Tracking/Rendering Workstation, which instantly retrieves the tracking and begins streaming out the data tracking and the final composite result. - Figure 8: The Camera/Computer Junction Box. Note that a version using several RJ45 cables is illustrated. All of these junction boxes also have a variant in which these cables are replaced by a single tactical fiber optic cable.

Detailed Description of the Invention

[036] The technology of the present invention relates to mixing computer-generated 3D objects in real time and transmitting real live video from a real camera (referred to as a film camera in the remainder of this document) at a filming location, TV broadcast or video game. The main objectives of this system are to: - Provide a real-time framing tool for virtual characters and environments for film directors. - View the real-time photorealistic result of virtual objects directly mixed with the real video. - Provide an accurate estimate of the position of a filming camera (called camera tracking), both for on-location recording and post-production. All film cameras are considered: crane cameras, Steadycam cameras, handheld cameras, cameras on tripods, etc. - Provide an approximation of 3D geometric modeling of the real scene, handling occlusions between real objects (and/or actors) and the virtual environment (3D objects, characters, etc.).

[037] The technology described here consists first and foremost of a powerful sensor fusion algorithm, thus mixing natural markers (i.e., not added manually to the physical scene) computed from 3 (in the stereoscopic case) or 2 (in the stereoscopic case). monoscopic case) different optical flows and the physical measurement of 4 other sensors (gyroscope, accelerometer, magnetometer, 3D range sensor). The technology described here also consists of a completely non-invasive system, which does not require the manual addition of any physical markers or other objects directly visible in the real scene, to carry out its work.

1 Hardware

[038] The technology presents 2 different possible hardware configurations, depending on the type of data that needs to be transmitted out.

[039] All of these 2 configurations have in common the following hardware, mounted on a device called a camera bar (see Figure 6 with detailed diagrams), as well as a separate connection box, which essentially consists of a junction box mixing or separating all the different types of cables into one or several independent harnesses (see Figure 8 for detailed diagrams): • One or two high-speed cameras (at least 100 fps), called test cameras, depending on the selected approach (monoscopic or stereoscopic). • Each test camera has a 180-degree fisheye lens mounted on it. • A 9-DOF sensor, including a gyroscope, magnetometer, and accelerometer (or a 6-DOF when the magnetometer cannot be used). • A 3D range sensor that captures the depth of a pixel.

[040] In addition to the camera bar, all configurations feature regular lens encoders mounted on the filming camera to read zoom, focus, and iris values. They also have in common a wireless inspection device (see Figure 7 with detailed diagrams) to inspect the scene and learn about it. However, note that in the case of a full stereoscopic approach (see section 2.5), and more precisely an instantaneous inspection (see section 2.5.1), this inspection device is not necessary.

[041] In addition to these shared features, the 2 configurations are described as follows: Configuration 1: Camera Transmission 1. Wireless Onboard Computer: (a) An Onboard Tracking Computer is mounted on the filming camera. It wirelessly transmits the translation and rotation of the film camera (RT) to an external rendering workstation, as well as the zoom, focus and iris (ZFI) originating from the encoders. (b) An External Content Generation Workstation. This machine typically runs computer graphics software to provide computer-generated 3D animations and relevant CG content for the final composite. It also transmits animation data to the external rendering workstation (1- lc). (c) An External Rendering Workstation, which on the one hand receives data from the onboard computer (1-la), and which on the other hand also manipulates the animation data coming from the content generation station (1-lb) . This rendering workstation uses the RT+ZFI movie camera from 1-la to display the 3D virtual environment from 1-lb, and mixes the result into the real video stream from the movie camera. It also transmits the final composite result back to the film camera's eyepiece or to a control monitor wirelessly. 2. Wired Onboard Computer: (a) An Onboard Tracking Computer is mounted on the filming camera. It wirelessly transmits film camera translation and rotation (RT) and encoder data (ZFI) to an external rendering workstation using tactical fiber optic cables. (b) Similar to Configuration 1-1b. (c) Similar to Setup 1-1c, except that the final composite result is sent back to the camera via an HD-SDI cable. Configuration 2: Full Transmission 1. Wireless embedded computer: (a) An embedded tracking computer is mounted on the filming camera. No actual processing is performed on this unit. It simply wirelessly transfers all data from the sensors to an external tracking station. 2 x video streams from the proof cameras, 1x9DOF data, 1 x range sensor data are streamed out. The total amount of data requires a minimum bandwidth greater than 142 MB/sec (proof cameras require 63 MB/sec, range sensors 74 MB/sec, and the 9DOF sensor 4.4 MB/sec). (b) An External Tracking Workstation that receives data from the onboard tracking computer. This unit processes all sensor data to calculate the rotation and translation matrices of the filming camera. It outputs the RT+ZFI Camera to the external rendering workstation. (2-ld). (c) A Content Generation Workstation. This machine typically runs computer graphics software to provide computer-generated 3D animations and CG content relevant to the final composite. It also outputs the animation data to the external rendering workstation (2-1d). (d) An External Rendering Station that on the one hand receives data from the onboard computer (2-1b), and on the other hand also processes the animated data coming from the content generation workstation (2-1c). This rendering workstation uses the RT+ZFI of the movie camera coming from 2-1b to display the 3D virtual environment coming from 2-1c, and mixes the result into the real video stream coming from the movie camera. It also transmits the final composite result back to the film camera's eyepiece or to a control monitor wirelessly. 2. Wired embedded computer: (a) An embedded tracking computer is mounted on the filming camera. It outputs the translation and rotation (RT) of the film camera to an external rendering workstation (2-2c) using tactical fiber optic cables or RJ45 Cat-6 multiple cables. Again, encoders are used for zoom, focus and iris (ZFI). (b) Similar to Configuration 2-1c. (c) Similar to Setup 2-1d, except the final composite result is sent back to the camera via an HD-SDI cable. 3. No onboard computer (see Figure 1 for a graphical illustration: all other versions can be easily derived from this one): (a) A single camera tracking device (camera bar) is attached to the filming camera, and connected to the junction box. (b) An external tracking workstation that receives data from the onboard tracking computer. This unit processes all sensor data to calculate the rotation and translation matrices of the filming camera. Encoders are again used for zoom, focus and iris (ZFI). It transmits the camera's RT+ZFI out to a 2-3d rendering workstation. (c) Similar to Configuration of 2-1c. (d) Similar to Setup 2-1d, except the final composite result is sent back to the camera via an HD-SDI cable.

[042] In addition to the previous configurations, each version also has two other variants. One of them has the Content Generation Workstation joined to the External Rendering Workstation, and the second variant has the External Rendering Workstation joined to the Tracking Workstation.

2 Software

[043] Several new scientific methods and software have been created to achieve accurate real-time tracking and augmented reality results [19]. These methods are described in the following sections.

2.1 Markless Monoscopic Tracking of the Video Signal 2.1.1 Process overview

[044] The monoscopic technique used in this technology is based on two separate steps that are, in fact, technically practically the same, except for the fact that one of them requires user interaction while the other is completely automatic.

[045] The first step is called inspection. A user uses a device built on the hardware described in the second paragraph of section 1 (see also Figure 7) to scan the real scene and the recording object. During this phase, natural markers are automatically learned and calculated in 3D by the system as the user scans the environment by walking around. As soon as the user starts with the inspection, he must also capture the registration object (see section 2.1.3.2) for the purpose of automatically estimating the scale, origin and orientation of the world. Once this process has been carried out, the user can continue to scan the rest of the scene to build the best possible 3D point cloud from natural markers (a natural marker is a point that is invisible to the human eye and is calculated by our algorithms ). The more thorough this inspection is, the better the final result will be. The inspection part is the most important task in the process and is highly critical.

[046] The second step consists of tracking the filming camera (see section 2.1.4) which is carried out from the tracking device that was placed on the filming camera. Different settings apply, following the description in the previous section titled "Hardware" and according to various situations. This step is completely automatic and generally does not require any human interaction.

2.1.2 Algorithm View

[047] The technology described here consists of an augmented reality framework based on a multiple sensor fusion technique (see section 2.3).

[048] In fact, it does not rely on the unique and usual optical flow data to track a camera and insert a virtual object. Such techniques have demonstrated their scientific and practical limitations in many cases. For example, if a user hides the camera used for tracking, then the tracking fails and is lost. This is exactly the same problem that arises when the camera points to the region that has not been inspected (see section 2.1.3). Furthermore, even if the filming camera is not moving, there are still some unavoidable issues of slight oscillations due to the accuracy of the natural markers detected and computed by the purely visual tracking algorithm. Data filtering partially solves the problem, but a perfectly stable virtual object is never obtained, especially when using very long focal lenses. As our technology uses other sensors to know whether the camera is moving or not, our tracking algorithm does not present this problem.

[049] Regular optical flow techniques such as SLAM [9], PTAM [8], RSLAM [13], etc., use information contained in the video captured by the camera to approximate its position and orientation. Our technology presents a similar approach to its visual tracking part (called visual tracking), except for the fact that, on the one hand, the core of the tracking does not consist of a simple search for keyframes when they are lost, and on the other hand, it uses all the data from the 7 available sensors (a film camera, two test cameras, a gyroscope, an accelerometer, a magnetometer and a 3D range sensor) to compute the precise position and orientation of the camera.

[050] We use an Extended Kalman Filter (EKF, see section 2.3.2) [7, 21] to integrate the following sensors: gyroscope, accelerometer, magnetometer sensors and probe cameras. The EKF technique is the heart of our software and all prediction/correction algorithms are based on a data fusion method that allows you to use the best of each sensor. This method provides unparalleled robustness for simple and critical cases where all other techniques have failed. In fact, when a sensor has lost track (currently meaning its measurements are no longer reliable), the EKF is still able to obtain a position/orientation estimate by mixing data from the other reliable and remaining sensors. Furthermore, as each sensor has a confidence level, this has an impact on the fusion algorithm in order to prevent the system from using inaccurate information. For example, a common gyroscope has a natural fault called drift, which tends to change its orientation even when it is not in motion. The more time passes, the greater the error generated by the gyroscope. Gyroscope drift correction is performed using visual tracking, and visual tracking is corrected by the gyroscope when its error becomes too large. The EKF is therefore able to find the best position and rotation of the Camcorder by predicting the movements of the Camcorder using previously correctly estimated values, and then correcting its prediction according to the new measurements provided by all sensors. .

[051] Furthermore, purer optical-based tracking techniques use keyframes (i.e., captures) to locate the camera when it is lost. If there is no keyframe of the current view you are pointing the camera at, then the recovery system fails. More robust techniques can naturally be used, such as SIFT [11] or SURF [1] to solve cases in which you simply rotated the camera vertically, for example. In fact, as all these systems compare the current frame with the most similar keyframe, it is very interesting to use rotation and scale invariant descriptors to obtain a better and faster match. However, if there is no keyframe for the position the camera is in, there is no way in which the system can recover its position. Our technique uses the best of each technique (note that we use ORB [18] as keypoint descriptors), and at the same time we perform three different tasks to recover the lost track (see section 2.1.4.4 for more details). The retrieval process can be viewed as a global algorithm based on a probabilistic and stochastic approach. When tracking is lost, 3D space is instantly sampled around the last known positions using an ellipsoidal approach, and all keyframes within this ellipsoid are compared to the current keyframe. Additionally, all other data coming from the remaining sensors (gyroscope, accelerometer, magnetometer, 3D range sensor) is still merged into the search, allowing our software to disambiguate all plausible good candidates. Of course, if no keyframe is a good enough candidate, then the system will use all other sensors besides visual tracking to calculate an approximation of the current position. The obvious consequence of this technique lies in the fact that our technology allows the user to point the camera at a location that has not been inspected, without losing tracking. However, our algorithms never stop sampling 3D space to find a keyframe (i.e. best approximation of RT) that matches the current capture. If a visual track is retrieved, all other sensor data is used and updated to correct its own information in addition to the camera information (see Extended Kalman Filter, section 2.3.2).

[052] The most recent (i.e. regular) version of our technology is stereoscopic (see section 2.5), which means that it uses two test cameras spaced 21 centimeters apart (see Figure 6). This technology does not require any calculation of a homography or any manual translation of a single proof camera (see section 2.1.3) to calculate the depth information of natural markers detected in the video signal. This method simplifies our main algorithm, making the inspection phase obsolete in most cases (see section 2.5). As with the monoscopic technique, visual tracking itself presents a complete subpixel approach, allowing the system to track the camera position at 4 different levels of an image pyramid (see Figure 4), whereas all other flow-based technologies optics mistakenly use a pixel approach at just two different image resolutions when searching for feature points.

2.1.3 1st Phase: Inspection 2.1.3.1 Initialization Step

[053] The first step consists of the user making a physical/manual horizontal translation with the device in order to capture two frames. The algorithm automatically detects natural markers (i.e., characteristic points) of images using an edge detection method called FASTER [17]. A match is applied to find the correspondence between pairs of points. For an image, all detected points must remain in the same 3D plane to allow calculation of a correct homography of that plane for the image. This provides rotation and translation of the camera in relation to the 3D plane. The second image follows the same principle and with this we obtain a second position and translation of the camera for the current image. It is now possible to compute the transformation from one camera to the other and obtain a first 3D point cloud.

[054] Let's consider two points of view (Left and Right) captured by a camera. For each view, the test camera was pointed at a point M located on a plane. ML and MR are the perspective projections of M in left and right views, respectively.

[055] We can write: where: • R is the rotation matrix by which ML is rotated with respect to MR. • N (A, B, C) is the normal vector of the plane to which point M belongs. • D is the distance to the plane such as Ax + By + Cz + D = 0.

[056] Solving H in equation 1 results in:

[057] Our algorithm requires a minimum of 4 coplanar points to perform a correct initialization. It is quite common that we currently have many more points than these. This is why we compute n possible homographies using 4 points among the entire set of points for each homography. We then use a RANSAC method [4] to find the best possible homography and construct a set of suitable and unsuitable candidates. Inadequate candidates will be rejected while suitable candidates will be refined by a Gauss-Newton method minimizing reprojection errors of points in both views. Having the best homography calculated from the filtered set of points is now necessary to estimate the rotation and translation matrices between the left and right views. This is easily achieved using the decomposition of singular values into two orthogonal matrices U, V and a diagonal matrix Y. The matrix H can now be written as: with:

[058] As we do not deal with cases of indeterminate homographies, Equation 3 presents two possible solutions: x1, x2 and x3 are the eigenvalues of the matrix H of equation 2, and classified as x1, x2 and x3 and x1 F x2 F x3. • Solution 2: x1, x2 and x3 are the eigenvalues of the matrix H of equation 2, and classified as x1, x2 and x3 and x1 F x2 F x3.

[059] In addition to the set of characteristic points, the algorithm also captures two key positions as snapshots of what the test cameras view from each point of view. Learning additional points always consists of the sum of three components: new detected points, key positions and fragments, which are described in section 2.1.3.3.

[060] Finally, it should be noted that in the case of a stereoscopic approach, (see section 2.5), the inspection phase becomes automatic and instantaneous. A manual inspection can still be used for extremely complex cases (experimentally decided on site), but this remains anecdotal.

2.1.3.2 Automatic scaling, origin and orientation of the 3D World

[061] During a monoscopic inspection, it is not possible to compute the exact origin, dimension and orientation of the world, as this process would require either prior knowledge of the real object including its shape and exact dimensions, or the distance used between the two views to compute the homography. Note that in the latter case this would not provide the origin or orientation of the world anyway. Most systems don't take this part into account. They often ask the user to scale manually, entering the dimensions of a 3D object visible in the videos. Other options allow a user to scale 3D virtual characters within specific modeling/animation software during filming, and also properly align it with the real world. However, any error in scale - even small - has dramatic impacts on the quality of the final composition, which is why the process is so carefully executed with our technology (see below). One of the most common observed consequences of poor sizing is a feeling for the user that virtual objects are sliding on the floor, instead of being perfectly coupled to it.

[062] We propose a new method that requires placing a proprietary recording object in the scene while performing the inspection. This object is automatically detected because it has a very well-known shape, and all its dimensions are also known. Absolutely no user interaction is required.

[063] Our registration object is produced from several parts, which essentially consist of spheres and legs, as shown in Figure 3. The spheres are essential parts of this object in our system, as they allow the use of blob detection algorithms to recognize the pattern through the video. The centers of gravity of the projected spheres (i.e. circles) are then computed, and because we precisely know the dimensions and distances between all the spheres, the system is able to provide a very accurate scale for the world. We also have a planar version of the registration object (see the right image of Figure 5), which is often more convenient to move from one place to another in a real scenario.

[064] Once this objective has been achieved, a user is able, for example, to place a virtual object 1.80 m high in a real video and ensure that it appears correctly and at this height. Other approaches fail to achieve this quality, due to the manual aspect of the task.

[065] Automatic sizing is something currently carried out during the inspection itself, but as it happens at the beginning of the inspection, it is important to also consider it as an initialization step. The computed scale is actually used in the following inspection steps to construct an accurate set of 3D natural markers. Let's now look, in detail, at the inspection itself.

2.1.3.3 Point Cloud Inspection and Construction Process

[066] Inspection requires the user to move through the scene using the inspection device. Because the system always computes new points based on its confidence in those previously learned, the inspection is always constructed in such a way that the test camera sees enough points from the previous set to add new candidates.

[067] During the movements of the test camera, the algorithm creates a map made of the three sets of data learned at the 4 different levels of an image pyramid (see Figure 4): • A key position. A key position contains a snapshot with 4 different resolutions of the image that the test camera sees. The bottom level of the pyramid consists of the original resolution of the test camera (640 x 480 in our case). This key position is also used by the algorithm to recover the camera position when tracking is becoming wrong. • A set of 2D feature points estimated by FASTER [17] in each of the key position images. FASTER is a well-known algorithm whose main characteristic is that it is an edge detector algorithm. Each key point also features a coupled ORB descriptor to ensure much faster recovery when the system loses track (see section 2.1.4.4). • A set of fragments (16 x 16 2D textures) centered on all characteristic points detected by the previous step. When detecting new points, there is no way to calculate their depth starting from a single view. This is the use of fragments. An epipolar search (see Figure 2, Point Cloud Builder rectangle) can be applied across the 4 key position images by finding a match between two fragments at two key positions as close as possible. Once a point (i.e., a texture) has been detected in both views, it is then possible to compute a 3D feature point. The set of 3D feature points is called Map. It is also important to understand that this search between the two key positions A and B is obtained by translating equal levels of the pyramid A and B, but also the sublevel of the pyramid B' (see Figure 4).

[068] During the construction of the map, the camera moves according to the movement of the inspection operator. Right now, our software only knows the camera parameters that have been previously computed (i.e., tracked). For the purpose of computing the new camera position, we need the current frame and the 3D point cloud of the natural markers. On the one hand, FASTER calculates a set of 2D markers at various levels of the image pyramid (current level and current level + 1), and on the other hand, the 3D point cloud is reprojected onto the current frame. This last step can only be achieved if we know the position of the camera as the points are being reprojected from its point of view. However, this is precisely what we are trying to compute. Therefore, our software uses a constant velocity model associated with the information provided by our 9-DOF sensor (see section 2.3) to predict the next camera position using the previously and correctly calculated position. Using this prediction, the 3D point cloud can be reprojected onto the current frame, and a point matching algorithm is applied to find a correspondence between the 2D points that were detected by FASTER and the projected points from the 3D point cloud. The error between the two sets of markers is minimized using a Levenberg-Marquardt algorithm [10, 12, 15], as it is known to be the best optimization algorithm for these types of problems. If the number of coincident points divided by the total number of projected points is greater than a given threshold, the system is able to successfully track the camera (tracking is good) and is allowed to add new key positions. 2D points detected by FASTER that did not find a match in the 3D point cloud are stored in memory for later use, as well as their related 2D 16 x 16 fragments. They will be needed to generate new 3D characteristic points (see the next paragraph).

[069] New key positions (and new characteristic points) will be added if three conditions are met. First of all, as we reported in the previous paragraph, the tracking must be good, in the sense that it is accurate enough or not lost. Second, the key position is added every 30 frames (1 per second) to avoid building an excessively large dataset. Thirdly, the new key position must be at least 10 cm away from the nearest key position. This method prevents the system from learning additional points when it is in stationary mode.

[070] When all these tests have been completed successfully, the software can add a new snapshot and new 3D feature points. The first step is to use FASTER again to detect new relevant 2D points from the new frame (snapshot). Since we have a set of non-coincident 2D feature points coming from the track, we now try to match the "old" 2D feature point with the newly computed one. This is accomplished through a classic epipolar search using the 2D texture fragments from the new set of points. Fragments are moved along epipolar lines (see Figure 2) trying to match the set of 2D fragments to the "old" set of points. Thanks to the epipolar constraint, it is possible to compute the depth of the 3D point from the correspondence of two 2D points. This process is in fact similar to the coincidence step of the homography phase (see section 2). If new points have been added to the map, a local pack adjustment is applied. After you have refined the position and rotation of the camera using these new feature points, a Levenberg-Marquardt optimization algorithm is used to refine the 3D position of all feature points. This is performed in a k + 1 frame window, which means that the minimization takes into account the current snapshot plus the k nearest ones to correctly estimate the current position and rotation of the camera (local packet adjustment). The value of k can be determined either empirically or can be computed adaptively according to a given error threshold for the minimization algorithm.

[071] In addition to the local package adjustment, a global package adjustment is also applied to all existing key positions. As the set of key positions becomes larger and larger over time, the system takes longer and longer to perform the global adjustment. If new key positions are being added and our software has not had enough time to perform the local and/or global package adjustments, the adjustments are then just cancelled, leaving priority for adding new key positions. They will be applied again as soon as new key positions are added.

[072] Once there is the presence of a point cloud, the camera can be tracked. This process is described in the next section.

2.1.4 2nd Phase: Filming Camera Tracking 2.1.4.1 Camera Geometric Calibration

[073] Camera calibration has several objectives. On the one hand, it gives us a mathematical relationship to describe how the image is created on the sensor. On the other hand, it calculates a transformation matrix between the 3D points and their projection onto the camera image plane.

[074] This important prerequisite of the inspection and tracking process means that we have to compute two different types of properties: intrinsic and extrinsic parameters. Intrinsic parameters are related to the properties of the camera and lens and move according to how they were physically constructed. They include the image format, the distance between the optical center of the camera and the image plane, and the main point. The image format is produced from two scale factors, relating pixels to distance. The main point is the projection of the optical center onto the image plane (preferably in the center of the image). Additionally, some lenses need to have their optical distortion computed and considered for tracking. During a calibration process, a flat surface with several circular patterns is successively captured by the camera (see Figure 5). For all positions, extrinsic parameters (i.e., camera position and rotation) and intrinsic parameters (including optical distortion) are computed via a Levenberg-Marquardt Minimization algorithm that minimizes reprojection error between all positions. snapshots.

[075] This process consists of a very famous technique based on work by Devernay and Faugeras [3].

2.1.4.2 Moving the Filming Camera

[076] The global system developed here requires the placement of a tracking device directly coupled to the target camera for filming. The position and orientation of the probe camera are estimated using the tracking technique previously described. However, there is a scaling problem when using the same standard for both film and proof cameras. In fact, because our proof cameras use 180-degree lenses, the circular shapes of the calibration pattern tend to appear very small in the proof camera view, consequently making them difficult to detect.

[077] We created our own calibration card that presents two different size and contrast standards. One part of the card has pure black circles on a pure white background (for the proof cameras). The second part of the card is made of pure white circles on a pure black background (Film Camera). We then use a very simple pattern detector to find both patterns for the proof and film cameras. Since we know the exact relative positioning of both patterns, the position and rotation of the film camera can be found by "propagating" the computed position and rotation of the proof camera and vice versa. In effect, since both cameras are directly connected together, we are able to use an alignment algorithm to accurately compute the 3D displacement (R, T) between them. We developed an alignment algorithm based on the previously described circular patterns. The patterns, whose exact sizes, shapes and positions are known, are recognized separately by both cameras, thus generating a trivial mathematical relationship (Rotation and Translation) between the proof and filming cameras.

2.1.4.3 Tracking Phase

[078] Separating inspection from the tracking process is just a pragmatic way of describing how the system is used at a location. In effect, the tracking process is no different from the tracking described in Figure 2 and used during an inspection. It, however, uses a simpler device (see Figure 6), which is very close to the inspection device, except for the fact that it does not include a touch screen for obvious reasons.

[079] The Visual Tracking rectangle in Figure 2 shows how a correspondence is found between the 2D natural markers detected by FASTER in a new frame and the existing 3D point cloud. The 2D texture fragments associated with each natural marker and originating from both datasets are compared linearly. Once all feature points have been processed, a Gauss Newton algorithm is applied to find the best possible match and compute the position and orientation of the camera. A minimization algorithm of this type generally requires approximately 10 iterations to converge to the solution. In order to increase the robustness of the minimization criteria, we used a statistical Tukey M-Estimator (maximum similarity type estimator) [6]. This also ensures that convergence is not disturbed by inappropriate criteria.

[080] Each tracking is performed twice and at two different levels of the pyramid before obtaining the final camera parameters. First, the convergence of the algorithm is greatly increased by computing an approximation of the camera position and orientation across a coarse level of the image pyramid, and using a subset of the natural markers. Secondly, the same computation is obtained on a much larger set (approximately 20 times larger) and being merged with the previous approximation to obtain a final accurate result.

2.1.4.4 Tracking and Recovery Failures

[081] During a camera tracking in a real scenario, many situations can occur in which visual tracking is lost. This often occurs, for example, when people stand in front of the test camera or the bar is simply hidden by objects. In all cases where we lose visual tracking, three different recovery tasks are started at the same time, and the one that achieves a recovery first is accepted.

[082] Using the Extended Kalman Filter prediction function (see section 2.3.2), it is possible to first attempt to recover the camera position by starting a search from the predicted EKF parameters. In effect, as EKF consists of a prediction-correction algorithm, it always tries to guess what the camera's next position will be before it actually moves to that position. When the prediction is correct it is therefore possible to obtain an instant recovery. In practice, this prediction works very well when the test camera is hidden and when the system is tracking using the IMU. However, it often fails when there has also been a significant translation of the filming camera when hiding the evidence cameras.

[083] The second level of recovery is very different. Each characteristic point has an invariable rotation descriptor associated with it, following the ORB (Oriented FAST and Rotated BRIEF) technique [18]. It is firstly based on the famous FAST technique [16] to compute multiscale features (i.e., at different levels of the image pyramid, see Figure 4), associated with a Harris scale [5]. The descriptor orientation is computed based on an intensity centroid. The second descriptor is rBRIEF, which consists of an rBRIEF [2] descriptor rotated following the orientation of the key point. Using these two descriptors, our technique is able to perform recovery from positions where regular visual tracking techniques fail. For example, let's assume we lose visual tracking for whatever reason. Suppose we now translate and rotate the camera by 90 degrees around the Z axis while it is still lost. The current image viewed by the camera has never been learned before, but the camera is still pointing in a direction in which we have added previously learned key positions and key points. However, as a regular recovery process is not rotation invariant, these systems cannot recover the trace. In our case, the use of invariant rotation descriptors associated with each key point allows a quick recovery (using a brute force matching technique) as soon as the filming camera is pointing somewhere from this previous position (the rotation does not matter). .

[084] Finally, the third retrieval task is more brutal and computationally intensive. An ellipsoid (3D ellipse) is constructed around the last known position of the camera. This ellipsoid has a size made up of the parameter covariance computed by the Extended Kalman Filter. Multiple samples are then generated within the ellipsoid following the EKF error covariance. A RANSAC algorithm is applied to this set with the purpose of finding the best possible match for the sought-after camera position and orientation. 2.2 Dynamic Zoom, Focus and Iris

[085] During the global tracking process, a director or director of photography, for example, may decide to zoom or focus on a character/object in the scene. Our technology integrates several regular encoders such as those proposed by Arri, but is also capable of working with any other encoder to capture a zoom, focus and iris value in real time. This process makes it possible to apply dynamic zoom and depth of field to a scene. 2.3 Sensor Fusion Technique

[086] All markerless techniques based on purely optical approaches may fail to track in many situations. The most common case is that there are not enough natural markers in the current proof camera view. In ideal situations this rarely happens, as the user usually pays attention not to obstruct the lens with their hand, for example. In real-life scenarios, this situation happens simply because the camera crew often needs to replace lenses, modify camera equipment, clean lenses, move filming equipment to another place, etc. In a regular marker-based and markerless (optical flow-based) technique, this situation prevents the tracking system from finding a good estimate of camera position and rotation.

[087] We created a sensor fusion method that automatically corrects the position and rotation of a test camera using multiple sensors: test camera, gyroscope, accelerometer and magnetometer. This technique is the heart of our technology. The first step consists of calibrating the system, that is, calibrating the 9-DOF IMU in relation to the test cameras. This method is a critical step to ensure that all sensors actually work in the same way. The relationship between the 9-DOF sensor and the test cameras is close to a technique called hand-eye [20], but we added several interesting scientific contributions to make it compatible with our tracking algorithm.

2.3.1 Calibration of the Test Chamber - IMU

[088] Let us consider the RIMU^ C transformation that computes the rotation of the 9-DOF sensor (IMU) for one of the two test cameras (C). RIMU is known in its local base and Rc is also known in its own local base. The objective is to solve RIMU^ C for Rc = RIMU^ C • RIMU.

[089] It is ensured that RIMU^ C is always constant as the relationship between IMU and cameras does not change over time (they are both coupled to the camera's static bar).

[090] Therefore, it is possible to write for ti, i [0, n]:

[091] From the previous equations we can deduce:

[092] The rotation measurement given by the gyroscope and visual tracking are superimposed on it. Therefore, it is logical to consider that RiMu and Rc describe the same rotation angles, but in different bases. Using the log map of the rotation matrix in Lie Algebra defined as Log : SO(3) ^ so(3), it is possible to convert the Rc matrix into an axis-angle representation with tr( Rc) being the matrix trace, such as .

[093] We can now write the following overdetermined equation system:

[094] As for any overdetermined system, it is possible to solve the previous rectangular matrix using a Singular Value Decomposition (SVD) as S = UrVt, with U being the m x n matrix of the eigenvectors S • St, V the m x n matrix of the eigenvectors St • S and D the n x n diagonal matrix of the sorted singular values of S.

[095] The solution to the previous equation is subject to interference from measurements. For this reason we apply a Levenberg-Marquardt minimization scheme to minimize the error:

[096] Now that we have a mathematical relationship between the IMU and the test cameras, it is possible to inject all measurements into the Extended Kalman Filter, with the purpose of extracting the best from each sensor and computing the Rotation and Translation of the filming camera .

2.3.2 Extended Kalman Filter

[097] The Extended Kalman Filter (EKF) consists of a well-known technique in Aerospace and Robotics research areas for merging different data from multiple different sensors. The aforementioned filter is essentially used to estimate the parameters of a nonlinear system from noisy measurements by computing an estimation error and applying two phases; prediction and correction. In the prediction phase, the Kalman filter computes the new parameters as well as an error related to a time step, using the parameters and the error estimated in the previous time step. The correction phase updates the forecast using new noisy measurements. These new measurements allow computing new estimated parameters and the error of the predicted parameters, and errors. If the error is greater than a given threshold, the parameters are corrected.

[098] In our present case, the EKF is developed in a very specific way, as we have several measurements related to 3D rotations (gyroscope + test cameras) and which cannot be interpolated easily.

[099] The idea of an EKF for our technology is to apply some type of confidence value to each measurement (test camera, gyroscope, magnetometer, accelerometer) and iteratively correct the position and current estimated rotation using the sensor which gets the best confidence score. In practice, EKF is a little more complicated than this, and can be described mathematically as a prediction-correction algorithm for nonlinear systems. Considering a state vector X (t) at a time step t, the algorithm applies the current physical model of our equations of motion to predict a new state of the vector X (t) becoming X(t)?f. The correction phase then generates a new state vector X(t+?t) in a time step t + ?t.

[0100] The EKF model can be written as follows, in our case: with h being the state equations for our observables and B is the global noise in the system.

[0101] If on the one hand we consider the measurements coming from visual tracking without markers (i.e., in the case of test cameras) as hwitness, and on the other hand the requirements coming from the IMU as hIMU, we can say that h is produced from two types of observables, hwitness and hIMU. By definition, h consists of a trivial function and its general form is given by the identification: Ywitness Depending on the type of sensors that are sending the measurements, we can now write two different versions of h and thus rewrite equation 11 for the case of visual tracking as: the observables being y (ye ^12) of visual tracking in a time step t for Translation (T, in meters), Rotation (R, in radians) Velocity (V, in m^s-1) and Angular Velocity ( Q, in rad • s-1).

[0102] In the case in which the sensor is a gyroscope, equation 11 becomes: being the IMU observables at a time step t for Rotation (R, in radians) and Angular Velocity (ß, in rad^s-1). In fact, there is no computable translation or velocity using a 9-DOF sensor. with (t)?f being the predicted state vector at time step t + ?t, and X(t) the estimated state vector at time step t.

[0103] Function f consists of the state vector update function, being defined as:

[0104] Note that the regular update function of the Kalman Filter is not usable for our case, as the Rotation Matrices cannot be interpolated and written in a linear A • X + B form.

Prediction Step

[0105] The prediction step is written using the Kalman filter definition: with: • Pf the covariant matrix of the predicted error at time step t + ?t • Pt the covariant matrix of the predicted error at time step t • The transition matrix as Jacobian matrix of fe computed as the sum of finite differences: • Q is the 12 x 12 covariant matrix of the state vector noise. Translation state vector noise is heuristically placed in 1.e-2, as well as rotation state vector noise in 1.e-3.

Correction Step

[0106] Following Kalman's definition, the correction in our case is written as: with: • I the identity matrix, • K ( t +?t) the Kalman gain at time step t + ?t, • H being the Jacobian matrix of the h function, being hwitness itself or hIMU depending on the currently selected sensor .

[0107] In our case, the Jacobian is computed as a sum of finite differences: • U(t + ?t) consists of the covariant matrix of noise measurements from the currently selected sensor (test camera or IMU). This value is modified depending on the quality of the measurement (confidence value). In the case of visual tracking, it is based on tracking error and:

[0108] It is even easier to determine U (t + ? t) for the IMU as noise errors are directly provided by the manufacturers of all sensors. The previous equation is directly replaced by these errors. • P (t + ? t) represents the error covariance matrix at time step t + ? t. • V and Q consist of the matrix representation of the velocity vector V and the angular velocity Q, respectively. • Jwitness|IMU from equation 20 is also written following the types of sensors: consist of measuring Translation, Rotation, Speed and Angular Speed, respectively. J.(ae {R, T, V, O}) is known as the Kalman innovation. 2.4 3D Reconstruction and Depth Keying

[0109] Real-time 3D reconstruction is achieved using a 3D range sensor integrated into our camera bar. This range sensor may consist of a structured light device (Microsoft Kinect, Asus Xtion, for example), or a time-of-flight camera (such as a TigerEye from Advanced Scientific Concept), also known as Flash Lidar. Both techniques have limitations and our technology currently uses only a low-range Asus Xtion sensor. As usual, a calibration step is necessary to determine the position of the Asus Xtion relative to the test cameras. We simply use exactly the same technique for the proof cameras as film alignment (see section 2.1.4.2). As the range sensor has a grayscale camera, we use our usual calibration standard so that it is detected on both cameras: the range sensor camera and one of our test cameras. This process is quite simple and similar to 2.1.4.2.

[0110] The range sensor is used for two purposes in our technology. On the one hand, it helps to eliminate bad characteristic points too far behind objects, for example. A simple reprojection of the feature points onto the swath sensor depth map can tell you whether the feature point is correct or not. During stereoscopic tracking (see section 2.5), each point created by the epipolar geometry may still be closer or further away than the estimate provided by the epipolar lines. We compare the reconstructed 3D point with the current value of its reprojection onto the swath sensor depth map, to improve its accuracy or simply reject it.

[0111] The second use of the range sensor is based on the depth map itself. As it provides depth with a maximum distance of 12 m, we are able to occlude all people (actors) in the scene within this range by virtual characters/objects and vice versa. This process is called dynamic occlusions and raises another level of realism for visual effects in film and live broadcasting. Our technology uses the main signal from the film camera (this could also be achieved by adding an additional high-resolution test camera) to improve the resolution of the range sensor, with depth reprojection onto the film camera and the refinement of results (edge enhancement). A technology of this type is capable of providing real-time depth switching, therefore eliminating the use of a green screen connection especially when a more powerful range sensor is used (Flash Lidar). 2.5 Markerless Stereoscopic Technique

[0112] The previously described technique using a monoscopic approach is actually not effective enough in real situations. Because it requires a known real object to be used as a record to compute the size, orientation, and position of the world, it is often difficult to use in movies when many people are always in your way when you are trying to record ( detect) the object. However, the main limitation of this monoscopic approach lies in the fact that it requires inspecting the scene. As a homography must be used to initialize the system, this fact implies that a technician has to travel to the location to begin inspecting the scenario. In practice, this operation is rarely carried out successfully, as the people working on the scene never leave the scene before the last few seconds.

[0113] We have constructed a stereoscopic approach to eliminate this limitation, bringing with it the entirely unique notion of instantaneous inspection and tracking. 2.5.1 Instant Inspection

[0114] The stereoscopic technique embedded in camera tracking is an important step forward compared to any other technique, including our previously described monoscopic approach. In fact, the use of 2 pre-calibrated test cameras provides several advantages. The first consists of Instant Inspection. Unlike the technique described in the monoscopic approach, there is no prior research required for the scene. As the exact distance between the two test cameras is perfectly known, and as they are synchronized using their internal genlock hardware capabilities, we are able to obtain a point cloud instantly without moving the bar. Therefore, the homography described in the paragraph is no longer used, as the 3D reconstruction of a point is carried out by the two test cameras using the Stereovision theory. The second important point is that there is no need for a manual scale. During the capture of the two video streams, we run a Levenberg-Marquardt minimization scheme to minimize the sum of pixel errors between the 3D point projection estimated by the epipolar geometry and the 2D feature point detected in the current video signal. 2.5.2 Automatic Registration

[0115] The use of an instant search raises, however, a new issue, which is the impossibility of using a recording object like the one detected in the monoscopic approach. The main reason for this limitation arises from the fact that the camera bar is directly connected to the filming camera, and it is not possible to ask the film crew to move their camera around to see the recording object. It is for this reason that we have added a second automatic registration, which does not require any physical registration objects to be added to the scene. It requires, however, that the user has two different interactions with the point cloud: point selection to inform the system which point we intend to use as a ground reference, and manual rotation around the Y axis ( vertical) to correctly orient the CG content in relation to the scene.

[0116] For the purpose of operating with just these two very fast and very simple user interactions, we must then compute all other left degrees of freedom, meaning computing an orthonormal basis having a perfectly vertical Y axis and correct scaling. Our technology simply takes advantage of the accelerometer built into the bar to compute the orientation of gravity, i.e. the Y axis. The magnetometer provides the north direction and therefore provides an approximate vector that allows us to compute an orthonormal basis using regular cross products.

[0117] Scaling is still much easier to compute. It is completely instantaneous and automatic and does not require any user interaction. As the distance between the test cameras is known exactly, all directly reconstructed 3D points have the correct dimension, and in this way there is no additional object to be detected, as in the pure regular framework of motion (SFM) algorithms. 2.5.3 Instant Tracking

[0118] The tracking process is slightly different from that described for the monoscopic approach in section 2.1.4.3. Each new point is created using epipolar geometry, which ensures that the dimension and depth of a 3D feature point are always correct. This means that, as soon as our system starts, it is already tracking thanks to this precise, high-speed and instantaneous inspection.

[0119] We compute the union of all 3D points reprojected and viewed from the left and right cameras to ensure maximum stability of the tracking process. The point cloud is simply generated by the 3D intersection of the epipolar lines combined with the ORB descriptor of the coincident features (see section 2.1.4.4). The feature matching and global minimization scheme remain similar to those used in the monoscopic approach.

[0120] In order to be able to rely on instant inspection, we have also significantly modified the way in which new points are constructed in the point cloud. In addition to the stereoscopic point reconstruction, we add the notion of grid dispersion and characteristic point uniformity. During a regular tracking algorithm, a 2D feature detection algorithm is launched and points are reconstructed in 3D as described in section 2.1.3.3 and 2.1.4.4. However, many of these points are unreliable (incorrect depth, edge points, moving points, shadow points, etc.), and the algorithm tends to learn a large number of points in high-contrast areas and none in low-contrast areas. contrast. This process generates a large amount of noise in the tracking data, which gives the sensation that the camera is shaking. The grid sparsity technique we created attempts to distribute the natural markers learned by the system in a roughly uniform manner. First, we start by using an extremely high threshold to generate a maximum of feature points. We then reduce the number of characteristic points according to a threshold provided by the user (16 x 16 grid, 2 samples per cell, for example), using a Harris score (we choose the 2 best points, for example). Each point therefore presents a quality criterion used to inform how reliable it is. If the uniformity is not good enough, the system also changes the contrast threshold locally to ensure that points can be captured even in low contrast areas. The fact that our algorithm distributes the characteristic points throughout the image also prevents the system from easily losing track. In fact, normal monoscopic approaches without a grid scattering technique like ours may have a large amount of tracking points in the lower left part of the image, for example. If an actor enters this area, all these natural markers will be hidden and tracking will be instantly lost, which is something that does not happen in our case, as we distribute the features throughout the image. 2.6 Augmenting the Real World 2.6.1 The Proof Camera Delay Problem

[0121] As the filming camera and the proof cameras run at different speeds, they can generate different delays in the frame acquisition process. The movie camera is the one that actually creates by far the most delay, forcing us to compute it for the purpose of properly augmenting the real world.

[0122] We again use our own calibration device to automatically compute the delay between two cameras. A small LED bright like the sun and blinking at a frequency of 1 Hz is placed in front of both cameras. A blob detector algorithm, enhanced by an intensity thresholding technique, was developed to detect the LED in both cameras. The related intensity curves, which are generated from capturing the LED intensity in the cameras, are compared using a minimization algorithm, with the purpose of finding the best temporal coherence between the samples. Several statistical parameters are minimized in this process, and a delay in milliseconds is automatically extracted from the calibration in approximately 10 seconds. 2.6.2 Rendering and Augmented Reality

[0123] Once we have a camera tracking in place, our software can receive data from different CG content generation software like Autodesk MotionBuilder, for example. This data is directly integrated and rendered within our software (in the case where the tracking and rendering workstation are merged, see Figure 1), to be composited with the real background or foreground (in the case of a green scenario).

[0124] Another interesting feature inherent to our technology is 3D Residue Coincidence. 3D Residue Matching is certainly not our invention, but it is carried out in an innovative way in our case. In fact, as our tracking algorithm generates an accurate 3D point cloud (thanks to the stereoscopic approach), all these points have a representation in both 3D and 2D when reprojected onto the Camcorder video. We have developed a simple user interface that allows the user to select multiple points within the 3D point cloud. With this process we create a 3D object, triangulating all selected points using a Triangulation Perception algorithm [14]. This newly created surface can now be activated as a Residue Coincidence Object, which means that every pixel covered by the 2D projection of this object onto the video camera is now replaced by the 3D virtual environment generated by the Rendering Workstation. 3D content (see section 1). Note that in the case of using a long-range depth sensor (i.e., at least 50 m) the Residual Coincidence technique becomes useless, as our depth switching method easily replaces it (see the section 2.4). Appendix 1 References [1] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: speeded up robust features. In Proceedings of the 9th European conference on Computer Vision - Volume Part I, ECCV'06, pages 404-417, Berlin, Heidelberg, 2006. Springer-Verlag. [2] Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua. Brief: binary robust independent elementary features. In Proceedings of the 11th European conference on Computer vision: Part IV, ECCV'1O, pages 778-792, Berlin, Heidelberg, 2010. Springer-Verlag. [3] Frederic Devernay and Olivier Faugeras. Straight lines have to be straight. In In SPIE, volume 2567, 2001. [4] Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981. [5] Chris Harris and Mike Stephens. A combined corner and edge detector. In In Proc. of Fourth Alve y Vision Conference, pages 147-151, 1988. [6] D. C. Hoaglin, F. Mosteller, and J. W. Tukey. Understanding Robust and Exploratory Data Analysis. Wiley series in probability and mathematical statistics: applied probability and statistics. John Wiley and Sons, 1983. [7] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASM E-Journal of Basic Engineering, 82(Series D):35-45, 1960. [8] Georg Klein and David Murray. Parallel tracking and mapping for small air workspaces. In Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR '07, pages 1-10, Washington, DC, USA, 2007. IEEE Computer Society. [9] J. J. Leonard and Durrant H. Whyte. Mobile robot localization by geometric tracking beacons. IEEE Transactions on Robotics and Automation, 7(3), 1991. [10] K . Levenberg. A method for the solution of certain problems in least squares. Quart. Appl. Math, 2:164-168, 1944. [11] David G. Lowe. Object recognition from local scaleinvariant features. In Proceedings of the International Conference on Computer Vision- Volume 2 - Volume 2, ICCV '99, pages 1150-, Washington, DC, USA, 1999. IEEE Computer Society. [12] D. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math, 11:431-441, 1963. [13] Christopher Mei, Gabe Sibley, Mark Cummins, Paul Newman, and Ian Reid. Rslam: A system for large-scale mapping in constant-time using stereo. Int. J. Comput. Vision, 94(2):198-214, September 2011. [14] Gary Hosler Meisters. Polygons have ears. Amer. Math. Monthly, 82:648-651, 1975. [15] J. J. More'. The Levenberg-Marquardt algorithm: implementation and theory, pages 105-116. Lecture Notes in Mathematics 630, SPRINGER, 1977. [16] Edward Rosten and Tom Drummond. Machine learning for high-speed corner detection. In Proceedings of the 9th European conference on Computer Vision - Volume Part I, ECCV'06, pages 430-443, Berlin, Heidelberg, 2006. Springer-Verlag. [17] Edward Rosten, Reid Porter, and Tom Drummond. Faster and better: A machine learning approach to corner detection. IEEE Trans. Pattern Analysis and Machine Intelligence, 32:105-119, 2010. [18] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In Proceedings of the 2011 International Conference on Computer Vision, ICCV '11, pages 2564-2571, Washington, DC, USA, 2011. IEEE Computer Society. [19] Ivan E. Sutherland. A head-mounted three-dimensional display. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I, AFIPS '68 (Fall, part I), pages 757-764, New York, NY, USA, 1968. ACM. [20] Roger Y. Tsai and Reimer K. Lenz. A new technique for fully autonomous and efficient 3d robotics hand-eye calibration. In Proceedings of the 4th international symposium on Robotics Research, pages 287-297, Cambridge, MA, USA, 1988. MIT Press. [21] J. K. Ulhmann. Algorithms for multiple target tracking. American Scientist, 80(2):128-141,

Claims (23)

1. MARKERLESS SYSTEM, the system including: (i) a video camera (j) ) sensors including an accelerometer and a gyroscope detecting in six degrees of freedom; (k) i) two test cameras that form a stereoscopic system; and (l) ) a processor; for mixing or compositing in real time, computer-generated 3D objects and a video signal from the camcorder, for generating real-time video with augmented reality for broadcast TV, cinema or video games, characterized by the fact that : (m) the video camera body can be moved in 3D and sensors, on or connected directly or indirectly to the video camera provide real-time positioning data that defines the 3D position and 3D orientation of the video camera, or that allow the 3D position and 3D orientation of the video camera to be calculated; (n) the two test cameras that form the stereoscopic system are fixed directly or indirectly to the video camera; (o) the system is configured to use this real-time positioning data automatically to create, recall, render or modify computer-generated 3D objects; (p) the system is configured to mix or composite the resulting computer-generated 3D objects with the video signal from the video camera to provide an augmented reality video for broadcast TV, cinema or video games; and wherein: (q) the system is configured to determine the 3D position and orientation of the video camera with reference to a 3D map of the real world, wherein the system is configured to generate the 3D map of the real world at least in part, using real-time 3D positioning data from sensors and also an optical flow in which the two test cameras that form a stereoscopic system inspect a scene and in which software running on the processor is configured to detect natural markers in the scene ('feature points') that have not been manually or artificially added to this scene; (r) the system is configured to use a constant velocity model associated with real-time 3D positioning data from the sensors to predict an upcoming position of the video camera using a previously computed or correctly confirmed position, and the system is configured to use this prediction to project a 3D point cloud onto a current proof camera frame and use a point matching algorithm to match points identified in a real-time video feed from the stereoscopic system and projected points in the projected 3D point cloud. 2. SYSTEM according to claim 1, wherein the sensors include a 3D range sensor, such as a structured light or time-of-flight camera, that captures the depth of each pixel of a video output from the camera of video. 3. SYSTEM according to claim 2, characterized by the fact that the depth of the edges is refined by back-projecting the depths of the 3D range sensor onto the high-resolution video signal of the video camera. 4. SYSTEM according to any one of the preceding claims, characterized in that the two test cameras forming a stereoscopic system operate at at least 100 fps to allow the system to be fully initialized without a separate stage of inspection of the scene only. be tracked (called "instant inspection"), but instead, inspection is performed continuously as the video camera is being used to capture video. 5. SYSTEM according to claim 4, characterized by the fact that the two stereoscopic test cameras that form the stereoscopic system allow the software to process the images and, even if the camera system is not moved as a whole, for the generation of an instantaneous 3D point cloud (for example, associating a large number of scene points with their position in 3D space using knowledge about the separation between the two cameras and epipolar geometry). 6. SYSTEM according to claim 5, characterized by the fact that the depth of each point in the 3D point cloud is obtained using fragments of corresponding 2D textures, obtained from each stereoscopic test camera and an algorithm line search epipolar. 7. SYSTEM according to any one of the preceding claims, characterized in that it executes a fusion algorithm that combines optical flow data from the test camera system with real-time positioning data from the sensors. 8. SYSTEM according to claim 7, characterized by the fact that the fusion algorithm is based on an Extended Kalman Filter prediction/correction technique to integrate the outputs of, and to recalibrate, all sensors, which may include a magnetometer, or a 3D range sensor, to determine the position and orientation of the video camera. 9. SYSTEM according to claim 8, characterized by the fact that the Extended Kalman Filter fusion algorithm uses confidence level data, associated with the output of each sensor, to determine the way to merge the data from each sensor. 10. SYSTEM according to any of the previous claims, characterized by the fact that the keyframes generated by the test cameras are part of the visual tracking process, and consist of real-time images computed with four different levels of resolution of the video signal from the test camera. 11. SYSTEM according to any one of the preceding claims, the system including a 3D range sensor, characterized in that the 3D range sensor is used to increase the accuracy of the depth measurement associated with a reconstructed 3D point obtained using the proof cameras that form the stereoscopic system, or to reject this reconstructed 3D point obtained using the proof cameras that form the stereoscopic system. 12. SYSTEM according to claim 11, characterized by the fact that the 3D range sensor is used for real-time depth switching to allow dynamic occlusion and suppress the eventual use of a green scene. 13. SYSTEM according to any one of the preceding claims, characterized by the fact that the system is configured to locally change a contrast threshold to include points even in areas of low contrast, in the point cloud. 14. SYSTEM according to any one of claims 4, 5 or 6, characterized by the fact that it includes a camera tracking system that combines instant inspections with video camera tracking as a director/camera operator tracks, pans and tilts the camera tracking system attached to the video camera. 15. SYSTEM according to claim 4, characterized in that the two test cameras that form the stereoscopic system allow continuous inspections of a scene in real time, to generate a cloud of points defining the scene. 16. SYSTEM according to any one of the previous claims, characterized by the fact that it couples rotation invariant descriptors, for example, using ORB, to characteristic points detected in the scene to facilitate tracking recovery. 17. SYSTEM according to any of the preceding claims, characterized by the fact that it uses a Levenberg-Marquardt minimization scheme for camera tracking to minimize the error between the points identified in the real-time video transmission of the stereoscopic system and the points projected onto the 3D point cloud. 18. SYSTEM according to any one of the preceding claims, characterized by the fact that a user can use the 3D point cloud generated by the camera tracking system to define 3D masks, such as 3D Residue Coincidence masks. 19. SYSTEM according to any one of the preceding claims, characterized by the fact that the video camera and a test camera are calibrated for a delay in acquiring frames using a modulated light source, such as with comparison of light intensity curves associated with a flashing LED. 20. SYSTEM according to any one of the preceding claims, characterized by the fact that the video camera consists of one of the following: crane camera; Steadicam camera; handheld camera; cart mounted camera, tripod mounted camera, smart phone, augmented reality glasses. 21. MARKERLESS METHOD FOR REAL-TIME MIXING OR COMPOSITION OF COMPUTER-GENERATED 3D OBJECTS AND A VIDEO SIGNAL FROM A VIDEO CAMERA, for generating real-time video with augmented reality for broadcast TV, cinema or video games , characterized by the fact that: (a) the body of the video camera can be moved in 3D and sensors, including an accelerometer and a gyroscope detecting in six degrees of freedom, on or connected directly or indirectly to the video camera provide data from real-time positioning that defines the 3D position and 3D orientation of the video camera, or that allows the 3D position and 3D orientation of the video camera to be calculated; (b) two test cameras forming a stereoscopic system are fixed directly or indirectly to the video camera; (c) this real-time positioning data is then automatically used to create, recall, render or modify computer-generated 3D objects; (d) the resulting computer-generated 3D objects are then mixed or composited with the video signal from the video camera to provide an augmented reality video for broadcast TV, cinema or video games; and wherein: (e) the 3D position and orientation of the video camera are determined with reference to a 3D map of the real world, wherein the 3D map of the real world is generated, at least in part, using the data real-time 3D positioning of sensors and also an optical flow in which the two test cameras that form a stereoscopic system inspect a scene and software running on a processor detects natural markers in the scene ('feature points') that do not were manually or artificially added to this scene; (f) a constant velocity model associated with real-time 3D positioning data from the sensors is used to predict the next position of the video camera using a previously calculated or correctly confirmed position, and this prediction is used to project a point cloud 3D in a current proof camera frame, and a point matching algorithm is used to match points identified in a real-time video feed from the stereoscopic system and projected points in the projected 3D point cloud. 22. METHOD according to claim 21, characterized by the fact that real-time positioning data is recorded and stamped with time codes to provide motion coincidence data for post-production processes. 23. METHOD according to claim 21 or 22, characterized by the fact that it is used for: (i) . Real-time tracking of studio cameras; or (ii). Real-time tracking from a Steadicam; or (iii). Real-time tracking of cameras mounted on cranes; or (iv). Real-time tracking of cart-mounted cameras; or V) . Real-time tracking of External Transmission (OB); I heard) . Real-time tracking implemented on a smart phone; or (vii). Real-time tracking implemented on a wearable computing device, such as augmented reality glasses; or (viii). Provide real-time tracking data for 2D post-production; or (ix). Provide real-time tracking data for post-conversion into 3D stereoscopic content; or (x). Provide real-time tracking data for native stereoscopic 3D content; or (xi). Insertion of 3D graphics; or (xii). Insertion of 3D graphics for placement in the studio or on location; or (xiii). Insertion of 3D graphics for OB; or (xiv). Insertion of 3D graphics in sponsored images; or (xv). Insertion of 3D graphics specific to user locations; or (xvi). Insertion of user-specific 3D graphics; or (xvii). Insertion of time-specific 3D graphics; or (xviii). Insertion of 3D graphics to fill crowd scenes; or (xix). Insertion of 3D graphics to replace green screens; or (xx). Insertion of 3D graphics with educational content to aid teaching, in museums and interpretation centers in cultural, historical and natural locations; or (xxi). Measuring the absolute or relative size of objects in a scene.