EP2791914A1

EP2791914A1 - System for filming a video movie

Info

Publication number: EP2791914A1
Application number: EP12813918.5A
Authority: EP
Inventors: Isaac PARTOUCHE; Jean-François SZLAPKA; Emmanuel LINOT
Original assignee: Solidanim
Current assignee: Oaro
Priority date: 2011-12-13
Filing date: 2012-12-13
Publication date: 2014-10-22
Also published as: US20140369661A1; CA2856464A1; KR20140100525A; MX341801B; US9756277B2; MX2014007075A; NZ624929A; US9648271B2; FR2984057B1; CN104094318A; WO2013088076A1; AU2012351392A1; FR2984057A1; IL232766A0; BR112014014173A2; JP2015506030A; US20150358508A1; HK1201625A1

Abstract

System for filming a video movie in a real space comprising: - a filming camera (9), - a sensor (16), - a computerized pinpointing module (27) for determining the location of the filming camera, - a monitoring screen (15), - a computerized compositing module (32) for generating on the monitoring screen (15) a composite image of the real image and of a projection of a virtual image, generated according to the filming camera (9) location data.

Description

VIDEO FILM TURNING SYSTEM.

The present invention relates to video film shooting systems.

We have been shooting video films for a long time. In the last few decades, augmented reality has increasingly been used to represent objects or events in the broadcast or projected video that would be difficult to film in the real world.

A classic method for building a video film with such an augmented reality sequence is to start by shooting actors in a calibrated neutral environment, for example in a studio with a monochrome background. A few weeks or months later, in post-production, three-dimensional animations are added, which give the illusion of interacting with the filmed actor.

It is difficult for actors and directors to play or film a realistic scene in a studio on a monochrome background. Therefore, it has recently been proposed a preview system by which a preliminary version of the animation is generated, and is shown to the director and actors before playing the stage. Thus, actors and directors can more easily imagine their virtual environment and / or the game of their virtual alter ego.

However, these systems are still very inadequate, and many adjustments have yet to be made in postproduction, to adapt the animations to the recorded film which is absolutely unalterable at this stage.

Systems using calibrated patterns are also known to try to get a better idea of where the camera is during filming. A system marketed under the name Lightcraft is an example. These systems are however very laborious to implement, because they require to equip the studio with sights, a complex operation, and they are also limited to be used in studio, or in spaces of limited spatial extent, that where the sights are placed.

The present invention is intended to overcome these disadvantages.

For this purpose, there is provided a system for shooting a video film in a real space defined in a real repository, comprising:

a shooting camera adapted to record a real image for a plurality of distinct time frames,

a tracking system in space, comprising:

. at least one sensor, presenting location data with respect to the turning camera known for each time frame, and adapted to transmit to a computer module for locating the natural topographic information detected by the sensor,

. a computerized tracking module adapted to determine, for each time frame, location data in the actual repository of the camera from the sensor location data, and a comparison between the natural topographic information and a model three-dimensional preset of real space,

a control screen,

a computerized composition module adapted to generate on the control screen, for each time frame, a composite image of the real image, and a projection of a virtual image, derived from a database of virtual animations, projection generated according to the location data in the real repository of the filming camera.

The display on the control screen is generated almost instantly, for example in the second, taking into account the possible processing time and latency due to the various components of the system.

Thanks to these provisions, one can directly visualize on the control screen at the time of shooting, the interactions between the real world and the virtual world. This allows if necessary to turn the same scene to obtain satisfaction.

By using natural topographical information, one avoids the problems related to the patterns mentioned above. We have more freedom to film.

In preferred embodiments of the invention, one or more of the following may also be used:

the sensor of the tracking system is an optical camera having at least one of the following characteristics:

a solid angle of view greater than a solid angle of view of the filming camera,

an acquisition frequency higher than an acquisition frequency of the filming camera,

- an acquisition in black and white,

a footprint at least two times smaller than a size of the filming camera;

an optical axis parallel to an optical axis of the filming camera,

an optical field superimposed on an optical field of the filming camera;

the tracking system comprises a computerized generation module adapted to generate said predetermined three-dimensional model of the real space, and said sensor is adapted to transmit topographic information detected by the sensor to the generation computer module;

- the sensor is adapted to transmit simultaneously with the computerized locating module and the computer module for generating the natural topographic information detected by the sensor, and wherein the computerized generation module is adapted to enrich said predetermined three-dimensional model of the real space from the natural topographical information detected by the sensor;

the topographic information includes information relating to real space geometric objects selected from points, lines, areas and volumes;

in turning configuration, the filming camera and the sensor are fixedly attached to each other;

the system further comprises a location system comprising a pattern adapted to be detected simultaneously by the filming camera and the sensor in a location configuration, and a computerized location module adapted to determine the respective location data of the sensor and the camera shooting from the simultaneous detection of the test pattern;

the system further comprises an optical calibration system comprising an optical calibration pattern adapted to be detected by the filming camera, in an optical calibration configuration, and the computerized registration module is adapted to determine, for each time frame, the location data in the actual repository of the filming camera further from optical calibration data of the filming camera determined by the optical calibration system;

the system further comprises at least one of the following entities:

an inertial sensor fixed on the filming camera, adapted to determine a movement of the filming camera, the computerized marking module being adapted to determine the location data in the actual repository of the filming camera further from data provided by the inertial sensor;

a standard, filmable by the filming camera, the computerized composition module being adapted to put a virtual image at the scale of the real space on the basis of an image of the standard acquired by the filming camera;

a system for determining a magnification parameter of the filming camera, the computerized composition module being adapted to generate the composite image by taking into account said parameter;

the system further comprises a computerized animation module, comprising a database of virtual animations, each animation comprising, for each of a set of time frames, a three-dimensional image expressed in a virtual repository, the computerized module of animations being adapted to transmit said three-dimensional images to the composition module;

the computerized tracking module is adapted to transmit the predetermined three-dimensional model of the real space to the computerized animation module;

the computerized composition module is adapted to generate on the control screen, for each time frame, a shadow of the virtual image, shadow generated according to the location data in the real repository of the filming camera, and data location of lighting in the real repository.

for a subsequent time frame, the computerized locating module is adapted to determine location data in the real repository of the filming camera also from location data in the real repository of the filming camera for a frame previous time; the computerized registration module comprises a selection module adapted to select from the geometric patterns geometric patterns of the three-dimensional model used to find the position of the filming camera in the 3D space;

the selection module compares geometric patterns of a subsequent image with geometric patterns of a previous image, associates geometric patterns present on the two images and immobile in the real space, and does not retain the other geometric patterns for comparison with the three-dimensional model;

the tracking system further comprises a second sensor having at least one characteristic different from the first sensor, chosen from the position, the orientation, the solid angle of view, the acquisition frequency, the optical axis, the optical field.

In some embodiments, it is thus possible to use a sensor dedicated to tracking, and optimized to implement it, which allows the filming camera to be dedicated exclusively to its primary filming function.

Other features and advantages of the invention will become apparent from the following description of one of its embodiments, given by way of non-limiting example, with reference to the accompanying drawings.

On the drawings:

FIG. 1 is a schematic view of a real space,

FIG. 2 is a schematic view of a turning system according to one embodiment of the invention,

FIG. 3 is a schematic view showing a use of the system of FIG. 2 in a learning configuration,

FIG. 4 is a perspective view of a three-dimensional model of real space,

FIG. 5 is a view similar to FIG. 2 in a location configuration,

FIG. 5a is a view similar to FIG. 5 for scaling,

FIG. 6a is a schematic view of the system in a turning configuration, at a first instant,

FIG. 6b is a representative diagram of an acquisition made by the filming camera at the instant shown in FIG. 6a,

FIG. 6c is a schematic view of a composite image produced on the control screen at the same time,

FIGS. 7a, 7b and 7c respectively correspond to FIGS. 6a, 6b and 6c for a second instant,

FIG. 8 is a schematic view of a screen of a programmable machine comprising an animation computer module,

FIG. 9 is a flowchart of a method for producing a video film using the objects described above, and

Figure 10 is a schematic view of an acquisition system according to an alternative embodiment.

In the different figures, the same references designate identical or similar elements.

Figure 1 schematically shows a part 1 of the real space. Figure 1 provides a very specific example of a real space 1. However, the present invention could be applied in a very large number of different real spaces.

An actual repository 2 is attached to the real space 1, and comprises for example an origin 0 and three orthonormal axes X, Y and Z. Thus, each point of the real space 1 has a unique set of coordinates in the Real repository 2.

In the example given purely by way of example, the real space 1 is an outdoor outdoor space, comprising a horizontal road 3 extending substantially along the Y axis and a building 4 located in the depth. Building 4 may include various windows 5a, 5b, 5c and doors 6a, 6b, 6c and the like. A sidewalk 7 extends for example between the road 3 and the building 4. It can for example note a car 8 parked.

As a real space, alternatively, an indoor space could be used, such as, for example, in a studio. Real space 1 has a number of natural topographic information. This information is for example related to geometric objects of real space, such as points, lines, surfaces and / or volumes. As a line, we can for example consider edges of a structure, and as point intersections of two of these edges. As a surface, it may for example consider solid surfaces, such as a car hood, or other. As volume, we can for example refer to objects, such as a car, or another object present in real space. Thus, the natural topographic information is distinguished from calibration markers reported by the fact (s) that:

they are arranged randomly, unordered,

they are arranged in a space of infinite dimensions, the whole world, and not limited to an area with markers,

they are highly heterogeneous, not just different from one another by a barcode type code,

they are available in a 3D volume space, not just on one or more plans,

they do not require complicated installation calibrated beforehand.

Referring now to Figure 2, there is described a video film shooting system, according to one embodiment, in a filming configuration. With regard to the video film, it is a series of images broadcast at a fast frequency (several images per second, for example 24 (cinema), 25 (PAL) or 30 (NTSC) images per second) to a spectator. This series of images is for example projected or broadcast as part of a film, a TV movie, an informative message, a video game, or other. In particular, this diffusion or projection can be delayed in time with respect to the turning.

This sequence of images relates an event taking place in real space 1.

As such, a filming camera 9 of any type suitable for filming such a scene is used. In particular, a digital camera that can acquire several images per second, for example 24 images per second, is used.

The camera 9 comprises an optical 10 that can acquire images in an optical field 11, and connected to a computer system 12. This connection is for example made by a cable 13 adapted or without cable, for example by radio transmission or other.

The turning camera 9 is of any known type suitable, but the invention is particularly suitable if it is possible to vary the optical field 11 during shooting. In particular, the optical field 11 can be varied by moving the filming camera 9 in the real space 1. This is particularly the case if the filming camera 9 is movable, in a guided manner, in the real space 1 , for example by being mounted on a rail or a crane having an articulated arm (not shown) defining a location of possibilities for the filming camera 9.

Alternatively, which is the alternative shown, a filming camera 9 is used which is sufficiently compact to be movable in the real space 1 simply by an operator (not shown).

According to an exemplary embodiment, the filming camera 9 comprises a monitor 14 mounted on the housing of the camera and having a control screen 15 visible by the film operator, and on which the optical field 11 acquired by the camera is displayed.

The turning system also comprises a system for locating in the space comprising, on the one hand, a sensor 16 and, on the other hand, a computerized registration module 17 of the computer system 12, and connected to the sensor 16 by a cable 18 or without a cable. , as indicated previously.

The sensor 16 has the particularity of having a known location at any time with respect to the filming camera 9. By location, here means that the position and the orientation of the sensor 16 with respect to the filming camera 9 are known to any moment. These are in particular the positions and relative orientations of the acquisition systems of the sensor and the camera 9 (CCD matrix for it). This can for example be achieved in a simple manner by fixing the sensor 16 rigidly to the turning camera 9, for example by means of a flange 19 or any other suitable mechanical system.

The sensor 16 is characterized in particular by an acquisition field 20. It is possible, for example, to place the sensor 16 so that no part of the filming camera 9 blocks a portion of the acquisition field 20, and that no part of the sensor 16 obstructs part of the optical field 11, as shown in FIG.

The sensor 16 is adapted to acquire information relating to the real space 1, so as to be able to determine the position of the sensor 16 in the real space, using the computerized registration module 17. In particular, it can be provided in a turning configuration, acquiring location data in the real space 1 with the sensor 16, and that the computer registration module 17 can determine, for an acquisition made by the sensor 16, with the aid of a predefined three-dimensional model 21 of the real space, the position of the sensor 16 in the real space. Thus, the registration module 17 will determine the most probable location of the sensor 16 in the real space, which makes it possible to match the data acquired by the sensor 16 and the predefined three-dimensional model 21 of the real space.

Knowing the position of the sensor 16 in the real space, and knowing the relative position of the filming camera 9 and the sensor 16, the tracking module 17 can thus determine the location data of the filming camera in the real repository.

It will be noted that, even if the method described above involves two successive steps of determining the position of the sensor 16, then determining the registration camera 9, one could alternatively directly determine the position of the filming camera 9 without an explicit determination of the location of the sensor 16.

It is intended to use a sensor 16 specifically dedicated to the tracking task, and having acquisition characteristics distinct from the filming camera 9. Thus, the filming camera 9 can be dedicated to its own task, which is to film , and the sensor 16 to its own task, which is to locate.

For example, the sensor 16 is a sensor optical. If it is intended to fix the sensor 16 to the filming camera 9, it is possible to provide in particular for the sensor 16 an optical camera of small size, in particular of a space at least two times smaller than the bulk of the filming camera 9. Thus, the inconvenience for the operator will be minimal.

As a sensor 16, it will be possible in particular to choose an optical camera specifically dedicated to obtaining the position of the filming camera 9 in real space. For example, it is possible to provide an optical camera having an acquisition frequency of at least an integer multiple of that of the filming camera 9, for example of the order of 100 images per second, thus making it possible to smooth out by calculation the position of the filming camera 9 in real space for each time frame.

In particular, it is also possible to choose an optical camera having an optical field (solid angle of view) greater than that of the filming camera 11, in order to maximize the information acquired from the real space 1 that can be used to calculate the positioning of the filming camera. Thus, one can for example use a wide angle lens ("fish eye" or "fish eye" in English) having an acquisition angle greater than 160 degrees.

You can also use a black and white camera, if necessary, as a tracking sensor. Thus the method described here can work even without acquiring color information.

The predefined three-dimensional model of the real space comprises, for example, natural topographical information of the real space 1. This space is for example available by any appropriate means. However, as shown in FIGS. 3 and 4, it is possible, for example, to use certain elements of the system that has just been described to generate the pre-established three-dimensional model of real space.

In particular, as shown in FIG. 3, during a preliminary step, in the training configuration, the three-dimensional model 21 is established. This step is for example performed shortly before the shooting, so that the real space , at the time of shooting, corresponds to the pre-established model.

During the learning step, a learning sensor 22 is moved into the real space 1. For a set of time frames, the learning sensor 22 transmits to the computer system 12, by any appropriate means, information acquired by the learning sensor 22. Thus, the computer system 12 comprises a computerized generation module 23 which, receiving information from the learning sensor 22 according to different angles of view, is capable of determining the three-dimensional model 21 ( to a scale factor close). Thus, by acquiring, with the learning sensor 22, the same natural topographical information of the real space 1 according to different angles of view, the generation module 23 is able to determine the three-dimensional position of a set of geometric objects of real space. As shown in FIG. 4, the three-dimensional model 21, represented as displayed projected in another perspective on a computer screen, consists of a set of geometric patterns (here dots). These points can be represented in any orientation, as in Figure 4, a perspective view of real space. In addition to the points 24, the three-dimensional model 21 could also consist of a set of other geometric objects, such as straight or curved lines, planar or non-planar surfaces, volumes, which are determined either by the generation module 23 itself, or by assistance of an operator of the generation module, the operator indicating to the generation module that a set of geometric objects belong to the same line / surface / volume.

As explained above, the three-dimensional model

21 thus generated will then be imported into the computerized tracking module, in order to identify at any time, in shooting configuration, the actual position in the real space of the filming camera.

In the example described, it is possible to use as the learning sensor 22, the same sensor 16 as that which is used in a turning configuration. Thus, the same algorithm is used to determine the three-dimensional position of a geometric object in real space in learning configuration, and to determine the position in real space of the location camera 16 from the positions in the space. real space geometric objects determined with this same camera. Moreover, by using the same sensor for both stages, one can, in filming configuration, continue to enrich the three-dimensional model, if this one were brought to change during filming (which can be the case in case external turning, or because of the presence of actor in the field of the sensor 16 in the turning configuration). Thus, in this case, the learning mode continues during filming.

As explained above, the predetermined three-dimensional model 21 can, if necessary, be made to a scale factor close. In this case, one can for example use a standard 25, of given length, which will be acquired with the learning sensor 22, thus making it possible to scale the three-dimensional model 21, as shown in FIG. 5a.

In addition, it is possible to implement a location configuration, in order to determine before shooting the respective location data of the filming camera 9 and the registration sensor 16. In particular, an example is given in the case where the sensor 16 is rigidly attached to the filming camera 9. During the location configuration, one filming a target 27 simultaneously with the filming camera 9 and the sensor 16. The information collected by the two tools is transmitted to a computerized location module 26 adapted to determine their relative position from the acquired images of the same pattern 27 by the two tools.

Turning now to Figure 2, the computer system 12 also includes a computer animation module 28. This animation module 28 may for example include an animation database 29 comprising one or more virtual animations. Each animation comprises for example, for each of a set of time frames corresponding to all or part of the duration of the video film to be rotated, characteristics of three-dimensional objects (point, line, surface, volume, texture, etc.). ) expressed in a virtual repository U, V, W 30. Each animation represents for example an augmented virtual reality event. For example, we can provide in the animation database, animations representing a three-dimensional virtual character, mobile or not, a special effect (rain, explosion, ...), or other. By way of example, FIG. 2 shows, for a given time frame, a virtual object 31, characterized by data expressed in the virtual space, identified by the virtual repository U, V, W. illustrative example very simple, we used a vertical column with a square base, fixed in time but, in practice, whether it will be for example a lion, walking, or other ....

As shown in FIG. 2, the computer system 12 includes a composition module 32. The composition module 32 imports an animation of the module animation 28 along a link 33. If necessary, if the animation is not already expressed in the real repository 2, the composition module 32 mathematically links the virtual repositories U, V, W and the real repository X, Y, Z by a suitable passage matrix (an example is described later).

Then, the computerized composition module 32 generates, for the time frame in question, a composite image of the real image acquired by the filming camera 9, and a projection of a virtual image, corresponding to the virtual object 31 for this same time frame, the projection being generated according to the location data in the real repository of the filming camera 9. Thus, the composite image comprises the superimposition of the real image, and the virtual image, as if this virtual image was the image of an object present in the real space, acquired, for this time frame, by the filming camera 9. The composite image is then displayed on the control screen 15. Thus , the operator, filming, can, for each frame of time, visualize, on its screen of control, the position and the orientation of the virtual object in the real space, according to its own angle of view, as if this virtual object was present in front of him. It can thus adapt if necessary the position of the filming camera with respect to the objects.

As a variant, the computer system 12 also comprises a monitor 15 'of a monitor 14' allowing, for the director, or for anyone interested, in real time, to view the composite image from the viewing angle of the filming camera.

An example is in particular given in Figures 6a to 7c. FIGS. 6a to 6c correspond to a first instant, in which an operator, not shown, films a portion 34 of the real space corresponding to the lower part rear of the car 8. The image 35 acquired by the filming camera 9 for this moment can be seen in Figure 6b. The position of the filming camera 9 for this time frame is determined by the tracking system. As represented in FIG. 6c, the composite image 36 generated on the control screen 15, 15 'comprises the superposition of the real image, and of the virtual object 31 seen according to the angle of acquisition of the camera 9. To do this, as explained above, knowing the positions, in real space, of the filming camera 9, and the virtual object 31, at this given instant, we can calculate a projection in the image 35 of this object.

Figs. 7a to 7c show a subsequent time frame (directly subsequent), and are explained with reference to Figs. 6a to 6c. Thus, the events shown in FIGS. 7a to 7c occur approximately at 1/24 second after those of the preceding figures. During this period of time, the viewing angle of a filming camera 9 has changed, so that the filming camera 9 now points more towards the top of the car 8. The portion 34 'imaged is also represented on the Figure 7a. The actual image acquired by the filming camera 9 is represented by reference 35 'in FIG. 7b. FIG. 7c represents the composite image 36 'corresponding to the superposition of the real image 35' and of the virtual object 31, expressed as a function of the location of the filming camera 9 for this time frame. Note that in this example, the virtual object 31 may be identical, on both time frames. Its projected representation for the two frames of time differs because of the difference of angle of view. However, in the case of an animation, the virtual object 31 may also be slightly different for the two time frames.

The above steps can be repeated in real time for each frame of time of the shoot and, where appropriate suitable for several filming cameras.

Referring again to FIG. 6a, for the time frame under consideration, the registration image 37 acquired by the sensor 16 may correspond to a larger volume of the real space, and the computerized registration module is adapted to extracting from this registration image 37 natural topographic information, and to determine the position in the real repository 2 of the filming camera 9, as explained above, from this detected natural topographical information, and the three-dimensional model 21. In particular, optical markers fixed in real space 1 can not be used, for a great simplicity of use. Only natural topographic information is then used, which avoids cluttering the turning space with artificial markers. However, the system described here is also compatible with artificial markers.

If the optical field of the sensor 16 were to be closed (indeed, the operator moves during the acquisition) by a real element of real space, the computerized locating module may have several options to determine at any time. the position in the actual space of the filming camera 9. For example, in the case where the computerized locating module fails to locate enough topographic information to determine for sure the position in the actual space of the camera. the filming camera 9, it can by default consider that the filming camera 9 is stationary during this time. In reality, when the two devices 9 and 16 are very close to each other, as in the embodiment presented, if the sensor 16 is not able to determine the topographic information, it is because the Optical field of the filming camera 9 is probably closed by a very close real object. At the next time frame where the sensor 16 can determine enough information in order to determine the position in the three-dimensional space of the filming camera 9, a composite image may again be generated for this position.

As such, it will be noted that the computerized registration module comprises a selection module adapted to select the geometric patterns of the three-dimensional model that can be used to find the position of the filming camera in the 3D space. On the one hand, the geometric patterns that may be in the field of the sensor 16 are selected, for example by means of an approximate knowledge of the position of the sensor from a previous time frame. On the other hand, if in a region of the image acquired by the sensor 16, the set of identified geometric patterns is too different from the three-dimensional model, these patterns are not taken into account for determining the position of the camera. shooting.

Thus, by comparing two near-time images acquired with the sensor 16, the geometric patterns present on the two images and immobile in the real space are associated in pairs. Other geometric patterns are considered moving in real space and are not preserved for comparison with the three-dimensional model.

As shown in FIG. 2, it is also possible, in these cases, to enrich the computerized locating module, by adding an inertial sensor 38 adapted to provide the computerized locating module with additional information on the position of the filming camera 9. By For example, the inertial sensor 38 is attached to the filming camera 9, or to the sensor 16 if it is attached to the filming camera 9. A matrix of passage between the filming camera and the sensor is associated with each magnification. In filming configuration, we use the information from the encoder to select the appropriate pass matrix.

According to an exemplary embodiment, as also represented in FIGS. 6a to 6c, the composition module 32 can also be adapted to generate a projected shadow of the virtual object 31 in the real space 1. As can be seen for example on FIG. 6a shows artificial (as shown) or natural lighting 39 whose position in the real reference frame 2 is known. Thus, as shown in FIG. 6b, the real image 35 comprises, in addition to an image of the object 8, an image 40 of its real shadow. As shown in FIG. 6c, the personalized three-dimensional model may comprise surface information on which the shadow 41 of the virtual object 31 will be projected, viewed from the shooting angle of the filming camera 9 The shadows of the virtual objects are calculated by taking into account the position in the real space of the virtual object 31, and the position in the real space of a surface on which the shadow of the virtual object is projected. 31, the position of the filming camera, and the position of the lights. Real and virtual shadows are also visible in Figure 7c.

The system which has just been described is of particular interest when the animation is displaced with respect to the optical field of the filming camera 9. In an exemplary embodiment, in a filming configuration, a fixed plane of a space will be rotated. real immobile, on which we will generate an animation whose form changes over time. Thus, we can verify that the animation is framed as desired during filming. Another example is to move the filming camera 9 in the real space 1, incorporating a mobile animation or, if necessary, still, to verify that it is framed as desired during acquisition. Referring back to FIG. 2, it is also possible for the system to include a means of taking into account a change in the focal length of the optics 10 of the filming camera 9.

Thus, in the example presented above, it can be considered that all the operations have been implemented for a fixed focal length.

Of course, if the focal length is changed during shooting, the actual images 35 and 35 'of Figs. 6b and 7b will be represented with a different magnification level. It is thus possible, as shown in FIG. 2, for the zoom 42 carried by the camera 9 to include an encoder 43 for detecting the degree of rotation of a magnification ring 42, and for the computerized locating module 17 to take the magnification level determined by data transmitted by the encoder 43 is counted. This can be done, for example by repeating the locating step of FIG. 5 for a plurality of different magnifications of the optics 10 of the filming camera. 9.

In the embodiments that have been described above, the virtual object 31 is expressed directly in the real referential 2 so as to be directly viewable from the angle of view of the filming camera 9. According to an exemplary embodiment it can be provided to couple the three-dimensional model generation module with the animation module 28. Thus, the link 33, which is described in connection with FIG. 2 for exporting animations from the animation module 28 to the module of composition 32 can also be used in the other direction, to transmit to the animation module 28 the three-dimensional model 21 established for the real space. Thus, as represented in FIG. 8, it is possible to represent on the screen 44 of the computerized animation module 28 the superposition of the virtual object 31 obtained from the database animation and the three-dimensional model 21. This superposition allows on the one hand to define the transition matrix between the virtual repositories U, V, W and real X, Y, Z in which are expressed, respectively, the virtual object 31 and the three-dimensional model 21. This can also be used to define or redefine the animation during shooting. Thus, the animation module 28 may include an application comprising a set of tools represented on the screen 44 by icons 45, and for pre-defining the animation. Thus, it is sufficient to have the three-dimensional points that are generated during the learning step to generate the animation directly in anticipation of its shooting in real space. By way of example, FIG. 8 shows thick arrows representing orders of movement or resizing of the virtual object 31 in the virtual space U, V, W. It is also possible to define a transformation of the virtual object over time between a starting object, represented by the reference 31, and an arrival object, represented by the reference 46. The deformation between these two representations of the virtual object over time can be parameterized. The system just described is, of course, simplified to the extreme to facilitate its understanding.

Thus, the system which has just been described makes it possible, if necessary, to retouch the animation directly at the time of shooting, in the real space, after acquisition by the system in a learning configuration, which still allows interaction increased between the real world and the virtual world.

As shown very schematically in FIG. 9, the turning system may, in one embodiment, be used as follows.

During a first step 101, the system is implemented in an optical calibration configuration of the filming camera 9, to determine the possible optical aberrations of the filming camera. This preliminary step is for example carried out using a test pattern, and collected information can be used thereafter by the computer system 12 to computerically correct the acquisition of the filming camera 9.

Then, in a step 102, the system is used in a learning configuration, in which a learning sensor is moved into real space, to generate a three-dimensional model of the real space. This three-dimensional model is also scaled.

Then, during a step 103, in a location configuration, the relative positions of the filming camera 9 and of a registration sensor 16 are determined.

Then, during a step 104, an animation is provided from a database of virtual animations, the animation being intended to cooperate with the actual space to be filmed.

During a step 105, the filming configuration system is used, and a composite image of the real image obtained by the optical camera 9 is generated on a control screen available at the location of the filming. projection generated, for the same time frame, on the real image, according to the location data in the real repository of the filming camera 9.

During a determination step 106, if the director considers that the shooting is satisfactory, (arrow 0), taking into account the composite images generated, it ends the shooting of the video film (step 107).

If the determination step 106 shows that the shooting is not satisfactory (arrow N), we can take advantage of having all the actors on site and operators to film the scene again (return to step 105). If necessary, it will be possible during this step to have modified the animation, as described above in relation to FIG. 8.

The computerized systems described above can be made by one or a plurality of programmable machines communicating with each other, via networks, allowing the remote importation of the animations from a database if necessary. 29 remote animations. Computer objects such as keyboard, screen, mouse, processor, cables, etc. may be of a classically known type. In particular, it can be provided that the animation resulting from the animation database corresponds to a simplified animation animation intended to be present in the final video film. Then, a few weeks later, in a post-production stage, we will be able to plan the final animation from the initial animation used during the shooting, and from the acquired film. Simplified animation includes a smaller amount of data (for example at least twice as small) as the final animation.

In the same way that we have described, in relation with FIGS. 6a-6c, the generation of a projected shadow of the image of the virtual object in real space, it will be possible to use the three-dimensional model, in particular the volume model. , to manage the occultations between the objects of the real space, and the virtual objects. Thus, if it is detected that according to the angle of view of the filming camera, a part of the virtual object 31 is located behind an opaque object the real space, as defined in the three-dimensional model, a computerized subtraction module can be used to, for this time frame, subtract from the composite image the hidden part of the virtual object 31. This is possible using the position in the actual space of the filming camera, the virtual object, and an opaque object as defined by the three-dimensional model. Thus, the operator or the director, seeing on his control screen 15, 15 ', that the virtual object 31 may not be visible in the way he wishes, can immediately adapt the position of the camera. shooting.

In the example presented, it has been described in FIG. 2 that the sensor 16 and the filming camera 9 had overlapping optical fields and / or acquisition axes relatively close to being parallel. However, this is not at all an obligation and, alternatively, the sensor 16 (also called camera witness) could for example film the ceiling or the floor of the real space, for example, while the optical axis of the filming camera 9 would be approximately horizontal.

According to an embodiment as shown in FIG. 10, the registration system comprises a second sensor 16 'having at least one characteristic different from the first sensor 16, chosen for example from the position, the orientation, the solid angle of shooting, acquisition frequency, optical axis, optical field. For example, a second sensor 16 'can be oriented towards the ceiling, and a third sensor 16' 'can be oriented laterally. Each sensor 16, 16 'and 16' 'transmits to the computerized locating module the natural topographic information that it detects. The computerized registration module 17 determines the location data in the real repository of the filming camera 9 from the location data of the sensors 16, 16 ', 16' '(taken together or separately), and a comparison between the natural topographic information and the predetermined three-dimensional model of the real space.

Various steps and methods described above are innovative, in addition to their use in the general process described, and the applicant reserves the right to protect them in any suitable manner.

Claims

A video filming system in real space defined in a real repository, comprising:

a shooting camera (9), adapted to record a real image for a plurality of distinct time frames,

a tracking system in space, comprising:

. at least one sensor (16), presenting location data with respect to the turning camera (9) known for each time frame, and adapted to transmit to a computer module for locating natural topographic information detected by the sensor,

a computerized registration module (17) adapted to determine, for each time frame, location data in the actual reel of the camera (9) from the location data of the sensor, and a comparison between the natural topographic information and a predetermined three-dimensional model (21) of the real space,

a control screen (15),

a computerized composition module (32) adapted to generate on the control screen (15), for each time frame, a composite image of the real image, and a projection of a virtual image, resulting from a database of virtual animations (29), projection generated according to the location data in the real repository of the filming camera (9).

The video film shooting system according to claim 1, wherein the sensor (16) of the registration system is an optical camera having at least one of the following features:

- a solid angle of view greater than one solid angle shooting camera shooting,

- an acquisition in black and white,

a footprint at least two times smaller than a size of the filming camera,

an optical axis parallel to an optical axis of the filming camera (9),

an optical field superimposed on an optical field of the filming camera (9).

A video film shooting system according to claim 1 or 2, wherein the registration system comprises a computer generated generation module (23) adapted to generate said predetermined three-dimensional model of the real space, and wherein said sensor (16) ) is adapted to transmit topographic information detected by the sensor to the generation computer module.

A video film shooting system according to claim 3, wherein the sensor (16) is adapted to simultaneously transmit to the computer registration module and the computer generated generation module the natural topographic information detected by the sensor, and wherein the module computer generated (23) is adapted to enrich said predetermined three-dimensional model of the real space from the natural topographic information detected by the sensor.

A video film shooting system according to any one of claims 1 to 4, wherein the topographic information comprises information relating to real space geometric objects selected from points, lines, areas and volumes. .

A video film shooting system according to any one of claims 1 to 5, wherein, in a filming configuration, the filming camera (9) and the sensor (16) are fixedly attached to each other.

The video film shooting system according to any of claims 1 to 6, further comprising a location system comprising a pattern (27) adapted to be simultaneously detected by the filming camera and the sensor in a location pattern. , and a computer location module (26) adapted to determine the respective location data of the sensor and the camera from the simultaneous detection of the test pattern.

The video film shooting system according to one of claims 1 to 7, further comprising an optical calibration system comprising an optical calibration pattern adapted to be detected by the filming camera, in an optical calibration configuration, and in an optical calibration system. which the computerized registration module is adapted to determine, for each time frame, the location data in the real repository of the filming camera (9) from further optical calibration data of the filming camera determined by the optical calibration system.

The video film shooting system according to any one of claims 1 to 8, further comprising at least one of the following entities:

an inertial sensor (38) fixed on the filming camera, adapted to determine a movement of the filming camera, the computerized positioning module being adapted to determine the location data in the actual repository of the filming camera furthermore to from data provided by the inertial sensor;

a standard (25), filmable by the filming camera, the computerized composition module being adapted to put a virtual image at the scale of the real space on the basis of an image of the standard acquired by the camera of filming; a system (43) for determining a magnification parameter of the filming camera, the computerized composition module (32) being adapted to generate the composite image by taking into account said parameter.

The video film shooting system according to any one of claims 1 to 9, further comprising a computer animation module (28), comprising a virtual animation database (29), each animation comprising, for each of a set of time frames, a three-dimensional image expressed in a virtual repository, the computerized animation module being adapted to transmit said three-dimensional images to the composition module (32).

The video film shooting system according to claim 10, wherein the computer registration module is adapted to transmit the predetermined three-dimensional model (21) of the real space to the computer animation module (28).

A video film shooting system according to claim 10 or 11, wherein the computerized composition module is adapted to generate on the control screen, for each time frame, a shadow (41) of the virtual image, shadow generated according to the location data in the actual reel of the camera (9), and lighting location data in the real repository.

A video film shooting system according to any one of claims 1 to 12, wherein for a subsequent time frame, the computer registration module (17) is adapted to determine location data in the real reference frame of the filming camera (9) also from location data in the actual repository of the filming camera (9) for a previous time frame.

14. Video film shooting system according to one of any one of claims 1 to 13, wherein the computerized registration module comprises a selection module adapted to select from the geometric patterns geometric patterns of the three-dimensional model used to find the position of the filming camera in 3D space.

A video film shooting system according to claim 14, wherein the selection module compares geometric patterns of a subsequent image with geometric patterns of a previous image, associates geometric patterns present on both images and motionless in the real space, and does not preserve the other geometric patterns for comparison with the three-dimensional model .16. A video film shooting system according to any one of claims 1 to 15, wherein the registration system further comprises a second sensor (16 ') having at least one characteristic different from the first sensor (16) selected from the position , orientation, solid angle shooting, acquisition frequency, optical axis, optical field.