FR3066304A1 - METHOD OF COMPOSING AN IMAGE OF AN IMMERSION USER IN A VIRTUAL SCENE, DEVICE, TERMINAL EQUIPMENT, VIRTUAL REALITY SYSTEM AND COMPUTER PROGRAM - Google Patents

Abstract

The invention relates to a method for composing an image of a user (U1) immersed in a virtual scene, said user being placed in a real space, at least one camera being arranged in a known manner and in such a way that user is placed in a field of said camera, the method comprising the following steps: - Segmentation (E2) of a user's contour in a color intensity image (IE) acquired by the camera; - Obtaining (E4) by tracking a position and an orientation of at least one key point (PC1) of the user in a repository of real space, called "world reference"; - Constructing (E5) a volume (VE) encompassing the user in the world repository from the at least one obtained key point and a predetermined pattern; - Construction (E8) of a depth map (CP) of the user by perspective projection of the volume constructed in a repository of an image plane of the camera and writing depth values of the points of the projected volume inside segmented contour. - Composition (E11) of an output image (IS) from the segmented image of the user and a perspective projection of the virtual scene in the repository of the image plane of the camera using the depth map built.

Description

Method for composing an image of a user immersed in a virtual scene, device, terminal equipment, virtual reality system and associated computer program 1. Field of the invention

The field of the invention is that of virtual reality, in particular the immersion of one or more users equipped with a head-mounted display (for “head-mounted display”, in English) in a virtual scene. The invention can in particular, but not exclusively, be applied to the case of a virtual reality system comprising a real space in which several users can physically coexist and share, thanks to their head-mounted display, the same collaborative virtual environment. 2. Presentation of the prior art

When a user is immersed in a virtual scene displayed on the screen of his head-mounted display, he has no perception of himself or of other possible users, which can disturb his immersive experience.

To remedy this drawback, it is common in the state of the art to represent other users by avatars. However, this method does not easily recognize a given user, is not very realistic and can even be disturbing.

We also know from the document of Suma et al. Entitled "Sharing Space in Mixed and Virtual Reality Environments Using a low-Cost Depth Sensor", published by the IEEE International Symposium on Virtual Innovation conference, in Singapore, in March 2011, a method allowing a user placed in a known real environment and immersed in a virtual scene to perceive another user placed in the same real environment and immersed in the same virtual scene as him. The proposed approach consists in placing an RGB-D camera on the user's augmented reality helmet according to the direction of his gaze, and in acquiring a color image and a depth image of the other user. A 3D point cloud of the real scene is constructed from these images. The point cloud obtained is positioned in the virtual scene relative to the position of the camera's depth sensor. This position is known because it is relative to the position of the virtual reality headset on which the camera is positioned, the position of the headset being followed by a tracking system, for example by infrared. A segmentation of the point cloud is implemented using information known from the real environment (height above ground, ceiling). The point cloud is then projected into the helmet image plane via a perspective projection. The management of occultations between the segmented silhouette of the second user and elements of the virtual scene is implicit since this point cloud is positioned in a 3D frame of reference of the scene.

An advantage of this method is that the depth maps make it possible to easily manage any occultations between the segmented silhouette of the second user and objects in the virtual scene.

One drawback of this method is that depth cameras produce depth maps, the accuracy of which drops when the distance between the user and the camera increases, typically from 4 or 5 meters or when it becomes small, typically below of 0.80 meters, which has a strong impact on the quality of the segmentation, especially of the hands.

Finally, from the document published under the number US2008 / 0030429, a virtual reality system is known, which, although devoid of a depth camera, generates a rendering image which mixes a segmented image of a part of the body of a user equipped with 'a head-mounted display, for example its legs, with the virtual scene. In this way, the user keeps a perception of his own body in the virtual environment.

A disadvantage of this second solution is that the real image is placed in the foreground of the composition, which may be suitable for the legs, but may pose a problem with the hands by creating a feeling of discomfort in the user. case of inconsistency between a poorly managed occultation and his perception of perspective (binocular perception). In addition, this method does not make it possible to offer a correct perception of other users present in the same real space. 3. Objectives of the invention The invention improves the situation. The invention particularly aims to overcome these drawbacks of the prior art.

More specifically, an objective of the invention is to propose a solution which makes it possible to insert the segmented image of a user into a virtual scene in a simple and effective manner, by managing possible occultations with objects of the virtual scene.

Another objective of the invention is to propose a solution which is robust to variations in distance between the camera and the user. 4. Statement of the invention

These objectives, as well as others which will appear subsequently, are achieved using a method of composing an image of a user immersed in a virtual scene, said user being placed in a real environment, at at least one camera being arranged in a known manner and such that the user is placed in a field of said camera, the method comprising the following steps:

Segmentation of a user contour in a color intensity image acquired by the camera;

Obtaining a position and an orientation of at least one key point of the user in a repository of the real scene, called world repository;

Construction of a volume including the user in the world repository from at least one key point obtained;

Construction of a user depth mask by projecting the volume constructed in a repository of a camera image plane and writing the depth values of the points of the volume projected inside the segmented contour; and

Composition of an output image from the segmented image of the user and a perspective projection of the virtual scene in the repository of the image plane of the camera using the constructed depth map. The invention provides a robust and effective solution for composing a real image of a user and the virtual scene in which he is immersed. It is based on an entirely new and inventive approach which consists in segmenting the silhouette of the user in the real image and in constructing, according to the perspective of the camera which acquired the real image, a map depth of a volume encompassing the user limited to the points of the volume corresponding to the interior of its segmented silhouette. This depth map is then used to position the segmented image of the user in front of or behind the projected elements of the virtual scene and thus compose a view of the user inserted into the virtual scene.

The encompassing volume therefore remains invisible in the final composition image, since it only intervenes to construct the depth map associated with the image of the segmented silhouette of the user.

Unlike the prior art based on a depth camera, the solution of the invention does not see its performance decrease when the distance between the user and the camera increases or decreases very strongly. In addition, it produces, thanks to prior knowledge of the real space in which the user operates, a segmentation of this user carried out in real time and with good quality.

By cleverly combining the information from the segmentation of a 2D image produced by the camera with that of a volume encompassing the user in the virtual scene, it builds a simplified depth map of the scene seen by the camera , which allows you to manage occultations between virtual and real objects in an uncomplicated and efficient way.

According to one aspect of the invention, the method comprises a step of constructing a mask image of the user by writing a first value for an image element located on or inside the outline and a second value, distinct from the first, for an image element situated outside the contour and the depth map is constructed by writing the depth values of the encompassing volume projected at the positions of the image elements associated with the first value in the image hides.

An advantage of this mask image is that it is very easy to handle and makes it possible to quickly construct the depth map of the segmented user.

According to another aspect of the invention, the composition step comprises, for a point of the virtual scene projected in the image plane, which is collocated with a point of the image of the segmented user, a comparison of their values depths, the intensity of the output image at this point being chosen equal to that of the point of the projected virtual scene or of the image of the segmented user which is associated with the lowest depth value.

In this way, the occultations are managed in a simple and efficient way.

According to another aspect of the invention, a second key point is obtained by following a lever held in hand by the first user.

An advantage is to increase the precision of the positioning of the volume including the user in the real space, without increasing the constraints which weigh on the user. Indeed, it needs this controller to interact with the virtual scene, for example to move objects of this scene.

According to yet another aspect of the invention, the volume model including the user is a box in the shape of a rectangular parallelepiped oriented according to an orientation of the at least one key point.

An advantage of this model is its simplicity of implementation.

According to another aspect of the invention, the volume model including the first user comprises a plurality of nested cylinders capable of moving with respect to each other according to predetermined mechanical stresses.

An advantage of this second model is that, although more complex, it allows more precise and more faithful modeling of the user. The volume thus constructed corresponds better to its silhouette, which makes it possible to limit errors when several users are present in the field of the camera. Indeed, if a part of a neighboring user is segmented at the same time as the first user in the input image, the fact of constructing a precise encompassing volume will make it possible to eliminate the segmented part corresponding to the neighboring user in the image masks the first user and therefore does not associate this part of the neighboring user with an erroneous depth.

According to yet another aspect, the real space is delimited by walls of monochromatic color and the segmentation step implements a technique called chroma-key segmentation.

An advantage is of this real space known beforehand is that the segmentation of the user or users is facilitated and can be carried out in real time and with precision.

According to a first embodiment of the invention, the at least one camera is placed outside the real space.

One advantage is being able to show other people what the first user immersed in the virtual scene does and sees. In this case, the display device is a screen located outside the real environment in which the first user operates.

According to a second embodiment, the at least one camera is placed on a head-mounted display of a second user located in real space.

An advantage is to allow the second user to see the first user move in the virtual scene as he perceives it. The display device is in this case the helmet screen placed in front of the user's eyes. The invention also relates to a device for composing an image of at least one user immersed in a virtual scene suitable for implementing the method according to any one of the particular embodiments defined above. This device could of course include the various characteristics relating to the composition process according to the invention. Thus, the characteristics and advantages of this device are the same as those of the composition process, and are not described in more detail.

According to a particular embodiment of the invention, such a composition device is included in a terminal equipment, comprising a transceiver module capable of receiving information relating to the virtual scene, the image acquired by the at least a camera, position and orientation tracking information for at least one key point of the user and a display device capable of displaying the composite image.

According to another aspect of the invention, the terminal equipment comprises a head-mounted display capable of being worn by a second user, a camera disposed on the head-mounted display, arranged in a direction of gaze of the second user and capable of acquiring the image of input and a screen placed before the eyes of the second user and on which the output image is restored.

In this way the second user visualizes the first user integrated into the virtual scene in which he himself is evolving.

Correlatively, the invention also relates to a virtual reality system comprising:

A tracking module arranged to follow a position and an orientation of at least one key point of the user in a system repository, called the world repository, said at least one key point being located on the head-mounted display;

At least one camera arranged so that the user is placed in a field of vision of said camera;

A module for generating a virtual scene SV;

Terminal equipment according to the invention; and

A first head-mounted display capable of being worn by the user and intended to restore to the user a visual representation of the virtual scene; The invention also relates to a computer program comprising instructions for implementing the steps of a composition method as described above, when this program is executed by a processor.

This program can use any programming language. It can be downloaded from a communication network and / or saved on a computer-readable medium. The invention finally relates to recording media, readable by a processor, integrated or not in the pose estimation device according to the invention, possibly removable, respectively storing a computer program implementing a composition method as previously described. 5. List of Figures Other advantages and characteristics of the invention will appear more clearly on reading the following description of a particular embodiment of the invention, given by way of simple illustrative and nonlimiting example, and appended drawings, among which: FIG. 1 schematically presents an example of a virtual reality system according to a first embodiment of the invention; FIG. 2 schematically presents an example of a virtual reality system according to a second embodiment of the invention; FIGS. 3A and 3B schematically show a front view and a side view of a stereo camera implemented by the virtual reality system according to an embodiment of the invention; FIG. 4 schematically presents the steps of a method of composing an image of a user with the virtual scene in which he is immersed, according to the invention; FIG. 5 illustrates an example of segmentation of the user on an image acquired by the stereo camera according to a technique called segmentation on a monochromatic background; FIG. 6 schematically illustrates an example of a mask image produced at the end of the segmentation step according to the invention; FIG. 7 schematically illustrates the step of obtaining position and orientation information from at least one key point of the user by monitoring, according to the invention; FIGS. 8A to 8D schematically illustrate views of a volume including the user based on a box model; FIGS. 9A and 9B schematically illustrate two views of a volume including the user based on a model of articulated cylinders; FIG. 10 schematically illustrates a virtual camera and its view pyramid; and FIG. 11 schematically illustrates an example of a material structure of a device for composing an image of the user immersed in the virtual scene according to the invention. 6. Description of a particular embodiment of the invention

As a reminder, the principle of the invention is firstly to segment the outline of a user on at least one image acquired by a camera in a real space, to calculate secondly a volume including this user in a repository real space, known as the world repository, using at least one positional and rotational tracking information for this user and a predetermined model. The segmented contour of the user in the acquired image and a projection of the volume calculated in a frame of reference of an image plane of the camera are then used to construct a depth map associated with the image of the user. This card allows when composing an image, seen by the camera, of the user immersed in the virtual scene, to take into account possible occultations of the user by objects of the 3D virtual scene.

In relation to FIG. 1, a first example of a virtual reality system or platform is presented according to a first embodiment of the invention. This platform 10 includes in particular:

A real space 20, for example a room whose walls are of a uniform monochromatic reference color, and in which evolves at least a first user Ui;

At least one camera 30 capable of acquiring a sequence of images of color intensities, or RGB camera (for “Red Green Blue”, in English) and arranged so that the first user Ui is present in its field of vision. For example, it is fixed to a mast at the edge of the real space 20, from which it has an external point of view on this space;

A module for generating a 3D virtual scene SV 40, for example integrated into a machine for calculating terminal equipment ET, such as a personal computer (for “Personal computer”, in English), a tablet, a telephone intelligent (for “smartphone”, in English) etc;

A 50i virtual reality or head-mounted headset with which the first user Ui is equipped.

This headset comprises at least one screen equipped with a lens system which is placed before the eyes of the user and on which it is capable of reproducing a 3D image of the virtual scene SV adapted from the point of view of the user. It also includes a module for transmitting / receiving data via wireless communication. In this way, it is able to receive information relating to the virtual scene, for example relating to the position of objects of this scene, coming from the generation module 40;

At least one joystick 60i intended to be held in hand by the first user Ui and by means of which it is able to interact with the virtual scene, which it visualizes on a display device integrated into its head-mounted display 50i;

A tracking system 70a, 70b capable of tracking a position and an orientation of at least one key point of the real space 20, such as for example a point PCi of the head-mounted display 50i, a point of the controller 60i or even a point PCo of the stereo camera 30. This system can be based for example on reflective balls fixed at the key points, these balls being locatable using infrared technology, or on inertial and magnetic sensors . It further comprises a transmission / reception module, for example based on wireless communication technology, capable of transmitting positional and orientational tracking information to an image composition device 100 according to the invention;

ET terminal equipment, in which the device 100 is integrated;

A display device capable of reproducing an output image which inserts the segmented user into the virtual scene SV. For example, as illustrated in Figure 1, this is that of the AND terminal equipment;

A frame of reference R known as “world frame of reference” of real space which coincides with that of the 3D virtual scene. We indeed make correspond a virtual reference frame of the virtual scene SV on the scale 1 with this world reference frame.

In this example, the stereo camera 30 is placed outside the real space 20 in which the first user Ui moves. It thus has an external point of view on the user Ui which evolves in real space, which will make it possible to reconstruct an image of the user Ui integrated into the virtual scene. In this case, to ensure a good registration between virtual scene and real space, it is necessary to perform a prior calibration of the real camera according to the Pinhole model (for “pinhole”, in English). This allows to initialize a projection model of a virtual camera which will be used in the following description to build the output image. One possible application is to show an outside audience what the user Ui sees immersed in the virtual scene SV.

In connection with FIG. 2, a second example of a virtual reality system according to the invention is presented. It is a platform 10 ′ similar to platform 10, except that it includes a stereo camera 302 which is placed on the virtual reality headset 502 of a second user U2 according to the direction of her stare. This second user U2 is located inside the real space 20 similar to that of the system 10. The camera 302 is arranged so that the first user Ui is placed within his field of vision. In this way, it is possible to reconstruct, from the point of view of the second user U2, an image of the first user Ui immersed in the virtual scene SV. In this second example, the module for generating the virtual scene 402 is advantageously placed in the head-mounted display 502 of the second user U2. Alternatively, it can be placed in the ET equipment. In this case, the description information of the virtual scene and their updates are transmitted to the head-mounted display 502 by a wireless connection. As regards the composition device 100 according to the invention, it can also be integrated into the head-mounted display 502.

In both embodiments, the camera 30, 302 is advantageously a stereo camera, as illustrated in FIGS. 3A (front view) and 3B (side view). Such a stereo camera 30, 302 actually comprises two cameras Ci, C2 arranged in relation to each other so as to produce, as the human visual system would do, two images of the same scene, according to two points of view slightly offset and allowing the reconstruction of a 3D representation of the scene. The distance d between the cameras Ci, C2 is known. It is possibly adjustable so as to better correspond to the characteristics of the user's visual system. This is particularly advantageous in the case of the head-mounted display, which includes a screen for each eye of the user and reproduces on each of them a different image.

In the following, for simplicity, we will consider the acquisition of a single image or of a single sequence of images by the camera 30, 302. Of course, if it is a stereo camera, the treatments described will be applied to each of the images or sequences of images acquired by the cameras C1, C2 and two output images will be composed, one for the left eye and one for the right eye of the user U2 or in the first example ( Figure 1) of a member of the public. In the case of a composition for a head-mounted display, these two images are distorted so as to take into account the optical parameters of the headphones and ensure good registration between real space and virtual scene on the screen.

In relation to FIG. 4, the steps of a composition process according to an embodiment of the invention are now described.

During a step Ei, an image of color intensities is obtained, called the input image IE, acquired using at least one camera 30, 302. In the following, we consider a single image IE. Of course, if you have a stereo camera, two images are obtained and the process steps are applied to each of them.

It is further assumed that at least the first user Ui is placed in the field of this camera. The acquired image therefore comprises, at least in part, the first user Ui. Of course, it can also include multiple users. For example, if it is a camera 30 located outside of the real environment 20, as illustrated in FIG. 1, it can show all the users present in the room 20.

In the following, it is assumed for simplicity that only the user Ui is visible on the input image IE. Of course, the invention is not limited to this example, but also relates to the case where several users operate in real space 20 and are simultaneously visible on an image acquired by the camera 30, 30.

During a step E2, illustrated in FIG. 5, a contour of the silhouette of the user Ui is produced on the input image IE. To do this, a segmentation technique known per se is used, such as for example the technique known as segmentation on a monochromatic background (for “chroma-key segmentation”, in English).

A simple version of the so-called “chroma-key” technique can be described as follows:

For each pixel of the color image, a distance is evaluated between the reference color intensity, which corresponds to that of the walls of the room 20, and the color intensity of the pixel. This distance can be measured in different color spaces, including RGB, YUV, YCbCr spaces.

If, for the current pixel, the measured distance is greater than a determined threshold, it is decided that it is not a background pixel:

A first value, for example equal to 1, is written at the position of this pixel in a mask image IM, illustrated by FIG. 6, of dimensions equal to those of the input image IE;

The color intensity value of the pixel is written in a buffer color image IC (for “buffer color image”, in English).

If, on the contrary, the distance measured is less than or equal to the determined threshold, it is decided that it is a background pixel:

A second value, for example equal to 0, is written in this pixel in the mask image IM;

Advantageously, it is assumed that the mask image IM and the buffer color image IC have been initialized with zero values. In this way, we only write to the positions of the pixels which do not belong to the background, but form part of the silhouette of the first user Ui.

As a variant, when the real space is not a room whose walls are of uniform monochromatic color, it is possible to segment the outline of the silhouette of the user Ui from a more complex background, using for example, a pattern recognition technique, based on classification and prior learning, as described for example in the document by Nakajima et al., entitled "Full-body person recognition system", published in the journal "Journal of the Pattern Récognition Society ”, in 2003, by Elsevier. At the end of this step E2, we therefore have: - the color buffer image IC of a virtual camera collocated with the camera 30, 302 where only the users appear and where all the background pixels have been deleted; the mask image IM whose “lit” pixels, that is to say of value 1, correspond to the silhouette of the user Ui. It will be noted that, according to a particular embodiment, the information contained in this mask image corresponds to the information generally stored in a buffer memory SB, (for “stencil buffer”, in English) and conventionally used by a graphical interface called API (for "Application Program Interface". This mask image makes it possible to delimit rendering areas in the color buffer image IC and to determine automatically, and very quickly, the pixels whose color intensities are to be written in a color rendering buffer ITC of an image of IS output.

During a step E3 illustrated in FIG. 7, position and orientation information is obtained from at least one key point of the user Ui and from at least one key point PCo of the camera 30, 302 in the world repository. This position and orientation information is advantageously delivered by the monitoring system 70. For example, we consider a first key point PCi placed on the head-mounted display 50i of the user Ui, a second key point PC2 placed on the controller 60i. Of course, additional key points can be used, for example a key point placed on a second controller 602 held by the other hand of the user Ui or else key points PC3, PC4 placed on devices fixed to his ankles.

In E4, a volume VE including the user Ui is calculated from the key point or points obtained and from a predetermined geometric and mechanical model. It is therefore with this volume to approximate the morphology of the user and to position him in the world R repository.

Several embodiments are envisaged:

According to a first approach of the invention, illustrated by Figures 8A and 8B, the calculated volume VE takes the form of a parallelepiped or box.

If only one key point PCi has been obtained, the box is positioned and oriented according to the position and the orientation of this key point. The dimensions (height, width and depth) of the box are chosen large enough to guarantee that the silhouette of the user remains contained in the box when he spreads or raises his arms. The lowest point of the box is located at ground level (minimum zero z coordinate in the world repository).

In relation to Figure 7, if three key points PCi, PC2 and PC3 are known, the dimensions of a box are calculated from the three key points. Using predetermined heuristic rules, which help define the model, we determine a minimum depth and width of the box (in case the key points are perfectly aligned). As an example, we now detail the calculation of a rectangle CR encompassing the three key points PCi, PC2, PC3 in the horizontal plane (fixed there): o This rectangle CR is centered and oriented according to the key point PCi of the head; o As illustrated by Figure 8C, we derive the positions (x ', z') of the other two key points PC2, PC3 in a frame (Ο ', χ', ζ ') centered in PCi and we calculate the bounds of the rectangle CR and the positions of its 4 vertices Tr, Tl, Br and Bl as follows: vmax.x '= Max (PC2.x', PC3.X ', Ow) vmax.z' = Max (PC2.z ' , PC3.Z ', Od) vmin.x' = Min (PC2.x ', PC3.X', -Ow) vmin.z '= Max (PC2.z', PC3.Z ', -Od) where Ow and Od respectively denote a minimum width and depth of a user's shoulders and vmin, vmax two vectors representing respectively a minimum distance and a maximum distance between the bounds of the rectangle and the key point PCi.

We thus obtain in the world reference frame R:

Tr.x = PCi.x-i- vmax.x '

Tr.z = PCi.z + vmax.z '

Tl.x = PCi.x + vmin.x '

Tl.z = PCi.z + vmax.z '

Br.x = PCi.x + vmax.x '

Br.z = PCi.z + vmin.z '

Bl.x = PCi.x + vmin.x1

Bl.z = PCl.x + vmin.z '

As illustrated in Figure 8D, we then build a box connecting two instances of this rectangle (on the vertical axis):

The base rectangle takes the ground level as value y = 0

That of the top takes a value in y such that (with PCi the key point of the head): y = Max (PCi.y + OH, PC2.y, PC3.y) OH is a predetermined value which refers to the deviation on the vertical axis between the key point PCi of the head and the skull of the user. It can be chosen wide enough to suit all users.

If 5 key points PCi to PC5 are known (heads, hands and feet), the preceding calculation is extended by taking into account the two additional points PC4, PC5.

According to a second approach to the invention, the encompassing volume model used is a set of articulated cylinders, as illustrated in Figures 9A and 9B. The articulation of the cylinders is subject to predetermined rules, intended to respect those of a human body. This more elaborate model allows to reproduce more precisely the morphology of the user.

If three key points PCi, PC2, PC3 are known at the level of the head and the hands, a first cylinder CYi is used to represent the trunk and the legs of the user. This cylinder starts on the ground and ends a little higher than the known position of the key point PCi of the head. Then two cylinders CY2, CY3 are used to model each of the arms in order to make the shoulder-elbow, elbow-hand links. The position of the shoulders and elbows are calculated using a technique known from the prior art, for example the document by Badler et al., Entitled "Real-Time Inverse Kinematics of the Human Arm", published by l 'University of Pennsylvania, http://repository.upenn.edu/hms/73. in 1996, which describes an inverse kinematics algorithm capable of taking into account the mechanical stresses of the joints of the arm of a human being. If 5 key points PCi to PC5 are known at the level of the head, hands and feet, the legs are separated from the trunk part by modeling each leg using two cylinders CY4, CY5. In the same way, the articulations between these two cylinders at the level of the pelvis, the knee and the foot, are calculated by the reverse kinematic technique previously mentioned. At the end of this step E4, there is a volume VE including the user Ui, whose position is known in the world reference system R.

During a step E5, the tracking information provided by the system 70a, 70b is used to position the key point PCo of the real camera 30, 302 in the world repository. It is assumed for example that the key point PCo is positioned on the top of the camera 30, 302 and that the position of this key point relative to the camera is known. In FIGS. 3A and 3B, the key point PCo has been shown placed at a height h above the two cameras Ci and C2, at a distance d / 2 from each of them. We understand that we can easily deduce from the knowledge of the coordinates of the PCO key point those of each of the cameras in the world reference system.

In relation to FIG. 10, a virtual camera Cv is then placed at the position obtained from the key point PCo and it is oriented in accordance with the real camera 30, 302. We also consider a reference frame Rc of this virtual camera. We then project in E6 the volume VE encompassing the user Ui previously constructed in the repository Rc of the virtual camera Cv using extrinsic parameters of the camera Cv, corresponding to its placement (rotation, translation) in the world repository R , then, according to a perspective projection model of the virtual camera and a PV view pyramid (for “frustum”, in English) of this camera, in an RI repository of a PI image plane of this camera. In FIG. 10, the plane PI is located at the level of the closest plane PP of the view pyramid. We understand that at this stage we only keep the points of the volume which are included in this pyramid and that we store in memory a depth of each of these points, this depth being linked to a distance between the camera and the point of the VE volume before projection.

We then build in E? a depth map image CP by writing at the position of each point projected in the plane PI its associated depth.

Two cases are possible:

In the IE input image, this point is located inside the segmented contour of the user Ui, that is to say that it corresponds to a value 1 in the mask image IM. In this case, the depth calculated for the volume point is written to the depth map CP; or

On the contrary, this point is outside the segmented contour of the user Ui, that is to say that it corresponds to a value 0 in the mask image IM. In this second case, nothing is written in the depth map CP. It is therefore assumed that this image is initialized to zero.

In E8, we obtain information describing the virtual scene SV, for example a set of mesh and texture of the 3D objects constituting this scene. For example, a facet of a mesh of an object is described by the coordinates of its vertices in a frame of reference for this object and a texture coordinate by vertex, that is to say a position in a texture image.

During a step Eg, the objects of the virtual scene are projected into the image plane PI of the virtual camera, in two steps, similar to that already described for the volume including the user. First we project the virtual objects into the reference frame Rc of the virtual camera using the pose of the camera 30, 302, then we use the perspective projection model of the virtual camera to project the facets of the mesh in the RI repository of the image plane, according to the camera view pyramid. It is a "rasterization" operation.

We calculate a depth information for each point of a facet projected on the screen. This depth is a function of a distance from the facet to the camera.

The color of the dots thus projected on the screen is then determined via various lighting and shading calculations, known to those skilled in the art.

In Eio, an IS output image is composed from the virtual scene projected in the screen repository and the image of the segmented user. For each point of the virtual scene thus projected, its depth is compared to that recorded at the same point in the depth map CP previously constructed. If its depth is greater than that of the depth map, this point of the virtual scene is placed behind the user Ui and does not obscure it; therefore, the color buffer image IC is not modified. If on the contrary, its depth is less than that of the mask image of depth IM, this point of the virtual scene is placed in front of the user Ui and the occult. Consequently, its color intensity is written in place of that of the input image in the color buffer image IC.

For example, we use a depth test algorithm (for "depth test" in English), described in the document by Catmull, entitled "A subdivision algorithm for computer display of curved surfaces", published by the University of Utah in 1974, pages 31 to 34 and available at the following address httBS.://staticJ1AÎsfls^cMxgm/static/f/552576Z6419248Z12705ŒZLZ3137Zcatmull thesis.pdf.

With the invention, the occultations between the user and the virtual scene are therefore very easily managed using the depth map CP.

An IS output image is thus obtained, which can be reproduced on a display device. In the first embodiment, the display device can be a monitor placed outside of room 20 or the screen of a terminal equipment AND, as previously described in relation to FIG. 1. The output image IS displayed thus shows the user Ui immersed in the virtual scene SV from the point of view of the camera 30.

According to the second embodiment of the invention, the camera 3Ü2 is that of the head-mounted display 5Ο2 of a second user U2. The rendering device is therefore the screen of his helmet placed in front of his eyes and the output image is presented to him presenting the first user Ui immersed in the virtual scene SV, the point of view of the camera 3Ü2 coinciding with his own point. of view.

It will be noted that the invention which has just been described, can be implemented by means of software and / or hardware components. In this perspective, the terms "module" and "entity", used in this document, can correspond either to a software component, or to a hardware component, or even to a set of hardware and / or software components, capable of implementing performs the function or functions described for the module or entity concerned.

In relation to FIG. 10, an example of a simplified structure of a device 100 for composing an image of at least one user immersed in a virtual scene according to the invention is now presented. The device 100 implements the composition method according to the invention which has just been described.

This figure 10 illustrates only one particular way, among several possible, of carrying out the algorithm detailed above. Indeed, the technique of the invention is carried out indifferently on a reprogrammable computing machine (a PC computer, a DSP processor or a microcontroller) configured to execute a program comprising a sequence of instructions, or on a dedicated computing machine ( for example a set of logic gates such as an FPGA or an ASIC, or any other hardware module).

In the case where the invention is implemented on a reprogrammable calculation machine, the corresponding program (that is to say the sequence of instructions) may be stored in a removable storage medium (such as for example a floppy disk, CD-ROM or DVD-ROM) or not, this storage medium being partially or completely readable by a computer or a processor.

For example, the device 100 comprises a processing unit 110, equipped with a processor pl, and controlled by a computer program Pgl 120, stored in a memory 130 and implementing the method according to the invention. On initialization, the code instructions of the Pgi computer program 120 are for example loaded into a RAM memory before being executed by the processor of the processing unit 110. The processor of the processing unit 110 puts implementing the steps of the method described above, according to the instructions of the computer program 120.

In this exemplary embodiment of the invention, the device 100 comprises a reprogrammable calculation machine or a dedicated calculation machine, capable of and configured for:

Obtain a color intensity image of at least one camera arranged in a known manner and so that the user is placed in his field;

Segment a contour of the user in the image of intensities of colors obtained;

Obtain a position and an orientation of at least one key point of the user in a repository of real space;

Build a volume encompassing the user in the scene repository from the at least one key point obtained;

Build a depth mask by projecting the volume built into a repository of an image plane of the camera and writing the depths of the volume projected at the points located inside the segmented contour; and

Compose the segmented image of the user with a projection of the virtual scene in the repository of the camera image plane, using the built-in depth mask.

The device 100 further comprises a Mi 140 storage unit, such as a memory, for example of the buffer memory type capable of storing for example the depth map CP constructed and, advantageously, the mask image IM of the segmented user .

These units are controlled by the processor μΐ of the processing unit 110.

Advantageously, such a composition device 100 can be integrated into an AND terminal equipment. In this case, the display device can be that of the terminal equipment ET, which is part of the virtual reality system 10, 10 ′.

The device 100 is then arranged to cooperate at least with the following modules of the terminal equipment AND: a data transmission / reception E / R module, by means of which it receives an image acquired by the camera or information tracking from the module 70a, 70b or even description information of the virtual scene SV; and - the display device DISP, configured to reproduce a composition of an output image IS.

Alternatively, the display device is for example a monitor or a television set located outside the real space 20.

According to the second embodiment of the invention, illustrated by FIG. 2, the terminal equipment ET is the head-mounted display 5Û2 of a second user U2 located in the real space 20. The device 100 is then integrated into the head-mounted display 5Û2 and the output image displayed on the head-mounted display.

Thanks to its good performance and its simplicity of implementation, the invention which has just been described allows several uses. A first application is to give a user of a virtual reality system, a perception of themselves and of possible other users present in the same real space.

A second application envisaged is to set up a “window” on the virtual scene in which one or more users of a virtual reality system are immersed, to allow an external audience to “share” their immersive experience in a way.

It goes without saying that the embodiments which have been described above have been given for purely indicative and in no way limitative, and that numerous modifications can be easily made by those skilled in the art without departing from the scope. of the invention.

Claims (15)

1. Method for composing an image of a user (Ui) immersed in a virtual scene (SV), said user being placed in a real space (20), at least one camera (30, 302) being arranged so known and such that the user is placed in a field of said camera, the method comprising the following steps: Segmentation (E2) of a contour of the user in an image (IE) of color intensities acquired by the camera ; Obtaining (E4) by monitoring a position and an orientation of at least one key point (PCi) of the user in a repository of real space, called "world repository"; Construction (E5) of a volume (VE) including the user in the world repository from at least one key point obtained and from a predetermined model; Construction (E8) of a depth map (CP) of the user by perspective projection of the volume in a frame of reference of an image plane of the camera and writing of depth values of the points of the volume projected inside the contour segmented, a depth value of a projected point being a function of a distance between a corresponding point in volume and the camera in the world reference frame; Composition (Eli) of an output image (IS) from the segmented image of the user and a perspective projection of the virtual scene in the repository of the image plane of the camera using the map of built depth. 2. Method according to claim 1, characterized in that it comprises a step (E3) of construction of a mask image (IM) of the user (Ui) by writing a first value for an image element located on or inside the contour and with a second value, distinct from the first, for an image element located outside the contour and in that the depth map (CP) is constructed by writing the depth values of the projected encompassing volume corresponding to the positions of the image elements associated with the first value in the mask image (IM). 3. Method according to one of claims 1 or 2, characterized in that the composition step comprises, for a point of the virtual scene projected in the image plane, which is collocated with a point of the image of the segmented user, a comparison of their depth values, the intensity of the output image at this point being chosen equal to that of the point of the projected virtual scene or of the image of the segmented user which is associated with the lowest depth value. 4. Method according to one of the preceding claims, characterized in that a second key point (PC2, PC3) is obtained by monitoring at least one lever held in hand by the first user. 5. Method according to one of the preceding claims, characterized in that the volume model including the user is a box in the shape of a rectangular parallelepiped localized according to a position and an orientation of the at least one key point. 6. Method according to one of claims 1 to 4, characterized in that the volume model including the first user comprises a plurality of nested cylinders capable of moving relative to each other according to predetermined mechanical stresses. 7. Method according to one of the preceding claims, characterized in that the real space comprises a space delimited by walls of monochromatic color and in that the segmentation step implements a technique known as chroma-key segmentation. 8. Method according to one of the preceding claims, characterized in that the camera (30) is placed outside the real space. 9. Method according to one of the preceding claims, characterized in that the camera (3Ο2) is placed on a virtual head-mounted display (5Ο2) of a second user (U2) located in real space (20). 10. Device (100) for composing an image of at least one user immersed in a virtual scene, said user being placed in a real space, said device being able to obtain (OBT. IE) an image of intensities of color of at least one camera arranged in a known manner and such that the user is placed in a field of said camera, the device comprising a calculation machine dedicated to or configured to: Segment (SEG) a contour of the user in the color intensity image obtained; Obtain (OBT. PC) a position and an orientation of at least one key point of the user in a repository of real space, called "world repository"; Construct (CONST. VE) a volume including the user in the scene reference frame from the at least one key point obtained; Construct (CONST. CP) a depth map of the user by projecting the volume constructed in a reference frame of an image plane of the camera and writing the depth values of the points of the volume projected inside the segmented contour; and Compose (COMP.) an output image (IS) from the segmented image of the user and a projection of the virtual scene in the repository of the image plane of the camera using the depth (CP) built. 11. Terminal equipment (ET), comprising a display device (DISP) and a data transmission and reception module (MER) via a communication network, characterized in that it further comprises a device (100) for composing an image of at least one user immersed (Ui) in a virtual scene (SV) according to claim 10, said transmit reception module being capable of receiving an input image acquired by at minus a camera, information relating to the virtual scene and information for tracking the position and orientation of at least one key point of the user in a repository of real space, known as the world repository, said device display being able to display said output image. 12. Terminal equipment according to claim 11, characterized in that it comprises a head-mounted display (502) able to be worn by a second user (U2) and a camera (302) arranged on the head-mounted display, arranged in a direction of gaze of the second user and able to acquire the input image (IE) and in that the head-mounted display comprises a screen placed before the eyes of the second user and on which the output image (IS) is restored. 13. Virtual reality system (10, 10 ′), comprising: A real space (20), in which at least one first user (Ui) operates; A head-mounted display (50i) able to be worn by the user (Ui); A tracking module (70a, 70b) arranged to follow a position and an orientation of at least one key point (PCi) of the user in a system repository, called world repository, said at least one key point being located on the head-mounted display; At least one camera (30) capable of being arranged so that the first user is placed in a field of vision of said camera; A module (40) for generating a virtual scene SV; and a terminal equipment (ET) according to claim 10. 14. Computer program (Pgl) comprising instructions for the implementation of the composition method according to any one of claims 1 to 9, when said program is executed by a processor. 15. A recording medium readable by a computer, on which a computer program is recorded comprising program code instructions for the execution of the steps of the method according to one of claims 1 to 9.