WO2015185537A1 - Procédé et dispositif pour la reconstruction du visage d'un utilisateur portant un visiocasque - Google Patents

Procédé et dispositif pour la reconstruction du visage d'un utilisateur portant un visiocasque Download PDF

Info

Publication number
WO2015185537A1
WO2015185537A1 PCT/EP2015/062229 EP2015062229W WO2015185537A1 WO 2015185537 A1 WO2015185537 A1 WO 2015185537A1 EP 2015062229 W EP2015062229 W EP 2015062229W WO 2015185537 A1 WO2015185537 A1 WO 2015185537A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
image
user
hmd
mounted display
Prior art date
Application number
PCT/EP2015/062229
Other languages
English (en)
Inventor
Xavier BURGOS
Nicolas Mollet
Julien Fleureau
Olivier Dumas
Thierry Tapie
Original Assignee
Thomson Licensing
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Licensing filed Critical Thomson Licensing
Publication of WO2015185537A1 publication Critical patent/WO2015185537A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/166—Detection; Localisation; Normalisation using acquisition arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to the domain of head-mounted display, especially when used for immersive experiences in gaming, virtual reality, movie watching or video conferences for example.
  • Head-mounted displays have undergone major design improvements in the last years. They are now lighter and cheaper and have higher screen resolution and lower latency, which makes them much more comfortable to use. As a result, HMD are now at a point where they will slowly start to affect the way we consume digital content in our everyday lives. The possibility of adapting the content being watched to the user's head movements provides a perfect framework for immersive experiences in gaming, virtual reality, movie watching or video conferences.
  • One of the main issues of wearing an HMD to this day is that they are very invasive, and hide the wearer's face. In some cases, this is not an issue since the wearer of the HMD is isolated in a purely individualistic experience. However, the recent success of HMD's suggests that they will soon play a part in social interactions.
  • One example can be collaborative 3D immersive games where two individuals play together and can still talk and see each other's faces.
  • Another example is video-conferencing, where switching from traditional screens to HMD can bring the possibility of viewing the other person (and his surroundings) in 3D as if he was really there. In both cases, not seeing the other person's face clearly damages the quality of the social interaction.
  • the purpose of the present disclosure is to overcome at least one of these disadvantages of the background.
  • one purpose of the present disclosure is to reconstruct the face of a user occulted at least in part by a head-mounted display.
  • the present disclosure relates to a method of reconstructing a face of a user wearing a head mounted display in at least a first image comprising the face of the user acquired with a camera. The method comprises:
  • the method further comprises selecting the second image from a set of second images of the face of the user acquired with different poses, a second information representative of the pose of the face being associated with each second image, the selected second image corresponding to the second image having an associated second information closest to the first information.
  • the second image corresponds to at least a part of a 3D model of the face of the user.
  • determining comprises tracking visible landmarks of the face of the user.
  • determining comprises tracking the head mounted display.
  • the at least a part of the face replaced with the at least a part of the 3D model corresponds to the part of the face occluded by the head mounted display.
  • the at least a part of the face replaced with the at least a part of the 3D model corresponds to the part of the face occluded by the head mounted display.
  • the method further comprises:
  • the method further comprises feathering edges of the at least a part of the second image.
  • the present disclosure also relates to a device configured for reconstructing a face of a user wearing a head mounted display in at least a first image comprising the face of the user acquired with a camera, the device comprising at least one processor configured for:
  • the at least one processor is further configured for tracking visible landmarks of the face of the user.
  • the at least one processor is further configured for selecting the second image from a set of second images of the face of the user acquired with different poses, a second information representative of the pose of the face being associated with each second image, the selected second image corresponding to the second image having an associated second information closest to the first information.
  • the second image corresponds to at least a part of a 3D model of the face of the user.
  • the at least one processor is further configured for tracking visible landmarks of the face of the user.
  • the at least one processor is further configured for tracking the head mounted display.
  • the at least one processor is further configured for:
  • the at least one processor is further configured for feathering edges of the at least a part of the second image.
  • the present disclosure also relates to a computer program product comprising instructions of program code for executing, by at least one processor, the abovementioned method of reconstructing a face of a user wearing a head mounted display, when the program is executed on a computer.
  • the present disclosure also relates to a (non-transitory) processor readable medium having stored therein instructions for causing a processor to perform at least the abovementioned method of reconstructing a face of a user wearing a head mounted display.
  • FIG. 1 A and 1 B show a method of reconstructing the face of a user wearing a head-mounted display, according to particular embodiments of the present principles
  • FIG. 2 and 3 show the face of the user of figures 1 A and/or 1 B before and after reconstruction of his face, according to particular embodiments of the present principles
  • FIG. 4 diagrammatically shows the structure of a device configured for implementing the method of reconstructing the face of the user, according to a particular embodiment of the present principles
  • FIG. 5 shows a method of reconstructing the face of a user wearing a head-mounted display, according to a particular embodiment of the present principles.
  • Figure 1 A shows an outline of a method of reconstructing the image of a the face of a user, according to a first exemplary embodiment.
  • the first step 1 0A is performed offline and consists in recovering 2D landmarks 1 03 from a video stream 1 01 acquired with a camera 1 2 (for example a webcam).
  • the video stream 1 01 comprises a sequence of images representing the face of a user (without any object occluding the face of the user) turning his head around to have frames belonging to many head poses (for example to all possible head poses or to all possible head poses in an angle range, the angle range being for example [-1 20° 1 20"] or [-100° 1 00"] or [-90°, 90°j with 0°corresponding to the head po se when the user faces the camera 12).
  • Landmarks of the face are estimated 102 for each frame of the video stream 101 by implementing any state-of-the-art algorithm such as the one described in "Robust face landmark estimation under occlusion", written by X.P. Burgos-Artizzu, P. Perona and P. Dollar, published in International Conference in Computer Vision (ICCV), IEEE, 2013. From the landmarks 103 is estimated the pose of the head for each frame of the video stream 101 by the way of a landmark to head pose converter 104, implemented under the form of any algorithm know from the skilled person in the art, for example a linear regression method.
  • the head pose parameters i.e.
  • the 3 degrees of freedom of the head pose (roll, pitch, yaw)) are for example estimated by learning a linear regression from the set of normalized 2D landmarks to the three orientation angles (roll, pitch, yaw), using a head-pose dataset.
  • Such an estimation of the head pose parameters is advantageously performed offline and the regression weights obtained from the estimation are stored in a memory and/or hard-coded.
  • the head-pose parameters are received from a remote unit and/or from a database.
  • a second step 1 1 A one or more images 1 12 of the user wearing the head-mounted display (HMD) are acquired, advantageously live by a camera 13 such as a webcam for example.
  • the camera 13 may be identical to the camera 12 but the camera 13 may also be different from the camera 12.
  • the HMD is tracked 1 1 1 to determine its pose parameters (i.e. the 3 degrees of freedom of the HMD's pose (roll, pitch, yaw)).
  • the HMD's pose is tracked in the one or more images 1 12 for example by using the information provided by the Internal Motion Unit (IMU) (i.e.
  • IMU Internal Motion Unit
  • a set of sensors comprising accelerometer, gyroscope and magnetometer
  • external tracking techniques adapted to estimate the head translation and orientation parameters (e.g. an external infrared camera tracking infrared emitters embedded into the HMD).
  • External tracking techniques may rely for example on recognizable patterns placed on the HMD to track the pose (i.e. the orientation) and the size of the HMD.
  • a face landmark estimation algorithm may be applied to track visible face landmarks (e.g. ears, mouth and chin).
  • the head (or HMD) pose of the current image 1 12 is estimated by the use of a Landmark to head pose converter 1 13 (that may be the same as the landmark to head pose converted 104).
  • the image (i.e. video frame) 1 03 presenting the best match in terms of head pose with respect to the current image 1 1 2 is retrieved.
  • the HMD (or head) pose determined with the current image 1 12 is compared with the head-pose parameters associated with each frame (image) 1 03.
  • the image 1 03 for which the associated head-pose parameters are the closest to the head-pose parameters of the current image 1 1 2 is then selected.
  • Each degree of freedom (roll, pitch and yaw) associated with the pose of the current image 1 1 2 is advantageously compared with the corresponding degree of freedom (i.e.
  • Such a process enables to replace the upper part of the face of a person wearing a HMD (or any device occluding at least partially the face) with the own unanimated face of the person.
  • Figure 1 B shows an outline of a method of reconstructing the image of a the face of a user, according to a second exemplary embodiment.
  • the first step 1 0B is performed offline and consists in recovering a 3D (three-dimensional) model 107 of the user's face from a set of 2D (two- dimensional) images 1 05 of the face of the user taken from multiple viewpoints using an uncalibrated camera.
  • a 3D model builder 1 06 is used for building the 3D model 1 07 of the face of the user from the set of images 1 05.
  • the 3D model builder implements methods well-known from the skilled person in the art, such as autocalibration or stratification methods. Non limitative of such methods are for example described in "Object modelling by registration of multiple range images" written by Y. Chen and G.
  • a second step 1 1 B one or more images 1 1 2 of the user wearing the head-mounted display are acquired, advantageously live by a camera 1 3 such as a webcam for example.
  • the orientation and the position, both corresponding to the pose, of the face in the world space, i.e. in the 3D space in which the user moves, are estimated.
  • the orientation and location of the face are for example determined by using a face landmarks estimation algorithm applied to track visible face landmarks, such as for example ears, mouth and/or chin.
  • the orientation and location of the face are determined by tracking the HMD worn by the user on his face. To that aim, information provided by the Internal Motion Unit (IMU) (i.e.
  • IMU Internal Motion Unit
  • a set of sensors comprising accelerometer, gyroscope and magnetometer
  • external tracking techniques adapted to estimate the head position and orientation in 3D.
  • External tracking techniques rely for example on recognizable patterns placed on the HMD to track the position, size and orientation of these patterns.
  • the HMD may also be tracked by using an external infrared camera tracking infrared emitters embedded in the HMD.
  • the 3D model is reprojected onto the image 1 1 2 at that location via a face replacer 1 14 to obtain the image 1 1 5, using the projection matrix associated with the camera 1 3 used for acquiring the image 1 1 2.
  • the face replacer 1 14 advantageously comprises a 3D reprojection algorithm and an in-painting algorithm.
  • the camera projection matrix describes a mapping from points in the 3D world (i.e. the space of the user) to 2D image coordinates (2D space of the image).
  • the camera projection matrix may be obtained for example via any chessboard calibration methods.
  • the image 1 15 corresponds to the image 1 1 2 onto which the part of the 3D model having the same pose as the pose of the face on the image 1 1 2 has been in-painted. The image may then be displayed.
  • FIG. 2 shows images of the face of the user wearing the HMD before and after face replacement, according to a first exemplary embodiment.
  • the image 20 (corresponding for example to the current image 1 12 of figure 1 A) represents the face 201 of the user wearing the HMD 202 acquired with a camera, for example live with a webcam.
  • Visible landmarks for example the chin 2020
  • Visible landmarks 2010 located on the HMD are also illustrated with white spots.
  • the pose of the face 201 is for example determined by using the visible landmarks of the face, as described for example in "Real time head pose estimation with random regression forests" written by Fanelli et al. and published in Computer Vision and Pattern Recognition, 201 1 .
  • the pose of the face may also be determined by using the landmarks of the HMD, in combination with information provided by the IMU associated with the HMD. According to a variant, the pose of the face is determined based only on the visible landmarks (of the face and/or of the HMD) or only on the information provided by the IMU.
  • the image 21 (corresponding for example to the final image 1 15 of figure 1 A) represents the result of the face reconstruction, according to a non-limitative example.
  • the face of the user occluded at least in part on the image 20 is replaced with a part 210 of the image selected (corresponding for example to the image 103 of figure 1 A) from the set of images of the face of the user without HMD.
  • the selected image is advantageously cropped around the pixels occupied by the HMD on the image 20 (these pixels being determined from the HMD tracking information).
  • the cropped part of the selected image is then paste on top of the current frame to obtain the final image 21 .
  • Texture information associated with the part 210 of the selected image is in-painted in place of the texture information of the HMD of the image 20, the in-painting process using the information about the location of the HMD.
  • the location of the HMD is determined (by video analysis, for example by using a machine learning method or by tracking landmarks located on the HMD, in combination with information provided by the IMU associated with the HMD).
  • the in-painting of the part of the selected image used for replacing the HMD onto the face of the user on image 20 advantageously uses information representative of the dimensions of the HMD (i.e.
  • Dimensions of the HMD are advantageously transmitted by the HMD to the computing device performing the face reconstruction.
  • the dimensions of the HMD are measured, for example based on video analysis or by tracking landmarks located for example at corners of the HMD. Then based on the information representative of the pose and location of the HMD and based on the information representative of the dimensions of the HMD, a part of the selected image corresponding to the part of the face occluded by the HMD 202 on image 20 is in-painted onto the face of the user 202, as illustrated on the image 21 .
  • the whole face of the user is replaced with texture information of the face of the selected image.
  • the result of the in-painting operation may be a patchy image, due to differences in color between the selected image and the image 20 and inconsistencies at the edges of the in-painted part(s) of the selected image.
  • a statistical color transfer is advantageously performed to alter the texture color of the selected image (or the selected parts of it) so that it matches that of the final image 21 provided by the current image 20, i.e. the texture color of the part 202 of the face not replaced by the part of the selected image.
  • ⁇ corresponding to the average and ⁇ to the standard deviation of the color information (for example RGB values associated with each pixel).
  • a p-norm feathering composition of the two layers may be performed to avoid visual inconsistencies at the borders and achieve a clean face replacement.
  • the texture color of the in-painted part of the selected image is close to the texture color of the part 202 of the face of the current image 20.
  • Figure 3 shows images of the face of the user wearing the HMD before and after face replacement, according to a second exemplary embodiment.
  • the image 30 (corresponding for example to the image 1 12 of figure 1 B) represents the face 301 of the user wearing the HMD 302 acquired with a camera, for example live with a webcam. Visible landmarks 3021 and 3022 of the face are illustrated with white spots on the face (respectively the chin and the mouth). Visible landmarks 3010 located on the HMD are also illustrated with white spots.
  • the pose of the face 301 is for example determined by using the visible landmarks of the face, as described for example in "Real time head pose estimation with random regression forests" written by Fanelli et al. and published in Computer Vision and Pattern Recognition, 201 1 .
  • the pose of the face may also be determined by using the landmarks of the HMD, in combination with information provided by the IMU associated with the HMD. According to a variant, the pose of the face is determined based only on the visible landmarks (of the face and/or of the HMD) or only on the information provided by the IMU.
  • the image 31 represents the result of the face reconstruction, according to a first non-limitative example.
  • the face of the user occluded at least in part on image 30 is replaced with a part 310 of the 3D model having a same pose information associated with than the determined face pose.
  • Texture information associated with the part 310 of the 3D model selected to replace the face of the user on the image is in-painted in place of the texture information of the face (including the HMD) of the image 30, the in-painting process using the information about the location of the face.
  • the whole face of the user is replaced with texture information of the 3D model.
  • the image 32 represents the result of the face reconstruction, according to a second non-limitative example.
  • only the part of the face occluded by the HMD on the image 30 is replaced with the corresponding part of the 3D model.
  • the location of the HMD is determined (by video analysis, for example by using a machine learning method or by tracking landmarks located on the HMD, in combination with information provided by the IMU associated with the HMD).
  • the in-painting of the part of the 3D model onto the face of the user on image 30 advantageously uses information representative of the dimensions of the HMD (i.e. length, width and depth) to replace only the part of the face occluded by the HMD.
  • Dimensions of the HMD are advantageously transmitted by the HMD to the computing device performing the face reconstruction.
  • the dimensions of the HMD are measured, for example based on video analysis or by tracking landmarks located for example at corners of the HMD. Then based on the information representative of the orientation and location of the face and based on the information representative of the dimensions of the HMD, a part 320 of the 3D model corresponding to the part of the face 301 occluded by the HMD 302 is in-painted onto the face 321 , as illustrated on the image 32.
  • HMD position information is advantageously used to select which part(s) of the 3D model is(are) kept and which removed.
  • the result of the in-painting operation is a patchy image 32, due to differences in color between the 3D model and the image 30 and inconsistencies at the edges of the in-painted part(s) of the 3D model.
  • a statistical color transfer is advantageously performed to alter the 3D model texture color so that it matches that of the target image, i.e. the texture color of the part 321 of the face not replaced by the 3D model.
  • a p-norm feathering composition of the two layers may be performed to avoid visual inconsistencies at the borders and achieve a clean face replacement.
  • the result of the statistical color transfer operation and of the feathering composition is illustrated with image 33.
  • the texture color of the in-painted part 330 of the 3D model is close to the texture color of the part 321 of the face of the original image 30.
  • Figure 4 diagrammatically shows a hardware embodiment of a device 4 configured for reconstructing at least a part of the face of a user wearing a Head-Mounted Display (HMD).
  • the device 4 is also configured for the creation of display signals of one or several images, the content of which integrating added part(s) of the face and/or original part(s) of the face acquired with any acquisition device such as a camera, for example a webcam.
  • the device 4 corresponds for example to a tablet, a Smartphone, a games console, a laptop or a set-top box.
  • the device 4 comprises the following elements, connected to each other by a bus 45 of addresses and data that also transports a clock signal:
  • microprocessor 41 or CPU
  • a graphics card 42 comprising:
  • GRAM Graphical Random Access Memory
  • ROM Read Only Memory
  • I/O devices 44 such as for example a tactile interface, a mouse, a webcam, etc. and - a power source 49.
  • the device 4 also comprises one or more display devices 43 of display screen type directly connected to the graphics card 42 to display images calculated in the graphics card, for example live.
  • the use of a dedicated bus to connect the display device 43 to the graphics card 42 offers the advantage of having much greater data transmission bitrates and thus reducing the latency time for the displaying of images composed by the graphics card.
  • a display device is external to the device 4 and is connected to the device 4 by a cable or wirelessly for transmitting the display signals.
  • the device 4, for example the graphics card 42 comprises an interface for transmission or connection (not shown in figure 4) adapted to transmit a display signal to an external display means such as for example an LCD or plasma screen or a video-projector.
  • register used in the description of memories 421 , 46, and 47 designates in each of the memories mentioned, both a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed).
  • the microprocessor 41 When switched-on, the microprocessor 41 loads and executes the instructions of the program contained in the RAM 47.
  • the random access memory 47 notably comprises:
  • 3D model for example mesh information, texture color information, etc.
  • the data 473 are for example received from a computing device via the RX/TX interface 48, for example via Wi-Fi or Ethernet.
  • the information 472 is for example received from the HMD and/or from a tracking unit via the RX/TX interface 48.
  • the algorithms implementing the steps of the method specific to the present disclosure and described hereafter are stored in the memory GRAM 421 of the graphics card 42 associated with the device 4 implementing these steps.
  • the graphic processors 420 of the graphics card 42 load these parameters into the GRAM 421 and execute the instructions of these algorithms in the form of microprograms of "shader" type using HLSL (High Level Shader Language) language or GLSL (OpenGL Shading Language) for example.
  • HLSL High Level Shader Language
  • GLSL OpenGL Shading Language
  • the random access memory GRAM 421 notably comprises:
  • a register 421 1 data representative of a new image of the face resulting from the combination of the data 471 of the original image with the data 421 0 of the selected part of the second image (e.g. the 3D model) to replace the whole face in the image 471 or a part of it, for example the part of the face occluded by the HMD.
  • the selected part of the second image e.g. the 3D model
  • a part of the RAM 47 is assigned by the CPU 41 for storage of the identifiers and the distances if the memory storage space available in GRAM 421 is insufficient.
  • This variant however causes greater latency time in the composition of an image comprising a representation of the environment composed from microprograms contained in the GPUs as the data must be transmitted from the graphics card to the random access memory 47 passing by the bus 45 for which the transmission capacities are generally inferior to those available in the graphics card for transmission of data from the GPUs to the GRAM and vice-versa.
  • the power supply 48 is external to the device 4.
  • Figure 5 shows a method of reconstructing the face of a user wearing a head-mounted display (HMD) implemented for example in a device 4, according to a non-restrictive advantageous embodiment.
  • HMD head-mounted display
  • the different parameters of the device 4 are updated.
  • the information representative of the pose (location and orientation) of the face and/or of the HMD are initialised in any way.
  • a step 51 information representative of the pose (orientation and location) of the face of a user is determined, the face of the user being comprised in at least one first image, also called original image, acquired with any acquisition device, for example with a webcam or a camera or a digital still camera. At least a part of the face is occluded by a HMD in the original image.
  • the information related to the orientation of the face is either determined based on video analysis (for example tracking of landmarks of the face or of the HMD) or based on the tracking of the HMD, in combination with information provided by the IMU of the HMD according to an optional variant.
  • the information related to the location of the face in the world space may also be determined from the face or from the HMD.
  • the information related to the location may be for example expressed with the coordinates of landmarks of the face and/or the HMD in the world space.
  • the whole face or only a part of it is replaced with at least a part of a second image representing the face of the user, by using information representative of the pose of the face, on information representative of dimensions of the head mounted display and on information representative of a location of the head mounted display provided by at least one inertial sensor of the head mounted display.
  • the whole face or only a part of it is replaced with a corresponding part of a second image, for example part of the 3D model of the user wearing the HMD.
  • the corresponding part of the 3D model is selected, the selection being based on the information related to the orientation and of the location of the face to apply the part of the 3D model matching to the orientation of the face.
  • the selected part of the 3D model is in-painted onto the original image by using the information related to the location of the face as to replace the part of the face to be replaced (either the whole face or only the part of the face occluded by the HMD).
  • the projective matrix associated with the acquisition device used for acquiring the original image is used to convert the 3D coordinates of the part of the 3D model into 2D coordinates of the original image.
  • the result of the combination of the original image and the selected part of the 3D model provides a face of the user reconstructed in a final image, i.e. without any occluded part.
  • the final image is displayed on a screen.
  • the second image is selected from a set of second images each representing the face of the user without HMD.
  • Each second image has been advantageously acquired with a different pose of the head of the user, head-pose parameters being associated with each second image to provide a second information representative of the pose of the face represented on the second image.
  • the selected second image corresponds to the second image of the set of second image for which the associated second pose information is the closest to the first pose information associated with the first image.
  • the whole face or only a part of it is replaced with a corresponding part of the selected second image.
  • the corresponding part of the selected second image is for example selected based on the information related to the location of the HMD.
  • the selected part of the selected second image is in-painted onto the first image by advantageously using the information related to the location of the HMD as to replace the part of the face to be replaced (either the whole face or only the part of the face occluded by the HMD).
  • the result of the combination of the first image and the selected second image provides a face of the user reconstructed in a final image, i.e. without any occluded part, the final image being advantageously displayed.
  • Steps 51 and 52 are advantageously reiterated for any image of a sequence of images acquired with the acquisition device.
  • the present disclosure is not limited to a method of displaying a video content but also extends to any device implementing this method and notably any devices comprising at least one GPU.
  • the implementation of calculations necessary to select the part(s) of the 3D model to be painted to the image of the face of the user is not limited either to an implementation in shader type microprograms but also extends to an implementation in any program type, for example programs that can be executed by a CPU type microprocessor.
  • the use of the methods of the present disclosure is not limited to a live utilisation but also extends to any other utilisation, for example for processing known as postproduction processing in a recording studio.
  • the implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or a device), the implementation of features discussed may also be implemented in other forms (for example a program).
  • An apparatus may be implemented in, for example, appropriate hardware, software, and firmware.
  • the methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, Smartphones, tablets, computers, mobile phones, portable/personal digital assistants ("PDAs”), and other devices that facilitate communication of information between end-users.
  • PDAs portable/personal digital assistants
  • Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information.
  • equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices.
  • the equipment may be mobile and even installed in a mobile vehicle.
  • the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD"), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory (“RAM”), or a read-only memory (“ROM”).
  • the instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination.
  • a processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
  • implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted.
  • the information may include, for example, instructions for performing a method, or data produced by one of the described implementations.
  • a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment.
  • Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal.
  • the formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream.
  • the information that the signal carries may be, for example, analog or digital information.
  • the signal may be transmitted over a variety of different wired or wireless links, as is known.
  • the signal may be stored on a processor-readable medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé de reconstruction du visage d'un utilisateur portant un visiocasque dans au moins une image comprenant le visage de l'utilisateur acquise à l'aide d'un appareil photographique. Le procédé consiste en la détermination de premières informations représentatives d'une pose du visage (201) et en le remplacement (42) d'au moins une partie du visage (201) de l'utilisateur avec au moins une partie (220) d'une deuxième image représentant le visage de l'utilisateur, le remplacement (42) utilisant lesdites informations représentatives de l'orientation et de la position du visage (201), sur des informations représentatives des dimensions du visiocasque (202) et sur des informations représentatives d'un emplacement du visiocasque (202) fournies par au moins un capteur inertiel du visiocasque (202).
PCT/EP2015/062229 2014-06-03 2015-06-02 Procédé et dispositif pour la reconstruction du visage d'un utilisateur portant un visiocasque WO2015185537A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP14305842 2014-06-03
EP14305841 2014-06-03
EP14305841.0 2014-06-03
EP14305842.8 2014-06-03

Publications (1)

Publication Number Publication Date
WO2015185537A1 true WO2015185537A1 (fr) 2015-12-10

Family

ID=53267396

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2015/062229 WO2015185537A1 (fr) 2014-06-03 2015-06-02 Procédé et dispositif pour la reconstruction du visage d'un utilisateur portant un visiocasque

Country Status (1)

Country Link
WO (1) WO2015185537A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180101227A1 (en) * 2016-10-06 2018-04-12 Google Inc. Headset removal in virtual, augmented, and mixed reality using an eye gaze database
CN109478344A (zh) * 2016-04-22 2019-03-15 交互数字Ce专利控股公司 用于合成图像的方法和设备
CN111936910A (zh) * 2018-03-13 2020-11-13 罗纳德·温斯顿 虚拟现实系统和方法
US11551427B2 (en) 2017-08-30 2023-01-10 Ronald H. Winston System and method for rendering virtual reality interactions

Non-Patent Citations (9)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "Positioning technology - Wikipedia, the free encyclopedia", 11 April 2013 (2013-04-11), XP055199758, Retrieved from the Internet <URL:https://en.wikipedia.org/w/index.php?title=Positioning_technology&oldid=549900889> [retrieved on 20150702] *
C. BAILLARD; A. ZISSERMAN: "A planesweep strategy for the 3D reconstruction of buildings from multiple images", INTERNATIONAL ARCHIVES OF PHOTOGRAMMETRY AND REMOTE SENSING, vol. 33, 2000, pages 56 - 62
FANELLI ET AL.: "Real time head pose estimation with random regression forests", COMPUTER VISION AND PATTERN RECOGNITION, 2011
MASAYUKI TAKEMURA ET AL: "Photometric inconsistency on a mixed-reality face", MIXED AND AUGMENTED REALITY, 2006. ISMAR 2006. IEEE/ACM INTERNATIONAL SYMPOSIUM ON, IEEE, PI, 1 October 2006 (2006-10-01), pages 129 - 138, XP031014660, ISBN: 978-1-4244-0650-0 *
MURPHY-CHUTORIAN E ET AL: "Head Pose Estimation in Computer Vision: A Survey", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE COMPUTER SOCIETY, USA, vol. 31, no. 4, 1 April 2009 (2009-04-01), pages 607 - 626, XP011266518, ISSN: 0162-8828, DOI: 10.1109/TPAMI.2008.106 *
O. FAUGERAS; Q.T. LUONG: "The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications", MIT PRESS, 2004
TAKEMURA M ET AL: "Diminishing head-mounted display for shared mixed reality", PROCEEDINGS / INTERNATIONAL SYMPOSIUM ON MIXED AND AUGMENTED RALITY : ISMAR 2002 ; SEPTEMBER 30 - OCTOBER 1,2002, DARMSTADT, GERMANY, IEEEE COMPUTER SOCIETY, LOS ALAMITOS, CALIF. [U.A.], 30 September 2002 (2002-09-30), pages 149 - 156, XP010620951, ISBN: 978-0-7695-1781-0 *
X.P. BURGOS-ARTIZZU; P. PERONA; P. DOLLAR: "Robust face landmark estimation under occlusion", INTERNATIONAL CONFERENCE IN COMPUTER VISION (ICCV), IEEE, 2013
Y. CHEN; G. MEDIONI: "Object modelling by registration of multiple range images", IMAGE AND VISION COMPUTING, vol. 10, no. 3, 1992, pages 145 - 155

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109478344A (zh) * 2016-04-22 2019-03-15 交互数字Ce专利控股公司 用于合成图像的方法和设备
US11568606B2 (en) 2016-04-22 2023-01-31 Interdigital Ce Patent Holdings Method and device for compositing an image
CN109478344B (zh) * 2016-04-22 2023-05-30 交互数字Ce专利控股公司 用于合成图像的方法和设备
US20180101227A1 (en) * 2016-10-06 2018-04-12 Google Inc. Headset removal in virtual, augmented, and mixed reality using an eye gaze database
US10269177B2 (en) 2016-10-06 2019-04-23 Google Llc Headset removal in virtual, augmented, and mixed reality using an eye gaze database
US11551427B2 (en) 2017-08-30 2023-01-10 Ronald H. Winston System and method for rendering virtual reality interactions
CN111936910A (zh) * 2018-03-13 2020-11-13 罗纳德·温斯顿 虚拟现实系统和方法

Similar Documents

Publication Publication Date Title
US11568606B2 (en) Method and device for compositing an image
US10055893B2 (en) Method and device for rendering an image of a scene comprising a real object and a virtual replica of the real object
US20180165830A1 (en) Method and device for determining points of interest in an immersive content
US20180144212A1 (en) Method and device for generating an image representative of a cluster of images
US11812066B2 (en) Methods, devices and stream to encode global rotation motion compensated images
WO2015185537A1 (fr) Procédé et dispositif pour la reconstruction du visage d&#39;un utilisateur portant un visiocasque
EP3236423A1 (fr) Procédé et dispositif pour composer une image
WO2018069215A1 (fr) Procédé, appareil et flux permettant de coder une transparence et des informations d&#39;ombre d&#39;un format vidéo immersif
WO2018086960A1 (fr) Procédé et dispositif de transmission de données représentant une image
EP3716217A1 (fr) Techniques pour la détection d&#39;une occlusion en temps réel
EP3310057A1 (fr) Procédé, appareil et flux permettant de coder des informations d&#39;ombrage et de transparence pour un format vidéo immersif
EP3310053A1 (fr) Procédé et appareil permettant de coder des informations de transparence pour un format vidéo immersif
WO2020012071A1 (fr) Procédé, appareil et produit programme informatique pour codage de vidéo volumétrique
EP3310052A1 (fr) Procédé, appareil et flux de format vidéo immersif
CN118196135A (zh) 图像处理方法、装置、存储介质、设备及程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15724730

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15724730

Country of ref document: EP

Kind code of ref document: A1