JP2019121074A - Selfie device, method and program in vr space - Google Patents

Abstract

To provide a device, a method, and a program capable of self-taking a picture as if a person, who is experiencing virtual reality by wearing a head-mounted display, were in a virtual reality space by combining an image taken by the person in real space with an image of the virtual reality space.SOLUTION: A selfie device includes a processor configured to place in a virtual reality (VR) space a body region image representative of a body in a real space of a user wearing a head mounted display (HMD), and superimpose an image of a face of the user who is not wearing the HMD on an image in the VR space in which the body region image is arranged.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technology for combining an image taken in a real space and an image of a space in virtual reality.

  Conventionally, by looking at an image projected on a head mounted display (hereinafter, also referred to as an HMD) or a huge screen, it is possible to experience as if it were actually in the space of virtual reality represented by the image. A technology of virtual reality (hereinafter also referred to as VR) that can be done has been developed. In the case of a VR using an HMD, a person wearing the HMD can further immerse in the VR world because it is disconnected from the real world.

  However, the image projected on the HMD can only be viewed by the person wearing the HMD. Therefore, no one other than the person wearing the HMD can know what kind of experience the person experiencing the VR is doing. That is, the space of the VR is a closed world that only the person wearing the HMD can know. Thus, it is difficult for those who wear the HMD to share the experiences experienced in the VR space with other people.

  By the way, in recent years, "self-portrait" which photographs oneself by oneself is popular. "Self-shooting" is a shooting method of photographing oneself as a subject while confirming the appearance of oneself on a display such as a smartphone. It is possible to shoot as you wish, so that others can easily understand the events you are experiencing. Therefore, the other person can easily know the event experienced by the person who made the selfie by looking at the photographed picture.

  Therefore, the present invention has been made in view of such problems, and the purpose of the present invention is that a person who wears an HMD and experiences VR is photographed in the real space by the person It is an object of the present invention to provide an apparatus, a method, and a program that can be taken by self-shooting as if the person is in the space of the VR by combining the image and the image of the space of the VR.

  In order to achieve such an object, a first aspect of the present invention is an apparatus. The apparatus arranges a body area image representing a body in a real space of a user wearing a head mounted display (HMD) in a virtual reality (VR) space, and the body area image is arranged. The processor comprises a processor configured to superimpose an image of a face of the user who is not wearing the HMD on an image in the VR space.

  A second aspect of the invention is a method implemented by the device. The method comprises the steps of: arranging a body region image representative of a body in real space of a user wearing a head mounted display (HMD) in a virtual reality (VR) space; Superimposing an image of a face of the user who is not wearing the HMD on the image in the VR space.

  A third aspect of the present invention is a program. This program is a program that causes a computer to function as the device of the first aspect.

  According to the present invention, an image of a person who is experiencing the VR wearing the HMD taking a self-portrait in the real space, and an image of a face where the person experiencing the VR is not wearing the HMD And the image of the VR space can be synthesized. Thus, a person who wears an HMD and experiences VR can take a picture as if he were in the space of the VR.

FIG. 7 is an image diagram of a self-portrait service in a VR space according to an embodiment of the present invention. FIG. 1 is a schematic overview of a selfie system in VR space according to an embodiment of the present invention. It is a functional block diagram of an image processing device of a self-portrait system in VR space concerning one embodiment of the present invention. It is a figure for demonstrating face composition concerning one embodiment of the present invention. It is a flowchart which shows an example of the processing flow in the image processing apparatus of the self-portrait system in VR space concerning one Embodiment of this invention.

  First, terms used in the present specification will be described. "Self-shooting" means that a person taking a picture takes himself or herself as a subject, and is also referred to as self-shooting, selfy, and the like. Also, “VR space” is a space of virtual reality represented by a video displayed on the HMD. Also, "real space" is a real space, that is, a real space. In the present invention, any HMD such as binocular, monocular, non-transmissive and transmissive types can be used.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is an illustration of a self-portrait service in a VR space according to an embodiment of the present invention. As shown in FIG. 1, in the real space 110, it is assumed that the user 100 wears the HMD 113 and is viewing an image displayed on the HMD 113. As a result, the user 100 can experience as if in the VR space 130. While looking at the inside of the VR space 130 from the viewpoint 131 (that is, with the HMD 113 attached), the user 100 faces the imaging device 112 and takes a self-portrait. Then, it is possible to acquire the self-portrait image 140 as if the user 100 in the VR space 130 was photographed by self-portrait. Hereinafter, FIG. 1 will be described more specifically.

<Composition of real space and VR space>
While watching the HMD image 111 displayed on the HMD 113, the user 100 faces the imaging device 112 and performs self-shooting. The HMD image 111 is an image of the VR space 130 spreading in front of the user 100's eyes. Furthermore, in the HMD video 111, a virtual camera 122 capable of moving in the VR space 130 appears as if the user 100 is holding it in his hand. The movement of the virtual camera 122 is linked to the movement of the imaging device 112. The user 100 can confirm his own appearance by looking at the real / VR composite image 121 displayed on the monitor of the virtual camera 122. The real / VR composite image 121 is an image captured by the virtual camera 122 in the real / VR composite space 120 which is a VR space obtained by combining the real space 110 and the VR space 130. That is, the real / VR composite image 121 is an image of the body of the user 100 captured by the imaging device 112 in the real space 110 and an image of a face on which the user 100 is not wearing the HMD 113 (details will be described below. Composition of Facial Image> and a video of the VR space 120 spreading on the back of the user 100 are composited. When the user 100 instructs the imaging device 112 to perform imaging, the virtual camera 122 in the real / VR space 120 performs imaging.

<Composition of face image>
The user 100 who is photographing the photographing device 112 in the real space 110 is wearing the HMD 113. Therefore, in the real / VR composite space 120, the image of the face of the user 100 wearing the HMD 113 in the real space 110, the image of the face of the user 100 not wearing the HMD 113 taken in advance It is overlapping. As described above, in the present invention, an image of the user 100 taken by self-portrait in the physical space 110, an image of a face where the user 100 is not wearing the HMD 113, and an image of the VR space 130 are synthesized. Ru.

  FIG. 2 is a schematic overview of a selfie system 200 in VR space according to an embodiment of the present invention. The schematic diagram includes an image processing apparatus 201, an imaging device 112, and a head mounted display (HMD) 113. The image processing apparatus 201 can transmit and receive data to and from each of the imaging device 112 and the HMD 113. The image processing apparatus 201 displays the image of the body of the person wearing the HMD 113 photographed by the photographing device 112 in a self-portrait, the image of the face on which the person is not wearing the HMD 113, and the HMD 113 It is possible to combine the image in the VR space.

  The head mounted display (HMD) 113 is any HMD such as binocular, monocular, non-transmissive, transmissive, and the like. The HMD 113 can transmit and receive data to and from the image processing apparatus 201. The HMD 113 can display the video received from the image processing apparatus 201 on the display.

  The imaging device 112 can include a camera function, a gyro sensor (angular velocity sensor), a depth sensor, and a button for instructing the virtual camera to perform imaging. The imaging device 112 may also include a memory for storing the completed image (ie, the self-portrait image 140 of FIG. 1). The imaging device 112 can transmit and receive data to and from the image processing apparatus 201. The imaging device 112 can transmit, to the image processing apparatus 201, an image captured by the camera function and information measured by the gyro sensor (angular velocity sensor) and the depth sensor. Also, the imaging device 112 can transmit, to the image processing apparatus 201, information indicating that the virtual camera is instructed to perform imaging. Also, the imaging device 112 can receive a completed image (that is, the self-portrait image 140 of FIG. 1) from the image processing apparatus 201.

The camera function of the imaging device 112 can capture a moving image like camera functions of conventional digital cameras and smartphones. The imaging device 112 can always transmit (in real time processing) a moving image being captured by the camera function to the image processing apparatus 201.

  A gyro sensor (angular velocity sensor) of the imaging device 112 can measure an angle or an angular velocity of the imaging device 112. The imaging device 112 can transmit (in real time processing) the information being measured by the gyro sensor (angular velocity sensor) to the image processing apparatus 201 at all times. Therefore, the image processing apparatus 201 can calculate the orientation of the imaging device 112.

  The depth sensor of the photographing device 112 can measure the distance between the photographing device 112 and a person who is photographing by the photographing device 112 in a self-shooting manner (that is, a person wearing the HMD 113). In addition, the depth sensor of the imaging device 112 can measure the distance between the imaging device 112 and an object other than the person who is photographing by the imaging device 112 in a self-portrait. The imaging device 112 can transmit (in real time processing) the information being measured by the depth sensor to the image processing apparatus 201 at all times. Therefore, the image processing apparatus 201 can obtain information on the depth of each point in the moving image captured by the camera function. As a result, the image processing apparatus 201 can three-dimensionally capture the body of the person who is photographing by the photographing device 112 and objects other than the person. In the image processing apparatus 201, the camera function captures an image of a person taking a picture with the imaging device 112 on the basis of information on the depth of each point in the moving image captured by the camera function. It is possible to identify in a moving image and generate a virtual three-dimensional object (hereinafter also referred to as a body region image) representing the person's body.

  The button of the imaging device 112 can identify that the virtual camera is instructed to perform imaging. For example, when the button is pressed, the imaging device 112 can determine that the virtual camera is instructed to perform imaging.

  FIG. 3 is a functional block diagram of the image processing apparatus 201 according to an embodiment of the present invention. The image processing apparatus 201 includes a physical space image acquisition unit 301, a physical / VR synthesis unit 302, a face synthesis unit 303, a self-portrait image transmission unit 304, a face image storage unit 305, and a self-portrait image storage unit 306. The image processing apparatus 201 is one or more computers including a processor and a memory. In addition, the image processing apparatus 201 executes a program for operating the physical space image acquisition unit 301, the physical / VR synthesis unit 302, the face synthesis unit 303, and the self-portrait image transmission unit 304, or a process flow to be described later. A storage medium storing the program of

  The real space image acquisition unit 301 receives, from the photographing device 112, a moving image in the real space transmitted by the photographing device 112 in real time processing, and information measured by the gyro sensor (angular velocity sensor) and the depth sensor. Can. The moving image in the real space is a moving image photographed by a person wearing the HMD 113 (hereinafter, also referred to as a user) using the photographing device 112 in a self-portrait. The real space image acquisition unit 301 stores the moving image in the real space received from the imaging device 112 and the information measured by the gyro sensor (angular velocity sensor) and the depth sensor so that the real / VR synthesis unit 302 can refer to them. (Not shown).

<Display of real / VR composite image>
The real / VR combining unit 302 combines a moving image in the real space photographed by the user with the photographing device 112 and an image of the VR space spreading on the back of the user, and combines the synthesized image. It can be displayed on the monitor of the virtual camera. At that time, the face synthesis unit 303 includes an image of the face of the user wearing the HMD 113 in the physical space, and an image of the face of the user 100 not wearing the HMD 113 stored in the face image storage unit 305. Synthesize Then, the virtual camera shoots. Hereinafter, the synthesis of the real space and the VR space will be described in detail with reference to FIG.

  In the real space 110 of FIG. 1, the user wearing the HMD 113 is photographing by the photographing device 112 by self photographing. The real / VR composite space 120 is an image obtained by combining the moving image in the real space 110 photographed by the user with the photographing device 112 and the image of the VR space 130 spreading on the back of the user (ie, real VR space for generating a VR composite image 121). The VR space 130 is a VR space represented by an image viewed by the user wearing the HMD 113. That is, the user wearing the HMD 113 is looking at the inside of the VR space 130 with the viewpoint 131 as the viewpoint, not the real / VR combined space 120.

  The real / VR combining unit 302 is a virtual three-dimensional object representing the user's body (that is, a body region based on the moving image in the real space received by the real space image acquiring unit 301 and the information measured by the depth sensor). Image) can be generated. In addition, the real / VR combining unit 302 can arrange a body region image in an arbitrary size in the real / VR combined space 120. At this time, based on the head tracking function (detection of the head movement of the user) of the HMD 113, the real / VR combining unit 302 determines the orientation of the user's face in the real space 110 and the body region in the real / VR space 120. The orientation of the face of the image can be matched. Also, based on the position tracking function of the HMD 113 (detection of the position of the HMD 113 in the real space 110), the real / VR combining unit 302 determines the position of the user in the real space 110 and the body in the real VR space 120. The position of the area image can be matched.

  Furthermore, the reality / VR combining unit 302 sets the reality in the position in the real / VR combined space 120 (that is, the hand of the body region image) corresponding to the position where the user holds the imaging device 112 in the real space 110. A virtual camera 122 for imaging in the VR composite space 120 can be arranged. Also, the real / VR combining unit 302 can arrange the monitor of the virtual camera 122 in the VR space 130. Also, the real / VR combining unit 302 causes the monitor of the virtual camera 122 in the VR space 130 to display an image in the real / VR combined space 120 that the virtual camera 122 in the real / VR combined space 120 is attempting to capture. be able to. In addition to the monitor of the virtual camera 122, a virtual three-dimensional object equivalent to the user's hand or a selfie stick is placed in the VR space 130 as if the user had a monitor in his hand. May be

  Here, the virtual camera 122 in the real / VR combined space 120 and the monitor of the virtual camera 122 in the VR space 130 will be described. The virtual camera 122 in the real / VR composite space 120 is for taking (storing) a two-dimensional image projected on the viewpoint with the position of the virtual camera 122 as a viewpoint. The monitor of the virtual camera 122 in the VR space 130 is for displaying an image in the real-VR composite space 120 that the virtual camera 122 in the real-VR composite space 120 is to capture.

  The position, the direction, and the motion of the virtual camera 122 in the real / VR composite space 120, the monitor of the virtual camera 122 in the VR space 130, and the imaging device 112 in the real space 110 are linked. Specifically, the real / VR combining unit 302 acquires, for example, information (such as a photographed image) measured by the position tracking device 114 (a camera or the like) in FIG. 1, and based on the information, the real space 110 The positional relationship between the imaging device 112 and the user can be calculated. Also, the real / VR combining unit 302 can calculate the direction of the imaging device 112 based on the information measured by the gyro sensor (angular velocity sensor) received by the real space image acquisition unit 301. Alternatively, the reality / VR combining unit 302 can calculate the orientation of the imaging device 112 based on the information (for example, a photographed image) acquired by the position tracking device 114 (for example, a camera or the like) in FIG. .

  The real / VR combining unit 302 is configured so that the positional relationship between the imaging device 112 and the user in the real space 110 and the positional relationship between the virtual camera 122 and the user in the real / VR space 120 match. The virtual camera 122 can be disposed in the VR space 120. In addition, the real / VR combining unit 302 can arrange the virtual camera 122 in the real / VR space 120 such that the direction of the imaging device 112 matches the direction of the virtual camera 122.

  In addition, the real / VR combining unit 302 is configured so that the positional relationship between the imaging device 112 and the user in the physical space 110 and the positional relationship between the monitor of the virtual camera 122 in the VR space 130 and the viewpoint 131 match. , And the monitor of the virtual camera 122 can be arranged in the VR space 130. Also, the real / VR combining unit 302 can arrange the monitor of the virtual camera 122 in the VR space 130 such that the direction of the imaging device 112 matches the direction of the monitor of the virtual camera 122.

<Shooting of a real / VR composite image>
The real / VR combining unit 302 can cause the virtual camera to shoot in response to the button of the shooting device 112 being pressed. Specifically, the real / VR combining unit 302 can receive, from the imaging device 112, information indicating that the virtual camera is instructed to perform imaging. In addition, the real / VR combining unit 302 displays the image displayed on the monitor of the virtual camera 122 in the VR space 130 when the shooting instruction is received (that is, the position of the virtual camera in the real / VR combined space 120 As a viewpoint, a two-dimensional image (projected on the viewpoint) can be acquired and stored in the self-portrait image storage unit 306.

  The face image storage unit 305 stores an image obtained by photographing the face of the user who is not wearing the HMD 113 from one or more angles. For example, the image of the face stored in the face image storage unit 305 is an image of the face taken from all directions (360 degrees). By photographing the face of the user from all directions, the face of the user can be captured stereoscopically. The image of the face has feature points in characteristic parts of the face (eg, nose, mouth, ears, jaws, etc.).

  The face combining unit 303 is an image of the face of the user wearing the HMD 113 in the real space 110 in the real-VR combining space 120 and a user not wearing the HMD 113 stored in the face image storage unit 305 One hundred face images can be combined. The face combining unit 303 will be described in detail with reference to FIG.

  FIG. 4 is a diagram for explaining face synthesis according to the embodiment of the present invention.

  The image a is a photographed image of the face of the user stored in the face image storage unit 305. As indicated by the asterisks in the image a, the image of the user's face image has one or more feature points. The feature point indicates the position of a characteristic portion (a portion that can not be covered by the HMD 113 such as a nose, a mouth, an ear, and a jaw) in the user's face.

  The image b is an image of a face shot of the user wearing the HMD 113 in the real space. That is, the image b is a body area image. As indicated by the asterisks in the image b, the body region image has one or more feature points.

  The image c is an image generated by the face synthesis unit 303. The face combining unit 303 can generate the image c by deforming the face of the image a so that the feature points in the image a match the feature points in the image b. Although the image c is an image of the upper half of the face (that is, a portion covered by the HMD 113), the image c may be an image of the entire face.

  The image d is an image generated by the face synthesis unit 303. The face synthesis unit 303 can generate the image d by superimposing the image c on the image b.

  In this specification, although an embodiment in which the face of the user who is not wearing the HMD 113 is superimposed on the face of the user in the real / VR composite space 120 has been described, the user in the image captured by the virtual camera 122 The face of the user who is not wearing the HMD 113 may be superimposed on the face of. That is, the virtual camera 122 shoots a user wearing the HMD 113 in the real / VR combined space 120. Then, the face synthesis unit 303 superimposes the face of the user who is not wearing the HMD 113 on the face of the user in the image captured by the virtual camera 122. Even in an embodiment in which the face of the user who does not wear the HMD 113 is superimposed on the face of the user in the image captured by the virtual camera 122, the user who does not wear the HMD 113 on the face of the user in the real / VR composite space 120 A self-portrait image 140 similar to that of the face superimposing embodiment can be obtained.

  Return to FIG. The self-portrait image storage unit 306 can store a self-portrait image captured by the virtual camera 122. The self-portrait image is an image as if the user in the VR space was photographed by self-portrait.

  The self-portrait image transmission unit 304 can transmit the self-portrait image in the self-portrait image storage unit 306 to the imaging device 112 or another device. Specifically, the self-portrait image transmission unit 304 can transmit a self-portrait image by e-mail or the like. Alternatively, the selfie image transmission unit 304 can download the selfie image to the imaging device 112 or another device.

  FIG. 5 is a flow chart showing an example of the processing flow in the image processing apparatus 201 of the selfie system in the VR space according to the embodiment of the present invention.

  In step 501, the image processing apparatus 201 causes the imaging device 112 to transmit a moving image in the real space 110 transmitted by the imaging device 112 in real time processing, and information measured by a gyro sensor (angular velocity sensor) and a depth sensor. To receive.

  In step 502, the image processing apparatus 201 generates a virtual three-dimensional object (body region image) representing the user's body based on the moving image in the physical space 110 received in step 501 and the information measured by the depth sensor. Generate

  In step 503, the image processing apparatus 201 places the body region image generated in step 502 in the real / VR combined space 120.

  In step 504, the image processing apparatus 201 sets the virtual camera at a position in the real / VR space 120 (that is, the hand of the body area image) corresponding to the position where the user holds the imaging device 112 in the real space 110. Place. Further, the image processing apparatus 201 arranges the monitor of the virtual camera at a position in the VR space 130 which corresponds to the position where the user holds the imaging device 112 in the real space 110. The image processing device 201 is based on, for example, information (such as a photographed image) measured by the position tracking device 114 (such as a camera) in FIG. 1 or information measured by the gyro sensor (angular velocity sensor) received in step 501. A virtual camera or a monitor of the virtual camera can be arranged.

  In step 505, the image processing apparatus 201 causes the monitor of the virtual camera arranged in step 504 to display an image in the real / VR space 120 which the virtual camera arranged in step 504 is to capture. Specifically, the image processing apparatus 201 uses the position of the virtual camera as a viewpoint, and displays a two-dimensional image projected on the viewpoint on the monitor of the virtual camera.

  In step 506, the image processing apparatus 201 captures an image with a virtual camera. Specifically, the image processing apparatus 201 displays an image displayed on the monitor of the virtual camera when receiving an instruction for photographing from the photographing device 112 (that is, with the position of the virtual camera as a viewpoint and projected on the viewpoint 2 Get the dimensional image) and store it in memory.

  In step 507, the image processing apparatus 201 transmits the self-portrait image captured by the virtual camera in step 507 to the imaging device 112.

  As described above, face synthesis may be performed in step 503 or may be performed after step 506.

  Note that the processing device and memory of the imaging device 112 or the HMD 113 are part or all of the physical space image acquisition unit 301 to the self-portrait image storage unit 306 of the image processing apparatus 201 described in this specification. It may be provided instead of Also, the imaging device 112 may be a dedicated device or a device such as a smartphone on which application software capable of executing the same processing as the imaging device 112 is installed.

  As described above, according to the present invention, it is as if a person who experiences virtual reality wearing the head mounted display takes a picture by himself in the space of the virtual reality without wearing the head mounted display. Selfie images can be generated.

  Although the embodiments of the present invention have been described above, the above embodiments are merely examples, and the present invention is not limited to the above-described embodiments, and can be implemented in various different forms within the scope of the technical idea thereof. It goes without saying that it is good.

  Further, the scope of the present invention is not limited to the illustrated and described exemplary embodiments, but also includes all the embodiments that bring about the same effects as the object of the present invention. Further, the scope of the present invention is not limited to the combination of the features of the invention as defined by the claims, but can be defined by any desired combination of specific features of all the disclosed respective features. .

100 User 110 Real Space 111 HMD Image 112 Shooting Device 113 HMD
114 Position tracking device 120 Real-VR composite space 121 Real-VR composite image 122 Virtual camera 130 VR space 131 Viewpoint 140 Self-portrait image 200 Self-portrait system 201 Image processing device 301 Real-space image acquisition unit 302 Real-VR composite unit 303 Face Composing unit 304 Self-portrait image transmission unit 305 Face image storage unit 306 Self-portrait image storage unit

Claims (6)

Place a body region image representing the body in real space of the user wearing the head mounted display (HMD) in a virtual reality (VR) space,
An apparatus comprising: a processor configured to superimpose an image of a face of the user who is not wearing the HMD on an image in the VR space in which the body region image is arranged. Place a virtual camera in the VR space,
Displaying on the monitor of the virtual camera an image in the VR space in which the body region image projected to the viewpoint at the position of the virtual camera is arranged;
The processor further includes a processor configured to acquire an image in the VR space in which the body region image projected to the viewpoint is arranged when receiving an instruction to shoot the virtual camera. The device according to claim 1.   The apparatus according to claim 2, wherein the motion of the virtual camera is linked to the motion of the user in the real space.   The body region image is generated on the basis of a moving image in the physical space of the user wearing the HMD, and information for calculating the depth of the moving image. A device according to any one of 3. A method implemented by the device,
Placing in a virtual reality (VR) space a body region image representative of a body in real space of a user wearing a head mounted display (HMD);
Superimposing an image of a face of the user who is not wearing the HMD on the image in the VR space in which the body region image is arranged.   A program for causing a computer to execute the method according to claim 5.

Images