JP2015172900A - Image presentation device and mask generation device to be used for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve an immersive feeling of a user who is viewing a projection image to be used for presenting an image in a virtual space to a user when the body of the user is reflected in the projection image.SOLUTION: A user attitude acquisition part 1 acquires the attitude of a user in a real space. A body shape calculation part 2 calculates the body shape of the user by using the acquired attitude of the user. A mask generation part 4 generates a mask for extracting an insertion image from the body image of the user in the real space on the basis of the body shape to be viewed from the viewpoint of the user in another space. Thus, it is possible to generate an insertion image by extracting at least a part of the body image of the user in the real space by using the mask. Also, it is possible to superimpose the insertion image on a viewpoint image in the other real space viewed from the viewpoint of the user.

Description

  The present invention relates to a technique for generating an image presented to a user. In particular, the present invention relates to a mask for cutting out an image in real space and combining it with an image in another space.

  Conventionally, in the field of games and simulations, a head mounted display (HMD) has been used in order for a user to immerse in a virtual space (Non-Patent Documents 1 and 2 below). In particular, recently, inexpensive HMDs have appeared and have been spurring popularity. In the present specification, the term “virtual” will be used hereinafter, which means “it does not actually exist but is equivalent to existing in function and effect”.

  To the user wearing the HMD, an image obtained by projecting a virtual space different from the real space in which the user exists is presented by a display provided in the HMD. On the other hand, the user's actual field of view is shielded and the real space cannot be seen. Thereby, the user can be immersed in the virtual space while being in the real space.

  In such a conventional HMD, in order to improve the feeling of immersion in the virtual space, the user's view is shielded as described above. Then, even if you can see your body (for example, limbs), you cannot see your body.

  On the other hand, there has also been proposed a method of causing a so-called avatar to appear in a virtual space, which can be called an own body in spite of its appearance, depending on computer graphics (Non-Patent Document 3 below). However, for example, even if you look at your hand holding something in the virtual space, you can see the image of the avatar's hand, not your hand. Since this situation differs from the user's sense of reality, there is a problem that the sense of immersion in the virtual space is reduced.

  Therefore, recently, using a video see-through HMD with an external camera attached to the front of the HMD, a real space image that should be visible to the user is acquired, and the body is cut out from the real space image and superimposed on the virtual space. A technique has been proposed (Non-Patent Document 4 below). Two methods, a chroma key method and a contour extraction method from video, are conceivable as methods for cutting the body from the image.

  In the chroma key method, only a body region is extracted by erasing or extracting only a region having a specific color. However, this method places restrictions on the colors that can be used. For this reason, special equipment is required, such as stretching a cloth with a uniform color on the cockpit side for measuring the user's body.

  In the method of extracting the contour from the video, it is necessary to cut out a body region from the image by image processing. In this method, there is a possibility that the accuracy to be obtained may fluctuate greatly depending on the state of the image to be used (for example, resolution) and the image processing algorithm to be used. In addition, since image processing is performed in units of pixels, there is a problem that real-time processing becomes difficult when a high-resolution image is used.

  Furthermore, when there is a shielding relationship between an object (so-called object) in the virtual space and the user's body, the body existing in front of the object can be seen, but the body existing behind the object needs to be hidden. The two methods described above have a problem that it is difficult to establish such a shielding relationship.

  As a technique for establishing the shielding relationship, a depth measurement method is considered. The depth measurement method is also called a Depsky method. This method estimates the occlusion relationship by accurately measuring the position information of the actual body in the three-dimensional space and mapping the position information to the position of the body in the virtual space. .

  However, it is difficult to accurately measure a very short distance (for example, a difference in the shape of a palm or a finger) with a distance sensor normally used at present. For this reason, the depth measurement method tends to have an inaccurate shape especially in the contour portion of the body, and as a result, there is a problem that it is difficult to create a highly accurate shielding relationship. In addition, there is a problem that it takes a considerable cost to arrange a high-precision distance sensor so that the position of each part of the body can be measured. Furthermore, in the method of obtaining depth information for each pixel and performing the calculation, there is a problem that the calculation cost increases exponentially when the image quality is improved (higher pixels).

S. Tachi, H. Arai and T. Maeda.Tele-existence master-slave system for remote manipulation.II.Proceedings of the 29th IEEE Conference on Decision and Control, pages 85-90 vol.1, 5-7, 1990. K. Suzuki, S. Wakisaka, and N. Fujii.Substitutional reality system: a novel experimental platform for experiencing alternative reality.Scientific reports, 2, 2012. S. Beck, A. Kunert, A. Kulik, and B. Froehlich. Immersive Group-to-Group Telepresence.IEEE Transactions on Visualization and Computer Graphics, 19 (4), pages 616-625, 2013. F. Steinicke, G. Bruder, K. Rothaus, and K. Hinrichs.Poster: A virtual body for augmented virtuality by chroma-keying of egocentric videos.In IEEE Symposium on 3D User Interfaces.3DUI 2009, pages 125-126.IEEE , 2009.

  The present invention has been made in view of the above situation. The main object of the present invention is to improve the immersive feeling of a user who is looking at a projection image when the user's body is reflected in the projection image used to present the image in the virtual space to the user. Is to provide technology for.

  Means for solving the above-described problems can be described as follows.

(Item 1)
To generate an insertion image by extracting at least a part of a user's body image in real space, and to generate a mask used to superimpose the insertion image on a viewpoint image in another space viewed from the user's viewpoint Equipment,
A user posture acquisition unit, a body shape calculation unit, and a mask generation unit;
The user posture acquisition unit is configured to acquire the posture of the user in the real space,
The body shape calculation unit is configured to calculate the user's body shape using the acquired posture of the user,
The mask generation unit is configured to generate a mask for extracting the insertion image from the body image of the user in the real space based on the body shape that should be seen from the viewpoint of the user in the different space. A mask generation device characterized by that.

  The separate space is, for example, a VR space, but may be a real space at a position spatially or temporally away from a point where the user exists. The user's body is, for example, the user's hand, arm, or leg, but may be another part or an article that the user can wear. The body shape calculation unit and the mask generation unit may realize their functions on a computer by one computer program, or realize the functions by one or a plurality of computers by a combination of a plurality of programs or modules. May be.

(Item 2)
It also has a viewpoint image generator,
The viewpoint image generation unit is configured to generate the viewpoint image when viewed from the viewpoint of the user in the different space,
Item 1. The mask generation unit is configured to generate the mask by using at least a part of the user's body part present in the viewpoint image as a body shape that should be visible from the user's viewpoint. The mask generating apparatus as described.

(Item 3)
Furthermore, a VR space model data storage unit is provided,
The mask generation device according to item 2, wherein the viewpoint image generation unit is configured to generate the viewpoint image based on VR space model data stored in the VR space model data storage unit.

(Item 4)
It also has a user viewpoint camera,
The user viewpoint camera is configured to acquire a body image of the user in the real space,
The attitude of the viewpoint of the user viewpoint camera matches the attitude of the user's viewpoint in the different space, or can be substantially matched by correcting an image viewed from either viewpoint. The mask generation device according to any one of items 1 to 3.

(Item 5)
Furthermore, it has a lighting condition calculation unit,
The mask generation device according to any one of Items 1 to 4, wherein the illumination condition calculation unit is configured to set a luminance or color condition in the inserted image based on an illumination condition in the different space. .

  The projection image presented to the user is generally generated using lighting conditions in another space. Using this lighting condition, the user feels by setting the brightness or color condition for the inserted image (in another space it should be viewed by the user as an avatar for the user or as part of it) Reality can be improved.

(Item 6)
Between at least a part of the body shape that should be seen from the user's viewpoint in the different space and some object in the different space, one is the other when viewed from the user's viewpoint in the different space Item 6. The mask generation device according to any one of Items 1 to 5, wherein the mask generation device is in a relationship of being shielded by.

(Item 7)
Comprising the mask generation device according to any one of items 1 to 6 and an insertion image generation unit;
The insertion image generation device is configured to generate the insertion image by applying the mask to the body image.

(Item 8)
The inserted image generating device according to item 7, a projection image generating unit, and an image presenting unit,
The projection image generation unit is configured to generate a projection image to be projected to the user by combining the insertion image with a viewpoint image in another space viewed from the user's viewpoint.
The image presenting device is configured to project the projection image from a viewpoint position optically conjugate with a user's viewpoint position in the real space.

(Item 9)
To generate an insertion image by extracting at least a part of a user's body image in real space, and to generate a mask used to superimpose the insertion image on a viewpoint image in another space viewed from the user's viewpoint A method using the apparatus of
Obtaining the posture of the user in the real space;
Calculating the user's body shape using the acquired posture of the user;
Generating a mask for extracting the insertion image from the body image of the user in the real space based on the body shape that should be seen from the viewpoint of the user in the different space. Mask generation method.

(Item 10)
A computer program for causing a computer to execute each step according to item 9.

  This computer program is stored in an appropriate recording medium (for example, an optical recording medium such as a CD-ROM or a DVD disk, a magnetic recording medium such as a hard disk or a flexible disk, or a magneto-optical recording medium such as an MO disk). Can be stored. This computer program can be transmitted via a communication line such as the Internet.

  According to the present invention, it is possible to extract a user's body image and incorporate it into the projected image so as to improve the immersive feeling of the user who is viewing the image projected from the virtual space.

  Furthermore, according to the present invention, a mask for extracting a user's body image can be generated with high accuracy and low cost.

  Moreover, according to the present invention, since the mask is used, the calculation cost can be kept low even when the number of pixels of the projected image is large. As a result, according to the present invention, it is possible to reduce the frame delay at the time of providing a moving image, and it is easy to provide an image with high image quality.

1 is a block diagram illustrating a schematic configuration of an image presentation device according to an embodiment of the present invention. It is a flowchart which shows the schematic procedure of the image presentation method which concerns on one Embodiment of this invention. It is a conceptual explanatory drawing about the bone data as body model data. It is a conceptual explanatory drawing about the external shape model as body model data. It is explanatory drawing which shows notionally the user viewpoint image in VR space for mask production | generation. It is explanatory drawing which shows notionally the user body image from the user viewpoint in real space. It is explanatory drawing which shows notionally the insertion image cut out from the user body image by mask application. It is explanatory drawing which shows notionally the user viewpoint image in VR space combined with an insertion image. It is explanatory drawing which shows notionally the projection image obtained by superimposing an insertion image on the user viewpoint image in VR space. It is a flowchart which shows the schematic procedure of the image presentation method which concerns on a modification. 1 is a plan view illustrating a schematic configuration of an HMD according to Embodiment 1. FIG. It is explanatory drawing which shows typically the reflective surface shape of the mirror used in HMD shown in FIG. 6 is a plan view showing a schematic configuration of an HMD according to Embodiment 2. FIG. FIG. 10 is an explanatory diagram illustrating an example of how to take parameters for image processing corresponding to a camera viewpoint offset in the second embodiment.

(One Embodiment of Image Presentation Device)
With reference to FIG. 1, an image presentation apparatus according to an embodiment of the present invention will be described.

  The image presentation apparatus according to the present embodiment includes a user posture acquisition unit 1, a body shape calculation unit 2, a viewpoint image generation unit 3, and a mask generation unit 4. Furthermore, the apparatus according to the present embodiment includes an insertion image generation unit 5, a projection image generation unit 6, a user viewpoint camera 7, and an image presentation unit 8. Furthermore, the apparatus of the present embodiment includes a body model data storage unit 9, a VR space model data storage unit 10, and an illumination condition calculation unit 11.

  The user posture acquisition unit 1 is configured to acquire a user posture in real space. As the user posture acquisition unit 1, a commercially available body motion measurement device can be used, and thereby relatively accurate measurement can be performed at a low cost.

  The body shape calculation unit 2 is configured to calculate the user's body shape using the user's posture acquired by the user posture acquisition unit 1. More specifically, in the present embodiment, the body shape of the user is calculated using body model data (described later) stored in the body model data storage unit 9.

  The viewpoint image generation unit 3 is configured to generate a viewpoint image when viewed from the user's viewpoint in another space (VR space in the present embodiment). That is, in the present embodiment, the user posture acquisition unit 1 can know the posture (direction and position) of the user viewpoint. Thereby, the viewpoint image generation unit 3 can map the position of the user viewpoint in the VR space and generate an image (that is, a viewpoint image) obtained from the virtual user viewpoint existing in the VR space. Here, in the present embodiment, a viewpoint image for generating a mask and a viewpoint image for generating a projection image described later can be generated separately. Then, in the viewpoint image for mask generation, a part corresponding to the body shape on the viewpoint image (this is in the VR space, so that the three-dimensional position can be accurately determined) so that the subsequent processing is easy. For example, it can be expressed by special color space information. A specific processing procedure will be described later.

  The mask generation unit 4 is configured to generate a mask for cutting out the insertion image from the body image of the user in the real space based on the body shape that should be seen from the user's viewpoint in another space. Here, “cut out” means “take out” or “extract”.

  More specifically, the mask generation unit 4 of the present embodiment generates a mask by using at least a part of the user's body part present in the viewpoint image as a body shape that should be visible from the user's viewpoint. It has become. Here, the viewpoint image is generated by the viewpoint image generation unit 3, and the user's body in the VR space is reflected in the viewpoint image.

  The insertion image generation unit 5 is configured to generate an insertion image by applying a mask to the body image. A specific example of processing for generating an insertion image will also be described later.

  The projection image generation unit 6 is configured to generate a projection image to be projected to the user by combining the insertion image with a viewpoint image in another space viewed from the user's viewpoint.

  The user viewpoint camera 7 is configured to acquire a user's body image in real space. Here, the posture of the viewpoint of the user viewpoint camera 7 matches the posture of the user's viewpoint in another space (that is, on another space coordinate). However, it is possible to substantially match these viewpoint positions by correcting an image viewed from either viewpoint.

  The image presenting unit 8 is configured to present the projection image to the user from a position optically conjugate with the viewpoint position of the user in the real space.

  Here, in this embodiment, the user viewpoint camera 7 and the image presentation unit 8 can be implemented as a so-called video see-through HMD. Further, the user viewpoint camera 7 and the image presentation unit 8 can be provided for both the left and right eyes, corresponding to the stereoscopic view. In the following description, basically, an image presented to one eye will be described, but the same processing is performed on an image presented to the other eye unless otherwise specified. Of course, an image in consideration of parallax for stereoscopic viewing is usually presented. It is also possible to present an image corresponding to monaural viewing.

  The body model data storage unit 9 is a functional element for storing data necessary for calculation of the body shape in the body shape calculation unit 2 described above. The body model data is, for example, bone data for representing the posture of the skeleton and outer shape model data for representing the outer shape of the body. Bone data represents the orientation and length of the main bone. The outline model data is applied to bone data and can represent the outline of the body. Specific examples of these data will also be described later.

  The VR space model data storage unit 10 is data for defining a virtual space (VR space), and normally includes data for configuring a virtual three-dimensional space. The viewpoint image generation unit 3 of the present embodiment is configured to generate a viewpoint image from the user viewpoint based on the VR space model data stored in the VR space model data storage unit 10.

  The illumination condition calculation unit 11 is configured to set the luminance or color condition in the inserted image based on the illumination condition for the avatar for the user in another space. In the VR space, the lighting conditions for the avatar can be treated as known. Of course, in general, lighting conditions in the VR space are determined not for the avatar alone but for the entire VR space. “Lighting conditions for avatars” means lighting conditions that relate to avatars (for example, those that change the brightness and color of the avatar surface or generate shadows by avatars). is there.

  A more detailed procedure for generating a mask and generating a projection image in this embodiment will be described later.

  An example of a video see-through HMD for mounting the user viewpoint camera 7 and the image presentation unit 8 will be described later.

  The user posture acquisition unit 1, the body shape calculation unit 2, and the mask generation unit 4 of the present embodiment correspond to an embodiment of the mask generation device of the present invention. Further, the insertion image generation unit 5 added to the mask generation apparatus corresponds to an embodiment of the insertion image generation apparatus of the present invention. Further, the addition of the projection image generating unit 6 and the image presenting unit 8 to the inserted image generating device corresponds to an embodiment of the image presenting device of the present invention.

(Projection image generation method)
Next, an example of a procedure for generating a mask using the above-described apparatus and generating a projection image using the mask will be described with further reference to FIG.

(Step SA-1 in FIG. 2)
First, the user posture acquisition unit 1 acquires the user posture. This acquisition is performed in parallel with the presentation of the image in consideration of real-time use. That is, while presenting the user's posture, the image presentation using the posture is performed, and the posture data and the projected image are sequentially updated. Moreover, a moving image can be provided to the user by sequentially updating the projection images.

(Step SA-2 in FIG. 2)
Next, the body shape calculation unit 2 calculates the user's body shape using the obtained user posture and the body model data stored in the body model data storage unit 9. Therefore, normally, before using this apparatus, basic body dimensions (distance between joints, hand shape, etc.) of the user are measured and stored in the body model data storage unit 9. preferable. However, since the human body dimensions are basically the same (for example, for adults), it is possible to use general dimension data not specific to an individual. The body shape to be calculated may be specified on the coordinates in the VR space or on the coordinates in the real space. This is because the mapping for converting these coordinate systems is usually known.

  FIG. 3 shows an example of a body dimension model as so-called bone data. FIG. 4 shows an example of a body outline model (in this example, a hand).

(Step SA-3 in FIG. 2)
Next, the viewpoint image generation unit 3 generates a viewpoint image viewed from the user viewpoint in the VR space, using the VR space model data stored in the VR space model data storage unit 10. Here, the posture of the user viewpoint in the VR space (the posture in this specification refers to the position and / or orientation) matches the posture acquired in the real space by the user posture acquisition unit 1 ( Match by mapping). Therefore, the body shape of the avatar that should be seen from the user's viewpoint in another space (VR space) is reflected in this viewpoint image (image in the VR space). The viewpoint image is acquired from the same viewpoint posture as the viewpoint image used as the projection image. As will be described later, the viewpoint image used for mask generation is generated as an image including the body, but the viewpoint image used as the projection image preferably does not include the body. Thereby, the operation | work which incorporates an insertion image in a viewpoint image can be facilitated. In addition, it is preferable that all viewpoint images are acquired at the same angle of view.

  An example of a viewpoint image for creating a mask is shown in FIG. In this example, the image portion corresponding to the user's body is white, and the others are black. More specifically, FIG. 5 defines a mask as an alpha channel, and expresses a portion to be left as white and the other as black. This avoids the problem of color interference that occurs when using a conventional color key and the color of the subject itself cannot be used, and creates a mask that does not restrict the subject color. be able to.

  In the VR space, the position of the user's body and the positions of other objects (virtual objects) are all known. For this reason, the above-described viewpoint image can accurately represent the shielding relationship between the two. In the example of FIG. 5, a part of the palm is hidden by a stick held in the hand (an example of a virtual object), and the right thumb in front of the stick appears in the image. Here, the object means a living body such as the body of another person.

  In generating a viewpoint image used as a projection image, the illumination condition obtained by the illumination condition calculation unit 11 and the positional relationship of each object (including its own body) in the VR space are used. It is also preferable to form a “shadow” that should be formed by the body. In this way, when a projection image is generated by inserting an insertion image (body image) into the viewpoint image, the reality in the projection image can be further improved.

(Step SA-4 in FIG. 2)
Next, the mask generation unit 4 generates a mask using the viewpoint image. The mask can be formed by extracting a portion corresponding to the user's body (which is known on the system side) from the viewpoint image shown in FIG. For example, the mask can be easily formed by appropriately setting the color of the body part on the viewpoint image and removing the part. Of course, the present invention is not limited to such an alpha-key technique, and a mask can be formed by distinguishing a body part from other parts by an appropriate technique.

In the present embodiment, an accurate occlusion relationship can be expressed on the viewpoint image without performing depth measurement for each pixel position. In the case of measuring the depth of the body position, there is a problem that not only the apparatus configuration becomes complicated and the cost is increased, but also the real-time property is deteriorated because the calculation process takes time. Furthermore, since accurate depth measurement is generally difficult, there is a problem that the accuracy of the obtained mask tends to be insufficient. On the other hand, in this embodiment, depth measurement for determining the body position for each pixel can be omitted.
・ The equipment configuration can be simplified;
・ Cost reduction can be realized;
・ Since the calculation process is simplified, real-time performance is improved;
-Since it is based on VR space data, there is an advantage that high mask accuracy can be realized.

  Moreover, in this embodiment, mask accuracy can be improved by improving the acquisition accuracy of the user posture. In this embodiment, since the user posture is acquired by measurement in real space instead of the image base, it is easy to improve the acquisition accuracy of the user posture, and therefore there is an advantage that the mask accuracy can be easily improved.

(Step SA-5 in FIG. 2)
On the other hand, the illumination condition calculation unit 11 calculates the illumination condition for the body of the avatar in the VR space. In the VR space, the lighting condition for the avatar body at any position can be easily calculated, and thus the detailed description of this calculation method is omitted. By applying this illumination condition to the mask, the cut out inserted image (described later) can be made natural in VR space (a feeling of uncomfortable feeling). The illumination condition can also be applied to the cut out inserted image.

(Step SA-6 in FIG. 2)
On the other hand, the user viewpoint camera 7 acquires a real image from a user viewpoint (this is a viewpoint in the real space, but is consistent with the user viewpoint in the VR space). FIG. 6 shows a schematic explanatory diagram of a real image. Here, the user's body may not be reflected in the real image, but whether the user's body is in the real image can be known from the VR space model data. Therefore, in such a case, the processing itself after the mask generation can be omitted. Hereinafter, since it is assumed that the user's body is included in the real image, the real image may be referred to as a body image. It should be noted that not only the posture of the user viewpoint camera 7 but also the angle of view preferably matches the viewpoint from the VR space. However, when the angle of view of the user viewpoint camera 7 is wider than the viewpoint in the VR space, there is no particular problem as long as mapping between the two in the image space is possible. In the opposite case, it is preferable to give consideration so as not to make the user feel uncomfortable.

(Step SA-7 in FIG. 2)
Next, the insertion image generation unit 5 applies the mask obtained in Step SA-4 to the above-described actual image (body image) to generate an insertion image. An example of the obtained insertion image is shown in FIG.

  In this embodiment, since an insertion image can be generated simply by applying a mask to an image, there is an advantage that the calculation process is simple. For this reason, even when an image with a large number of pixels is used, there is an advantage that the calculation cost can be kept low and real-time processing becomes easy.

(Step SA-8 in FIG. 2)
Next, the projection image generation unit 6 combines (for example, superimposes) the projection image obtained in Step SA-7 on the viewpoint image (see FIG. 8) obtained by the viewpoint image generation unit 3 in Step SA-3. Unlike the viewpoint image for mask generation, the viewpoint image used here is generated assuming that the body of the avatar does not exist. Since such processing is in the VR space, it can be easily performed. In addition, this makes it possible to generate a projection image to be projected to the user simply by superimposing the images. An example of the obtained projection image is shown in FIG.

(Step SA-9 in FIG. 2)
Next, the image presentation unit 8 presents the obtained projection image to the user. This projected image is preferably observable at an equivalent angle of view from a position optically equivalent to the user viewpoint camera 7 (so-called optical conjugate). Thereby, a projection image can be presented to a user without a sense of incongruity.

  In the present embodiment, a part of an actual user's body image can be cut out and incorporated into a projection image obtained from the VR space without contradiction, and presented to the user as a projection image. For this reason, according to the present embodiment, there is an advantage that a high immersive feeling can be given to the user when his / her body appears in the projection image.

  Furthermore, by adding natural lighting conditions to the inserted image in the VR space, it is possible to further reduce the sense of incongruity of the projected image, and to further improve the immersive feeling felt by the user.

(Modification)
Next, an example in which the above-described embodiment is modified will be described with reference to FIG. In the description of this modification, steps that perform substantially the same processes as those in the above-described embodiment are denoted by the same reference numerals, thereby avoiding complicated description.

  In the above-described embodiment, using the user's body shape acquired in Step SA-2, in Step SA-3, the user viewpoint image in the VR space including the user's body shape (in the above example, for mask generation) Generated). And the mask was produced | generated in step SA-4 using this user viewpoint image.

  On the other hand, in this modification, a mask is generated in step SA-4 using the user's body shape calculated in step SA-2. This process can be performed at high speed because the user's body position and orientation with respect to the user viewpoint are known. In addition, this process can be implemented, for example, in one of the body shape calculation unit 2 and the mask generation unit 4 or in another functional element.

  In this modification, the process of generating the user viewpoint image using the VR space model data can be omitted, so that the calculation process can be further speeded up. However, in this modification, since VR space model data is not used, it is difficult to calculate the shielding relationship between the object in the VR space and the user's body as described above.

  Also in this modified example, the user viewpoint image used in step SA-8 is generated in the same manner as in the above embodiment (for example, as shown in FIG. 8) in step SA-3.

  Other configurations and advantages of the modification are basically the same as those of the above-described embodiment, and thus further description thereof is omitted.

(Example 1 of HMD: No viewpoint position correction)
Next, Example 1 will be described as an example of a head mounted display (HMD) on which the user viewpoint camera 7 and the image presentation unit 8 are mounted with reference to FIG.

  In the first embodiment, different images (that is, images given parallax for stereoscopic viewing) can be presented to the left eye 121 and the right eye 122, respectively.

  Specifically, in the first embodiment, the user viewpoint camera 7 includes a left-eye camera 71 and a right-eye camera 72. Further, in the first embodiment, the image presenting unit 8 includes a left eye presenting unit 81 and a right eye presenting unit 82.

  The left-eye camera 71 is disposed at a position optically conjugate with the left eye 121, and an image in real space (for example, a body image including the user's body) is obtained via the left-eye mirror 131. You can shoot. Similarly, the right-eye camera 72 is disposed at a position optically conjugate with the right eye 122, and can capture a real space image via the right-eye mirror 132.

  The left-eye presentation unit 81 is configured to present a left-eye image to the user's left eye 121 via the left-eye mirror 131. In addition, a left-eye lens 141 for adjusting the focal length is disposed on the optical path between the left-eye presenting unit 81 and the left eye 121, whereby the presenting unit 81 can be used for the left-eye. It is arranged at a position optically conjugate with the camera 71. Similarly, the right-eye presentation unit 82 is configured to be able to present a right-eye image to the user's right eye 122 via the right-eye mirror 132. In addition, a right-eye lens 142 for adjusting the focal length is disposed on the optical path between the right-eye presentation unit 82 and the right eye 122, whereby the presentation unit 82 is used for the right-eye. It is arranged at a position optically conjugate with the camera 72.

  Moreover, in Example 1, it is preferable to use the trapezoidal mirror schematically shown in FIG. 12 as the left-eye mirror 131 and the right-eye mirror 132. By using this shape, a rectangular VR space image can be presented to the user, and a rectangular real space image can be acquired by the camera.

  In the first embodiment, the user does not actually see the real space, but only the projection image generated in the VR space. Here, the image of the user's body is acquired by a camera at a position conjugate with the user's eyes, and since this image is combined with the projection image, the user has a high feeling of being buried in the VR space. be able to.

(Example 2 of HMD: with viewpoint position correction)
Next, Example 2 will be described as another example of a head mounted display (HMD) on which the user viewpoint camera 7 and the image presentation unit 8 are mounted with reference to FIG. In the description of the second embodiment, the same reference numerals are used for the components that are basically the same as those in the first embodiment, thereby avoiding complicated description.

  In the first embodiment described above, the left and right cameras 71 and 72 are disposed at positions optically conjugate with the left and right eyes 121 and 122. On the other hand, in the second embodiment, the left and right cameras 71 and 72 are arranged to be offset forward from the left and right eyes 121 and 122, and therefore, from a position optically conjugate with these eyes. It is off.

  For this reason, in Example 2, the left and right cameras 71 and 72 can directly acquire a real space image without using a mirror (see FIG. 13). On the other hand, the left and right presentation units 81 and 82 present images to the left and right eyes 121 and 122 via the left and right mirrors 131 and 132, respectively.

  Therefore, in Example 2, there exists an advantage that the mechanical structure as HMD can be simplified.

  On the other hand, in the second embodiment, as described above, there is an offset between the camera viewpoint and the user viewpoint. This offset relationship is schematically shown in FIG. In this example, the angle of view of the user viewpoint is θ1, the angle of view of the camera is θ2, the width of the projection plane viewed from the user viewpoint is W1, and the width of the projection plane viewed from the camera is W2. Further, the offset amount between the user viewpoint and the camera viewpoint is D, and the distance (focal length) from the user viewpoint to the projection plane is F.

  As in the second embodiment, when there is an offset between the user viewpoint and the camera viewpoint, mapping between the camera image and the user viewpoint image (that is, the VR space image) is performed, and matching between the two can be maintained. preferable. For example, by performing coordinate conversion of a camera image (real space image), consistency with the viewpoint image in the VR space can be maintained. Of course, on the contrary, the VR space image can be coordinate-transformed. In this way, even when there is an offset between the viewpoints, it is possible to present the image without causing the user to feel uncomfortable. Note that the calculation of mapping between images can be geometrically performed if the parameters as shown in FIG. 14 are known, and thus detailed description thereof will be omitted.

  Other configurations and advantages of the second embodiment are basically the same as those of the first embodiment, and thus detailed description thereof is omitted.

  The calculation processing in the above-described embodiment can be implemented by incorporating appropriate computer software into the computer.

  The contents of the present invention are not limited to the above embodiment. In the present invention, various modifications can be made to the specific configuration within the scope of the claims.

  For example, each component described above may exist as a functional block, and may not exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing or cloud computing.

  Furthermore, in the present embodiment, a VR space is assumed as another space, but the present invention can be applied to a real space separated from a place where the user exists as another space. In this case, the camera viewpoint in another space (that is, the remote space) is matched with the user viewpoint as in the above embodiment. In addition, camera parameters in another space match the user-side camera parameters or can be matched by correction.

DESCRIPTION OF SYMBOLS 1 User posture acquisition part 2 Body shape calculation part 3 Viewpoint image generation part 4 Mask generation part 5 Insertion image generation part 6 Projection image generation part 7 User viewpoint camera 71 Camera for left eyes 72 Camera for right eyes 8 Image presentation part 81 Left eye Presentation unit 82 Right-eye presentation unit 9 Body model data storage unit 10 VR space model data storage unit 11 Illumination condition calculation unit 121 Left eye 122 Right eye 131 Left-eye mirror 132 Right-eye mirror 141 Left-eye lens 142 Right Ophthalmic lens

Claims (10)

To generate an insertion image by extracting at least a part of a user's body image in real space, and to generate a mask used to superimpose the insertion image on a viewpoint image in another space viewed from the user's viewpoint Equipment,
A user posture acquisition unit, a body shape calculation unit, and a mask generation unit;
The user posture acquisition unit is configured to acquire the posture of the user in the real space,
The body shape calculation unit is configured to calculate the user's body shape using the acquired posture of the user,
The mask generation unit is configured to generate a mask for extracting the insertion image from the body image of the user in the real space based on the body shape that should be seen from the viewpoint of the user in the different space. A mask generation device characterized by that. It also has a viewpoint image generator,
The viewpoint image generation unit is configured to generate the viewpoint image when viewed from the viewpoint of the user in the different space,
2. The mask generation unit is configured to generate the mask using at least a part of the body part of the user present in the viewpoint image as a body shape that should be visible from the viewpoint of the user. The mask generator described in 1. Furthermore, a VR space model data storage unit is provided,
The mask generation apparatus according to claim 2, wherein the viewpoint image generation unit is configured to generate the viewpoint image based on VR space model data stored in the VR space model data storage unit. It also has a user viewpoint camera,
The user viewpoint camera is configured to acquire a body image of the user in the real space,
The attitude of the viewpoint of the user viewpoint camera matches the attitude of the user's viewpoint in the different space, or can be substantially matched by correcting an image viewed from either viewpoint. The mask production | generation apparatus of any one of Claims 1-3. Furthermore, it has a lighting condition calculation unit,
The mask generation according to any one of claims 1 to 4, wherein the illumination condition calculation unit is configured to set a luminance or color condition in the inserted image based on an illumination condition in the different space. apparatus. Between at least a part of the body shape that should be seen from the user's viewpoint in the different space and some object in the different space, one is the other when viewed from the user's viewpoint in the different space The mask generation device according to any one of claims 1 to 5, wherein the mask generation device is shielded by the mask. Comprising the mask generation device according to any one of claims 1 to 6 and an insertion image generation unit;
The insertion image generation device is configured to generate the insertion image by applying the mask to the body image. The insertion image generation device according to claim 7, a projection image generation unit, and an image presentation unit,
The projection image generation unit is configured to generate a projection image to be projected to the user by combining the insertion image with a viewpoint image in another space viewed from the user's viewpoint.
The image presenting device is configured to project the projection image from a viewpoint position optically conjugate with a user's viewpoint position in the real space. To generate an insertion image by extracting at least a part of a user's body image in real space, and to generate a mask used to superimpose the insertion image on a viewpoint image in another space viewed from the user's viewpoint A method using the apparatus of
Obtaining the posture of the user in the real space;
Calculating the user's body shape using the acquired posture of the user;
Generating a mask for extracting the insertion image from the body image of the user in the real space based on the body shape that should be seen from the viewpoint of the user in the different space. Mask generation method.   A computer program for causing a computer to execute each step according to claim 9.

Images