JP2021034744A - Image presentation device and program - Google Patents

Abstract

To provide an image presentation device capable of displaying a human body or a material neighbor a viewer user, or the like in a virtual space.SOLUTION: An image presentation device comprises: a reproduction video image acquisition part that acquires a reproduction video image as a video image for a reproduction; a circumference video image acquisition part that acquires a circumference video image as a video image of the circumference of an own device; a recognition part that recognizes a prescribed subject contained in the circumference video image; a distance information acquisition part that acquires distance information corresponding to the circumference video image; a mask generation part that generates mask information expressing whether or not it is a region which has to present the reproduction video image on the basis of the distance information, and expressing whether or not it is a region where the subject recognized by the recognition part exists; and a presentation part that outputs at least the reproduction video image on the basis of the mask information generated by the mask generation part.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video presentation device and a program.

Research and development on virtual reality is in progress. As virtual reality type content, many live-action content shot around the entire circumference using a 360-degree camera has come to be seen. These have also made it possible to provide users with the experience of being in another location. For example, a head-mounted display (HMD) or the like is used to provide virtual reality type contents.

In addition, a technology has been proposed for multiple users to experience the virtual space provided by virtual reality type contents together. By allowing multiple users to experience the virtual space together, the user can experience the virtual space while sharing the enjoyment with other users, rather than the experience of only one person covering the entire field of view with a head-mounted display. You can also.

For example, in the techniques shown in Non-Patent Document 1 and Non-Patent Document 2, the avatars of other users are displayed in the virtual space. Further, in the technique described in Non-Patent Document 3, it is also proposed to display another user in a remote place cut out from a live-action video in a virtual space.

In addition, a device has also been developed that generates and presents an image according to the position and posture of a user who views virtual reality type content. For example, Patent Document 1 describes a configuration example of a stereoscopic image display device. The stereoscopic image display device includes a position / posture measuring device that measures the individual position / posture of the head-mounted display device worn by the user. Then, the binocular image generation means has a left eye image that enters the left eye of each user and a right eye that enters the right eye when the user observes the three-dimensional original image according to the position / posture measured by the position / posture measuring device. Generate video. As a result, the stereoscopic image display device displays an image according to the position and posture of the user.

Further, Patent Document 2 describes an image processing device that determines the transparency of a generated image according to the moving speed of an imaged object in the real space.

Japanese Unexamined Patent Publication No. 2016-0119170 Japanese Unexamined Patent Publication No. 2018-063567

“Facebook Spaces”, Beta, Facebook, Inc., updated 2019, downloaded August 8, 2019, URL https://www.facebook.com/spaces "Have you not used it yet? The possibility of VR opened up by" Facebook Spaces "", Posted by caug5, CAPA Co., Ltd., May 8, 2018, August 8, 2019 Download, URL https: // www. capa.co.jp/archives/22169 Simon N.B Gunkel, Marleen D.W. Dohmen, Hans Stokking, Omar Niamut, "360-degree photo-realistic VR conferencing", the 26th IEEE Conference on Virtual Reality and 3D User Interfaces, posters, 2 pages, Mar. 2019.

However, for the purpose of further immersing yourself in the world of content, you can not only enjoy and utilize the virtual space with other users, but also exist together in the real space. It is effective to create the feeling of sharing a virtual space with other users. For that purpose, at least a part of the user's own body to watch, the situation of the place where the user is, and the appearance of other users who are watching the same content at the same time are presented as a part of the content. Then, it is expected that the effect as if immersed in the world of the content can be obtained.

The present invention has been made based on the above-mentioned problem recognition, and at least a part of the viewer's own body, the place where the viewer is present, and the appearance of other users who are watching at the same time. It is intended to provide a video presentation device and a program capable of displaying an image in a virtual space.

[1] In order to solve the above problems, the image presenting apparatus according to one aspect of the present invention includes a reproduction image acquisition unit that acquires a reproduction image that is a reproduction image, and a peripheral image that is an image around the own device. A peripheral image acquisition unit that acquires an image, a recognition unit that recognizes a predetermined subject included in the peripheral image, a distance information acquisition unit that acquires distance information corresponding to the peripheral image, and the distance information based on the distance information. A mask generation unit that generates mask information indicating whether or not the image for reproduction should be presented, and mask information indicating whether or not the subject is an area recognized by the recognition unit. And a presentation unit that outputs at least the reproduction video based on the mask information generated by the mask generation unit.

[2] Further, one aspect of the present invention is a synchronization unit that synchronizes the playback position of the playback video output by the presentation unit in the time direction with another video presentation device in the above-mentioned video presentation device. Is further prepared.

[3] Further, in one aspect of the present invention, in the above-mentioned image presenting apparatus, the presenting unit is at least one of the reproduction image and the peripheral image for each area in the screen based on the mask information. Is output to display.

[4] Further, one aspect of the present invention further includes a presentation area extraction unit that cuts out only a part of the peripheral image acquired by the peripheral image acquisition unit in the image presentation device, and the recognition unit. Is to recognize the predetermined subject based on the peripheral image before the presentation area extraction unit cuts out, and the presentation unit outputs the peripheral image cut out by the presentation area extraction unit.

[5] Further, in one aspect of the present invention, in the video presenting apparatus, the mask generation unit includes information on a mixing ratio in a region where the playback video and the peripheral video are mixed and presented. In the region where the mask information is generated and the reproduction image and the peripheral image are mixed and presented, the reproduction image and the peripheral image are mixed based on the information of the mixing ratio. It outputs as if it were.

[6] Further, in one aspect of the present invention, in the above-mentioned video presenting apparatus, the presenting unit has a function of changing the transparency of the playback video, and the transparency is adjusted according to the mask information. It outputs video for playback.

[7] Further, in one aspect of the present invention, in the above-mentioned image presenting apparatus, the transmittance is a real number of 0 or more and 1 or less, and the presenting unit reproduces the reproduction with the transmittance corresponding to the mask information. It outputs the video for the purpose.

[8] Further, one aspect of the present invention is the above-mentioned image presenting device, in which the image presenting device is a type of device worn on the head of a mammalian organism (for example, a human), and the position of the image presenting device. The presenting unit further includes a position and a posture detecting unit for detecting the posture, and the presenting unit is a partial image corresponding to the position and the posture detected by the position and the posture detecting unit in the reproduction image which is an all-around image. Is the one that outputs.

[9] Further, one aspect of the present invention includes a playback image acquisition process for acquiring a reproduction image which is a reproduction image, and a peripheral image acquisition process for acquiring a peripheral image which is an image around the own device. Is it a recognition process for recognizing a predetermined subject included in the peripheral image, a distance acquisition process for acquiring distance information corresponding to the peripheral image, and an area for presenting the reproduction image based on the distance information? A mask generation process that generates mask information indicating whether or not the mask information indicates whether or not the subject exists in the area recognized by the recognition unit, and a mask generated in the mask generation process. Based on the information, it is a program that causes a computer to execute at least the presentation process of outputting the playback video and the processing.

According to the present invention, it is possible to further increase the immersive feeling in the world of virtual reality.

It is a block diagram which shows the schematic functional structure of the image presenting apparatus according to 1st Embodiment of this invention. It is a schematic diagram which shows the configuration example of the system in which a plurality of image presentation devices by 1st Embodiment cooperate with each other. Regarding the video presented by the video presentation device according to the first embodiment, a schematic diagram showing a distinction for each region, whether the video acquired by the peripheral video acquisition unit is presented or the video acquired by the playback video acquisition unit is presented. (A plan view of a three-dimensional space as a plan view). It is the schematic which shows the example of the image processing (cutting out) by the presentation area extraction part by 1st Embodiment. It is a schematic diagram which shows the example of the presentation method of the image acquired by the peripheral image acquisition unit, and the image acquired by the reproduction image acquisition unit 5 by the image presenting apparatus according to 1st Embodiment. (A) is a view of the three-dimensional space from the side, and (B) is a graph showing the relationship between the distance from the image presenting device and the value of the mask data. It is the schematic which shows the structural example of the image presented by the image presenting apparatus by 1st Embodiment. It is a flowchart which shows the procedure of processing about one frame of the image presented by the image presenting apparatus by 1st Embodiment. It is a block diagram which shows the schematic functional structure of the image presenting apparatus according to 2nd Embodiment of this invention. It is the schematic which shows the structural example of the image presented by the image presenting apparatus 51 by 2nd Embodiment.

[First Embodiment]
Next, the first embodiment of the present invention will be described with reference to the drawings. The video presentation device according to the first embodiment uses a video see-through method. The video see-through method will be described later.

FIG. 1 is a block diagram showing a schematic functional configuration of the image presentation device according to the present embodiment. As shown in the figure, the image presentation device 1 includes a processing unit 2, a peripheral image acquisition unit 3, a distance information acquisition unit 4, a playback image acquisition unit 5, a position / attitude acquisition unit 6, and a synchronization unit 7. , And a display device 9. Further, the processing unit 2 includes a recognition unit 21, a presentation area extraction unit 22, a mask generation unit 23, and a presentation unit 24. At least a part of the functions of the above-mentioned functional units included in the image presenting device 1 can be realized by using an electronic circuit. Further, a part or all of each of these functional parts may be realized by using a computer and a program. Each functional unit has a storage means, if necessary. The storage means is, for example, a flip-flop that maintains a predetermined state on an electronic circuit, a variable on the program when a program is used, or a memory that is allocated by executing the program. Further, if necessary, a non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used. The functions of each part are as follows.

The image presentation device 1 is realized as, for example, a type of device worn on the head of a mammalian organism (for example, a human). That is, the image presentation device 1 can be realized as a head-mounted display in which an image display device, an external measurement device, and a function for processing an image (for example, realized by using a computer) are integrated. The image presentation device 1 is used for superimposing an image (reproduction image) on a real space by using a video see-through or optical see-through method. In the present embodiment, the video presentation device 1 realizes a video see-through method.

The processing unit 2 processes information from the peripheral image acquisition unit 3, the distance information acquisition unit 4, the playback image acquisition unit 5, the position / attitude acquisition unit 6, and the synchronization unit 7, and displays the information on the display device 9. Calculate and output the video to be displayed. The processing unit 2 may be built in the head-mounted video presentation device 1, or may be realized as a separate body (for example, as a PC) from the head-mounted main body.

The peripheral image acquisition unit 3 acquires peripheral images that are images around the image presenting device 51 (own device). The peripheral image acquisition unit 3 is realized by using, for example, a stereo camera. Further, the image acquisition means used by the peripheral image acquisition unit 3 may be a so-called fisheye camera having a wide viewing angle.

The distance information acquisition unit 4 acquires the distance information (depth map) corresponding to the image (peripheral image) acquired by the peripheral image acquisition unit 3. The distance information acquisition unit 4 may acquire a distance image corresponding to the peripheral image as a device separate from the peripheral image acquisition unit 3. Further, the distance information acquisition unit 4 may obtain a fisheye image from the peripheral image acquisition unit 3 realized by using a fisheye stereo camera and calculate the distance information. It should be noted that the acquisition of the distance image itself can be realized by using the existing technology.

That is, the peripheral image acquisition unit 3 and the distance information acquisition unit 4 are combined from the viewpoint of the own device (that is, from the viewpoint of the user who wears the image presentation device 1 on the head), RGBD (R: red). , G: green, B: blue, D: distance) Information is acquired.

The playback video acquisition unit 5 acquires the playback video. The playback video is, for example, the content of a 360-degree live-action video shot of the entire sky (or the entire circumference). Further, the playback video may be a video using computer graphics in part or in whole. The playback video acquisition unit 5 acquires playback video from a recording medium such as a DVD, a Blu-ray disc, or a hard disk device. Alternatively, the playback video acquisition unit 5 may acquire the playback video delivered by a signal such as communication or broadcasting. The playback video consists of a time-series frame image and audio that is appropriately required.

The position / posture acquisition unit 6 detects the position and posture of its own device (image presentation device 1). Here, the position is information represented by the position coordinates in the three-dimensional space. Further, the posture is information representing the orientation of the image presenting device 1 by, for example, three-dimensional angle information. The position / posture acquisition unit 6 may acquire the position and posture by incorporating, for example, a gyro sensor. Further, since the self-position can be estimated from the image of the stereo camera, the position / posture acquisition unit 6 may calculate the position and the posture from the image acquired by the peripheral image acquisition unit 3. Further, the position / attitude acquisition unit 6 receives a beacon signal from the outside, or receives a signal such as infrared rays emitted from the outside in synchronization with the clock to identify a place in the real space. , The position and posture may be acquired. Further, in the position / posture acquisition unit 6, a plurality of cameras provided outside (for example, in a room which is a viewing space for contents) take a picture of the own device (image presenting device 1), and the position and posture obtained are obtained. Information may be received.

The synchronization unit 7 exchanges information at any time by communicating with another video presenting device 1. When the own device (video presentation device 1) reproduces the same video content in synchronization with another video presentation device 1, the synchronization unit 7 displays the playback video on the display device 9 in the time direction. Is synchronized with the other video presentation device 1. Specifically, for example, information on the relative time position of the content to be reproduced is exchanged between the synchronization units 7 of the video presentation device 1 at predetermined time intervals. The video presentation device 1 adjusts the output timing of the video content to be reproduced based on the time information received from the other device by the synchronization unit 7 of the own device.

Further, when the synchronization unit 7 starts playing the same video content in synchronization with the other video presentation device 1, the synchronization unit 7 transmits the position and posture information of the own device to the other video presentation device 1. Further, the synchronization unit 7 receives information on the position and orientation of the device from the other video presentation device 1. Here, the number of "other devices" may be one or two or more. In this way, the synchronization unit 7 acquires the position and orientation information of the other video presenting device 1 to cut out (based on the position and orientation) the video of the virtual reality content to be reproduced by the other video presenting device. It is possible to match with 1. The configuration for cooperative operation by the plurality of video presentation devices 1 will be described later.

The display device 9 displays the image output by the presentation unit 24. The video output by the presentation unit 24 may include both a peripheral video and a playback video. The display device 9 displays an image passed from the presentation unit 24 for each area on the screen. The display device 9 may perform stereo display for stereoscopic viewing, for example.

The recognition unit 21 performs a process of recognizing a predetermined subject (for example, a person) in the peripheral image acquired by the peripheral image acquisition unit 3. The recognition unit 21 has learned in advance the characteristics of a predetermined subject in the video by machine learning. By referring to the trained model, the recognition unit 21 identifies a part (information such as coordinates of a region in the image) in which the subject is reflected in the peripheral image, and outputs the information. As a result of the recognition process, the recognition unit 21 passes the position information of the area in the peripheral image to the mask generation unit 23.

The presentation area extraction unit 22 cuts out an image (image) of the presentation area from the peripheral image acquired by the peripheral image acquisition unit 3. The image in the presentation area may be only a part of the entire peripheral image (for example, a part near the center). As a result, the viewing angle of the image acquired by the peripheral image acquisition unit 3 and the viewing angle of the image displayed on the display device 9 can be matched. The presentation area extraction unit 22 passes the cut out video (image) to the presentation unit 24. The method by which the presentation area extraction unit 22 extracts the video of the presentation area will be described later with reference to another figure. The process of the presentation area extraction unit 22 may include the process of central projection, as will be described later.

The presentation area extraction unit 22 also passes information on the positional relationship between the peripheral image before extraction and the extracted and cut out image to the mask generation unit 23. In other words, the presentation area extraction unit 22 positions the image acquired by the peripheral image acquisition unit 3 and the image matched to the viewing angle of the display device 9 when the image is cut out by using a technique such as central projection. The correspondence between the above is transmitted to the mask generation unit 23. As a result, the mask generation unit 23 can generate mask information suitable for the coordinate system for presentation.

When a wide-angle lens such as a fisheye lens is selected as the camera used by the peripheral image acquisition unit 3, even if the head of the user wearing the image presentation device 1 moves, the recognition unit 21 captures a wide range of images. Increases the possibility that a predetermined subject can be correctly recognized. That is, when the recognition unit 21 recognizes a subject such as a person, it becomes easy to track the subject. That is, the recognition accuracy is stable. On the other hand, if the viewing angle is too wide, it may not match the viewing angle of the display device 9, but the processing of the presentation area extraction unit 22 makes it possible to match the viewing angles of both.

The mask generation unit 23 generates mask information. The mask information indicates whether or not it is an area in which a playback image should be presented based on the distance information passed from the distance information acquisition unit 4. Further, the mask information indicates whether or not the subject recognized by the recognition unit 21 exists in the region. In the present embodiment, the mask information is information indicating whether the peripheral image should be displayed or the reproduction image should be displayed. The mask generation unit 23 may generate mask information including information on the mixing ratio in the region where the playback image and the peripheral image are mixed and presented.

As described above, the mask generation unit 23 generates a mask based on the distance and a mask based on the recognition result. As a result, the presentation unit 24 can make the following presentations. For example, the body of oneself or the body of another person who is experiencing the same virtual reality content together in the same space can be presented in the virtual reality image. With respect to a predetermined subject (person or the like) recognized by the recognition unit 21, the peripheral image can be displayed regardless of the distance from the own device. For other than a specific subject (person, etc.), the presentation is based on the distance. That is, an object in a range relatively close to the own device is presented in the virtual reality space in a form included in the peripheral image. In addition, objects that exist in a range relatively far from the own device are not presented in the form included in the peripheral image. In such an area where an object existing in a range relatively far from the own device exists, a playback image is presented.

Here, the "relatively close range" is, for example, a range in which a person can reach out and touch without moving from the place (for example, while sitting). For example, the range is within 1 meter. On the contrary, the "relatively distant range" is, for example, a range of about 2 meters or more. In the middle distance range (1 meter or more and 2 meters or less), it is conceivable to present an image in which a short-distance peripheral image and a long-distance playback image are mixed.

The presentation unit 24 outputs at least a playback image based on the mask information generated by the mask generation unit 23. Also, based on the mask information, the peripheral image is output in the short distance area. When the presentation unit 24 presents the playback video, an appropriate portion of the video is cut out from the entire playback video based on the position and orientation of the video presentation device 1 and displayed. That is, the presentation unit 24 may output a partial image corresponding to the position and the posture detected by the position / posture acquisition unit 6 in the reproduction image which is the all-around image.

In the region where the playback video and the peripheral video are mixed and presented, the presentation unit 24 outputs them so that the playback video and the peripheral video are mixed based on the value of the mixing ratio of the mask information. .. That is, the presentation unit 24 outputs at least one of the reproduction video and the peripheral video for each area in the screen based on the mask information. Further, the presentation unit 24 may output both images in a mixed manner based on the above mixing ratio.

FIG. 2 is a schematic diagram showing a configuration example of a system in which a plurality of video presentation devices 1 cooperate with each other. As shown in the figure, the plurality of video presentation devices 1 can operate in cooperation with each other while communicating with each other. Although the figure shows a situation in which three video presentation devices 1 are operating at the same time as an example, the number of video presentation devices 1 that cooperate with each other is arbitrary. In the illustrated situation, the three video presentation devices 1 operate simultaneously in the same space (for example, the same room). Each video presentation device 1 is used by one user. In the illustrated example, the three video presentation devices 1 exchange information with each other through the wireless router 31 and via the server device 32. Specifically, the synchronization units 7 of each video presentation device 1 exchange information with each other by communication.

The main types of information exchanged between the plurality of video presentation devices 1 are the following two types.
The first is the position and orientation of the video presentation device 1. At the start of content reproduction, the video presentation device 1 notifies another video presentation device 1 of information on the position and posture of its own device. At the same time, the image presenting device 1 receives information on the position and orientation of the other device. In this way, when each of the plurality of video presentation devices 1 acquires the position and orientation information of the other video presentation devices 1 so that the video presentation devices 1 simultaneously reproduce the same content, the playback video can be used. The partial video to be cut out can be matched between the video presentation devices 1.
The second is information for matching the reproduction timing between the image presentation devices 1 when reproducing the reproduction image. Specifically, for example, the video presentation device 1 exchanges information on the time position of content reproduction with each other. Such information exchange may be performed at predetermined time intervals. As a result, the same content can be reproduced at the same timing between the plurality of video presentation devices 1.

Although FIG. 2 shows a configuration in which a plurality of video presentation devices 1 operate in cooperation with each other, the video presentation device 1 does not operate in cooperation with other video presentation devices 1 but also operates in a single form. It is possible. Further, for example, one of the plurality of video presentation devices 1 has a server function to collect information on each video presentation device 1 and distribute the collected information to each video presentation device 1. It may be in the form of a server.

FIG. 3 shows a distinction for each region between the video acquired by the peripheral video acquisition unit 3 and the video acquired by the playback video acquisition unit 5 with respect to the video presented by the video presentation device 1. It is a schematic diagram. The figure is a plan view of a space (for example, inside a room) in which the image presenting device 1 is used.

In the figure, reference numeral 100 is an area such as a room. The area 100 may be surrounded by, for example, a wall or the like. Reference numeral 101 is a user who intends to view a video by using the video presentation device 1 in the area 100. Further, each of the reference numerals 102 and 103 is a person different from the user 101. The person 102 is relatively close to the user 101. The person 103 exists at a position relatively far from the user 101. Reference numeral 108 is a table placed on the floor in the area 100. Further, reference numeral 121 is a sub-region within the range 100 and within a predetermined distance from the user 101. The sub-region 121 is indicated by a broken line.

In the present embodiment, as described above, the presentation unit 24 is acquired by the peripheral image acquisition unit 3 with respect to the area where the object within a predetermined distance exists and the area recognized as a person regardless of the distance. Present the video. Further, the presentation unit 24 acquires the area other than the above area (the area where an object within a predetermined distance exists and the area recognized as a person regardless of the distance) by the playback video acquisition unit 5. Present the video. That is, with respect to the area shown by hatching in FIG. 3, the presentation unit 24 of the video presentation device 1 used by the user 101 presents the video acquired by the peripheral video acquisition unit 3. Here, the "regions indicated by hatching" are a region inside the boundary line 131 and within the region 100 and a region inside the boundary line 132 (note that all the regions are within the region 100). ) And.

The video presenting device 1 displays which video, the video acquired by the peripheral video acquisition unit 3 or the video acquired by the playback video acquisition unit 5, is displayed based on the distance in the three-dimensional space. To control. Further, the image presenting device 1 sets the image acquired by the peripheral image acquisition unit 3 and the image for reproduction according to the recognition result (whether or not the area includes a person) of the image captured in the three-dimensional space. It controls which of the images acquired by the acquisition unit 5 is displayed. FIG. 3 described here shows a state of being projected onto a plan view by omitting the dimension in the height direction in the three-dimensional space.

FIG. 4 is a schematic view showing an example of image processing by the presentation area extraction unit 22. FIG. 3A shows an example of an image at a viewing angle (FOV, Field of View) taken by the peripheral image acquisition unit 3. FIG. (B) is an example of an image at a viewing angle as a result of extraction by the presentation area extraction unit 22 based on the image of FIG. (A). As shown in the figure, the peripheral image acquisition unit 3 may use, for example, a lens having a short focal length (for example, a fisheye lens or a lens close thereto) to capture an image (image) having a very wide viewing angle. .. The presentation area extraction unit 22 cuts out an image having such a wide viewing angle according to the viewing angle of the display device 9 (for example, a head-mounted display) by, for example, central projection. The projection method here is not limited to the central projection, and for example, a parallel projection or the like may be used.

FIG. 5 is a schematic view showing an example of a method of presenting the video acquired by the peripheral video acquisition unit 3 and the video acquired by the playback video acquisition unit 5 by the video presentation device 1. In the figure, the image presenting device 1 presents an image (referred to as “real space image” for convenience) acquired by the peripheral image acquisition unit 3, and the image acquired by the playback image acquisition unit 5. An area for presenting an image (referred to as "virtual image" for convenience) and an area for presenting an image (referred to as "mixed image" for convenience) in which these real space images and virtual images are mixed are shown. ..

FIG. (A) is a schematic diagram showing the relationship between the distance from the user 101 who views the content using the video presentation device 1 and the video to be presented. The horizontal axis in this figure represents the distance from the user 101. It is divided into three areas, R1, R2, and R3, according to the distance d (unit: meter (m)) from the user 101. The range of 0 ≦ d <1 is the region R1. With respect to the region R1, the video presenter 1 presents a real real space video. The range of 1 ≦ d <2 is the region R2. With respect to region R2, the video presenter 1 presents a mixed video. The range of 2 ≦ d is the region R3. With respect to the area R3, the video presentation device 1 presents a virtual video (virtual).

FIG. 3B is a graph showing an example of mask data for realizing the display method shown in FIG. The horizontal axis of this graph is the distance from the user of the video presentation device 1. The vertical axis of this graph is the mask data value m. m = 0.0 corresponds to displaying only the real space image (that is, the ratio of the virtual image is 0.0). m = 1.0 corresponds to displaying only the virtual image (that is, the ratio of the virtual image is 1.0). M in the range of 0.0 <m <1.0 represents the ratio of the virtual image in the display of the mixed image. In the example shown in FIG. 3B, m = 0.0 when 0 ≦ d <1 (region R1). Further, when 1 ≦ d <2 (region R2), 0.0 <m <1.0 and m is variable. As an example, m = d-1.0. Further, when 2 ≦ d (region R3), m = 1.0.

That is, the mask generation unit 23 generates a mask image in which the value of the mask value (ratio of virtual images) m corresponding to the distance d is the pixel value based on the distance image passed from the distance information acquisition unit 4. Further, the presentation unit 24 appropriately produces a real space image, a virtual image, or a mixed image (the mixing ratio depends on m) according to the value (m value) of each pixel of the mask image passed from the mask generation unit 23. Present.

Here, in FIG. 5, the result of the recognition process by the recognition unit 21 is omitted. Actually, based on the result of the recognition process of the recognition unit 21, the real space image is presented in the area where a person exists, regardless of the distance from the image presenting device 1. That is, with respect to the region where a person exists, m = 0.0 in the mask image generated by the mask generation unit 23.

In FIG. 5, m = d-1.0 is set as an example in the region for displaying the mixed image. However, the relationship between m and d is not limited to the relationship expressed by this mathematical formula. m and d may be other correspondence relationships. The value of m may be a monotonous increase in a broad sense with respect to the value of d.

Further, in FIG. 5, a region R2 is provided as a region for displaying the mixed video. However, the area for displaying the mixed image does not necessarily have to be provided. As an example, in the range of d ≦ 1.5, the area R1 (the area for displaying the real space image, m = 0.0) is set, and in the range of 1.5 <d, the area R3 (virtual image is displayed). It is an area to be displayed, and m = 1.0).

FIG. 6 is a schematic view showing a configuration example of a video presented by the video presentation device 1 of the present embodiment. As shown in the figure, the presentation unit 24 of the video presentation device 1 displays a presentation video that is the result of synthesizing the playback video (virtual video video) of (1) and the peripheral video (real space video) of (2). Present. In the figure, reference numeral 301 is a reproduction video. The playback video 301 is a video obtained by cutting out a part of the original all-sky video. Reference numeral 302 is a peripheral image. Further, reference numeral 303 is a mask image generated based on the distance from the image presenting device 1 and the recognition result of the peripheral image. A masked peripheral image 304 is obtained by synthesizing the peripheral image 302 and the mask image 303. The white portion of the mask image 303 is an area allocated for the peripheral image 302. The hatched portion of the mask image 303 is an area allocated for the reproduction image 301. Then, the presentation image 305 is obtained by synthesizing the reproduction image 301 and the masked peripheral image 304. Note that the size of the playback image 301 is larger than the size of the masked peripheral image 304. The masked peripheral image 304 is combined with a predetermined partial area of the reproduction image 301. The presentation unit 24 displays the presentation video synthesized by the video see-through method on the display device 9 in this way. In this video see-through method, the presentation unit 24 includes an image of the user's own body of the image presenting device 1 and the periphery of the user's own vicinity (within a predetermined distance) in the all-sky image (reproduction image). The image and the image of the body of a person (for example, a person who is simultaneously viewing and experiencing the same content) existing in the same space (inside the room) are superimposed. For example, a head-mounted display that is worn on the user's head is suitable for displaying such video see-through type images.

FIG. 7 is a flowchart showing a processing procedure for one frame of the video presented by the video presentation device 1. As a specific processing procedure, another procedure having the same content as the processing presented in the figure may be used. Hereinafter, the procedure will be described according to this flowchart.

In step S11, the peripheral image acquisition unit 3 acquires the peripheral image for one frame and transfers the image to the processing unit 2.
In step S12, the distance information acquisition unit 4 acquires a distance image (depth map) corresponding to one frame of the video, and transfers the distance image to the processing unit 2.

In step S13, the recognition unit 21 performs recognition processing based on one frame of the peripheral image passed from the peripheral image acquisition unit 3, and identifies an area in which a person is reflected. The recognition unit 21 outputs the position information of the human area in the peripheral image. The recognition unit 21 passes the result of the recognition process to the mask generation unit 23.

In step S14, the presentation area extraction unit 22 converts an image for one frame of the peripheral image passed from the peripheral image acquisition unit 3 in step S11 into a central projection image.
In step S15, the presentation area extraction unit 22 extracts the presentation area from the central projection image which is the result of the conversion in step S14. The presentation area is an area of a portion of the central projected image to be displayed on the display device 9. The presentation area extraction unit 22 passes the extracted image of the presentation area to the presentation unit 24. Further, the presentation area extraction unit 22 passes information regarding the position of the extracted presentation area to the mask generation unit 23.

In step S16, the mask generation unit 23 creates a mask for presenting an image based on the distance image received from the distance information acquisition unit 4 and the position information of the area in which the person is reflected received from the recognition unit 21. Generate. The mask may be, for example, a matrix of virtual image ratio values (m; 0.0 ≦ m ≦ .0) for each pixel. Further, the mask data may be represented in another form. The mask is, in other words, a matrix of transparency values when synthesizing a virtual image and a real space image. The mask generation unit 23 passes the generated mask to the presentation unit 24.

In step S17, the presentation unit 24 has an image (real space image) acquired by the peripheral image acquisition unit 3 and an image acquired by the playback image acquisition unit 5 based on the mask data passed from the mask generation unit 23. Combine with (virtual image). At this time, the presentation unit 24 selects only a predetermined portion of the all-around video frame passed from the playback video acquisition unit 5 based on the position and orientation of the video presentation device 1 itself passed from the position / posture acquisition unit 6. , Cut out for presentation. Then, the presentation unit 24 performs mixing according to the above mixing ratio (m) for each pixel.

In step S18, the presentation unit 24 causes the display device 9 to display the image synthesized in the process of step S17.

As described above, the series of processes shown in this flowchart is the process for one frame of the video. The video presentation device performs this series of processing every frame. In this way, the presentation unit 24 synthesizes the playback video acquired by the playback video acquisition unit 5 and the peripheral video acquired by the peripheral video acquisition unit 5 and presents the video as a moving image.

An example of a variation of the processing shown in this flowchart is as follows.
For example, the recognition process in step S13 may be performed based on the central projection image that is the result of conversion in step S14.
For example, the conversion process to the central projected image in step S14 may not be performed.
For example, in the mask generation process of step S16, a mask based on the distance and a mask based on the recognition result may be created separately and then combined later, or both masks may be combined from the beginning. It may be created as mask data.
For example, in the mask generation process of step S16, there may be no region (region where 0.0 <m <1.0) for displaying the mixed video. In that case, the value of m changes discontinuously at the boundary between the region of m = 0.0 and the region of m = 1.0 according to the distance and the image recognition result.
Further, for example, the order of each process shown in the flowchart may be changed within a range in which no logical contradiction occurs.

The configuration and processing of the video presentation device 1 according to the present embodiment are summarized as follows.

The image presentation device 1 includes at least a playback image acquisition unit 5, a peripheral image acquisition unit 3, a recognition unit 21, a distance information acquisition unit 4, a mask generation unit 23, and a presentation unit 24. The playback video acquisition unit 5 acquires a playback video, which is a playback video. The peripheral image acquisition unit 3 acquires a peripheral image which is an image around the own device. The recognition unit 21 recognizes a predetermined subject (for example, a person) included in the peripheral image. The distance information acquisition unit 4 acquires the distance information corresponding to the peripheral image. The mask generation unit 23 generates mask information. The mask information indicates whether or not the reproduction video should be presented based on the distance information. Further, the mask information indicates whether or not the subject recognized by the recognition unit 21 is in the region where the subject exists. The presentation unit 24 outputs at least the reproduction video based on the mask information generated by the mask generation unit 23. The mask information indicates, for example, whether it is an area in which a playback image should be presented or an area in which a peripheral image (or a peripheral situation) should be presented, corresponding to a position on the screen. Further, the mask information may have information (numerical value) of the mixing ratio of the playback image and the peripheral image. In a typical case, a peripheral image is presented in a relatively short-distance region and an region in which a predetermined subject is recognized. In addition, the playback video will be presented in a relatively long-distance region. In addition, in the region of the intermediate distance between them, both images may be mixed and presented. The mixing ratio may be, for example, a linear value according to the distance.

The image presenting device 1 may include a synchronization unit 7. The synchronization unit 7 synchronizes the playback position of the playback video output by the presentation unit 24 in the time direction with another video presentation device. Information can be exchanged at any time between the self-image presenting device and another image presenting device by communication.

Based on the mask information, the presentation unit 24 outputs so as to display at least one of the reproduction video and the peripheral video for each area in the screen. The presentation unit 24 may output both images in a mixed manner based on the above mixing ratio.

The image presentation device 1 may include a presentation area extraction unit 22. The presentation area extraction unit 22 has a function of cutting out only a part (for example, a central portion) of the peripheral image acquired by the peripheral image acquisition unit 3. In this case as well, the recognition unit 21 may perform the subject recognition process based on the peripheral image before the presentation area extraction unit 22 cuts out. On the other hand, the presentation unit 24 outputs so as to display the peripheral image of the portion cut out by the presentation area extraction unit 22.

The mask generation unit 23 may generate the mask information including the information of the mixing ratio in the region where the reproduction image and the peripheral image are mixed and presented. In the region where the playback video and the peripheral video are mixed and presented, the presentation unit 24 outputs the playback video and the peripheral video so as to be mixed based on the value of the mixing ratio of the mask information. To do.

As an example, the image presentation device 1 is a type of device worn on the head of a mammalian organism (for example, a human being). The image presentation device 1 may further include a position and posture detection unit (position / posture acquisition unit 6) for detecting the position and posture of the image presentation device 1. The presentation unit 24 may output a partial image corresponding to the position and the posture detected by the position / posture acquisition unit 6 in the reproduction image which is the all-around image.

As described above, according to the present embodiment, when presenting virtual reality video content (for example, virtual reality of a live-action 360-degree video), the viewer's own body, an object in the vicinity of the viewer, or the like , Present the body of another person (for example, a person who watches the same content) together in a natural way. This makes it possible to enjoy the immersive feeling in the content and the sharing of the situation with other people.

That is, for the user who uses the image presentation device 1, the user's own body is displayed on the screen as it is, so that a more realistic experience can be experienced in the virtual reality space. Users can also share their virtual reality experience with others in the same space. Further, when displaying in stereo, the content can be presented while maintaining the actual sense of distance. Further, in the case of the head-mounted device configuration, the user can wear the video presentation device 1 to view the content, and the free action is not hindered. For example, the range of action of the user is not restricted due to the convenience of an external sensor for measuring the user's own body. In addition, the user can operate a real object that is relatively close to himself / herself while getting the feeling of actually touching it. This enables "watching while watching", for example, eating while watching a virtual reality image.

[Second Embodiment]
Next, the second embodiment of the present invention will be described. The matters already described in the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described. The image presenting apparatus according to the second embodiment uses an optical see-through method. The optical see-through method will be described later.

FIG. 8 is a block diagram showing a schematic functional configuration of the image presentation device according to the present embodiment. The video presentation device 51 has a configuration partially similar to that of the video presentation device 1 described in the previous embodiment. The video presentation device 51 includes a processing unit 52 in place of the processing unit 2 in the video presentation device 1. Further, the image presenting device 51 includes a display device 59 instead of the display device 9 in the image presenting device 1. A feature of the processing unit 52 of the present embodiment is that it has a presentation unit 74 instead of the presentation unit 24 in the first embodiment. Further, the processing unit 52 has a presentation area extraction unit 22 instead of the presentation area extraction unit 22 in the first embodiment.

The display device 59 is a type of device suitable for presentation by an optical see-through method. That is, the display device 59 displays the playback video acquired by the playback video acquisition unit 5 in a part of the entire screen. Further, the display device 59 enables the user who uses the video presentation device 51 to optically see the surroundings of the user himself / herself in the area where the playback video is not displayed. As one form, the display device 59 displays a part of the reproduction image (the pixel part) based on the RGB signal for each of the pixels of the display screen, or sets the pixels in a transparent state and shows the surrounding situation with the eyes of the user. It has a function of controlling whether or not to make the image visible.

The presentation area extraction unit 72 extracts an area to be presented in the image acquired by the peripheral image acquisition unit 3. The presentation area extraction unit 72 cuts out an image having a wide viewing angle acquired by the peripheral image acquisition unit 3 according to the viewing angle of the display device 59, for example, by center projection. In the present embodiment, the presentation area extraction unit 72 does not pass the cut-out video itself to the presentation unit 74. As described above, since the optical see-through method is used in this embodiment, it is not necessary to display the image itself acquired by the peripheral image acquisition unit 3 on the display 59. However, the presentation area extraction unit 72 masks the positional correspondence between the image acquired by the peripheral image acquisition unit 3 and the image adjusted to the viewing angle of the display device 59 when the image is cut out by the central projection. Notify the generation unit 23.

The presentation unit 74 presents the video acquired by the playback video acquisition unit 5 based on the mask generated by the mask generation unit 23. In the present embodiment, the presentation unit 74 controls the transmittance (m) of the playback image instead of displaying the peripheral image on the display device 59, so that the viewer can see the transparency represented by the mask in the real space. To be able to see.

That is, the presentation unit 74 has a function of changing the transparency of the playback video. The presentation unit 74 outputs a playback image with transparency according to the mask information. Where the situation corresponding to the peripheral image should be presented, the presentation unit 74 transmits the reproduction image. This allows the viewer to visually recognize the real space. Further, the transparency may be a real number of 0 or more and 1 or less. In the intermediate region (the region where the transmittance is larger than 0 and smaller than 1), the presentation unit 74 outputs the playback video with the transmittance corresponding to the numerical value of the mask information. That is, the presentation unit 74 controls the video output so that the viewer can see the playback video and the real space in a mixed state at a predetermined mixing ratio. By having such a function, the presentation unit 74 realizes optical see-through. In other words, the presentation unit 74 masks the body of the person who is experiencing together with his / her own body and the body of the person who is experiencing together in the same space, and the vicinity portion of the whole sky image to be presented, and presents the image for reproduction. ..

The processing procedure of the image presentation device 51 according to the present embodiment is basically the same as the processing procedure in the previous embodiment. However, in the present embodiment, instead of displaying the peripheral image acquired by the peripheral image acquisition unit 3 on the display device, the image presenting device 51 allows the viewer to see the real space in the area in the image. The presentation of the image is controlled so as to.

FIG. 9 is a schematic view showing a configuration example of a video presented by the video presentation device 51 of the present embodiment. As shown in the figure, the presentation unit 74 of the image presentation device 51 is a combination of the real space of (1) and the reproduction image (with a masked portion) of (2), and the presentation image (optical) of (3). See-through method) is presented to the user. In the figure, reference numeral 311 is a real space that is visible to the user through the display device 59. Reference numeral 312 is a reproduction video. The playback video 312 is a video obtained by cutting out a part of the original all-sky video. Further, reference numeral 313 is a mask image generated based on the distance from the image presenting device 51 and the recognition result of the peripheral image. The white portion of the mask image 313 is an area allocated so that the user can see the real space (reference numeral 311). The hatched portion of the mask image 313 is an area allocated for the reproduction image 312. The presentation unit 74 controls to mask the masked portion (the area shown in white in the masked image 313) of the reproduced image 312. That is, in the display device 59, nothing is displayed on the mask portion of the reproduction video 312 (state of reference numeral 314), and the portion is in a transparent state. As a result, the user visually recognizes the real space (reference numeral 311) of the mask portion through the screen of the display device 9 which is in the transparent state. That is, the user visually recognizes the image in which the masked reproduction image 314 and the real space of the masked portion are combined as the presented image (reference numeral 315). Note that the size of the playback video 312 is different from the size of the playback video 312 and the size of the mask video 313, and the size of the playback video 312 is larger. The mask image 313 is assigned to a predetermined partial area of the reproduction image 311. The presentation unit 74 displays the presentation image (reference numeral 315) synthesized by the optical see-through method on the display device 59 in this way. In this optical see-through method, the presentation unit 74 transmits the user's own body (display device 59) of the image presentation device 1 in the image cut out from the all-sky image (reproduction image) as an effect obtained. It exists in the same space (inside the room) as the real space (the real space that can be visually recognized through the display device 59) and the real space (the real space that can be visually recognized through the display device 59) in the vicinity (within a predetermined distance) of the user himself / herself. The body of a person (for example, a person who is viewing and experiencing the same content at the same time) (a real space that can be visually recognized through the display device 59) is presented in a transparent and visible state. For example, a head-mounted display that is worn on the user's head is suitable for displaying such an optical see-through type image. In particular, it is a head-mounted display capable of controlling whether each pixel on the display screen is in a transparent state or a state in which a pixel in a reproduction image is displayed.

The configuration peculiar to this embodiment is summarized as follows.

The presentation unit 74 has a function of making the transparency of the playback video variable. The presentation unit 74 outputs a playback image with transparency according to the mask information. Where the situation corresponding to the peripheral image should be presented, the presentation unit 74 transmits the reproduction image. This allows the viewer to visually recognize the real space. Further, the transparency may be a real number of 0 or more and 1 or less. In the intermediate region (the region where the transmittance is larger than 0 and smaller than 1), the presentation unit 74 outputs the reproduction video with the transmittance corresponding to the numerical value of the mask information. That is, the presentation unit 74 controls the video output so that the viewer can see the playback video and the real space in a mixed state at a predetermined mixing ratio.

As described above, according to the present embodiment, it is possible to realize a content presentation method similar to or similar to the first embodiment by the optical see-through method.

It should be noted that at least a part of the functions of the video presentation device in each of the above-described embodiments can be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, or a USB memory, or a storage device such as a hard disk built in a computer system. Say that. Furthermore, a "computer-readable recording medium" is a device that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In that case, it may include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client. Further, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

It should be noted that a modified example of the above embodiment may be implemented.
As an example of the modification, a video presentation device that does not have the synchronization unit 7 may be implemented. In this case, it is not possible to exchange information between a plurality of video presentation devices and achieve synchronization for presenting the same content at the same time.
As another example of the modification, the presentation area extraction unit 72 may not extract the presentation area. For example, when the viewing angle of the peripheral image acquired by the peripheral image acquisition unit 3 and the viewing angle when displaying the peripheral image on the display device are the same or close to each other, the viewing angles of both are matched. The processing of the presentation area extraction unit 72 of the above can be omitted.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

The present invention can be used, for example, in a device for presenting video content, a service for presenting video content, and the like. However, the scope of use of the present invention is not limited to those exemplified here.

1 Video presentation device 2 Processing unit 3 Peripheral image acquisition unit 4 Distance information acquisition unit 5 Playback video acquisition unit 6 Position / orientation acquisition unit 7 Synchronization unit 9 Display device 21 Recognition unit 22 Presentation area extraction unit 23 Mask generation unit 24 Presentation unit 31 Wireless router 51 Video presentation device 52 Processing unit 59 Display device 72 Presentation area extraction unit 74 Presentation unit

Claims (9)

A playback video acquisition unit that acquires playback video, which is playback video,
A peripheral image acquisition unit that acquires peripheral images that are images of the surroundings of the own device,
A recognition unit that recognizes a predetermined subject included in the peripheral image,
A distance information acquisition unit that acquires distance information corresponding to the peripheral image,
Mask information indicating whether or not the playback image should be presented based on the distance information, and mask indicating whether or not the subject recognized by the recognition unit exists in the area. A mask generator that generates information and
A presentation unit that outputs at least the playback video based on the mask information generated by the mask generation unit.
A video presentation device including. A synchronization unit that synchronizes the playback position of the playback video output by the presentation unit in the time direction with another video presentation device.
The video presentation device according to claim 1, further comprising. Based on the mask information, the presenting unit outputs at least one of the playback video and the peripheral video for each area in the screen.
The video presentation device according to claim 1 or 2. A presentation area extraction unit that cuts out only a part of the peripheral image acquired by the peripheral image acquisition unit,
With more
The recognition unit recognizes the predetermined subject based on the peripheral image before the presentation area extraction unit cuts out.
The presentation unit outputs the peripheral image cut out by the presentation area extraction unit.
The video presentation device according to claim 3. The mask generation unit generates the mask information including the information of the mixing ratio in the region where the playback image and the peripheral image are mixed and presented.
In the region where the playback video and the peripheral video are mixed and presented, the presentation unit outputs the playback video and the peripheral video so as to be mixed based on the information of the mixing ratio.
The video presentation device according to claim 4. The presenting unit has a function of changing the transparency of the playback video, and outputs the playback video with the transparency according to the mask information.
The video presentation device according to claim 1 or 2. The transparency is a real number of 0 or more and 1 or less.
The presenting unit outputs the playback video with transparency according to the mask information.
The video presentation device according to claim 6. The image presentation device is a type of device that is worn on the head of a mammalian organism.
Position and posture detection unit that detects the position and posture of the video presentation device,
With more
The presenting unit outputs a partial image corresponding to the position and posture detected by the position and posture detecting unit in the playback image which is an all-around image.
The video presentation device according to any one of claims 1 to 7. The playback video acquisition process to acquire the playback video, which is the playback video,
Peripheral image acquisition process to acquire peripheral image which is the image around the own device,
A recognition process for recognizing a predetermined subject included in the peripheral image and
The distance acquisition process for acquiring the distance information corresponding to the peripheral image and
Mask information indicating whether or not the playback image should be presented based on the distance information, and mask indicating whether or not the subject recognized in the recognition process exists in the area. The mask generation process to generate information and
Based on the mask information generated in the mask generation process, at least the presentation process of outputting the playback video and the presentation process.
A program that causes a computer to perform the processing of.

Images