JP6717486B1 - Extended virtual space providing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a user with a higher quality extended virtual space. SOLUTION: A measurement unit 10 that acquires measurement information including at least information that can recognize a direction and a position of a user, a photographing unit 20 that acquires a raw video image that includes a direction of a user's X line of sight, and a basic live video image. A video processing unit 30 that generates a processed video from which an arbitrary part is removed from the input video, a material video including a captured video, a CG video, or a combination thereof, and a processed video are combined based on measurement information. It has at least a combining unit 40 that generates a combined image and a display unit 50 that displays the combined image to the user X. The image processing unit 30 performs a depth-based removal process of extracting an extraction space Z by removing a portion having depth information of a predetermined distance or more based on depth information of an input image based on a raw image. Further, chroma-based removal processing for removing a portion having predetermined color space information may be performed on the input video based on the raw video. [Selection diagram] Figure 2

Description

The present invention relates to a system in which a user can visually recognize an extended virtual space that is a combination of a plurality of videos, and more specifically, video processing is performed on a real video captured around a user in a virtual space composed of material videos in real time. The present invention relates to an augmented virtual reality providing system capable of displaying a user's limbs and the like in a virtual space by combining processed images so that the virtual space can be recognized as if a real user were present.

"VR", "AR", "MR" and the like are known as technologies with "R (Reality)".
“VR” is an abbreviation for “Virtual Reality” and is also called “virtual reality”. This makes it possible to provide an experience in which the user himself/herself actually exists in the virtual space by displaying the virtual space on a head mounted display (HMD) worn by the user.

Next, “AR” is an abbreviation for “Augmented Reality” and is also called “augmented reality”. This is to add digital information created by CG or the like to the real world. For example, in a game application of a smart phone, it is configured to display a virtual reality character together with the scenery of the real world using the position information of the smart phone. By doing so, it is possible to provide an experience through the screen of a smartphone as if the character of the game were present on the spot.
As described above, the "VR" is mainly in the virtual space, whereas the "AR" is mainly in the real space.

Further, “MR” is an abbreviation for “Mixed Reality” and is also called “mixed reality”. This allows the user to experience a world in which the virtual world is superimposed on the real world through transparent goggles or smart glasses worn by the user.
This “MR” is a concept that mainly uses the physical space, like the above-mentioned “AR”.

By the way, the applicant has invented the technology described in Patent Document 1 below.
The image-chart type simulation service providing system according to Patent Document 1 provides real image information (corresponding to “real space”) and three-dimensional virtual image information (corresponding to “virtual space”) of a motion capture device. It is configured to be able to display combined image information obtained by combining processing based on measurement information.
In the present technology, a user wearing the HMD models the room by combining CG images of windows, furniture, home appliances, and the like with a CG image of a room having a predetermined size in the real world and displaying the combined image on the HMD. It is structured so that it can be experienced as a room.
It can be said that this is a technique in line with the concept of “AR” or “MR” among the “VR”, “AR”, and “MR” described above.

JP, 2017-146762, A

As a concept different from any of the above-mentioned "VR", "AR" and "MR", the applicant newly conceived and devised a concept "AVR".
The “AVR” is an abbreviation for “Advanced Virtual Reality”, and is a concept called “augmented virtual reality”.
This is to experience a space (hereinafter referred to as "extended virtual space") in which a virtual space such as "VR" is mainly used and a user or the like in the real space is superimposed on the virtual space.
Therefore, “AR” and “MR” are mainly in the real space, whereas “AVR” is mainly in the virtual space, and the two have different basic ideas.
In the extended virtual space by this “AVR”, by setting the user's proximity area or other arbitrarily set area as the real space, the user's body or an object held by the user exists in the real space. Since it is possible to display things in the virtual space, it has been convinced that the user can experience a feeling different from the conventional VR.

In realizing the concept of "AVR" described above, at least one of the problems described below was found.
(1) When extracting a video part such as a user's body or an object held by the user for synthesis, it is desired to more beautifully extract the video part.
(2) When the depth information obtained from the captured image of the camera or the depth information obtained from the 3D scanner is used in extracting the user's body or the like from the captured image of the reality, the user's feet and the floor of the room are clearly separated. Separately, I want to more beautifully extract only the user's feet as a real space.
(3) It is desirable to reduce the load of the process of generating the extracted video from the actual captured video, which is one of the synthesis targets.
(4) I want to reduce the information processing load of the composition work.

Therefore, an object of the present invention is to provide a means capable of providing a user with a higher quality extended virtual space.

The first invention of the present application made to solve the above problems is an extended virtual space providing system for allowing a user to experience an extended virtual space generated by combining a plurality of videos, and recognizes the direction and position of the user. A measurement unit that acquires measurement information including at least possible information, a capturing unit that acquires a raw video including the user's approximate line-of-sight direction, and an arbitrary portion with respect to an input video based on the raw video Generating a processed image from which a processed image is removed, a combined image including a material image including a video image, a captured image, a CG image or a combination thereof, and the processed image is generated based on the measurement information. a combining unit, for displaying the composite image to the user, includes a display unit, at least, the video processing unit, against the input image based upon the raw image, the value of the depth information of the input image A depth-based removal process that is a removal process based on the input image, and a chroma-based removal process that is a removal process based on the value of the color space information of the input image, for the input image that has been subjected to the depth-based removal process. It is what
A second invention of the present application is the same as the first invention, wherein, in the chroma removal processing, the HSV color space information included in each pixel of the input image has a hue within a predetermined range and a predetermined saturation or more. The process is to remove the pixel.
A third invention of the present application is characterized in that, in the first invention or the second invention, a plurality of hues or saturations in the predetermined range used for the chroma removal processing are set. ..

According to the present invention, at least one of the effects described below is exhibited.
(1) By performing the depth system removal processing, the user's body can be beautifully extracted and incorporated into the composite video.
(2) By performing chroma removal processing, the user's feet and the floor of the room can be clearly distinguished, and the user's feet can be extracted neatly.
(3) Since the depth system removal process is a simple process of comparing the depth information with the set predetermined distance, the removal process speeds up.
(4) By performing the depth system removal processing and the chroma system removal processing in parallel, the processing speed of the combining operation becomes faster.
(5) VR sickness can be reduced because the user's limbs can be visually recognized as they are, instead of the virtual limbs.
(6) Since the synthetic video is generated at a high speed, it is possible to suppress the delay between the actual movement of the limbs of the user and the movement of the limbs of the user on the synthetic video.

The functional block diagram of the extended virtual space provision system which concerns on this invention. The apparatus block diagram of the extended virtual space provision system which concerns on this invention. FIG. 6 is an image diagram showing a generation example (1) of an extended virtual space. The image figure which shows the example (2) of generation of extended virtual space. FIG. 5 is an image diagram when the removal processing in the video processing unit is performed in parallel.

Embodiments of the present invention will be described below with reference to the drawings.

<1> Functional configuration (Fig. 1)
FIG. 1 shows a functional block diagram of an extended virtual space providing system according to the present invention.
The extended virtual space providing system according to the present invention is a system installed in a user's living space, and by allowing the user to visually recognize a composite video formed by combining a plurality of videos, the user feels different from the conventional VR. It is a system that makes it possible for anyone to experience.
The present system includes a measuring unit 10 that acquires measurement information of a user, a photographing unit 20 that acquires a raw image of a user's approximate line-of-sight direction, and an image processing unit that performs image processing based on the raw image acquired by the photographing unit 20. 30 and at least a function as a synthesizing unit 40 that generates a synthetic image that is a combination of a plurality of images, and a display unit 50 that displays the synthetic image to the user.
The details of each unit will be described below.

<2> Measuring unit (Fig. 1)
The measuring unit 10 is a device having a function of acquiring measurement information of a user.
The “measurement information” includes at least information that allows the orientation and position of the user to be recognized. Based on this measurement information, a synthetic image is generated or a synthetic image to be visually recognized by the user is controlled. You can
The measurement unit 10 can use an angle sensor, an acceleration sensor, a sensor group such as a gyroscope, a motion capture device, and a GPS device.
Further, the measuring unit 10 includes all types such as a type installed in a living room, a type built in a head mounted display worn by a user, a type separately mounted on a user's body, and a type held by a user.

<3> Imaging unit (Fig. 1)
The image capturing unit 20 is a device having a function of acquiring a live image including the direction of the user's line of sight.
The "live image" is an image obtained by shooting a real space including the user's line-of-sight direction, that is, an image showing the user's living space.
The imaging unit 20 may be a camera built in the head mounted display, a camera externally attached to the head mounted display, or a camera that can be worn by the user.
Further, in the present invention, the photographing unit 20 is sufficient as long as it is a camera capable of photographing a raw image at least in the direction of the line of sight, but a camera capable of photographing the entire circumference of the user as the photographing unit 20 is not excluded.

<4> Video processing unit (Fig. 1)
The image processing unit 30 is a device having a function of generating an image (processed image) on the real space side that is a synthesis target from an input image based on a raw image acquired by the image capturing unit 20.
The “input video” is a video based on the raw video acquired by the shooting unit 20, and specifically, the live video itself acquired by the shooting unit 20 or the live video by the video processing unit 30. It also includes images that have been subjected to image processing.
The video processing unit 30 can be configured with hardware such as a CPU, a memory, and a GPU included in the information processing device under the control of software.

<4.1> Overview of Processes In the present embodiment, the video processing performed by the video processing unit 30 is divided into two types: depth-based removal processing and chroma-based removal processing.
In the present invention, these video processing may be performed in any order and regardless of the number of times to collectively generate one processed video E. Alternatively, the same input video may be separately processed in parallel with the depth removal processing. The two processed images E may be generated by preparing the respective processed images and the chroma removal processing.
The details of each video process will be described below.

<4.2> Depth-based removal processing The depth-based removal processing is processing for performing removal on an input video based on a raw video based on the depth information value of the input video.
As the depth information of the input video, data embedded in the raw video by the image capturing unit 20, data separately generated by image analysis of the raw video, data obtained from a separately provided 3D scanner, or the like can be used.

<4.2.1> Example of Depth System Removal Process As an example of the depth system removal process, a process of removing a portion having depth information of a predetermined distance or more based on the depth information of the input image from the input image. Is possible.
In addition, according to the present invention, not only the information on the relative distance from the user but also the direction (angle information) of the user and their absolute values may be used to specify the depth information to be removed.
For example, the removal process can be performed so that the pillar space that exhibits a substantially fan shape in a plan view in the range of a predetermined angle from the user is the real space and the remaining space is the virtual space.
Further, it is also possible to perform the removal processing so that the pillar space having a rectangular shape in a plan view with the user substantially in the center is the real space and the remaining space is the virtual space.
These may be appropriately determined depending on the situation of the extended virtual space that the user experiences.

<4.3> Chroma-based Removal Processing The chroma-based removal processing is processing for removing the input video based on the raw video based on the value of the color space information of the input video. More specifically, This is a process for removing a portion corresponding to the uniform color of the background material arranged in the user's living room from the input image and allowing another image to be combined with the removed portion.
The chroma removal processing is based on well-known color space information such as RGB, RGBA, CMY, HSV, and HLS, and sets a predetermined range among the elements and numerical ranges defined in each color space to achieve a specific homogeneity. The part corresponding to the color is removed.
In this embodiment, the chroma system removal processing is performed using the HSV color space information. The HSV color space information is information including hue (hue: 0° to 360°), saturation (saturation: 0% to 100%), and lightness (value: 0% to 100%).
In the course of trial and error in developing the system according to the present invention, the present inventor responds to a uniform color of the background material when a predetermined range of hue and saturation is set regardless of the lightness. It was found that the part was removed quickly and cleanly.
Therefore, in the present invention, the chroma removal processing may be processing to remove the pixel when the HSV color space information for each pixel of the input image has a hue in a predetermined range and a predetermined saturation or more. preferable.

<4.4> Preferable Example of Video Processing As described above, the order and the number of times of each removal processing are not limited, and a plurality of processed videos may be generated.
Among them, if the performance and cost of each device that composes the system at the current stage is balanced, the procedure to perform chroma removal processing on the video that has been subjected to depth removal processing on the raw video is adopted. It was found that the procedure to be performed has an excellent balance between high-speed generation and clear extraction of the processed image corresponding to the real space side of the synthesis target.
This is because image processing is performed by removing a portion of a space apparently separated by a predetermined distance or more from the user by depth processing in advance and not performing chroma removal processing on the removed portion. It is considered that this is due to the fact that the speed was improved.
This is effective in the case where only the vicinity of the user is set as the real space and the space away from the user by a predetermined distance or more is visually recognized in the virtual space.

<5> Synthesis part (Fig. 1)
The synthesizing unit 40 is a device having a function of generating a synthesized image G by combining a plurality of images based on the measurement information obtained by the measuring unit 10.
The image information to be combined includes at least one or more processed images generated by the image processing unit 30 and a material image that is stored in advance or generated separately from the raw image.
As the material image, a captured image obtained by capturing an actual place, a CG image, or a combination image thereof can be used.
The processed image corresponds to the image on the real space side in the extended virtual space, and the material image combined with the portion removed by the image processing unit 30 corresponds to the virtual space in the extended virtual space.
The synthesizing unit 40 can be configured with hardware such as a CPU, a memory, and a GPU included in the information processing device under software control.

<6> Display (Fig. 1)
The display unit 50 is a device having a function of displaying the combined video generated by the combining unit 40 to the user.
For the display unit 50, a head mount display, goggles, or the like that can be used in VR can be used, and in addition, a mobile terminal such as a smartphone that can be housed in a housing and made into goggles can be used.

<7> Device configuration example (Fig. 2)
FIG. 2 shows a device configuration diagram of the extended virtual space providing system according to the present invention.
In FIG. 2, the extended virtual space providing system according to the present invention is configured with a head mounted display A worn by a user X, an information processing device B installed in a living room space Y, and a background material C appropriately placed in the living room space Y. doing.
Then, an arbitrary region in the direction of the line of sight of the user X is set as the extraction space Z included in the combined video forming the extended virtual space.
The extraction space Z is partitioned by the set value of the predetermined distance used in the depth removal processing in the video processing unit described above.

<7.1> Head mounted display (Fig. 2)
The head-mounted display A has a function of a measurement unit 10 that acquires measurement information of the user X, a photographing unit 20 that acquires a raw image of the user X in a substantially line-of-sight direction, and a display unit 50 that displays a composite image to the user X. Have at least.
In this embodiment, the measuring unit 10 uses a sensor group such as an angle sensor, an acceleration sensor, and a gyroscope built in the head mounted display A, and the photographing unit 20 is externally attached to the front surface of the head mounted display A. A dual camera is used, and the display portion 50 uses the display portion in the head mounted display A.
For a raw video image captured by the dual cameras constituting the image capturing unit 20, depth information can be generated by using the parallax of the video image captured by each camera.

<7.2> Information processing device (FIG. 2)
The information processing apparatus B has at least the functions of a video processing unit 30 that performs video processing on the basis of the raw video acquired by the imaging unit 20, and a synthesis unit 40 that generates a synthetic video that combines a plurality of videos. There is.
In addition, the information processing apparatus is configured to be able to store or externally acquire in real time a material video composed of a captured video, a CG video, or a combination thereof, which is one of the composition targets.

<7.3> Background material (Fig. 2)
The background material C is an object having a function of specifying a region in which pixels are removed by the chroma removal processing.
The background material C in the present invention is not limited to the sheet-like material shown in FIG. 2 and includes tapes and paints.
For example, an object (for example, a gun or a sword for a game, a hand cutter or a handle of a heavy machine in a construction site simulation) held by the user X is wrapped with a tape having a uniform color, or a paint having a uniform color is applied. If this is the case, it is also possible to perform processing such as synthesizing the material images that are virtual images of these objects while removing the portions by the chroma removal processing.
Further, a chair on which the user X sits, a jig for suspending the user X, and the like become the background material C when the coloring process corresponding to the uniform color used for the chroma removal processing is performed.

<8> Generation example of extended virtual space (1) (FIG. 3)
FIG. 3 shows an example (1) of generating an extended virtual space using this system.
In FIG. 2 described above, the background material C is arranged on the wall of the living room Y, and the user X faces the wall of the living room Y and holds his or her hands in front of the head mounted display A. Presents.
At this time, the raw image D captured by the image capturing unit 20 shows a background covered with a uniform color as shown in FIG. 3A, and both hands of the user X are in front of them.
When depth processing is first performed on the raw image D by the image processing unit 30, depth information of a predetermined distance or more from the user X (more accurately, the camera) is obtained as shown in FIG. 3B. The processed image E1 from which the held portion is removed is generated.
At this time, if the accuracy of the depth information based on the raw image D is low, the gap between the fingers may not be properly removed, and a uniform color of the background material C at the back may remain.
Therefore, when the chroma removal processing is further performed on the processed video E1 subjected to the depth removal processing, the processed video E2 from which the uniform color portion between the fingers is removed is generated as shown in FIG. 3C. ..
The combining unit 40 generates a combined image G by combining the processed image E2 illustrated in FIG. 3C and the material image F illustrated in FIG. 3D based on the measurement information acquired by the measuring unit 10. Then, it is transmitted to the display unit 50 of the head mounted display A.

<8.1> Reasons for not using only chroma removal processing (from a practical point of view)
In the image diagram shown in FIG. 3, even if the chroma removal processing is performed immediately from the raw image D shown in FIG. 3A without performing the depth removal processing according to FIG. It is possible to obtain the same composite image G as in 3(d).
However, in a system that is premised on performing only the chroma removal processing, it is necessary to install the background material C over the entire circumference of the living room space Y, which is not convenient. For example, the work of arranging the background material C on the ceiling portion of the living room space Y is laborious, and is one of the work to be avoided.
Therefore, if the background material C is not arranged on the ceiling portion of the living room space and the processed image E subjected to the depth system removal processing is provided on at least the ceiling portion, the composite image having a relatively high composite quality is obtained. The image G can be generated.

<9> Generation example of extended virtual space (2) (FIG. 4)
FIG. 4 shows an example (2) of generating an extended virtual space using this system.
From FIG. 2 described above, it is assumed that the user X faces the floor side of the living space Y.
The background material C is partially installed on the floor of the living room space Y where the user X stands.
At this time, as shown in FIG. 4A, the raw image D captured by the image capturing unit 20 shows the portion of the background material C covered with the uniform color and the floor Y1.
When the depth processing removal process is first performed on the raw image D by the image processing unit 30, as shown in FIG. 4B, a depth of a predetermined distance or more from the user X (more accurately, the imaging unit 20). A processed image E1 from which the information-bearing portion has been removed is generated.
At this time, since there is no great difference between the distance from the user X to the foot portion and the distance to the floor, even if the depth system removal processing is performed, only the foot portion cannot be extracted well, and the surroundings of the foot portion cannot be extracted. The uniform color of may remain.
Therefore, when the chroma removal processing is further performed on the processed video E1 that has been subjected to the depth removal processing, a processed video E2 from which the uniform color portion remaining around the feet is removed is generated as shown in FIG. 4C. To be done.
FIG. 4D is a composite video G in which the material video F is combined with the processed video E2 of FIG. 4C. This material image F is an image when the user X standing on the edge of a high-rise building is looking down on the ground. In such a situation, if the foot portion of the user X is beautifully extracted and combined, there is a great advantage from the viewpoint of immersive feeling.

<10> Features of Extended Virtual Space Features of the extended virtual space according to the present invention will be described.
In the expanded virtual space composed of the composite video G shown in FIGS. 3 and 4, while the virtual space of the material video F is expanded around, the limbs of the user X himself/herself which are photographed in real time in the area near the user X and Since the clothes are reflected, the user X can experience the impression that he/she moved to the virtual space with his/her own body.

[Improvement of immersion]
As a result, the extended virtual space according to the present invention is more immersive for the user X than in the conventional case where the posture and movement of the user X are detected by a motion capture device and the virtual limbs are displayed in the virtual space. Contributes to the improvement of

[Reduction of VR sickness]
In addition, the effect of reducing VR sickness of the user X has been confirmed in the expanded virtual space according to the present invention as compared to the case where virtual limbs are displayed in the virtual space like the conventional VR.
Although the clear reason for this effect has not been clarified yet, the following reasons are considered, for example.
Reason 1: Because the limbs of the user X can be visually recognized as they are instead of the virtual limbs, the space can be recognized with a feeling closer to the real space.
Reason 2: The depth system removal process is a simple process of comparing the values between the depth information and the set predetermined distance and has a high processing speed. Therefore, the actual movement of the limbs of the user X and the composite image G Since there is little delay between the movement of the limbs of the user X and excellent followability, the user X does not feel uncomfortable.

[Parallel execution of each removal process] (Fig. 5)
In the present invention, the depth-based removal processing and the chroma-based removal processing in the video processing unit 30 may be performed in parallel, and the respective results may be subject to synthesis as processed video.
FIG. 5 shows a generation image of the composite video G according to this embodiment.
As shown in FIG. 5, the depth image removing process and the chroma image removing process are separately performed on the raw image D acquired from the imaging unit 20, and the generated image images E1 and E2 and the material image F are generated. By synthesizing, a synthetic image G can be generated.
At this time, the uniform color portion of the background material C remaining in the processed image E2 on which the depth system removal processing has been performed may be controlled so as not to be reflected in the combined image G.
For example, with respect to a part removed in one processed image E, even if there is a part left in the other processed image E, a method of controlling so that the part is not reflected in the combined image G, or a plurality of processed images E1 , E2 may be set with respective priorities.

[Multiple chroma processing removal processes] (not shown)
In the present invention, the video processing unit 30 may be configured to prepare a plurality of setting values with different color space information and execute the chroma removal processing a plurality of times.
For example, by dividing the background material C arranged in the living room space Y into, for example, a blue background material C and a green background material C, the background material C is combined with the location where the bluish chroma removal processing is performed. It is possible to change the image and the image to be combined with the location where the green chroma removal processing has been performed.

[Depth removal only] (not shown)
In the present invention, the video processing unit 30 may be configured to perform only the depth system removal processing.
For example, when the expanded virtual space to be experienced by the user X is expanded in the circumferential direction and the upward direction when viewed from the user X, a relatively high quality composite image G can be obtained by the user X even by only the depth system removal processing. Can be visually confirmed.

10 Measuring Unit 20 Photographing Unit 30 Image Processing Unit 40 Synthesizing Unit 50 Display Unit X User Y Living Room Space Y1 Floor Z Extraction Space A Head Mounted Display B Information Processing Device C Background Material D Raw Video E Processed Video F Material Video G Composite Video

Claims (3)

An extended virtual space providing system for allowing a user to experience an extended virtual space created by combining a plurality of images,
A measurement unit that acquires measurement information including at least information that can recognize the orientation and position of the user,
An image capturing unit that acquires a live image including the user's approximate line-of-sight direction,
An image processing unit that generates a processed image from which an arbitrary portion has been removed, with respect to the input image based on the raw image,
A composition unit that generates a composite image in which a material image including a captured image, a CG image, or a combination thereof and the processed image are combined based on the measurement information.
A display unit for displaying the composite video image to the user,
Has at least
The video processing unit,
Against the input image based upon the raw image, and depth-based removal process is a removal process based on the value of the depth information of the input image,
The input image on which the depth system removal processing has been performed is subjected to a chroma system removal process that is a removal process based on the value of the color space information of the input image to obtain the processed image .
Extended virtual space providing system. The chroma removal processing is
When the HSV color space information included in each pixel of the input image has a hue in a predetermined range and a predetermined saturation or higher, the pixel is removed.
The extended virtual space providing system according to claim 1 . Hue or saturation of the predetermined range used for the chroma removal processing, characterized in that a plurality is set,
The extended virtual space providing system according to claim 1 .

Images