JP2016019170A - Stereoscopic video display device and video generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereoscopic video display device capable of simultaneously presenting the same stereoscopic video image to a plurality of observers without performing complicated adjustment, and a video generation program.SOLUTION: In a stereoscopic video display device 1, a left-eye video image and a right-eye video image for each observer are generated in accordance with a positional posture measured by a positional posture measuring device 10 by a video image generation device 20, the left-eye video image and the right-eye video image which are read out at a predetermined frame rate are alternately switched and rearranged in accordance with time series to generate a time-series video image. Thereafter, a buffer control signal is generated in accordance with switch timing of a left-eye frame and a right-eye frame. Further, a left-eye field video image and a right-eye field video image are captured from the time-series video image displayed on a video display device 30, the left-eye field video image and the right-eye field video image are held in accordance with the buffer control signal, and the held left-eye and right-eye field video images are then displayed to the observer by a head-mounted display device 40.SELECTED DRAWING: Figure 2

Description

  The present invention relates to the field of stereoscopic video display technology, and relates to a stereoscopic video display device and a video generation program for displaying a stereoscopic video on a single video display device in a real space shared by a plurality of observers.

  Conventionally, as a stereoscopic image display technology, a pair of left-eye and right-eye images with binocular parallax are displayed alternately in a time-division manner, and the left-eye image and the right-eye image are presented using the liquid crystal shutter glasses, respectively, There is a liquid crystal shutter system that obtains a stereoscopic effect by doing so.

  In 2010, the liquid crystal shutter type 3D television was marketed by a plurality of electric manufacturers, and the spread of the 3D television to general households was expected, but it did not spread so widely. The reason for this is, first, the trouble of wearing liquid crystal shutter glasses, second, the lack of motion parallax that the appearance of the three-dimensional image does not change when the observer moves his head left and right, and third, observation For example, visual fatigue due to a mismatch between the position of the point where the eyes of both eyes of the person intersect (convergence position) and the position where the eye is in focus (focus adjustment position) is mentioned.

  In order to solve these three problems, image reproduction type stereoscopic video technologies such as an integral imaging method and an electronic holography method are currently being studied. These techniques are techniques for reproducing light rays emitted from a three-dimensional image using a minute lens or a diffraction grating, and ideally can reproduce light exactly the same as light emitted by a real object. For this reason, the light entering the eyes of the observer and the light emitted by the entity are exactly the same, and therefore the observer cannot visually distinguish between the real object and the stereoscopic image.

  However, in order to reproduce light exactly the same as the light emitted by a real object, extremely fine lenses and optical elements are required. However, the current miniaturization technology and display panel technology are completely different from the light emitted by a real object. The image does not reach the same light, and the image entering the single eye does not reach the VGA image quality (640 × 480). Therefore, the design guideline such as how much sample the light emitted by the real object is sampled to obtain a natural stereoscopic effect cannot be verified. Note that the “sampling” described above means, for example, when reproducing a stereoscopic image, when reproducing a stereoscopic image, such as how many element images are allocated to each element lens and reproduced using which direction of light. It means a condition. In addition, in the image reproduction type 3D image technology, all the light in a certain range emitted by the real object is reproduced, so that the light to the space where the observer does not exist is reproduced, which is useless for the observer. Information.

  On the other hand, in the virtual reality technology, right and left images of a stereoscopic image are generated according to, for example, the position and orientation of the observer's head and are placed in front of the left and right eyes attached to a head-mounted display (HMD). By displaying on the display panel, the light emitted from the real object can be efficiently directed to the eyes of the observer. In such virtual reality technology, unlike the image reproduction type stereoscopic video technology described above, it is possible to freely set how much the light from the stereoscopic image entering the left and right eyes of the observer changes when the head position and orientation change. Therefore, it is possible to evaluate the sampling level that the observer feels natural. Note that “changing the sampling level of a stereoscopic image” means changing the number of directions of light that reproduces the stereoscopic image.

  Furthermore, in Mixed Reality (MR) technology, which is an advancement of this technology, the real world in front of the viewer is synthesized by combining the stereoscopic image and the real world in front of the viewer. It is possible to generate a sensation that a three-dimensional image exists. By using this mixed reality to simulate an image reproduction type 3D image, it is possible to evaluate the sampling parameters of the image reproduction type 3D image technology in the actual viewing environment, which cannot be verified with the current miniaturization technology and display panel technology. It becomes. However, since the distance between the eyes of the observer is different, it is difficult to naturally synthesize the real world and the stereoscopic image.

  Therefore, for example, in Patent Document 1, a real-world image captured by two cameras attached to a head-mounted display and a stereoscopic image generated by computer graphics (CG) are combined. A method has been proposed.

JP 2006-285609 A

  However, the invention proposed in Patent Document 1 has a problem that the adjustment when synthesizing the real-world video and the stereoscopic video is complicated, and the processing is troublesome.

  The present invention has been made in view of such points, and provides a stereoscopic video display device and a video generation program capable of simultaneously presenting the same stereoscopic video to a plurality of observers without complicated adjustments. The task is to do.

  In order to solve the above problems, a stereoscopic video display apparatus according to the present invention includes a video generation apparatus that generates a time-series video from an input three-dimensional original video, and the time-series video generated by the video generation apparatus. A video display device to display, a plurality of head-mounted display devices respectively mounted on the heads of a plurality of observers observing the time-series video displayed by the video display device, and the video display device A position and orientation measurement device for measuring individual position and orientation of a head-mounted display device, wherein the image generation device includes a binocular image generation means, an image switching means, a visual field Video control means, and the head-mounted display device includes a left-eye visual image photographing means, a right-eye visual image photographing means, a left-eye visual image buffer means, a right-eye visual image buffer means, And eye view image display unit, and the right-eye view image display means, the arrangement comprising a.

  According to such a configuration, the stereoscopic image display device is configured such that when a plurality of observers observe the three-dimensional original image according to the position and orientation measured by the position and orientation measurement device by the binocular image generation unit. A left eye image entering the left eye of each observer and a right eye image entering the right eye of each observer are generated. Further, the stereoscopic video display device reads out the left-eye video and the right-eye video generated by the binocular video generation unit at a predetermined frame rate by the video switching unit, and the read left-eye frame of each observer. Then, the time-series video is generated by alternately switching the right-eye frame for each observer and rearranging the right-eye frame along the time series. Then, the stereoscopic video display device acquires the switching timing of the left-eye frame and the right-eye frame for each observer in the time-series video generated by the video switching unit by the visual field video control unit, and the switching timing To generate a buffer control signal. Further, the stereoscopic image display device captures a left-eye visual field image indicating the visual field of the left eye of the observer by the left-eye visual field image capturing unit, and a right-eye visual field image indicating the visual field of the observer's right eye by the right-eye visual field image capturing unit. Shoot. In the stereoscopic video display device, the left eye visual image buffer means holds the left eye visual image captured by the left eye visual image photographing means in accordance with the buffer control signal input from the visual field video control means. In the stereoscopic video display device, the right eye visual image buffer means holds the right eye visual image captured by the right eye visual image photographing means in accordance with the buffer control signal inputted from the visual field video control means. The stereoscopic image display device displays the left eye field image held by the left eye field image buffer unit by the left eye field image display unit, and is held by the right eye field image buffer unit by the right eye field image display unit. The right eye visual field image is displayed.

  In order to solve the above-mentioned problem, the video generation program according to the present invention is the stereoscopic video display device, wherein the computer of the video generation device includes the binocular video generation unit, the video switching unit, and the visual field video control. It was decided to function as a means.

  According to the present invention, video corresponding to the position of the observer is arranged in time series and displayed on the video display device as time series video, and the time series video displayed on the video display device by the head-mounted display device. By capturing and holding the captured left-eye and right-eye view images as appropriate, it is possible to adjust the observation position of multiple observers without adjusting the real-world image and the stereoscopic image. Accordingly, stereoscopic images can be presented individually.

It is a figure which shows typically the structure of the three-dimensional video display apparatus concerning this invention. 1 is a block diagram illustrating a configuration of a stereoscopic video display device according to a first embodiment of the present invention. It is a figure which shows typically the structure of the time-sequential image | video in the three-dimensional video display apparatus concerning this invention. It is explanatory drawing for demonstrating the visual field image in the three-dimensional video display apparatus which concerns on this invention, (a) is a figure which shows the positional relationship of a 1st observer, an nth observer, and an image display apparatus, (b) is. FIG. 4 is a diagram showing m frames of a left eye field image and a right eye field image of the first observer; FIG. 5C is a diagram showing m frames of a left eye field image and a right eye field image of the nth observer; It is. 4 is a control timing chart of a visual field image in the stereoscopic image display device according to the present invention. It is explanatory drawing for demonstrating the visual field image in the three-dimensional image display apparatus which concerns on this invention, (a) is m frame of the 1st observer's left-eye image hold | maintained by the buffer hold control signal, and m frame of the right-eye image (B) is a diagram showing m frames of the left-eye image and m-frame of the right-eye image held by the buffer hold control signal, and (c) is held by the buffer hold control signal. FIG. 6D shows m + 1 frame of the left eye image and m + 1 frame of the right eye image of the first observer, and FIG. 8D shows m + 1 frame of the left eye image and m + 1 frame of the right eye image held by the buffer hold control signal. FIG. It is a flowchart which shows the processing content of the stereoscopic video display apparatus which concerns on 1st Embodiment of this invention, (a) is a flowchart which shows the processing content of a position and orientation measurement apparatus, (b) is the processing content of a video generation apparatus. (C) is a flowchart showing the processing content of the video display device, and (d) is a flowchart showing the processing content of the head-mounted display device. It is explanatory drawing for demonstrating the angle of view of the visual field imaging | photography means in the three-dimensional display apparatus which concerns on this invention, (a) is a case where a video display is seen from the side, Comprising: The vertical image of a visual field imaging | photography means The figure which shows an angle | corner, (b) is a figure which is a case where a video display apparatus is seen from the top and shows the horizontal field angle of a visual field video imaging | photography means. It is a block diagram which shows the structure of the three-dimensional video display apparatus concerning 2nd Embodiment of this invention.

  Hereinafter, a stereoscopic image display apparatus according to the present invention will be described with reference to the drawings. In the following description, the same names and symbols indicate the same configuration as a general rule, and detailed description will be omitted as appropriate.

<First Embodiment>
The configuration of the stereoscopic video display device 1 according to the first embodiment of the present invention will be described with reference to FIGS. As shown in FIG. 1, for example, as shown in FIG. 1, n stereoscopic viewers (n = 2 in FIG. 1) respectively wear head mounted display devices 40 on their heads to display images. When the display image of the device 30 is observed, a stereoscopic image corresponding to each position is displayed for each observer. In addition, as shown in FIG. 1, the stereoscopic image display apparatus 1 includes a position / orientation measurement apparatus 10 and a head-mounted display apparatus 40 that are connected wirelessly. The position and orientation of the part-mounted display device 40 can be measured. Further, as shown in FIG. 1, the stereoscopic image display device 1 includes a position / orientation measurement device 10, a video generation device 20, a video generation device 20, and a video display device 30, which are connected to each other by wire, so that data can be input and output. It is configured to be possible.

  Here, although the invention proposed in Patent Document 1 described above was able to present stereoscopic images individually according to the observation positions of a plurality of observers, images installed in the real world as in the present invention When displaying on the display device 30, since a plurality of observers see the same image, it is necessary to change the stereoscopic image according to the observation position.

  For example, in the example illustrated in FIG. 1, the first observer is sitting on the left side with respect to the video display device 30, and the nth observer is standing on the right side with respect to the video display device 30. Therefore, the stereoscopic image display device 1 displays a stereoscopic image corresponding to a state in which the stereoscopic image is observed from the sitting position on the left side for the first observer, and stands on the right side for the nth observer. A stereoscopic image corresponding to a state in which the stereoscopic image is observed from the position is displayed. That is, the stereoscopic video display device 1 displays a stereoscopic video corresponding to each position for a plurality of observers who observe the display video of the same video display device 30. As shown in FIG. 2, the stereoscopic video display device 1 includes a position and orientation measurement device 10, a video generation device 20, a video display device 30, and a head-mounted display device 40.

  The position / orientation measurement apparatus 10 measures the position / orientation of the head-mounted display device 40. Specifically, the position / orientation measurement apparatus 10 is a head-mounted display device 40 for the video display device 30 (first-header-mounted display device 40A to n-th head-mounted display device 40B). Each of the individual positions and orientations is measured. Here, the above-described “position and orientation” specifically means six degrees of freedom (three translational degrees of freedom and three degrees of freedom of rotation) of the head-mounted display device 40. As shown in FIG. 2, the position / orientation measurement apparatus 10 outputs the position / orientation measurement result to the binocular image generation means 21 of the image generation apparatus 20.

  The specific configuration of the position / orientation measurement apparatus 10 is not particularly limited as long as it can acquire the six degrees of freedom of the head-mounted display apparatus 40, and is well-known such as a magnetic sensor, an optical sensor, and an ultrasonic sensor. Technology can be used. Further, the position where the position / orientation measuring apparatus 10 is disposed is not particularly limited, and the position / orientation measuring apparatus 10 may be located in the image generation apparatus 20, the head-mounted display apparatus 40, or the outside thereof as shown in FIG. It may be arranged.

  The video generation device 20 generates a time-series video from the input three-dimensional original video. Here, the above-mentioned “three-dimensional original image” means an image having three-dimensional information such as computer graphics or an image taken by integral photography. In addition, the above-described “time series video” means a video in which videos that enter the left eye and the right eye of a plurality of observers are alternately arranged along the time series. The specific structure of the time-series video will be described later (see FIG. 3). As shown in FIG. 2, the video generation device 20 includes a binocular video generation unit 21, a video switching unit 22, and a visual field video control unit 23.

  The binocular image generation means 21 (first observer binocular image generation means 21A to n-th observer binocular image generation means 21B) is a three-dimensional original image and the position and orientation of the head-mounted display device 40. From the above, an image entering the left eye and the right eye of each observer is generated. When the number of observers of the video display device 30 is n, the binocular video generation means 21 is a first-viewer binocular video generation means 21A to an n-th binocular video generation means as shown in FIG. 21B. The first observer binocular image generation means 21A to the nth observer binocular image generation means 21B have the same functions. Therefore, the binocular image generation means 21 including these will be described below in place of the individual descriptions of the first observer binocular image generation means 21A to the nth observer binocular image generation means 21B. I will do it.

  Specifically, as shown in FIG. 2, the binocular image generation unit 21 is configured to detect the position and orientation of the head-mounted display device 40 measured by the position and orientation measurement device 10 from a three-dimensional original image input from the outside. Accordingly, a left eye image and a right eye image of a plurality of observers are generated. Here, the above-mentioned “left-eye image” means an image that enters the left eye of each observer when the observer looks at the image display device 30 and observes a three-dimensional original image. The “right-eye video” means a video that enters the right eye in the same case.

  As will be described later, the video actually displayed by the video display device 30 is not a three-dimensional original video but a time-series video. Therefore, more specifically, the binocular video generation means 21 obtains the left eye video and right eye video of the observer when the three-dimensional original video is displayed at the position of the video display device 30 by calculation. The right-eye video and the left-eye video here are parallax images in consideration of the positions of the right eye and left eye of each observer with respect to the three-dimensional original video. As shown in FIG. 4C, portions other than the video are not included.

  In the binocular image generating means 21, the first observer binocular image generating means 21A generates the left eye image and the right eye image of the first observer and outputs them to the image switching means 22 for both the nth observer images. The eye image generation unit 21 </ b> B generates a left eye image and a right eye image of the n th observer and outputs them to the image switching unit 22. That is, the first observer binocular image generation means 21A to the nth observer binocular image generation means 21B generate the corresponding left eye image and right eye image of the first observer to the nth observer, respectively. Output to the video switching means 22.

  Here, the binocular video generation unit 21 can generate the left-eye video and the right-eye video using a process of rendering based on a virtual camera in computer graphics, for example. In this case, the binocular image generation means 21 first calculates the positions of the left eye and the right eye in the position and orientation measured by the position and orientation measurement apparatus 10 in the virtual space, and places the virtual cameras at the positions of the left eye and the right eye, respectively. To do. Next, the binocular video generation means 21 can generate a left-eye video and a right-eye video by rendering a three-dimensional original video input from the outside by a virtual camera. The above-mentioned “left eye and right eye positions” specifically refer to the optical center of the left eye visual field image capturing means 41a and the optical center of the right eye visual field image capturing means 41b in the head-mounted display device 40 described later. Means. The binocular image generation means 21 can accurately and simply generate the left-eye image and the right-eye image for each observer through the above processing.

  In addition to the rendering based on the virtual camera described above, the binocular video generation unit 21 performs rendering based on the optical characteristics of an image reproduction type stereoscopic video display device such as integral photography (IP), for example. The left-eye image and the right-eye image described above can be generated. In this case, the binocular video generation means 21 first calculates the positions of the left eye and the right eye in the position and orientation measured by the position and orientation measurement device 10 in the virtual space. Next, the binocular video generation means 21 calculates an element image that can be observed on each element lens at each position of the left eye and the right eye in accordance with an IP setting parameter indicating a setting value of the element image and the element lens group in integral photography. Thus, a left eye image and a right eye image are generated.

  That is, the binocular video generation means 21 confirms that the element image reproduced by the light having the direction corresponding to the light generated from a certain point of the three-dimensional original image in the virtual space is arranged behind the virtual element lens. Suppose. Next, the binocular video generation means 21 sets the IP setting parameters such as the number and arrangement of the element images and the curvature of the virtual element lens to determine which element image light enters the observer's eyes. Calculation is performed based on the positions of the left eye and the right eye. Since which element image light enters at which eye position depends on the above-described IP parameter, the element image entering the left eye and the right eye can be determined for each virtual lens by the above method. it can. Then, by performing calculations for all virtual element lenses, it is possible to generate a two-dimensional image corresponding to an arbitrary eye position, that is, a left-eye image and a right-eye image. The binocular image generation means 21 can reproduce the appearance when, for example, some parameters of the image reproduction type stereoscopic image apparatus are changed by performing the above-described processing. The left-eye image and the right-eye image can be accurately and easily generated.

  In addition, since the above method can generate a left-eye image and a right-eye image from an input IP image without using an actual element lens or optical element, it can be used as in a conventional image reproduction type stereoscopic image technology. Ultra-fine lenses and optical elements are not required, and light that is exactly the same as the light emitted by real objects can be reproduced beyond the limits of such hardware. Therefore, by using the above method, it is possible to demonstrate design guidelines such as how much sample the light emitted by a real object, which was impossible in the past, should be sampled so that the observer can obtain a natural stereoscopic effect. Become. Furthermore, unlike the conventional image reproduction type stereoscopic video technology, the above method does not reproduce light in a space where there is no observer, and thus does not generate useless information for the observer.

  When the left eye video and the right eye video are generated by rendering based on the optical characteristics of the image reproduction type stereoscopic video device, as shown in FIG. IP configuration parameters are also entered. This “IP setting parameter” refers to various setting values in integral photography such as the number, diameter, thickness, pitch, aperture ratio, element image size, panel resolution, etc. of element lenses in integral photography. It means that.

  The video switching means 22 generates a time-series video by switching the left-eye video and the right-eye video of a plurality of observers for each readout frame. Specifically, as shown in FIG. 2, the video switching unit 22 temporarily stores the left-eye video and the right-eye video of each observer generated by the binocular video generation unit 21 in a memory (not shown). Next, the video switching means 22 reads the left eye video and right eye video of each observer stored in the memory at a predetermined frame rate. As a result, the left eye image of each observer is cut out into a plurality of left eye frames, and the right eye image of each observer is cut out into a plurality of right eye frames. Then, the video switching means 22 generates a time-series video by alternately switching the read left-eye frame and right-eye frame of each viewer for each viewer and rearranging them in time series. Further, the video switching unit 22 outputs the generated time-series video to the video display device 30 as shown in FIG.

  Here, the frame rate for reading the left-eye image and the right-eye image is, for example, that the number of frames of the three-dimensional original image input from the outside is “k frames / second” and the number of observers is n. In some cases, “2 × k × n frames / second” is preferable. When the upper limit value of the frame rate is determined by the video display device 30 or the head-mounted display device 40, the number n of observers is reduced to the above-mentioned range of “2 × k × n frames / second”. Or the frame rate may be set to the upper limit value of the video display device 30 or the head-mounted display device 40.

  Further, the time series video described above is arranged in time series for each read frame (m, m + 1,...) Read by the video switching means 22, as shown in FIG. The observer has a configuration in which left-eye frames and right-eye frames are alternately arranged. For example, when the number of observers is n, the video switching unit 22 reads “m-frame first left-eye frame” and “m-frame first” from the memory (not shown) as shown in FIG. "1 observer's right eye frame", "m frame n left observer's left eye frame", "m frame n right observer's right eye frame", "m + 1 frame first observer's left eye frame" , “M + 1 frame for the right eye of the first observer”, “m + 1 frame for the left eye of the nth observer”, and “m + 1 frame for the right eye of the nth observer” in this order. In this way, a time-series video is generated.

  The visual field image control means 23 controls the visual field image captured by the head-mounted display device 40. Specifically, the visual field video control unit 23 acquires the switching timing of the left eye frame and the right eye frame for each observer in the time-series video generated by the video switching unit 22. Then, the visual field image control means 23 generates a buffer control signal according to the switching timing described above and, as shown in FIG. 2, the buffer control signal is used as the left eye visual field video buffer means 42a and the right eye visual field visual field of the head-mounted display device 40. Output to the video buffer means 42b.

  Here, the “switching timing” is the timing at which the readout frame (m frame, m + 1 frame,...), The observer, the left eye frame, and the right eye frame are switched in the time-series video shown in FIG. "M-frame first observer's left eye frame", "m frame first observer's right eye frame", and m frame's n-th observer left eye as described above. "Frame for the right eye of the nth observer of the mth frame", "frame for the left eye of the first observer of the (m + 1) th frame", "frame for the right eye of the first observer of the (m + 1) th frame", " This means information indicating the order of “the frame for the left eye of the n-th observer” and “the frame for the right eye of the n-th observer of the (m + 1) th frame”.

  The “buffer control signal” is specifically a signal composed of a buffer refresh control signal and a buffer hold control signal. The buffer refresh control signal is imaged by the left eye visual image photographing means 41a of the head-mounted display device 40 to be described later, and refreshes (erases) the left eye visual image already held by the left eye visual image buffer means 42a. The right eye visual image captured by the right eye visual image capturing means 41b of the head-mounted display device 40 and already held by the right eye visual image buffer means 42b is refreshed. The buffer hold control signal holds (accumulates) the left-eye field image captured by the left-eye field image capturing unit 41a in the left-eye field image buffer unit 42a, and the right-eye field image captured by the right-eye field image capturing unit 41b. It is held in the right eye visual field image buffer means 42b. Details of control of the visual field image by the buffer control signal and details of the visual field image will be described later (see FIGS. 4 and 5).

  The video display device 30 displays the time-series video generated by the video generation device 20. The specific configuration of the video display device 30 is not particularly limited as long as it can display a time-series video, and for example, a general liquid crystal display or the like can be used. The video display device 30 is preferably configured with a frame rate as high as possible.

  The head-mounted display device 40 (the first observer's head-mounted display device 40A to the nth observer's head-mounted display device 40B) shoots a field image and displays it to each observer. Is. As shown in FIG. 1, the head-mounted display device 40 is mounted on each of the heads of a plurality of observers who observe time-series images displayed by the image display device 30. When the number of observers of the video display device 30 is n, the head-mounted display device 40 has a head-mounted display device 40A for the first observer to a head mounted for the n-th observer as shown in FIG. It comprises a mold display device 40B. The first observer head-mounted display device 40A to the n-th observer head-mounted display device 40B have the same functions. Therefore, in the following description, instead of individual descriptions of the first observer head-mounted display device 40A to the n-th observer head-mounted display device 40B, the head-mounted display device 40 including them will be described. I will explain.

  As shown in FIG. 2, the head-mounted display device 40 includes a left-eye visual field image capturing unit 41a, a right-eye visual field image capturing unit 41b, a left-eye visual field image buffer unit 42a, a right-eye visual field image buffer unit 42b, and a left-eye visual field image buffer unit 42b. Video display means 43a and right eye visual field video display means 43b are provided. Here, the head-mounted display device 40 is configured to be able to transmit signals to the position and orientation measurement device 10 wirelessly.

  The left eye visual field image capturing means 41a captures a left eye visual field image indicating the visual field of the left eye of the observer. As shown in FIG. 2, the left-eye visual field image capturing means 41a outputs the captured left-eye visual field image to the left-eye visual field buffer means 42a. The right eye visual field image capturing means 41b captures a right eye visual field image showing the visual field of the observer's right eye. As shown in FIG. 2, the right eye visual field image capturing means 41b outputs the captured right eye visual field image to the right eye visual field image buffer means 42b. These left-eye visual field image capturing means 41a and right-eye visual field image capturing means 41b can use an imaging device such as a general video camera, for example.

  Here, the aforementioned “field-of-view image” refers to the entire field of view of the observer including the time-series image displayed by the image display device 30 in consideration of the position and orientation of the head-mounted display device 40 with respect to the image display device 30. It means a video. In the following, for example, as shown in FIG. 4A, a visual field image when the first observer sits at the left position of the video display device 30 and the nth observer stands at the right position of the video display device 30. Details will be described.

  In this case, as shown in FIG. 4B, the left eye visual field image of the first observer is an image viewed from the left eye of the first observer, and the first observer's head worn by the first observer. From the relationship between the position and orientation of the wearable display device 40A and the position of the left eye of the first observer, the video display device 30 in the left-eye visual field image is located at the upper right of the video. Further, the right eye visual field image of the first observer is an image viewed from the right eye of the first observer as shown in FIG. 4B, and the first observer's head wearing that the first observer wears. From the relationship between the position and orientation of the mold display device 40A and the position of the right eye of the first observer, the video display device 30 in the right eye visual field image is the video display device 30 in the left eye visual field image of the first observer. It is located on the left compared to the position of.

  Further, the left eye visual field image of the nth observer is an image viewed from the left eye of the nth observer as shown in FIG. 4C, and the head wearing for the nth observer worn by the nth observer. From the relationship between the position and orientation of the mold display device 40B and the position of the left eye of the n-th observer, the video display device 30 in the left-eye visual field image is positioned at the lower left of the video. The right eye visual field image of the nth observer is an image viewed from the right eye of the nth observer as shown in FIG. 4C, and the head wearing for the nth observer worn by the nth observer. From the relationship between the position and orientation of the mold display device 40B and the position of the right eye of the nth observer, the video display device 30 in the right eye visual field image is the video display device 30 in the left eye visual field image of the nth observer. It is located on the left compared to the position of.

  As shown in FIG. 4A, the first observer and the n-th observer are at positions shifted left and right with respect to the center of the video display device 30. 4 (b) and FIG. 4 (c), it is not directly in front, but is tilted to the left and right according to the position of each observer. Yes.

  The left-eye visual field buffer means 42a is for accumulating the left-eye visual field image photographed by the left-eye visual field photographing means 41a in accordance with the buffer control signal input from the visual field image control means 23. As the left eye visual field buffer means 42a, a general memory or the like can be used. Specifically, as shown in FIG. 2, according to the buffer refresh control signal input from the visual field image control means 23, the left eye visual field image photographing means. Refreshes (deletes) the previous frame of the left-eye visual field image input from 41a and already held (accumulated), and shoots by the left-eye visual field image capturing unit 41a according to the buffer hold control signal input from the visual field image control unit 23 Hold the current frame of the left-eye view image.

  The right-eye visual image buffer means 42b accumulates the right-eye visual image captured by the right-eye visual image photographing means 41b in accordance with the buffer control signal input from the visual image control means 23. The right eye visual field image buffer means 42b can use a general memory or the like. Specifically, as shown in FIG. 2, the right eye visual field image photographing means is in accordance with the buffer refresh control signal input from the visual field image control means 23. Refreshes (deletes) the previous frame of the right-eye visual field image input from 41b and already held (accumulated), and shoots by the right-eye visual field image capturing unit 41b according to the buffer hold control signal input from the visual field image control unit 23 Hold the current frame of the right-eye view image. Here, details of the refresh process and hold process of the visual field video by the left eye visual field buffer means 42a and the right eye visual field buffer means 42b will be described later (see FIG. 5).

  The left eye field image display means 43a displays the left eye field image held by the left eye field image buffer means 42a. The right eye visual image display means 43b displays the right eye visual image held by the right eye visual image buffer means 42b. As the left eye visual field image display means 43a and the right eye visual field image display means 43b, a general small liquid crystal display or the like can be used, which are arranged in front of the left eye and the right eye of the first to nth observers, respectively. In addition, it is preferable that the left eye visual field image display means 43a and the right eye visual field image display means 43b are configured with a frame rate as high as possible.

  Hereinafter, details of the control of the visual field image by the buffer control signal will be described with reference to FIG. Here, it is assumed that a time series video is output from the video switching unit 22 and the video display device 30 displays the time series video. In this case, as shown in FIG. 5, the timing at which the output of the first observer's left eye frame (Sub.1_L) in the m frames of the time series video by the video switching unit 22 ends, that is, the time series video by the video display device 30. At the timing when the display of the left-eye frame for the first observer in the m-th frame ends, a buffer hold control signal (L-1 control signal: falling) and a buffer refresh control signal (R-1 control signal) are output from the visual field image control means 23. : Falling) is output. The buffer hold control signal (L-1 control signal) is input to the left eye visual field image buffer means 42a of the first observer head-mounted display device 40A, and the buffer refresh control signal (R-1) described above. Control signal) is input to the right eye visual field image buffer means 42b of the head-mounted display device 40A for the first observer.

  Next, according to the buffer hold control signal (L-1 control signal), the left eye visual field buffer means 42a holds the current frame (Sub.1_L (m frame)) of the left eye visual field image as shown in FIG. And rewriting is prohibited. As a result, the left eye field image display means 43a of the first observer's head-mounted display device 40A remains in the state where the current frame (Sub.1_L (m frame)) of the left eye field image is displayed.

  At the same time, according to the buffer refresh control signal (R-1 control signal), the right eye visual field image buffer means 42b, as shown in FIG. 5, the previous frame (Sub.1_R (m-1 frame)) of the right eye visual field image. Refresh and erase. Then, as shown in FIG. 5, immediately after the first observer's right eye frame (Sub.1_R) in m frames of the time-series video by the video switching means 22 is output, that is, the time-series video by the video display device 30. Display the right frame of the first eye in the right eye visual image display means 43b of the head-mounted display device 40A for the first observer. 1_R (m frame)) is displayed.

  Next, the timing at which the output of the first observer's right eye frame (Sub.1_R) in the m frames of the time-series video by the video switching means 22, that is, the first observation in the m frames of the time-series video by the video display device 30 is performed. The buffer hold control signal (R-1 control signal: falling) is output from the visual field image control means 23 at the timing when the display of the right eye frame is completed. The buffer hold control signal (R-1 control signal) is input to the right eye visual field image buffer means 42b of the first observer head-mounted display device 40A.

  Next, as shown in FIG. 5, the right-eye visual field image buffer means 42b, according to the buffer hold control signal (R-1 control signal: falling), presents the current frame (Sub.1_R (m frame) of the right-eye visual field image. ) Is held and rewriting is prohibited. As a result, the current frame (Sub.1_R (m frame)) of the right-eye visual field image is still displayed on the right-eye visual field image display means 43b of the head-mounted display device 40A for the first observer. As shown in FIG. 5, the left-eye visual field buffer unit 42a and the right-eye visual field buffer unit 42b perform the above processing for each frame (m, m + 1,...) Of time-series video and for each observer. .

  For example, when there are two observers (n = 2), the left-eye visual image buffer means 42a and the right-eye visual image buffer means 42b are configured according to the buffer hold control signal input from the visual image control means 23 as shown in FIG. a) to FIG. 6 (d), [1] m frame of left eye visual field image of the first observer, [2] m frame of right eye visual field image of the first observer, [3] nth observer. [4] m frame of the right eye field image of the nth observer, [5] m + 1 frame of the left eye field image of the first observer, and [6] the right eye field image of the first observer. The left eye field image and the right eye field image are held in the order of m + 1 frame, [7] m + 1 frame of the left eye field image of the nth observer, and [8] m + 1 frame of the right eye field image of the nth observer. Thereby, a stereoscopic image corresponding to each position is displayed to the first observer and the nth observer. Note that the left-eye visual field image and the right-eye visual field image include images of portions other than the screen of the video display device 30, as shown in FIGS. 6 (a) to 6 (d). The video of this part is a real-world video, but unlike the above-described Patent Document 1, the present invention does not require processing for synthesizing a real-world video and a stereoscopic video.

  As described above, the left-eye field image buffer unit 42a (or the right-eye field image buffer unit 42b) is configured to store the left-eye field image (or right-eye field image) already held by the buffer refresh control signal input from the field image control unit 23. The previous frame is erased, and immediately after that, the left-eye visual image (or right-eye video) input from the left-eye visual image capturing means 41a (or right-eye visual image capturing means 41b) to the left-eye visual image buffer means 42a (or right-eye visual image buffer means 42b). Is rewritten by the buffer hold control signal input from the visual field image control means 23 immediately after that. Therefore, the left-eye visual field image display means 43a (or the right-eye visual field image display means 43b) of the first observer head-mounted display device 40A to the n-th observer head-mounted display device 40B has the observer Only the left-eye video and the right-eye video generated by the corresponding first-viewer binocular image generation unit 21A to n-th binocular image generation unit 21B are displayed.

  According to the stereoscopic video display device 1 having the above configuration, as shown in FIG. 1, the visual field images captured by a plurality of cameras (the left eye visual field image capturing unit 41a and the right eye visual field image capturing unit 41b) are displayed. By observing one video display device 30 shared by a plurality of viewers through the head-mounted display device 40, it becomes possible for the plurality of viewers to simultaneously view one stereoscopic video. That is, the stereoscopic video display device 1 arranges videos according to the position of the observer in time series and displays them on the video display apparatus 30 as time series videos, and displays them on the video display apparatus 30 by the head-mounted display device 40. By capturing the real world including the time-series video that has been recorded, and appropriately holding and displaying the captured left-eye and right-eye view images, a plurality of real-world images and stereoscopic images can be displayed without adjustment. A stereoscopic image can be presented individually according to the observation position of the observer.

  In addition, according to the stereoscopic video display device 1 according to the present invention based on the mixed reality technology, a simulation can be performed when an image reproduction type stereoscopic video display device installed in an actual viewing environment is observed by a plurality of observers. This makes it possible to verify the parameters of the image reproduction type stereoscopic video display device, which was impossible with the current miniaturization technology and display panel technology. When the head-mounted display device 40 according to the present invention can be made compact enough not to be a burden on the observer, the conventional light that reproduces the light to the space where the observer does not exist is reproduced. Unlike the image reproduction type 3D image technology, it can be used as an efficient 3D image display apparatus that does not consume useless information.

  Hereinafter, an overall processing flow of the stereoscopic image display apparatus 1 according to the first embodiment will be described with reference to FIG. First, as shown in FIG. 7A, the position and orientation measurement device 10 measures the individual position and orientation of the head-mounted display device 40 with respect to the video display device 30 (step S10). Next, as shown in FIG. 7B, the left-eye video and the right-eye video of each observer are generated by the binocular video generation means 21 of the video generation device 20 according to the position and orientation of the head-mounted display device 40. (Step S20). Next, as shown in FIG. 7B, the video switching means 22 of the video generation device 20 generates a time-series video in which the left-eye frame and the right-eye frame are switched alternately for each time series and for each observer. (Step S21). Next, as shown in FIG. 7B, the visual field video control means 23 of the video generation device 20 acquires the switching timing of the left-eye frame and the right-eye frame for each observer in the time-series video, and the buffer control signal Is generated (step S22).

  Next, as shown in FIG. 7C, the video display device 30 displays the time-series video (step S30). Next, as shown in FIG. 7 (d), the left eye field image and the right eye field image are captured by the left eye field image capturing unit 41a and the right eye field image capturing unit 41b of the head-mounted display device 40 (step S40). . Next, as shown in FIG. 7 (d), the left-eye field image buffer unit 42a and the right-eye field image buffer unit 42b of the head-mounted display device 40 hold the left-eye field image and the right-eye field image according to the buffer hold control signal. Alternatively, the left eye field image and the right eye field image are refreshed in accordance with the buffer refresh control signal (step S41). Finally, the held left eye field image and right eye field image are displayed by the left eye field image display means 43a and the right eye field image display means 43b of the head-mounted display device 40 (step S42). Through the above processing, a stereoscopic image corresponding to each position is displayed for each observer.

Second Embodiment
Here, the stereoscopic image display apparatus 1 according to the first embodiment described above is based on the premise that the screen of the image display apparatus 30 is always included in the visual field image of each observer (see FIGS. 4 and 6). ), It is also assumed that the field of view of each observer is out of the image display device 30.

For example, as shown in FIG. 8 (a), if a vertical photographing angle of view image capturing means of the head-mounted display device 40 (the left-eye view image capturing means 41a and a right-eye view image capturing means 41b) was theta _V The video display device 30 is within the range of the vertical shooting angle of view θ _V of the n-th observer's visual field imaging means, but is outside the range of the vertical imaging angle of view θ _V of the first observer's visual field imaging means. It has become. Accordingly, the view image of the nth observer does not include the screen of the video display device 30.

Further, as shown in FIG. 8 (b), if the horizontal shooting angle of view image capturing means of the head-mounted display device 40 (the left-eye view image capturing means 41a and a right-eye view image capturing means 41b) was theta _H The video display device 30 is within the range of the horizontal shooting angle of view θ _H of the n-th observer's visual field imaging means, but is outside the range of the horizontal shooting angle of view θ _H of the first observer's visual field imaging means. It has become. Therefore, also in this case, the screen of the video display device 30 is not included in the visual field image of the nth observer. Hereinafter, the configuration of the stereoscopic video display apparatus 1A according to the second embodiment that enables processing in such a situation will be described with reference to FIG.

  As shown in FIG. 9, the stereoscopic image display apparatus 1 </ b> A according to the second embodiment includes a position / orientation measurement apparatus 10, an image generation apparatus 20 </ b> A, an image display apparatus 30, and a head-mounted display apparatus 40. ing. As shown in FIG. 9, the video generation device 20 </ b> A includes a photographing determination unit 24 in addition to the binocular video generation unit 21, the video switching unit 22, and the visual field video control unit 23. Here, the configuration of the stereoscopic video display device 1A other than the imaging determination unit 24 in the video generation device 20A is the same as that of the stereoscopic video display device 1 described above, and thus the description other than the imaging determination unit 24 is omitted.

The photographing determination unit 24 determines whether or not the video display device 30 is photographed (included) in the left-eye visual field video and the right-eye visual field video. The imaging determination unit 24 is in the head-mounted display device 40 (first observer head-mounted display device 40A to nth observer head-mounted display device 40B) measured by the position / orientation measuring apparatus 10. The left-eye visual field image and the right-eye visual field are determined by the positions and orientations of the left-eye visual field image capturing unit 41a and the right-eye visual field image capturing unit 41b, and the field angles θ _V and θ _H of the left-eye visual field image capturing unit 41a It is sequentially determined whether the screen of the video display device 30 is captured in the video. The photographing determination unit 24 then outputs the determination result to the binocular video generation unit 21. The above-mentioned “shooting angle of view θ _V , θ _H ” is a value set in advance for the left eye visual field image capturing unit 41 a and the right eye visual field image capturing unit 41 b.

  The binocular video generation unit 21 does not generate the left-eye video and the right-eye video while the imaging determination unit 24 determines that the video display device 30 has not been shot. Accordingly, the video switching unit 22 generates a time-series video without including the left-eye video and the right-eye video. Note that while the video determination device 24 determines that the video display device 30 has not been shot, nothing is included in the time-series video and the video display device 30 is not displaying anything. Become. However, in this case, since the observer does not look at the video display device 30 in the first place, the observer does not feel any sense of incongruity even when nothing is included in the time-series video as described above. In addition, when the left-eye video and the right-eye video are generated and included in the time-series video even when the observer's visual field is out of the video display device 30, a stereoscopic video that is not observed by the viewer is displayed, which is wasteful. Processing occurs.

  For example, as shown in FIG. 8A or FIG. 8B, the visual field image of one observer (first observer) does not include the video display device 30, and the other observer (n-th observation). When the video display device 30 is included in the viewer's visual field video, the time-series video displayed by the video display device 30 does not include the left-eye video and the right-eye video of the first observer. The left eye image and the right eye image of the two observers are included. Therefore, even if one observer is not looking at the video display device 30, the other observer can observe the stereoscopic video without being affected at all. The binocular video generation unit 21 generates the left-eye video and the right-eye video as usual while the video determination unit 24 determines that the video display device 30 is being shot, and the video switching unit 22 also operates normally. Process.

  The stereoscopic image display apparatus 1A having the above-described configuration does not generate a left-eye image and a right-eye image when the observer's field of view is out of the image display apparatus. Video can be displayed.

  Here, the video generation devices 20 and 20A of the stereoscopic video display devices 1 and 1A described above can be realized by operating a general computer with a program that functions as each of the above-described units and units. This program can be distributed via a communication line, or can be written on a recording medium such as a CD-ROM for distribution.

  In this case, the video generation program for the stereoscopic video display device 1 includes a video generation device 20, a video display device 30, a head-mounted display device 40, and a position / orientation measurement device 10. The computer of the video generation device 20 is caused to function as the above-described binocular video generation means 21, video switching means 22, and visual field video control means 23. In addition, the video generation program for the stereoscopic video display device 1A is a stereoscopic video display device 1A that includes the video generation device 20A, the video display device 30, the head-mounted display device 40, and the position and orientation measurement device 10. The computer of the video generation device 20A is caused to function as the above-described binocular video generation means 21, video switching means 22, and visual field video control means 23.

  As described above, the stereoscopic video display device and the video generation program according to the present invention have been specifically described in the form for carrying out the invention, but the gist of the present invention is not limited to these descriptions, and the claims It should be interpreted broadly based on the scope description. Needless to say, various changes and modifications based on these descriptions are also included in the spirit of the present invention.

  For example, as shown in FIG. 1, the above-described stereoscopic image display device 1 includes a position and orientation measurement device 10 and a head-mounted display device 40 that are wirelessly connected to each other. The device 20 and the video display device 30 are connected by wire, but the connection method is not particularly limited. That is, in the stereoscopic video display device 1, for example, the position / orientation measurement device 10 and the head-mounted display device 40 are connected by wire, and the position / orientation measurement device 10, the video generation device 20, the video generation device 20, and the video display device 30 are connected. Each may be connected wirelessly.

1, 1A stereoscopic image display device 10 position and orientation measurement device 20, 20A image generation device 21 binocular image generation means
21A First-viewer binocular image generation means 21B n-th observer binocular image generation means 22 Image switching means 23 Field-of-view image control means 24 Imaging determination means 30 Image display device 40 Head-mounted display device 40A First observation Head-mounted display device 40B for the viewer n-th head-mounted display device 41a Left-eye visual image capturing means 41b Right-eye visual image capturing means 42a Left-eye visual image buffer means 42b Right-eye visual image buffer means 43a Left-eye visual image display means 43b Right eye visual field image display means

Claims (5)

A video generation device that generates a time-series video from the input three-dimensional original video, a video display device that displays the time-series video generated by the video generation device, and the time displayed by the video display device A plurality of head-mounted display devices respectively mounted on the heads of a plurality of observers observing the sequence video, and a position and orientation for measuring individual position and orientation of the head-mounted display device with respect to the video display device A stereoscopic image display device comprising a measurement device,
The video generation device includes:
When a plurality of observers observe the three-dimensional original image according to the position and orientation measured by the position and orientation measurement device, the left eye image entering each observer's left eye and the right eye entering each observer's right eye Binocular video generation means for generating video,
The left-eye video and the right-eye video generated by the binocular video generation means are read at a predetermined frame rate, and the read left-eye frame and right-eye frame of each observer are alternately switched for each observer. Video switching means for generating the time-series video by rearranging along the time-series,
Visual field video control means for obtaining a switching timing of the left-eye frame and the right-eye frame for each observer in the time-series video generated by the video switching means, and generating a buffer control signal according to the switching timing; Prepared,
The head-mounted display device is
Left-eye visual field image capturing means for capturing a left-eye visual field image indicating the left eye field of the observer;
A right-eye visual field imaging means for capturing a right-eye visual field image showing the visual field of the right eye of the observer;
In accordance with the buffer control signal input from the visual field image control means, left eye visual image buffer means for holding the left eye visual image captured by the left eye visual image capturing means;
A right-eye visual image buffer means for holding the right-eye visual image captured by the right-eye visual image capturing means according to the buffer control signal input from the visual-field video control means;
Left eye field image display means for displaying the left eye field image held by the left eye field image buffer means;
Right eye visual image display means for displaying the right eye visual image held by the right eye visual image buffer means;
A stereoscopic video display device comprising:   The binocular video generation means calculates the positions of the left eye and the right eye in the position and orientation measured by the position and orientation measurement device in a virtual space, and places virtual cameras at the positions of the left eye and the right eye, respectively. The stereoscopic image display apparatus according to claim 1, wherein the left-eye image and the right-eye image are generated by rendering the three-dimensional original image with a virtual camera.   The binocular video generation means calculates the positions of the left eye and the right eye in the position and orientation measured by the position and orientation measurement device in a virtual space, and sets the set values of the element image and the element lens group in integral photography. 2. The stereoscopic image according to claim 1, wherein the left-eye image and the right-eye image are generated by calculating an element image that can be observed in each element lens at each position of the left eye and the right eye according to an IP setting parameter that is indicated. Display device. The video generation device includes the position and orientation of the left-eye visual image photographing means and the right-eye visual image photographing means in the head-mounted display device measured by the position / orientation measurement device, and the predetermined left-eye visual image photographing. Photographing determination means for sequentially determining whether or not the video display device is photographed in the left visual field video and the right visual field video according to the photographing field angle of the means and the right eye visual field photographing means,
The binocular video generation means does not generate the left-eye video and the right-eye video while it is determined by the shooting determination means that the video display device is not shot.
The stereoscopic video display device according to any one of claims 1 to 3, wherein the video switching unit generates the time-series video without including the left-eye video and the right-eye video. A video generation device that generates a time-series video from the input three-dimensional original video, a video display device that displays the time-series video generated by the video generation device, and the time displayed by the video display device A plurality of head-mounted display devices respectively mounted on the heads of a plurality of observers observing the sequence video, and a position and orientation for measuring individual position and orientation of the head-mounted display device with respect to the video display device A left-eye visual field imaging means for capturing a left-eye visual field image that indicates the visual field of the left eye of the observer, and a right-eye visual field image that indicates the visual field of the right eye of the observer A right-eye visual image photographing means for photographing the left-eye visual image photographed by the left-eye visual image photographing means in accordance with a buffer control signal input from the video generation device. Left-eye view image buffer means for holding, right-eye view image buffer means for holding the right-eye view image photographed by the right-eye view image photographing means in accordance with the buffer control signal inputted from the image generation device, and the left-eye view image 3D image display apparatus comprising: left eye field image display means for displaying the left eye field image image held by the buffer means; and right eye field image display means for displaying the right eye field image image held by the right eye field image buffer means. And the computer of the video generation device,
When a plurality of observers observe the three-dimensional original image according to the position and orientation measured by the position and orientation measurement device, the left eye image entering each observer's left eye and the right eye entering each observer's right eye Binocular video generation means for generating video,
The left-eye video and the right-eye video generated by the binocular video generation means are read at a predetermined frame rate, and the read left-eye frame and right-eye frame of each observer are alternately switched for each observer. Video switching means for generating the time-series video by re-arranging along the time-series,
The switching timing of the left-eye frame and the right-eye frame for each observer in the time-series video generated by the video switching means is acquired, and the buffer control signal is generated according to the switching timing to generate the left-eye visual image buffer And visual field video control means for outputting to the right eye visual video buffer means,
Video generation program to function as

Images