JP2018142162A - Image forming system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image forming system capable of allowing an observer to view see-through and three-dimensional images without requiring troublesome adjustment.SOLUTION: An image forming system comprises: a projector that projects a predetermined projected image on a projection surface; a wearable terminal that includes a transmission type display and an image generation device for generating a display image to be displayed on the transmission type display and can be worn by an observer; and a position information output device that outputs position information indicating the position of the observer to the transmission type display. The image forming system adjusts the attitude and/or angle of view of the image to be displayed on the transmission type display on the basis of the input position information. The observer can simultaneously observe the display image displayed via the transmission type display and the projected image on the projection surface.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image forming system using a transmissive display.

  For example, planetariums that reproduce the starry sky by projecting stars, planets, the sun, the moon, planetary satellites, and comets are known. A general planetarium projects a celestial object on a dome screen formed on the inner surface of the dome by a planetarium projection device arranged in the dome so that multiple observers sitting on a seat in the dome can look up and observe. I have to.

  By the way, in recent planetariums, a program for watching a story about a celestial object is also made to be viewed by an observer simultaneously with the projection of the celestial object. Along with this, there are cases where it is desired to display a spaceship or the like in addition to the celestial body, for example, according to the contents of the viewing program. In such a case, a planetarium projection device can be used to simultaneously display an image of a spacecraft or the like on a dome screen. However, since the image is two-dimensional, there is a problem that it is difficult for an observer to feel reality.

  On the other hand, as shown in Patent Document 1, it is possible to view a stereoscopic video by simultaneously projecting a digital image for the left eye and a digital image for the right eye using, for example, special glasses worn by the viewer. A system has already been developed. By using such a system, it is possible to make an observer visually recognize a stereoscopic image of a spacecraft or the like.

JP 2005-354644 A

  However, in the technique described in Patent Document 1, a stereoscopic effect is most effectively obtained at the viewing position assumed at the time of video production, and a sufficient stereoscopic effect can be obtained depending on the position of the seating seat. Therefore, there is a problem that unpleasant color shift occurs. Further, since the entire apparatus is large-scale, there is a problem that it is costly to be used by many observers.

  On the other hand, a wearable terminal having a display capable of displaying a stereoscopic video is already on the market. Therefore, in a state where the wearable terminal is attached to each observer, it is possible to make the observer visually recognize a stereoscopic image of a spacecraft or the like superimposed on the celestial image projected on the dome screen. However, if the same image is displayed on each wearable terminal, all viewers view the same image in the same direction, which causes a deviation from the projected image projected on the dome screen. Therefore, there is a fear that the contents of the viewing program cannot be appropriately expressed. In particular, when a huge spacecraft or the like is spatially displayed as a stereoscopic image, it is difficult to obtain the stereoscopic effect. On the other hand, it may be possible to secure a three-dimensional effect by manually adjusting the wearable terminal according to the viewing angle of the observer, but this is not practical due to the problem of time and effort. A similar problem can occur when an attendee makes a presentation or the like using a wearable terminal in a conference room.

  The present invention is intended to solve the above-described problems, and provides an image forming system that allows an observer to visually see a three-dimensional image without seeing through troublesome adjustment. Objective.

The image forming system of the present invention includes:
A wearable terminal equipped with a transmissive display and an image generation device that generates a display image to be displayed on the transmissive display;
A position information output device that outputs position information indicating the position of the observer to the wearable terminal,
The image generation device is configured to adjust the posture and / or angle of view of an image displayed on the transmissive display based on the input position information,
The observer can simultaneously observe a display image displayed through the transmissive display and an external image seen through the transmissive display.

The image forming system of the present invention includes:
A projector that projects a predetermined projection image on the projection surface;
A wearable terminal equipped with a transmissive display and an image generation device that generates a display image to be displayed on the transmissive display;
A position information output device that outputs position information indicating the position of the observer to the wearable terminal,
The image generation device is configured to adjust the posture and / or angle of view of an image displayed on the transmissive display based on the input position information,
The observer can simultaneously observe a display image displayed via the transmissive display and a projected image on the projection surface.

  According to the present invention, it is possible to provide an image forming system in which an observer can visually see a three-dimensional image without seeing through troublesome adjustment.

1 is a schematic diagram illustrating an image forming system according to a first embodiment. It is a figure which shows the observer equipped with the head mounted display which is a wearable terminal. It is a block diagram which shows the structure of a head mounted display. 3 is a flowchart showing the operation of the image forming system according to the present embodiment. It is a figure which shows a part of image forming system concerning a modification. It is the schematic which shows the image forming system which concerns on 2nd Embodiment.

(First embodiment)
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic diagram showing an image forming system according to the first embodiment. FIG. 2 is a diagram showing an audience wearing a head-mounted display that is a wearable terminal. FIG. 3 is a block diagram showing the configuration of the head mounted display.

  In FIG. 1, the planetarium facility PN includes a hemispherical dome screen (projection surface) DS, a projector PR disposed in the center of the dome screen DS, and a plurality (represented here) disposed around the projector PR. Only two seats ST1 and ST2. Audience (observers) OB1 and OB2 are seated on the seats ST1 and ST2, respectively. Here, it is assumed that a three-dimensional virtual coordinate system with the origin O as the center of the projector PR is defined.

  As shown in FIG. 2, each audience OB1 (or OB2) wears a head-mounted display HMD on the head, and the transmissive display DPL covers both eyes in the worn state. The audience OB1 (or OB2) wearing the glasses-type transmissive display DPL can simultaneously view the image displayed on the transmissive display DPL and the outside through the transmissive display DPL.

  The audience OB1 (or OB2) receives the head-mounted display HMD at the entrance of the planetarium facility PN, for example, and sits on an arbitrary seat ST1 (or ST2) while wearing the head-mounted display HMD to watch the planetarium viewing program. To do. The image forming system according to the present embodiment includes a planetarium facility PN, a head mounted display HMD, and a seat information output device IOD.

  The head-mounted display HMD attached to the audience OB1 (or OB2) has a common configuration, and as shown in FIG. 3, an input device IID for inputting seat information, a motion sensor MS including an acceleration sensor, a gyroscope, etc. And a memory MR for storing display image data, an image generation device PROC for generating a three-dimensional display image, and a transmissive display DPL.

  On the other hand, as shown in FIG. 2, a seat information output device (position information output device) IOD for outputting seat information is embedded in the back of the seat ST1 (or ST2) where the audience OB1 (or OB2) sits. . Assume that the seat information output device IOD stores the coordinates (position information) of the embedded seat ST1 (or ST2) in the internal three-dimensional virtual coordinate system as seat information in its built-in memory.

  The seat information can include calibration data when a standard figure person (observer) is seated in front of the seat. The calibration data is, for example, a transformation matrix for transforming the coordinate system of the transmissive display DPL itself in the head-mounted display HMD into a three-dimensional virtual coordinate system in the planetarium facility PN, and 6-dimensional information of position and orientation Is preferably provided. The head-mounted display HMD is provided with a motion sensor MS, and can acquire a posture in accordance with the inclination of the audience's head. Based on this information, the position / posture of the three-dimensional display image in the three-dimensional virtual coordinates can be changed. Note that the position information included in the seat information is used because the position cannot be acquired by the head mounted display HMD alone. However, when the spectator sits on the reclining seat and tilts the backrest, the position changes, and therefore it is preferable to perform position correction as described later with reference to FIG.

The seat information written in the seat information output device IOD of each seat can be generated in advance by the following method, for example.
(A) Based on the design information, the relative position / posture of the dome screen DS and the seat ST1 (or ST2) (coordinates in the three-dimensional virtual coordinate system, etc.) is calculated and stored in the built-in memory of the seat information output device IOD Write.
(B) A pattern image projected by the projector PR by installing a camera on the seat or by seating a human body or doll with a calibration head-mounted display HMD equipped with a camera on the seat ST1 (or ST2), Alternatively, a plurality of (usually 6 or more) light sources such as LEDs for calibration are photographed by a camera, and the dome screen DS and the transmissive display DPL of the head mounted display HMD for calibration are obtained based on the obtained pattern. The relative position / posture is calculated and written to the built-in memory of the seat information output device IOD.

  The seat information output device IOD and the input device IID are preferably communicable using a short-range wireless communication system. Thereby, it is possible to prevent erroneous seat information from being acquired due to interference between devices in the presence of a plurality of seats. In this case, it is preferable to use an existing short-range wireless communication technology such as Bluetooth (registered trademark) or NFC (Near field radio communication). A wireless communication device is incorporated in the tip of the head of the head-mounted display HMD, the back of the headband, etc., so that it can communicate with the seat information output device IOD embedded in the seat ST1 (or ST2) in a non-contact manner. It becomes possible to do. As a result, the audience OB1 (or OB2) wearing the head mounted display HMD can read out the necessary seat information simply by sitting on the seat ST1 (or ST2).

  FIG. 4 is a flowchart showing the operation of the image forming system according to the present embodiment. First, before starting the viewing program, the audience OB1 (or OB2) wears the head-mounted display HMD (step S101), and further sits on the seat ST1 (or ST2), so that the seat information output device IOD and the input device IID Thus, the head-mounted display HMD can recognize that the audience OB1 (or OB2) has been seated on the seat ST1 (or ST2) (YES in step S102).

  Next, in step S103, the input device IID of the head mounted display HMD acquires unique seat information from the seat information output device IOD. In step S104, the motion sensor MS detects the posture of the audience OB1 (or OB2), and transmits posture information to the image generation device PROC accordingly. At this time, the seat information from the seat information output device IOD and the posture information from the motion sensor MS are input to the image generation device PROC. In this state, the audience OB1 (or OB2) waits for the start of the viewing program.

  At the start of the viewing program, an image (star image) such as a celestial object is projected from the projector PR onto the dome screen DS according to a control device (not shown). At the same time, viewing program start information (trigger) is input to the input device IID via the seat information output device IOD. In response to this, the image generation device PROC reads out the image data from the memory MR, and generates a three-dimensional image 3D such as a spacecraft, for example. The three-dimensional image 3D is processed based on the seat information and the posture information. (Step S105). Specifically, the three-dimensional image 3D is adjusted so that the audience OB1 looks up at the bottom surface of the spacecraft at a relatively close position with the backrest of the seat ST1 tilted, while the audience OB2 is the backrest of the seat ST2. The three-dimensional image 3D is adjusted so that the spaceship at a relatively far position can be seen from the side in a state where is set up. That is, the three-dimensional image 3D is processed based on the seat information and the posture information so as to have a posture and / or angle of view suitable for each observer to observe.

  Since the processed image is displayed (output) on the transmissive display DPL of the head mounted display HMD in step S106, the observer OB1 (or OB2) can view the astronomical image (projected) on the dome screen DS. The three-dimensional image 3D (display image) displayed on the transmissive display DPL can be observed on the back of the image.

  Thereafter, when it is determined in step S107 that the viewing program is not terminated in the head mounted display HMD (determination NO), it is detected in the subsequent step S108 whether or not the posture of the observer OB1 (or OB2) has changed. . The presence or absence of this posture change is detected according to the output of the motion sensor MS. For example, when the observer OB1 (or OB2) changes the angle of the backrest of the seat ST1 (or ST2), or the observer OB1 (or OB2) For example, when the direction of the head is changed, a signal corresponding thereto is output from the motion sensor MS to the image generation device PROC.

  When the signal output from the motion sensor MS to the image generation device PROC changes (YES in Step S108), the image generation device PROC acquires new posture information in Step S104, and processes the image in Step S105. The image processed in step S106 is output. On the other hand, when the signal output from the motion sensor MS to the image generation device PROC does not change (NO in Step S108), the output of the same posture image is continued in Step S106.

  If it is determined in step S107 that the viewing program is terminated in the head mounted display HMD (determination YES), image output is stopped in step S109.

  According to the present embodiment described above, the head mounted display HMD does not have seat information in advance, and the seat information is acquired only when the audience OB1 (or OB2) is seated. When distributing the system, individual settings and management are not required, and facility operation is simplified. Further, according to the present embodiment, the display image displayed by the transmissive display DPL of the head mounted display HMD is relatively fixed in space with respect to the celestial image projected by the projector PR. . As a result, a three-dimensional image (stereoscopic image) for enjoying a planar image such as a planetarium more deeply is superimposed on the celestial image, and is observed by the observer as intended by the producer according to the viewing program. For example, it will be possible to provide powerful video experiences by displaying the spacecraft and planets within reach, and display avatars and characters to provide familiar and secure content. Become.

  Furthermore, since the image displayed on the transmissive display DPL is always displayed at a fixed position with respect to the dome screen DS, it is possible to enjoy the same content again from a different angle. To contribute. In addition, by inputting information about the audience wearing the transmissive display DPL via the input device IID, it is also possible to additionally display individual contents according to the age, sex, nationality, etc. of each audience. . For example, when information indicating that a child is wearing a head-mounted display HMD is input, an image of the character running around can be displayed via the transmissive display DPL, and a foreigner wears the head-mounted display HMD. When information indicating this is input, it is possible to simultaneously display captions and explanations.

  FIG. 5 is a diagram illustrating a part of an image forming system according to a modification. In a general reclining seat, the backrest tilts according to a uniaxial rotational movement, and therefore it is easy to estimate the position from the tilt. Therefore, position correction can be performed based on the motion sensor MS provided in the head mounted display HMD. However, in order to improve the estimation accuracy, an angle sensor AS that measures the amount of rotation of a rotary encoder or the like is separately provided on the seat ST1 on the rotation axis of the backrest as shown in FIG. 5, and the output signal of the angle sensor AS is used as the seat information. It is also possible to input to the output device IOD, whereby the angle of the backrest, that is, the reclining state can be transmitted to the head mounted display HMD, and the display image can be processed by using it as auxiliary posture information.

  More specifically, the seat information output device IOD inputs the position of the seat ST1 and the exact backrest angle detected by the angle sensor AS to the input device IID of the head mounted display HMD as seat information. The three-dimensional image 3D displayed on the transmissive display DPL is processed based on this seat information. In the case of this modification, the motion sensor MS may be omitted from the head mounted display HMD, which contributes to downsizing and power saving of the head mounted display HMD.

  Even when seated on the seat, a human can change the position of the head by a certain amount of upper body movement. For this reason, it is difficult to follow the display of the three-dimensional image with respect to an operation such as looking into the image only by calibration data and correction by the angle sensor. Therefore, for example, using a fixed camera or the like installed in the planetarium facility PN, the marker mounted on the head mounted display HMD mounted by each spectator can be photographed, and the position and orientation can be corrected by using the information. It is difficult and costly to install a camera for acquiring individual movements in a space where seats are densely arranged, such as planetarium facilities, movie theaters, and conference rooms. It is also possible to place a marker such as a chess pattern and read it with the camera built in the head mounted display to estimate the position / posture. However, if it is difficult to place the marker in a planetarium or a movie theater, etc. Is not realistic. In this embodiment, in view of this point, sitting on the seat deeply reduces the movement of the upper body of the audience, and by using only the posture change, it is possible to easily display a three-dimensional image in a fixed space in the facility. Can do.

(Second Embodiment)
FIG. 6 is a schematic diagram showing an image forming system according to the second embodiment. This image forming system can be used, for example, in the case of a presentation performed in a conference room or the like. In FIG. 6, the purpose of the presentation and the performance and specifications of the agenda product can be projected on the screen (projection surface) SC of the conference room using a projector (not shown). It is not always necessary to provide the screen SC.

  Here, attendees (observers) OB <b> 1 to OB <b> 3 wearing the head mounted display HMD are seated in the seats ST <b> 1 to ST <b> 3 of the conference room, respectively. Again, it is assumed that a three-dimensional virtual coordinate system with the center of the conference room as the origin is defined.

  On the other hand, as shown in FIG. 2, a seat information output device IOD for outputting seat information is embedded in the backrest of the seats ST1 to ST3 where the attendees OB1 to OB3 sit. The seat information output device IOD is assumed to store the coordinates (seat information) of the embedded seats ST1 to ST3 in the prescribed three-dimensional virtual coordinate system in the built-in memory. Since other configurations are the same as those of the above-described embodiment, description thereof is omitted.

  In the present embodiment, the image generation device PROC reads out image data from the memory MR, generates a 3D image 3D such as an engine, and displays it on the transmissive display of the head mounted display HMD. 3D is processed in advance based on seat information and posture information. Specifically, the 3D image 3D of the engine viewed from the left is displayed on the transmissive display DPL of the attendee OB1, and the 3D of the engine viewed from the front is displayed on the transmissive display DPL of the attendant OB2. The image 3D is displayed, and the image is processed so that the three-dimensional image 3D of the engine viewed from the right side is displayed on the transmissive display DPL of the attendee OB3.

  For each seat ST1 to ST3 fixed in the conference room, the image displayed on the transmissive display of the head mounted display HMD worn by the attendees OB1 to OB3 is relatively on the space based on the screen SC. Fixed. This assumes usage in a conference room where a plurality of fixed seats ST1 to ST3 face each other. When the attendees OB1 to OB3 wear the head-mounted display HMD and sit on the seats ST1 to ST3, the seat information is automatically output from the seat information output device IOD to the head-mounted display HMD, and thus transmitted. A three-dimensional image 3D floating in the conference room space is displayed on the type display. At this time, it appears as if the same product is observed from the seats ST1 to ST3 according to the positions and postures of the seats ST1 to ST3. For example, as shown in FIG. 6, by displaying a three-dimensional image 3D of the engine, the explainer can actually explain the characteristics and functions of each part using his / her hand. Further, since the three-dimensional image 3D is displayed using the transmissive display, each other's face and facial expression (external image) that can be seen through the transmissive display can be clearly identified, so that the conference can proceed smoothly.

  In the conference room, while the attendee is sitting on the seat, the posture correction is performed by the head mounted display HMD, and when the attendee stands for explanation, the position is switched to the conventional position correction (that is, the image processing is performed). Not to use). In this case, the input device IID of the head mounted display HMD performs close proximity communication with the seat information output device IOD built in the seat, and when the attendee starts up, the input device IID and the seat information output device IOD It becomes possible to determine that the person is away from the seat by detecting that communication is impossible. At this time, the head mounted display HMD may autonomously switch to a communication means such as Wi-Fi and receive a position correction service by a fixed camera (not shown) provided in the conference room. In addition, the fixed camera discriminates the attendees who have stood up and acquires their positions, so that only a signal for image processing is sent from the facility side only at the head-mounted display HMD of the attendees who have stood up. Also good.

  As a modification, in the head mounted display HMD in the above embodiment, for example, a camera CA as an imaging unit can be provided as shown by a one-dot chain line in FIG. For example, an image generation apparatus PROC as an image processing apparatus captures an image of a specific two-dimensional code CD pasted on the back of the seat before the audience or attendee is seated or on a table with the camera CA, and the image signal is an image processing apparatus. By performing image processing, position information (here, seat information) included in the image of the two-dimensional code CD may be read and a display image may be processed based on the position information.

  The present invention is not limited to the embodiments described in the specification, and it is apparent to those skilled in the art from the embodiments and ideas described in the present specification that other embodiments and modifications are included. It is. The description and examples are for illustrative purposes only, and the scope of the present invention is indicated by the claims. For example, the projection surface may be a curved surface or a flat surface. Further, the transmissive display may be arranged only in front of one eye of the observer, and the display image may be a two-dimensional image.

AS angle sensor CA camera CD two-dimensional code DPL transmissive display DS dome screen HMD head mounted display IID input device MR memory MS motion sensor OB1, OB2 audience OB1-OB3 attendee PN planetarium equipment PR projector PROC image generator SC screen ST1 ~ ST3 Seat

Claims (7)

A wearable terminal equipped with a transmissive display and an image generation device that generates a display image to be displayed on the transmissive display;
A position information output device that outputs position information indicating the position of the observer to the wearable terminal,
The image generation device is configured to adjust the posture and / or angle of view of an image displayed on the transmissive display based on the input position information,
The image forming system in which the observer can simultaneously observe a display image displayed through the transmissive display and an external image visible through the transmissive display. A projector that projects a predetermined projection image on the projection surface;
A wearable terminal equipped with a transmissive display and an image generation device that generates a display image to be displayed on the transmissive display;
A position information output device that outputs position information indicating the position of the observer to the wearable terminal,
The image generation device is configured to adjust the posture and / or angle of view of an image displayed on the transmissive display based on the input position information,
The image forming system in which the observer can simultaneously observe a display image displayed via the transmissive display and a projected image on the projection surface.   The image forming system according to claim 2, wherein the projection surface is a dome screen, and the projector projects a star image on the dome screen.   The image forming system according to claim 1, wherein the position information output device is disposed in a seat on which the observer sits, and outputs the position information wirelessly to the wearable terminal.   The wearable terminal includes a motion sensor that detects a posture of the observer wearing the wearable device and outputs posture information related to the posture, and the image generation device is configured to transmit the transmission type based on the inputted posture information. The image forming system according to claim 1, wherein the posture and / or the angle of view of an image displayed on the display is adjusted.   And an angle sensor that detects and outputs an angle of a backrest of a seat on which the observer is seated, and the image generation device is configured to display an attitude of an image displayed on the transmissive display based on the input angle and / or The image forming system according to claim 1, wherein the image angle is adjusted.   The wearable terminal includes an imaging unit and an image processing device. When the imaging unit images a predetermined marker and outputs an imaging signal to the image processing device, the image processing device is positioned based on the imaging signal. The image forming system according to claim 1, wherein the image generating apparatus acquires information and adjusts a posture and / or an angle of view of an image displayed on the transmissive display based on the position information.