JP2016161611A - Display system and display control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display system and a display control method which can simply perform operation, such as instructing a position and the like, by a device a user wears.SOLUTION: A display system 1 is provided with an HMD 100 and a projector 300 performing display to a screen SC. The HMD 100 has an image display part 20 which is worn by a user and can transmit an outside scene; and a motion detection part detecting motions of the image display part 20. The projector 300 is provided with a processing part which displays a positioning guide to the screen SC and coordinates a motion amount of the image display part 20 and a moving amount in the screen SC on the basis of a detection result of the motion detection part when the user moves the image display part 20 in accordance with a display position of the positioning guide.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display system and a display control method.

  2. Description of the Related Art In recent years, so-called wearable devices that are used by users on their bodies and personal items are known. As an example of the wearable device, there is a display device called a head mounted display (HMD) that is used by being worn on the head (see, for example, Patent Document 1). The HMD described in Patent Document 1 is a transmissive display that a user wears on the head and has a function of detecting the user's line of sight, and information is matched to the state of the user's line of sight. indicate.

JP 2012-108379 A

As described in Patent Document 1, it is difficult to use a conventional input device such as a keyboard or a mouse in a state where a transmissive HMD is mounted. Further, when using a non-transparent HMD and other wearable devices, not limited to a transmissive HMD, the types and modes of input operations are limited, and improvement in convenience has been demanded.
The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide a display system and a display control method capable of easily performing an operation such as instructing a position by a device worn by a user. To do.

In order to achieve the above object, a display system of the present invention includes a display unit that is worn by a user and configured to transmit an outside scene, and a terminal device that includes a motion detection unit that detects the movement of the display unit, A display device that performs display on a display surface, wherein the display device displays a display surface marker on the display surface, and the user moves the display unit according to a display position of the display surface marker. A processing unit is provided that associates the amount of movement of the display unit with the amount of movement on the display surface based on the detection result of the motion detection unit.
According to the present invention, since the movement amount on the display surface can be obtained from the movement of the terminal device worn by the user, it becomes possible to input the amount on the display surface by moving the terminal device. . For this reason, the input operation when using the terminal device can be facilitated, and the convenience can be improved.

In the display system according to the present invention, the display device displays the display surface marker at a plurality of different positions on the display surface, and the processing unit indicates the movement amount on the display surface. It is obtained from a plurality of display positions.
According to the present invention, it is possible to quickly associate the movement of the terminal device worn by the user with the amount of movement on the display surface.

Moreover, the present invention is characterized in that, in the display system, the terminal device displays a user-side marker on the display unit so as to be superimposed on the display surface marker on the display surface.
According to the present invention, since the user visually recognizes the user side marker using the terminal device, the movement of the terminal device and the amount of movement on the display surface can be more accurately associated. In addition, the operation burden on the user can be reduced.

Further, the present invention is the display system, wherein the terminal device includes an acquisition unit that acquires a display position at which the display device displays the display surface marker, and the processing unit. The amount of movement of the display unit is obtained from the detection result of the movement detection unit, and the amount of movement of the display unit and the display surface are determined based on the obtained amount of movement and the display position of the display surface marker acquired by the acquisition unit. And performing a process of associating the amount of movement with.
ADVANTAGE OF THE INVENTION According to this invention, the movement amount of a terminal device and the movement amount on a display surface can be matched by the function of a terminal device.

In the display system according to the aspect of the invention, the display device may acquire a motion amount of the display unit obtained from the detection result of the motion detection unit of the terminal device or the detection result of the motion detection unit. An amount acquisition unit, and the processing unit, the processing unit based on the movement amount acquired by the movement amount acquisition unit and the display position where the display surface marker is displayed, the movement amount and the display surface And performing a process of associating the amount of movement with.
According to the present invention, the amount of movement of the terminal device and the amount of movement on the display surface can be associated by the function of the display device.

In the display system according to the present invention, the display device includes a correction processing unit that performs geometric correction of an image displayed on the display surface, and the correction processing unit is based on the amount of movement of the display unit. It is characterized in that at least one of the correction amount and the correction direction of the image is obtained.
According to the present invention, it is possible to correct an image by an operation using the terminal device by using the association between the movement of the terminal device and the movement amount on the display surface.

In the display system according to the aspect of the invention, the processing unit included in the display device may include the detection result of the motion detection unit of the terminal device acquired by the motion amount acquisition unit or the detection result of the motion detection unit. The position or distance on the display surface is obtained based on the amount of movement obtained from the above.
ADVANTAGE OF THE INVENTION According to this invention, input operation can be performed with a terminal device using matching with the movement of a terminal device, and the movement amount on a display surface.

In the display system according to the present invention, the terminal device includes the display unit that is worn on a head of the user, and the display device is configured separately from the terminal device. The display surface marker is displayed on a surface.
ADVANTAGE OF THE INVENTION According to this invention, the movement of the terminal device with which a user wears to a head can be matched with the movement amount in a display surface, and intuitive operation is realizable.

In order to achieve the above object, the display control method of the present invention includes a display unit configured to transmit an outside scene, and a motion detection unit that detects the movement of the display unit, and is worn by a user. Using a terminal device and a display device for displaying on a display surface, the display device displays a display surface marker on the display surface, and the user moves the display unit according to a display position of the display surface marker. The movement amount of the display unit and the movement amount on the display surface are associated with each other based on the detection result of the motion detection unit at the time.
According to the present invention, since the movement amount on the display surface can be obtained from the movement of the terminal device worn by the user, it becomes possible to input the amount on the display surface by moving the terminal device. . For this reason, the input operation when using the terminal device can be facilitated, and the convenience can be improved.
In addition, the present invention may be configured as a program that realizes the above display control method or a recording medium that stores the program.

Explanatory drawing which shows schematic structure of the display system which concerns on embodiment. The functional block diagram of a head mounted type display apparatus. The functional block diagram of a projector. The flowchart which shows operation | movement of a display system. Explanatory drawing of operation | movement of a display system. Explanatory drawing of operation | movement of a display system. The flowchart which shows operation | movement of a display system. Explanatory drawing of operation | movement of a display system.

FIG. 1 is a diagram showing a schematic configuration of a display system 1 according to an embodiment to which the present invention is applied.
The display system 1 includes a head-mounted display device (hereinafter referred to as an HMD) 100 worn by a user on the head, and an image (screen) on a screen SC (projection surface) at a position where the user wearing the HMD 100 can visually recognize. And a projector 300 that projects the projection image P). The HMD 100 corresponds to a terminal device, and the projector 300 corresponds to a display device.

  The HMD 100 includes an image display unit 20 (display unit) that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. The control device 10 also functions as a controller for the user to operate the HMD 100.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses-shaped frame in the present embodiment. The frame has a right holding part 21 and a left holding part 23. The right holding unit 21 and the left holding unit 23 are positioned on the user's side when the user wears the image display unit 20, and are placed on the user's head like a temple of glasses. The image display unit 20 is held.

The right holding unit 21 is provided with a right display driving unit 22, and the left holding unit 23 is provided with a left display driving unit 24. The frame of the image display unit 20 is provided with a right optical image display unit 26 and a left optical image display unit 28 located in front of the user's eyes.
The right optical image display unit 26 is positioned in front of the user's right eye when the user wears the image display unit 20, and the left optical image display unit 28 is when the user wears the image display unit 20. It is located in front of the user's left eye. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20. A camera 61 is disposed at this connecting portion.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

  The right display drive unit 22 forms an image based on the image data as will be described later, and this image is displayed on the right optical image display unit 26. The left display driving unit 24 forms an image based on the image data, and the image formed by the left display driving unit 24 is displayed on the left optical image display unit 28. As a result, images are displayed for the left and right eyes of the user.

The right optical image display unit 26 and the left optical image display unit 28 are configured as a see-through display device that transmits an outside scene. For this reason, the user visually recognizes the images displayed by the right optical image display unit 26 and the left optical image display unit 28, and also visually recognizes the outside scene through the right optical image display unit 26 and the left optical image display unit 28. To do. In addition, by displaying an image having parallax on the right optical image display unit 26 and the left optical image display unit 28, it is possible to display a 3D (stereoscopic) image that is viewed stereoscopically.
In the present embodiment, the user can visually recognize the image displayed on the image display unit 20 while wearing the image display unit 20 and can visually recognize the projection image P projected on the screen SC.

The camera 61 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28. In a state where the user wears the image display unit 20, the position of the camera 61 is substantially in the middle of both eyes of the user in the horizontal direction and above the eyes of the user in the vertical direction. The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera.
The camera 61 images at least a part of the outside scene in the front side direction of the HMD 100, in other words, the user's view direction when the HMD 100 is worn. The width of the angle of view of the camera 61 can be set as appropriate, but the imaging range of the camera 61 is a range that includes the outside world visually recognized by the user through the right optical image display unit 26 and the left optical image display unit 28. preferable. The camera 61 executes imaging in accordance with control of the control unit 140 (FIG. 2) of the control device 10 and outputs captured image data to the control unit 140.

  The image display unit 20 is connected to the control device 10 via a connection cord 48. The connection cord 48 branches left and right, is connected to the right holding unit 21 and the left holding unit 23 of the image display unit 20, and transmits image data to each of the right display driving unit 22 and the left display driving unit 24.

  The connection cord 48 is connected to the right earphone 32 and the left earphone 34, and the connection cord 48 is provided with a microphone 63. For example, as shown in FIG. 1, the microphone 63 is arranged such that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs the sound signal to the sound processing unit 187 ( Output to FIG.

The control device 10 that controls the HMD 100 includes a lighting unit 12 that functions as an indicator that indicates the operating state of the HMD 100, a track pad 14 and a key operation unit 15 for performing an input operation, and a power switch 18. The lighting unit 12 includes a light source such as an LED (Light Emitting Diode), and notifies the operation state (for example, power ON / OFF) of the HMD 100 according to the lighting state of the light source. The key operation unit 15 includes various input elements such as a determination key, a direction key, and a menu key.
The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted. The power switch 18 is a switch for switching on / off the power of the HMD 100.

FIG. 2 is a functional block diagram showing the configuration of the HMD 100. As shown in FIG. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
The right display driving unit 22 and the left display driving unit 24 project liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”) and images formed by the LCDs 241 and 242, respectively. Projection optical systems 251 and 252. The right projection optical system 251 projects image light onto the right light guide plate 261 included in the right optical image display unit 26, and the left projection optical system 252 projects image light onto the left light guide plate 262 included in the left optical image display unit 28. .

The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes.
The right display drive unit 22 includes a right backlight control unit 201, a right LCD control unit 211, and a right backlight 221, and the left display drive unit 24 includes a left backlight control unit 202, a left LCD control unit 212, and a left display. A backlight 222 is provided.

  The left backlight 222 has a light source such as an LED and a diffusion plate. The left LCD 242 is a transmissive liquid crystal panel that is arranged on the optical path of light emitted from the diffusion plate of the left backlight 222 and has a plurality of pixels arranged in a matrix. The left projection optical system 252 includes a lens group that guides image light transmitted through the left LCD 242.

  The left projection optical system 252 includes a collimator lens that converts the image light emitted from the left LCD 242 into a parallel light flux. The image light that has been converted into a parallel light beam by the collimator lens is incident on the left light guide plate 262. The left light guide plate 262 is a prism formed with a plurality of reflecting surfaces that reflect image light, and the image light is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 262. A half mirror positioned in front of the left eye LE is formed on the left light guide plate 262. The image light reflected by the half mirror is emitted from the left optical image display unit 28 toward the left eye LE, and this image light forms an image on the retina of the left eye LE so that the user can visually recognize the image.

  The right display driving unit 22 is configured to be symmetrical with the left display driving unit 24. The right backlight 221 includes a light source such as an LED and a diffusion plate. The right LCD 241 is a transmissive liquid crystal panel that is arranged on the optical path of light emitted from the diffusion plate of the right backlight 221 and has a plurality of pixels arranged in a matrix. The right projection optical system 251 includes a lens group that guides image light transmitted through the right LCD 241.

  The right projection optical system 251 includes a collimator lens that converts the image light emitted from the right LCD 241 into a light beam in a parallel state. The image light converted into a parallel light beam by the collimator lens is incident on the right light guide plate 261. The right light guide plate 261 is a prism formed with a plurality of reflecting surfaces that reflect image light, and the image light is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 261. A half mirror located in front of the right eye RE is formed on the right light guide plate 261, and image light reflected by the half mirror is emitted from the right optical image display unit 26 toward the right eye RE, and the image light is An image is connected to the retina of the right eye RE, and the user is made to visually recognize the image.

  Image light reflected by the half mirror and external light transmitted through the right optical image display unit 26 and the left optical image display unit 28 are incident on the user's right eye RE and left eye LE. The HMD 100 superimposes the image light of the internally processed image and the external light on the user's eyes and allows the user to see the outside scene through the right optical image display unit 26 and the left optical image display unit 28. The image by the image light is visually recognized over the outside scene. As described above, the HMD 100 functions as a see-through display device.

  A 9-axis sensor 66 (motion detection unit) is built in the frame of the image display unit 20. The 9-axis sensor 66 is a unit incorporating an acceleration sensor, a magnetic sensor, and an angular velocity sensor (gyro sensor). For example, the 9-axis sensor 66 is built in the right holding unit 21 or the left holding unit 23 shown in FIG. The 9-axis sensor 66 is fixed to the frame of the image display unit 20. When the frame is displaced, the movement and the posture of the frame are detected by the 9-axis sensor 66. The position of the 9-axis sensor 66 is arbitrary and it is sufficient that it is arranged so as not to be displaced with respect to the frame.

The angular velocity sensor of the nine-axis sensor 66 is a rotation around the X axis (pitch), a rotation around the Y axis (yaw), and a rotation around the Z axis (roll) at the measurement reference point of the built-in detection mechanism. Is detected. As shown in FIG. 1, the 9-axis sensor 66 detects the angular velocity with respect to the head of the user wearing the image display unit 20 in the left-right direction as the X axis, the front-rear direction as the Y axis, and the vertical direction as the Z axis. And More specifically, the X axis corresponds to the front horizontal direction including the right optical image display unit 26 and the left optical image display unit 28 in the frame of the HMD 100. The Y axis corresponds to the front-rear direction of the frame, and the Z axis corresponds to the height direction of the frame.
The acceleration sensor of the nine-axis sensor 66 is a three-axis acceleration sensor that detects acceleration in the X-axis direction, the Y-axis direction, and the Z-axis direction described above. The nine-axis sensor 66 is a three-axis magnetic sensor that detects magnetic fields in the X-axis direction, the Y-axis direction, and the Z-axis direction described above.

  The control device 10 of the HMD 100 includes an interface 114 for connecting various external devices OA that are content supply sources. As the interface 114, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface can be used. The external device OA is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used.

The control device 10 further includes a control unit 140, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.
The input information acquisition unit 110 (input unit) is connected to the operation unit 135. The operation unit 135 includes the track pad 14, the key operation unit 15, and the like, and the input information acquisition unit 110 acquires input content based on a signal input from the operation unit 135. The control device 10 includes a power supply unit 130 and supplies power to each unit of the control device 10 and the image display unit 20.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store still image data or moving image data to be displayed on the image display unit 20.
The storage unit 120 stores setting data 121. The setting data 121 includes setting values set in advance for various processes executed by the control unit 140.
In addition, the storage unit 120 stores content data 122. The content data 122 includes image (still image or moving image) data and / or audio data of content displayed on the image display unit 20.

  A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 acquires the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the control device 10. The GPS 115 outputs the current position obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
The communication unit 117 can be used when receiving content data by communicating with the external device OA. When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and causes the image display unit 20 to display an image. On the other hand, when the external device OA is wired to the interface 114, the control unit 140 acquires content data from the interface 114 and causes the image display unit 20 to display an image. The communication unit 117 and the interface 114 function as a data acquisition unit DA that acquires content data from the external device OA, and the acquired content data is stored as content data 122 in the storage unit 120.

  In the present embodiment, the communication unit 117 performs data communication with the projector 300. The communication unit 117 may directly communicate with the projector 300, or may communicate with the projector 300 via a network configured using a wireless LAN access point (not shown).

  The control unit 140 stores a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a basic control program and data executed by the CPU in a nonvolatile manner. ROM (not shown). The control unit 140 controls each unit of the HMD 100 by executing a control program by the CPU. In addition, the control unit 140 reads out and executes the computer program stored in the storage unit 120 and realizes various functions of the control unit 140. That is, it functions as an operating system (OS) 150, an image processing unit 160, a display control unit 170, a communication control unit 181, an operation detection unit 182, a content output control unit 183, a calibration control unit 184, and an audio processing unit 187. .

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the RAM of the control unit 140 for each frame as image data of the target image (Data in the figure). This image data is, for example, RGB data.
Note that the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24 as necessary. The image processing unit 160 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing is performed to generate 3D image data from 2D image data. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the RAM to the image display unit 20 via the transmission units 51 and 52, respectively. The transmission units 51 and 52 function as a transceiver and execute serial transmission between the control device 10 and the image display unit 20. The image data Data transmitted via the transmission unit 51 is referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is referred to as “left eye image data”.

  The display control unit 170 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates and generates image light by each of the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control injection. Specifically, the right LCD 241 is controlled to be turned on / off by the right LCD control unit 211, and the right backlight 221 is turned on / off by the right backlight control unit 201. In addition, the display control unit 170 controls the driving of the left LCD 242 by the left LCD control unit 212 and the driving of the left backlight 222 by the left backlight control unit 202.

The communication control unit 181 controls communication by the communication unit 117, and establishes communication with the external device OA and receives content data from the external device OA. In addition, the communication control unit 181 controls the communication unit 117 to establish communication with the projector 300 and communicate control data with the projector 300.
The operation detection unit 182 controls the input information acquisition unit 110 to acquire operation data of the operation detected by the operation unit 135 and determine the operation content.

  The content output control unit 183 reads the content data 122 stored in the storage unit 120, controls the image processing unit 160 and the display control unit 170, and causes the image display unit 20 to display an image based on the image data included in the content data 122. Display. Further, when the content data 122 includes audio data, the content output control unit 183 controls the audio processing unit 187 to output the audio of the content from the right earphone 32 and the left earphone 34.

  For example, the content output control unit 183 displays the AR content while the user is looking at the object through the image display unit 20. The content output control unit 183 provides information about the object by performing AR display that displays an image, a character, or the like at a position corresponding to the object, or the shape of the object seen through the image display unit 20 Change how it looks. The AR content includes image and character data displayed at a position corresponding to the object. In addition, the AR content may include data for specifying an object and data regarding the display position of an image or a character. The display position of the AR content may be a position overlapping the target object or may be around the target object. The target object may be an object, may be a real estate such as a building, may be a moving body such as an automobile or a train, and may be a living body such as a human being or an animal. The projection image P projected by the projector 300 is the target. That is, the display position of the image displayed on the image display unit 20 by the content output control unit 183 is a position that matches the position of the projection image P that the user visually recognizes through the image display unit 20. As a result, the display system 1 displays an image linked with the projection image P projected by the projector 300 on the HMD 100.

In the present embodiment, as an example, the projector 300 controls the projection position of the projection image P and the content of the projection image in accordance with the position and orientation of the head of the user wearing the image display unit 20.
In order to perform this control, in the HMD 100, the HMD 100 and the projector 300 cooperate to execute calibration. In the HMD 100, the calibration control unit 184 controls the operation of the HMD 100 in calibration that cooperates with the projector 300.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

  As described above, the nine-axis sensor 66 is an inertial sensor (motion sensor) that detects acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes). When the image display unit 20 is mounted on the user's head, the control unit 140 detects the movement of the user's head based on the detection value of the 9-axis sensor 66. In addition, the control unit 140 uses the communication control unit 181 to generate data indicating the detection value of the 9-axis sensor 66 or data regarding the movement of the image display unit 20 generated from the detection value of the 9-axis sensor 66. Send to.

The image display unit 20 includes an interface 25. The interface 25 includes a connector to which the connection cord 48 is connected, and is connected to the control device 10 via the connection cord 48. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission units 51 and 52 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 170 to the corresponding reception units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface for connecting the camera 61 and the 9-axis sensor 66. The interface 25 controls data relating to detection values of acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes) by the nine-axis sensor 66, captured image data of the camera 61, the camera 61, and the nine-axis sensor 66. Control data is transmitted to and received from the control unit 140.

  The reception unit 53 operates as a receiver corresponding to the transmission unit 51 and executes serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. .

The reception unit 54 operates as a receiver corresponding to the transmission unit 52 and executes serial transmission between the control device 10 and the image display unit 20. The left backlight control unit 202 drives the left backlight 222 based on the input control signal. The left LCD control unit 212 drives the left LCD 242 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the left-eye image data Data input via the reception unit 54. .
The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

FIG. 3 is a functional block diagram illustrating the configuration of the projector 300.
The projector 300 includes an interface (I / F) unit 311 that connects the image supply device 2 that outputs image data. A plurality of image supply apparatuses 2 can also be connected to the I / F unit 311.

  Examples of the image supply device 2 include an image reproducing device such as a DVD player, a broadcast receiving device such as a digital TV tuner, and an image output device such as a video game machine and a personal computer. The image supply device 2 may be a communication device that communicates with a personal computer or the like and receives image data. The image supply device 2 is not limited to a device that outputs digital image data, and may be a device that outputs an analog image signal. In this case, an analog / digital conversion device that generates digital image data from an analog image signal may be provided on the output side of the image supply device 2 or the I / F unit 311 of the projector 300. The specific specifications and number of connectors and interface circuits included in the I / F unit 311 are arbitrary.

  The image supply device 2 outputs digital image data in a data format that can be supported by the I / F unit 311. The data input from the image supply device 2 to the projector 300 may be a still image or a moving image (video) as long as the data format is compatible with the I / F unit 311. In the following description, data input from the image supply device 2 to the projector 300 is referred to as image data.

  The projector 300 includes a control unit 320 that controls each unit of the projector 300 and a projection unit 330 that displays (projects) an image based on image data input to the I / F unit 311 on the screen SC. The I / F unit 311 is connected to an image processing unit 312 (image processing apparatus) that processes image data and outputs a display image signal to the projection unit 330.

The projection unit 330 includes a light source 331, a modulation unit 332, and a projection optical system 333. In addition, a light source control unit 314 is connected to the light source 331, and a display driving unit 313 is connected to the modulation unit 332.
The light source 331 includes lamps such as a xenon lamp and an ultrahigh pressure mercury lamp, or a solid light source such as an LED and a laser light source. The light source 331 is turned on by the electric power supplied from the light source control unit 314 and emits light toward the modulation unit 332. The light source control unit 314 can adjust the light emission luminance of the light source 331 under the control of the control unit 320.

The modulation unit 332 modulates the light emitted from the light source 331 to generate image light, and irradiates the projection optical system 333 with the image light. In the present embodiment, the modulation unit 332 includes three liquid crystal light valves corresponding to red (R), green (G), and blue (B) colors, and the light emitted from the light source 331 is used as the liquid crystal light valve. The structure which permeate | transmits is illustrated.
A display driving unit 313 is connected to the three liquid crystal light valves of the modulation unit 332. The display driving unit 313 drives each pixel of the liquid crystal light valve based on the image signal output from the image processing unit 312 and draws an image on the liquid crystal light valve in units of frames (screens).

  The optical path between the light source 331 and the modulation unit 332 or the modulation unit 332 may be provided with a reflector, a lens group (not shown), a polarizing plate, a light control element, and the like. In addition, the modulation unit 332 can have a configuration using a reflective liquid crystal panel. In this case, the modulation unit 332 reflects the light emitted from the light source 331 to the liquid crystal panel and guides the reflected light to the projection optical system 333. Further, the modulation unit 332 may be configured to use a digital mirror device (DMD), and may be configured to include one DMD and a color wheel.

  The projection optical system 333 guides the image light modulated by the modulation unit 332 toward the screen SC, and forms an image on the screen SC. The projection optical system 333 may be configured to include, for example, optical elements such as a prism that synthesizes light that has passed through three liquid crystal light valves, a lens group that guides image light, and a mirror. Further, the projection optical system 333 includes a zoom lens for enlarging / reducing the image on the screen SC, a focus lens for adjusting the focus, a zoom adjusting motor for adjusting the degree of zoom, a focus adjusting motor for adjusting the focus, and the like. May be.

  The image processing unit 312 performs processing such as color tone correction on the image data input to the I / F unit 311 under the control of the control unit 320. The image data processed by the image processing unit 312 is converted into an image signal for each frame and input to the display driving unit 313. The image processing unit 312 can also display the image data stored in the storage unit 325. In this case, the image data stored in the storage unit 325 is input from the control unit 320 to the image processing unit 312, and the image processing unit 312 performs processing on the image data and outputs an image signal to the display driving unit 313. .

The projector 300 includes a communication unit 318. The communication unit 318 performs wireless data communication conforming to standards such as wireless LAN (WiFi) and Bluetooth. The communication unit 318 transmits / receives control data to / from the HMD 100 according to the control of the control unit 320.
The communication unit 318 may directly communicate with the HMD 100 or may communicate with the HMD 100 via a network configured using a wireless LAN access point (not shown).

  The control unit 320 includes a processor that controls each unit of the projector 300. For example, the control unit 320 includes a CPU, a ROM, and a RAM (not shown), and the CPU executes a program stored in the ROM, thereby allowing each unit of the projector 300 to be executed. Control.

The control unit 320 functions as a projection control unit 321 and a calibration control unit 322 by executing the program.
The projection control unit 321 controls execution timing, execution conditions, and the like of processing executed by the image processing unit 312. Further, the projection control unit 321 controls the light source control unit 314 of the projection unit 330 to adjust the luminance of the light source 331 and the like.

  The calibration control unit 322 executes a calibration process for aligning the position of the image displayed by the HMD 100 and the projection image P projected by the projector 300. The calibration control unit 322 causes the projection unit 330 to project a calibration image onto the screen SC, and receives data related to the movement of the image display unit 20 transmitted from the HMD 100 via the communication unit 318. The calibration control unit 322 generates data that associates the projection image P with the movement of the image display unit 20 based on the data received from the HMD 100 and stores the data in the storage unit 325 as calibration data 327.

  The control unit 320 is connected to an input unit 315 that receives user input operations. The input unit 315 is connected to an operation panel 316 provided with a switch, and detects an operation on the operation panel 316. The input unit 315 is configured as an infrared light receiving unit that receives an infrared signal transmitted from the remote controller 317 and detects an operation on the remote controller 317. Input unit 315 outputs operation data indicating operations on operation panel 316 and remote controller 317 to control unit 320.

  A storage unit 325 is connected to the control unit 320. The storage unit 325 is a storage device that stores programs and data executed by the CPU of the control unit 320 in a nonvolatile manner. The storage unit 325 stores a control program for the control unit 320 to control the projector 300, various setting data processed by the control program, and the like. Further, the storage unit 325 stores calibration image data 326 of an image projected by the projection unit 330 in the calibration process, and calibration data 327 generated in the calibration process.

Next, the operation of the display system 1 will be described.
FIG. 4 is a flowchart showing the operation of the display system 1, and particularly shows the calibration process. 4A shows the operation of the projector 300, and FIG. 4B shows the operation of the HMD 100. In the operation of FIG. 4, the calibration control unit 322 functions as an input detection unit, a motion amount acquisition unit, and a processing unit.

  5 and 6 show examples of images displayed on the display system 1 in the calibration process. 5A and 5E show examples of the projection image P projected by the projector 300 onto the screen SC, and FIGS. 5B and 5C show the display state of the HMD 100. FIG. FIG. 5D shows the visual field V of the user wearing the HMD 100. 6A to 6C show the visual field V of the user, and the projection image P that the user sees through the image display unit 20 is shown together.

  In the display system 1, when an instruction to start calibration is detected by the input information acquisition unit 110 of the HMD 100 or the input unit 315 of the projector 300, the HMD 100 and the projector 300 transmit and receive control data related to this instruction. The calibration control unit 322 of the projector 300 starts calibration processing (step S11). The calibration control unit 322 reads out the calibration image data 326 from the storage unit 325, and causes the projection unit 330 to display a calibration image (step S12). This calibration image is an image composed of figures and characters for alignment, and is called an alignment guide here. In a state where the alignment guide is displayed, the calibration control unit 322 waits until input information transmitted by the HMD 100 is received (step S13; NO).

  FIG. 5A shows a state where the alignment guide is displayed on the screen SC in step S12. 5A, the cross-shaped alignment guide MP is positioned at the center of the projection image P, and the alignment guides M1 to M4 are positioned at the four corners of the projection image P, respectively. The calibration image data 326 may be image data of the entire projection image P including the alignment guides MP and M1 to M4, for example. The alignment guides MP and M1 to M4 correspond to display surface markers.

FIG. 5B shows the state of the HMD 100 at the start of the calibration process. FIG. 5B shows a surface of the image display unit 20 facing the user's eyes. The right optical image display unit 26 is positioned on the right side, and the left optical image display unit 28 is positioned on the left side.
The calibration control unit 184 of the HMD 100 starts the calibration process (step S31), reads the calibration image data 123 from the storage unit 120, and causes the image display unit 20 to display the calibration image (step S32). . This calibration image is an image composed of figures and characters for alignment, and is called a guide here.
FIG. 5C shows an example in which guides MR and ML are displayed by the image display unit 20. The right optical image display unit 26 corresponding to the right eye displays a right eye guide MR, and the left optical image display unit 28 corresponding to the left eye displays a left eye guide ML. The display position of the guide MR in the right optical image display unit 26 and the display position of the guide ML in the left optical image display unit 28 are associated with each other in advance, so that an image of one guide can be seen by the user.

  FIG. 5D shows the user's visual field V when the user visually recognizes the guides MR and ML shown in FIG. As shown in FIG. 5D, it appears to the user that the guides MR and ML form one guide M image. That is, the display positions of the guides MR and ML are adjusted according to the characteristics of the image display unit 20 and the user. The characteristics include, for example, the convergence angle, the size of the half mirror that is the display area of the right optical image display unit 26 and the left optical image display unit 28, the distance between the user's eyes and the half mirror, and the distance between the pupils of the user. Etc. The guides ML and MR, and the guide M in the form in which these are visually recognized, correspond to the user side marker.

  A user wearing the image display unit 20 performs an input operation on the control device 10 when the guide M is visible. This input operation is an operation meaning that the guide M has been confirmed, and may be a simple operation. For example, an operation of pressing any key provided in the control device 10 or an operation of touching the track pad 14 can be performed. This operation serves as a trigger for instructing the start of calibration based on the position of the image display unit 20.

  The calibration control unit 184 of the HMD 100 waits until the input information acquisition unit 110 detects an input operation after displaying the guide MR and ML in step S32 (step S33; NO). When the input operation of the user is detected (step S33; YES), the calibration control unit 184 acquires the detection value of the 9-axis sensor 66 included in the image display unit 20. Furthermore, the calibration control unit 184 generates input information including data (motion data) related to the detection value of the 9-axis sensor 66 and transmits it to the projector 300 (step S34). The motion data is data relating to the detection value of the 9-axis sensor 66, and includes at least data relating to the detection value of the acceleration sensor. In this case, the projector 300 can obtain information regarding the attitude of the image display unit 20 based on the motion data included in the input information transmitted in step S34. After transmitting the input information in step S34, the calibration control unit 184 waits until an input operation is performed (step S35; NO).

  When the calibration control unit 322 of the projector 300 receives the input information transmitted by the HMD 100 (step S13; YES), the calibration control unit 322 acquires motion data included in the input information and temporarily stores it in the storage unit 325 (step S14). .

Subsequently, the calibration control unit 322 changes the color of the alignment guide to be aligned among the alignment guides MP and M1 to M4 displayed on the screen SC. In the present embodiment, it is set so that the alignment is first performed on the alignment guide MP at the center of the projection image P, and then at least one of the alignment guides M1 to M4 is aligned. When step S15 is executed first, the calibration control unit 322 changes the display color of the alignment guide MP. As a result, as shown in FIG. 5E, the display state of the center alignment guide MP appears to the user to change, so that the user can be guided to view the alignment guide MP.
After changing the display state of the alignment guide (here, the display color), the calibration control unit 322 waits until receiving input information transmitted by the HMD 100 (step S16; NO).

  The user recognizes that the display color of the alignment guide MP has changed in the projected image P on the screen SC, and places his head so that the guide M displayed by the image display unit 20 overlaps the alignment guide MP. move. Then, as illustrated in FIG. 6A, the input operation of the control device 10 is performed in a state where the guide M in the visual field V appears to overlap the alignment guide MP of the projection image P. This operation is an operation for instructing that the position has been adjusted, and may be a simple operation similar to the operation when the guide M is visually recognized. For example, an operation of pressing any key provided in the control device 10 or an operation of touching the track pad 14 can be performed.

  When the calibration control unit 184 of the HMD 100 detects a user's input operation (step S35; YES), the calibration control unit 184 acquires a detection value of the 9-axis sensor 66 included in the image display unit 20. Further, the calibration control unit 184 generates input information including motion data related to the detection value of the 9-axis sensor 66 and transmits it to the projector 300 (step S36). Further, the motion data included in the input information transmitted by the calibration control unit 184 in step S36 may include not only the detection value of the 9-axis sensor 66 at that time but also data related to the history of the detection value of the 9-axis sensor 66. Good. That is, statistical data such as a change in the detection value of the 9-axis sensor 66 after the acquisition of the detection value of the 9-axis sensor 66 in the previous step S35 or step S36, or an integrated value of the detection values may be included. Further, the calibration control unit 184 may periodically acquire and accumulate the detection values of the 9-axis sensor 66 at predetermined time intervals, for example, by associating the detection time with the detection value. May be stored. In this case, the calibration control unit 184 may include the detection value and the detection time of the 9-axis sensor 66 in the plurality of detections in association with each other in step S36.

  After transmitting the input information, the calibration control unit 184 determines whether control data for notifying completion of calibration has been received from the projector 300 (step S37). If the calibration control unit 184 does not receive control data for notifying completion of calibration within a preset time (step S37; NO), the calibration control unit 184 returns to step S35 and waits for the user's input operation.

When receiving the input information from the HMD 100 (step S16; YES), the calibration control unit 322 of the projector 300 acquires the motion data included in the input information and temporarily stores it in the storage unit 325 (step S17).
Here, the calibration control unit 322 determines whether the calibration is completed (step S18). The calibration is completed when the operations of steps S15 to S17 are performed for all the alignment guides set out of the alignment guides MP and M1 to M4. Therefore, the calibration control unit 322 determines whether or not the operations in steps S15 to S17 have been completed for all the alignment guides set out of the alignment guides MP and M1 to M4.

When there is an alignment guide for which the operations of steps S15 to S17 are not performed among all the alignment guides that have been set, the calibration control unit 322 determines that the calibration has not been completed (step S18; NO). . In this case, the calibration control unit 322 returns to step S15 to change the display state (display color) of the alignment guide to be aligned next, and to change the display state (display color) of the aligned alignment guide to the original. Return to. Note that the alignment guide that has been aligned may be changed to a display state (display color) indicating that the alignment has been completed, instead of returning the display state (display color) to the original.
FIG. 6B shows an example in which the alignment guide M4 is aligned next to the alignment guide MP. In the example of FIG. 6B, the display state of the alignment guide M4 changes. Since the display state of the alignment guide M4 in the upper left corner appears to the user to change, the user can be guided to view the alignment guide M4.

  After the calibration control unit 322 changes the display color of the alignment guide M4 in step S15, the user performs an operation of aligning the guide M with the alignment guide M4 as shown in FIG. Perform the operation. This input operation is detected by the calibration control unit 184 in step S35.

  As described above, the alignment guides set out of the alignment guides MP and M1 to M4 are aligned, and the motion data is stored. After the alignment of the set alignment guide is completed, the calibration control unit 322 determines that the calibration is completed (step S18; YES). The calibration control unit 322 generates control data that notifies the HMD 100 of the completion of calibration, and transmits the control data to the HMD 100 (step S19).

  In step S37, the calibration control unit 184 of the HMD 100 waits for control data for notifying completion of calibration from the projector 300. When the calibration control unit 184 receives control data for notifying completion of calibration (step S37; YES), the calibration control unit 184 ends the operation of FIG.

  Further, after transmitting the control data to the HMD 100, the calibration control unit 322 of the projector 300 associates the display position on the screen SC with the amount of movement of the HMD 100 (step S20). When it is determined in step S18 that the calibration is completed, the storage unit 325 stores the motion data received from the HMD 100 in association with the set alignment guide. This motion data includes data relating to acceleration, angular velocity, and magnetism detection values detected by the 9-axis sensor 66.

  The calibration control unit 322 processes motion data corresponding to a plurality of alignment guides stored in the storage unit 325. Specifically, the calibration control unit 322 selects two alignment guides from the alignment guides aligned by calibration. Subsequently, the amount of motion of the image display unit 20 when the line-of-sight direction of the image display unit 20 moves between the two alignment guides is obtained from the difference in motion data corresponding to the selected alignment guide. The calibration control unit 322 calculates the difference between the display positions of the two alignment guides. The display position of the alignment guide is represented by, for example, coordinates in a liquid crystal light valve or DMD included in the calibration control unit 322 or coordinates in frame data generated by the image processing unit 312. In the example shown in FIGS. 5E and 6A to 6C, the calibration control unit 322 obtains a coordinate difference between the alignment guide MP and the alignment guide M4. Further, the amount of movement of the image display unit 20 may be obtained as a change in the attitude of the image display unit 20 based on the detection value of the 9-axis sensor 66, but based on the integral value of the detection value of the 9-axis sensor 66, the image display It is preferable to obtain the amount of change in the angle and position of the portion 20. For example, the calibration control unit 322 changes the angle and position of the image display unit 20 based on the detection values of the 9-axis sensor 66 from the state of FIG. 6A to the state of FIG. That is, the movement amount of the image display unit 20 is calculated. Then, the calibration control unit 322 generates data that associates the unit amount of the display position of the alignment guide with the amount of movement of the image display unit 20 (step S21). This data can be, for example, a parameter, a table including parameters, or a function or matrix that associates the unit amount of the display position of the alignment guide with the amount of movement of the image display unit 20. This data is called calibration data. In step S <b> 21, the calibration control unit 322 stores the generated calibration data in the storage unit 325 as calibration data 327.

  After the calibration data 327 is generated, the amount of movement when the image display unit 20 moves can be converted into the amount of movement of the display position in the projector 300. Therefore, an operation in which the user intentionally moves the image display unit 20 can be used as an operation for specifying the movement amount.

  The calibration data 327 can be used not only when geometric correction is performed according to the movement of the HMD 100, but also for an operation of moving a cursor in a projected image on the screen SC (display surface), for example. It can also be used for operations such as selecting menus and icons displayed on the screen SC, and operations for instructing deformation (enlargement, reduction, rotation) and movement of the projected image.

  Note that when the display state of the alignment guide is changed in step S15, changing the display color of the alignment guide is an example, and for example, the display size or shape may be changed. In addition, the display state of the alignment guide to be changed is maintained, and the display state of other alignment guides such as deleting the display of other alignment guides or blinking other alignment guides is displayed. The target alignment guide may be made conspicuous by the changing method.

FIG. 7 is a flowchart showing the operation of the display system 1 and shows the operation when geometric correction is performed using an operation of moving the image display unit 20. FIG. 7A shows the operation of the projector 300, and FIG. 7B shows the operation of the HMD 100. In the operation of FIG. 7, the calibration control unit 322 functions as a correction processing unit.
FIG. 8 shows an example of an image displayed on the display system 1 in the calibration process. 8A and 8B show the visual field V of the user, and the projection image P that the user sees through the image display unit 20 is shown together.

  When an instruction to start geometric correction is detected by the input information acquisition unit 110 of the HMD 100 or the input unit 315 of the projector 300, the HMD 100 and the projector 300 transmit and receive control data relating to this instruction.

The calibration control unit 184 of the HMD 100 starts the geometric correction process (step S71), reads the calibration image data 123 from the storage unit 120, and causes the image display unit 20 to display an image for geometric correction (step S72). . The image for geometric correction is an image composed of figures and characters for performing geometric correction. In the present embodiment, as an example, the image is similar to the calibration image shown in FIGS. Including a guide. As the geometric correction image used by the MD 100, a previously stored image may be used separately from the calibration image.
In addition, the calibration control unit 322 of the projector 300 starts the geometric correction process (step S51), reads the calibration image data 326 from the storage unit 325, and causes the projection unit 330 to display the geometric correction image ( Step S52). In this embodiment, the calibration image data 326 that is a calibration image is used as the geometric correction image. However, the geometric correction image data different from the calibration image data 326 is used. Also good.

As shown in FIG. 8A, alignment guides MP, M1 to M4 are arranged in the projected image P on the screen SC. The guide M is displayed in the visual field V of the user, and the user can visually recognize the alignment guides MP and M1 to M4 of the projection image P through the image display unit 20. The projected image P in FIG. 8A is a square image for geometric correction. In the correct projection state, the geometric correction projection image P is rectangular, but in the example of FIG. 8A, trapezoidal distortion has occurred and is not rectangular. The purpose of the operation shown in FIG. 7 is to generate parameters for deforming the projection image P so as to be rectangular.
In the geometric correction, an operation of moving any one of the alignment guides M1 to M4 of the projection image P to a position where the projection image P becomes a rectangle is performed. The number of the alignment guides M1 to M4 to be moved is arbitrary. When the user determines that the projection image P is rectangular and the geometric correction has been completed, the geometric correction process ends. The calibration control unit 322 generates data for performing geometric correction based on the amount of movement of the display position of the alignment guide according to the user's operation. By correcting the projection image based on this data, the trapezoidal distortion of the image projected thereafter can be corrected.
Therefore, the alignment guides M1 to M4 at the four corners that are the operation target points to be operated by the user are displayed on the projection image P that is the geometric correction image displayed by the calibration control unit 322. Note that the center alignment guide MP of the projection image P can be set as an operation target.

As shown in FIG. 8A, the user moves the head so that the guide M overlaps one of the alignment guides of the projection image P (in this example, the alignment guide M4), and the control device 10 moves to the control device 10. Perform the operation. This input operation is an operation meaning that the guide M has been confirmed, and may be a simple operation.
The calibration control unit 184 of the HMD 100 waits until the input information acquisition unit 110 detects an input operation after displaying the guide MR and ML in step S72 (step S73; NO). Therefore, when the user performs an operation on the control device 10, this operation is detected (step S73; YES), and the operation is resumed. The calibration control unit 184 acquires the detection value of the 9-axis sensor 66 included in the image display unit 20. Further, the calibration control unit 184 generates input information including data (motion data) related to the detection value of the 9-axis sensor 66 and transmits it to the projector 300 (step S74). The motion data is the same as the data described in the operation of FIG. After transmitting the input information in step S74, the calibration control unit 184 waits until an input operation is performed (step S75; NO).

The calibration control unit 322 of the projector 300 waits until input information is received from the HMD 100 after displaying the geometric correction image in step S52 (step S53; NO). When receiving the input information transmitted by the HMD 100 (step S53; YES), the calibration control unit 322 acquires motion data included in the input information and temporarily stores it in the storage unit 325 (step S54). The motion data stored here is data indicating the orientation of the image display unit 20 corresponding to the geometric correction start position. Therefore, in step S54, the calibration control unit 322 specifies the position of a target point (correction point) on which geometric correction is performed based on the received motion data. Specifically, the calibration control unit 322 converts the motion data included in the input information received in step S52 into display position data using the calibration data 327, and corrects the converted display position. The position of the point.
The calibration control unit 322 waits until new input information is received from the HMD 100 in a state where the geometric correction image is displayed on the screen SC (step S55; NO).

The user wearing the image display unit 20 places his head so that the guide M of the image display unit 20 overlaps the position where the alignment guide (alignment guide M4) selected in FIG. move. In FIG. 8A, the projection image P has a trapezoidal distortion, and the user performs an operation of moving the alignment guide M4 to correct the trapezoidal distortion. This operation is an operation of aligning the guide M with the position of the movement destination for moving the alignment guide M4, and specifically, an operation in which the user moves the head.
The user performs an input operation to the control device 10 after aligning the guide M with the position of the movement destination. This operation may be a simple operation as described above. When the calibration control unit 184 detects an input operation by the user (step S75; YES), the calibration control unit 184 acquires a detection value of the 9-axis sensor 66, generates input information including motion data, and transmits the input information to the projector 300 (step S75). S76).

  When receiving the input information from the HMD 100 (step S55; YES), the calibration control unit 322 of the projector 300 acquires the motion data included in the input information (step S56). Further, the calibration control unit 322 specifies the display position of the movement destination to which the correction point specified in step S53 is moved based on the motion data and the calibration data 327. Then, the calibration control unit 322 generates geometric correction data based on the positions before and after the correction point is moved. The geometric correction data includes at least one of data indicating the correction amount of the projection image and data indicating the correction direction.

  Then, the calibration control unit 322 performs geometric correction of the projection image P based on the generated data (step S57). In step S57, the geometric correction data generated in step S56 is set in the image processing unit 312 to execute geometric correction. Thereby, the projected image P for geometric correction displayed on the screen SC is corrected as shown in FIG. In FIG. 8B, the position of the alignment guide M4 is moved, and the entire projection image P is deformed along with the movement of the alignment guide M4, and the trapezoidal distortion is corrected.

  Here, the user performs an input operation indicating that a desired correction has been made, that is, an instruction operation for completion of correction, in response to the change in the projected image P as shown in FIG. 8B. After transmitting the input information, the calibration control unit 184 of the HMD 100 waits for an input operation to the control device 10 (step S77), and if there is no correction completion instruction operation within a predetermined time (step S77; NO), The process returns to step S73. When the correction completion instruction operation is detected (step S77; YES), the calibration control unit 184 generates control data notifying that the correction is completed, and transmits the control data to the projector 300 (step S78). finish.

After executing geometric correction, the image processing unit 312 waits for reception of control data notifying completion of correction (step S58), and if control data is not received within a predetermined time (step S58; NO), step S53. Return to. When control data notifying completion of correction has been received (step S58; YES), the calibration control unit 322 includes geometric correction data in the calibration image data 326 and causes the storage unit 325 to store the data (step S58). This process is terminated.
Note that the geometric correction data generated by the calibration control unit 322 can be, for example, a parameter, a table including parameters, or a function or matrix that converts coordinates before geometric correction into coordinates after geometric correction. .

  In this way, the user wearing the image display unit 20 performs an operation of moving the head, and only by performing an operation on the control device 10 at the start and end of the operation, the user's head movement can be achieved. It is detected as an input operation for indicating a position, and processing is performed. For this reason, the input operation by the movement of the image display unit 20 can be realized.

  As described above, the display system 1 according to the embodiment to which the present invention is applied includes the HMD 100 and the projector 300 that performs display on the screen SC. The HMD 100 includes an image display unit 20 that is worn by a user and configured to transmit an outside scene, and a 9-axis sensor 66 that detects the movement of the image display unit 20. The projector 300 displays an alignment guide on the screen SC, and the image display unit based on the detection result of the 9-axis sensor 66 when the user moves the image display unit 20 according to the display position of the alignment guide. A control unit 320 that performs processing for associating 20 movement amounts with the movement amount on the screen SC is provided. Thereby, the movement amount in the screen SC can be obtained from the movement of the HMD 100 worn by the user. Therefore, by moving the HMD 100, that is, by moving the head on which the image supply device 2 is mounted, it is possible to perform input regarding the amount on the screen SC. For this reason, input operation when using the HMD 100 can be facilitated, and convenience can be improved.

Further, the projector 300 displays the alignment guide at a plurality of different positions on the screen SC, and obtains the movement amount on the screen SC from the plurality of display positions of the alignment guide. For this reason, it is possible to quickly associate the movement of the HMD 100 worn by the user with the amount of movement on the screen SC.
In addition, since the HMD 100 displays the guide M for visual recognition on the alignment guide of the screen SC on the image display unit 20, the movement of the HMD 100 and the amount of movement on the screen SC are more accurately associated. Can do. Thereby, the burden of a user's operation can be reduced.
Further, the calibration control unit 322 of the projector 300 detects that the input is acquired by the input information acquisition unit 110 of the HMD 100, and the movement obtained from the detection value of the 9-axis sensor 66 of the HMD 100 and the detection result of the 9-axis sensor 66. Get the data. Then, the calibration control unit 322 performs processing for associating the motion data with the movement amount on the screen SC based on the acquired motion data and the display position of the alignment guide. For this reason, the amount of movement of the HMD 100 and the amount of movement on the screen SC can be associated by the function of the projector 300.

In addition, the calibration control unit 322 performs geometric correction of the image displayed on the screen SC, and obtains data for geometric correction including at least one of the correction amount and the correction direction of the image based on the motion amount of the HMD 100. In this way, the image can be corrected by an operation using the HMD 100 by using the association between the movement of the HMD 100 and the movement amount on the screen SC.
Further, the calibration control unit 322 can obtain the position or distance on the screen SC based on the calibration image data 326 obtained from the motion data related to the detection result of the 9-axis sensor 66 of the HMD 100. For this reason, it is possible to perform an input operation on the HMD 100 using the association between the movement of the HMD 100 and the movement amount on the screen SC.

  In the display system 1, the guide M is displayed on the image display unit 20 worn on the user's head, and the alignment guide is displayed on a screen SC configured separately from the HMD 100. In this way, an intuitive operation can be realized by associating the movement of the HMD 100 with the amount of movement on the screen SC.

In the above embodiment, an example in which the calibration control unit 322 of the projector 300 performs a process of associating the motion data of the image display unit 20 with the display position on the screen SC in steps S20 to S21 in the operation of FIG. . The present invention is not limited to this, and the control unit 140 of the HMD 100 may perform processing corresponding to steps S20 to S21.
Similarly, in the operation of FIG. 7, an example is shown in which the calibration control unit 322 of the projector 300 performs a process of generating data for performing geometric correction in step S <b> 256. The present invention is not limited to this, and the control unit 140 of the HMD 100 may perform processing for generating data for performing geometric correction. In this case, data for performing geometric correction is transmitted from the HMD 100 to the projector 300, and the control unit 320 may set the geometric correction parameters of the image processing unit 312 based on this data.

  In this case, the calibration control unit 184 of the HMD 100 functions as an acquisition unit that acquires a display position where the projector 300 displays the alignment guide. Further, the calibration control unit 184 functions as a control unit. When the input information acquisition unit 110 receives an input, the control unit obtains a motion amount from the detection result of the 9-axis sensor 66. Further, based on the obtained amount of movement and the display position of the alignment guide acquired by the acquisition unit, processing for associating the amount of movement with the amount of movement on the screen SC is performed. In this case, the amount of movement of the HMD 100 and the amount of movement on the screen SC can be associated by the function of the HMD 100.

In addition, this invention is not restricted to the structure of the said embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
For example, the image display unit 20 included in the HMD 100 may be configured to be worn by the user while moving with the user's body, and is not limited to the eyeglass type. For example, you may employ | adopt the image display part of the shape worn like a hat. For example, you may attach to body accessories, such as a helmet, clothing, or accessories, such as a decoration. Further, the display device is not limited to the configuration in which image light is emitted to both eyes of the user as in the HMD 100 so that the image is visually recognized.

  Further, in the above embodiment, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection cord 48 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head. 4 and 7 is not limited to the operation performed on the track pad 14 or the key operation unit 15 of the control device 10, but for example, an operation of moving the image display unit 20 in a predetermined pattern. It may be.

  The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. Further, the control device 10 may be configured separately from the image display unit 20, and various signals may be transmitted and received between the control device 10 and the image display unit 20 by wireless communication.

  For example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.

  The optical system that guides the image light to the user's eyes may include an optical member that transmits external light that is incident on the apparatus from the outside and that is incident on the user's eyes together with the image light. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes. For example, the present invention can also be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eyes, and the laser light is incident on the user's eyes to scan the retina and form an image on the retina. A configuration that allows the user to visually recognize an image may be employed. Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, a light emitting unit includes a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system. You may prepare. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

  Moreover, in the said embodiment, although the calibration control part 322 was set as the structure which displays a some alignment guide, this invention is not limited to this, For example, you may display one alignment guide. In this case, the correspondence with the movement in the HMD 100 can be obtained by moving the display position of the alignment guide. For example, a user who wears the HMD 100 may move the HMD 100 so as to move one alignment guide and follow the movement of the alignment guide.

  In the above-described embodiment, the configuration in which the HMD 100 and the projector 300 are connected by wireless communication is illustrated, but a wired connection may be used, and the connection form is arbitrary.

In addition, at least a part of the functional blocks illustrated in FIGS. 2 and 3 may be realized by hardware, or may be configured by cooperation of hardware and software. That is, the present invention is not limited to the configuration in which independent hardware resources are arranged as shown in FIGS. Further, the program executed by the control unit 140 may be stored in the storage unit 120 or a storage device in the control device 10, or the program stored in an external device is acquired via the communication unit 117 or the interface 114. It is good also as a structure to execute. Similarly, the program executed by the control unit 320 may be stored in the storage unit 325, or may be supplied from another storage unit or an external device.
Of course, other detailed configurations can be arbitrarily changed.

  DESCRIPTION OF SYMBOLS 1 ... Display system, 2 ... Image supply apparatus, 10 ... Control apparatus, 14 ... Trackpad, 15 ... Key operation part, 20 ... Image display part, 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding , 24 ... Left display drive unit, 26 ... Right optical image display unit, 28 ... Left optical image display unit, 48 ... Connection code, 61 ... Camera, 66 ... 9-axis sensor (motion detection unit), 100 ... HMD (terminal) Device), 110 ... input information acquisition unit (input unit), 117 ... communication unit, 120 ... storage unit, 121 ... setting data, 122 ... content data, 123 ... calibration image data, 135 ... operation unit, 140 ... control 160, image processing unit, 170 ... display control unit, 181 ... communication control unit, 182 ... operation detection unit, 183 ... content output control unit, 184 ... calibration control unit, 187 ... audio processing unit 261 ... Right light guide plate, 262 ... Left light guide plate, 300 ... Projector (display device), 312 ... Image processing unit, 313 ... Display drive unit, 315 ... Input unit, 316 ... Operation panel, 317 ... Remote control unit, 318 ... Communication unit 320, control unit, 321 ... projection control unit, 322 ... calibration control unit (input detection unit, motion amount acquisition unit, processing unit, correction processing unit), 325 ... storage unit, 326 ... calibration image data, 327 ... calibration data, 330 ... projection unit, M, ML, MR ... guide (user side marker), MR, M1 to M4 ... position alignment guide (display surface marker), SC ... screen (display surface).

Claims (9)

A display unit that is worn by a user and configured to transmit the outside scene, and a terminal device that includes a motion detection unit that detects the movement of the display unit;
A display device for displaying on the display surface;
With
The display device displays a display surface marker on the display surface, and the display based on a detection result of the motion detection unit when the user moves the display unit according to a display position of the display surface marker. A processing unit that performs processing for associating the amount of movement of the unit with the amount of movement on the display surface;
A display system characterized by The display device displays the display surface marker at a plurality of different positions on the display surface;
The processing unit obtains a movement amount on the display surface from a plurality of display positions of the display surface marker;
The display system according to claim 1. The terminal device displays on the display unit a user-side marker for visual recognition superimposed on the display surface marker on the display surface;
The display system according to claim 1, wherein: The terminal device
An acquisition unit for acquiring a display position at which the display device displays the display surface marker;
The processing unit;
With
The processing unit obtains the amount of movement of the display unit from the detection result of the motion detection unit, and based on the obtained amount of movement and the display position of the display surface marker acquired by the acquisition unit, Performing a process of associating the amount of movement with the amount of movement on the display surface;
The display system according to claim 1, wherein: The display device
A detection result of the motion detection unit of the terminal device, or a motion amount acquisition unit that acquires a motion amount of the display unit obtained from the detection result of the motion detection unit;
The processing unit;
With
The processing unit performs a process of associating the amount of movement of the display unit with the amount of movement on the display surface based on the amount of movement acquired by the amount of movement acquisition unit and the display position where the display surface marker is displayed. about,
The display system according to claim 1, wherein: The display device includes a correction processing unit that performs geometric correction of an image displayed on the display surface,
The correction processing unit obtains at least one of a correction amount and a correction direction of the image based on a movement amount of the display unit;
The display system according to claim 5. The processing unit included in the display device is based on a detection result of the motion detection unit of the terminal device acquired by the motion amount acquisition unit or a motion amount obtained from a detection result of the motion detection unit. Determining the position or distance on the display surface,
The display system according to claim 5 or 6. The terminal device includes the display unit mounted on the user's head,
The display device displays the display surface marker on the display surface configured separately from the terminal device;
The display system according to claim 1, wherein: A display unit configured to transmit an outside scene, and a motion detection unit that detects the movement of the display unit, using a terminal device worn by a user and a display device that displays on a display surface,
The display device displays a display surface marker on the display surface;
The movement amount of the display unit and the movement amount on the display surface are associated with each other based on the detection result of the movement detection unit when the user moves the display unit according to the display position of the display surface marker. ,
A display control method characterized by the above.

Images