JP2016110489A - Display device, and method and program for calibrating display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display device that can specify the position of a user and the direction of the user's face more accurately than a prior art without using a plurality of sensors.SOLUTION: A display device 200 including a display 230 and used while attached to the head of a user, during execution of a calibration mode, displays an AR image for calibration at a position on the display 230 corresponding to a predetermined reference position S by using real space information 421 acquired from an information providing server 140, and when receiving an input indicating that the AR image for calibration is seen overlapped on the predetermined reference position S from the user, performs calibration with a direct viewing direction defined by a signal output from an inertia sensor 240 as a reference direction S.SELECTED DRAWING: Figure 12

Description

  The present invention relates to a display device, a display device calibration method, and a calibration program.

  In recent years, a technology called augmented reality (AR) that superimposes and displays additional information in an actual environment has attracted attention. If a head mounted display (such as Smart Glass) that employs AR technology is used, AR images such as guidance and advertisements can be superimposed and displayed in the actual environment.

  In order to make the user feel as if the AR image is present in the real environment, the display position of the AR image, depending on the position of the user and the orientation of the face (hereinafter also referred to as “direct viewing direction”), The size, shape, etc. need to be changed from time to time. For this purpose, it is necessary to accurately specify the position of the user and the orientation of the face when generating the AR image.

  In order to meet such a demand, for example, a technique for specifying a user's position and face orientation using a gyro sensor has been developed (Patent Document 1). However, generally, if the user's position and face orientation are specified based only on the output of the gyro sensor, errors are accumulated in the identification result, and a positional deviation occurs between the actual environment and the AR image. .

  In view of this, a technique has been developed in which a sensor other than the gyro sensor is further mounted to correct the position of the user and the orientation of the face (Patent Document 2).

JP 2008-299669 A JP-A-8-289226

  However, if a plurality of sensors are mounted to accurately identify the user's position and face orientation, the cost, weight, and power consumption increase as compared to the case where only the gyro sensor is mounted.

  An object of this invention is to provide the display apparatus etc. which can pinpoint a user's position and face direction more correctly than before without using a some sensor.

(1) A display device that includes a display unit and is used by being mounted on a user's head, wherein the information acquisition unit acquires real space information about at least the position of the user and a predetermined reference position; A sensor that outputs a signal corresponding to the movement of the head, a calibration unit that calibrates the direct viewing direction determined by the signal output from the sensor, the real space information acquired by the information acquisition unit, and the sensor output A display control unit that displays an AR image on the display unit based on a direct viewing direction determined by a signal to be performed, and when executing the calibration mode, the display control unit acquires the real space acquired by the information acquisition unit. Using the information, an AR image for calibration is displayed at the position of the display unit corresponding to the predetermined reference position. ,
When the calibration unit receives an input from the user indicating that the calibration AR image appears to overlap the predetermined reference position, a direct viewing direction determined by a signal output from the sensor is used as a reference direction. Display device to be calibrated.

  (2) The display device according to (1), wherein the predetermined reference position is a position of a predetermined screen, and an image including a graphic having a predetermined shape is displayed on the screen.

  (3) The display device according to (2), wherein the AR image for calibration includes a figure having a shape corresponding to the figure displayed on the screen.

  (4) When the calibration unit receives an input from the user indicating that the graphic displayed in the AR image for calibration appears to overlap the graphic displayed on the screen, the sensor outputs The display device according to (3), wherein calibration is performed using a direct view direction determined by a signal to be performed as a reference direction.

  (5) The display device according to any one of (2) to (4), wherein the position of the user is a position of a seat on which the user is seated.

  (6) The display device according to (1), wherein the predetermined reference position is a position of an object having a predetermined shape.

  (7) The display device according to (6), wherein the AR image for calibration includes a figure having a shape corresponding to the object.

  (8) The calibration unit receives a signal from the sensor when receiving an input from the user indicating that the figure included in the calibration AR image appears to overlap the shape of the object. The display device according to (7), wherein calibration is performed using a fixed direct view direction as a reference direction.

  (9) The display device according to any one of (1) to (8), wherein the information acquisition unit acquires from a server independent of the display device.

  (10) The display device according to any one of (1) to (8), further including a storage unit that stores the real space information, wherein the information acquisition unit acquires the real space information from the storage unit. .

  (11) The input of the user received by the input receiving unit is performed by pressing a predetermined button or by holding the user's head stationary for a predetermined time or longer. The display device described.

  (12) A display device calibration method that includes a display unit and is used by being worn on a user's head, wherein the display device acquires real space information relating to at least the position of the user and a predetermined reference position. An information acquisition unit, a sensor that outputs a signal according to the movement of the user's head, a calibration unit that calibrates a direct view direction determined by a signal output from the sensor, and the information acquired by the information acquisition unit A display control unit that displays an AR image on the display unit based on real space information and a direct viewing direction determined by a signal output from the sensor, and (a) the display control unit when executing the calibration mode. The display corresponding to the predetermined reference position using the real space information acquired by the information acquisition unit (B) the calibration unit accepts an input from the user indicating that the calibration AR image appears to overlap the predetermined reference position. A calibration method for a display device, comprising: calibrating with a direct viewing direction determined by a signal output from the sensor as a reference direction.

  (13) A calibration program for causing a computer of a display device used by being mounted on a user's head to have a display unit, the display device including at least the position of the user and a predetermined reference position An information acquisition unit that acquires real space information about the sensor, a sensor that outputs a signal according to the movement of the user's head, a calibration unit that calibrates a direct viewing direction determined by a signal output by the sensor, and the information A display control unit that displays an AR image on the display unit based on the real space information acquired by the acquisition unit and a direct viewing direction determined by a signal output from the sensor. a) The display control unit uses the real space information acquired by the information acquisition unit. (B) displaying a calibration AR image at a position of the display unit corresponding to the predetermined reference position; and (b) the calibration unit overlaps the calibration AR image at the predetermined reference position. A calibration program for causing the computer to execute a step of calibrating with a direct view direction determined by a signal output from the sensor as a reference direction when an input indicating that the image is visible is received from the user.

  According to the present invention, the orientation of the user's face (direct viewing direction) is calibrated using at least real space information regarding the user's position and a predetermined reference position. Therefore, the user's position and face orientation can be specified more accurately than before without using a plurality of sensors, and a low cost, light weight, low power consumption display device or the like can be provided.

It is a block diagram which shows the example of whole structure of the display system which concerns on this embodiment. (A) It is a top view of the inside of a movie theater where a screening system is arranged. (B) It is a perspective view of the inside of a movie theater where a screening system is arranged. It is a block diagram which shows the hardware structural example of an information provision server. It is a figure which shows the example of an external appearance of a display apparatus. It is a block diagram which shows the hardware structural example of a display apparatus. It is a block diagram which shows the function structural example of a screening system and a display apparatus. It is a figure which shows the example of the schematic data structure of real space information. It is a figure for demonstrating the target object in the movie theater part where position information is given as real space information. It is a figure which shows the example of the schematic data structure of AR image information. It is a flowchart which shows the procedure of the projection process performed in a screening apparatus. It is a figure which shows the example of the projection content for the calibration projected on the screen. It is a flowchart which shows the procedure of the calibration process performed in a display apparatus. (A) It is a figure for demonstrating the production | generation method of AR image for a calibration. (B) It is a figure which shows the example of the AR image for calibration displayed on the display of a display apparatus. It is a figure which shows a state when AR image for a calibration cannot be seen overlapping with the projection content C. It is a figure which shows a state when the AR image for calibration appears to overlap with the projection content C. It is a flowchart which shows the procedure of the information provision process performed in an information provision server. It is a figure for demonstrating the production | generation method of AR image at the time of a screening process. (A) It is a figure which shows the modification of AR image for calibration. (B) It is a figure which shows the state of a user's gaze direction when calibrating using the AR image for calibration of FIG. 18 (A).

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation, and may be different from the actual ratios.

  FIG. 1 is a block diagram illustrating an example of the overall configuration of a display system 1 according to the present embodiment. FIG. 2A is a top view of the movie theater where the screening system 100 is arranged. FIG. 2B is a perspective view of the movie theater where the screening system 100 is arranged. FIG. 3 is a block diagram illustrating a hardware configuration example of the information providing server 140. FIG. 4 is a diagram illustrating an appearance example of the display device 200. FIG. 5 is a block diagram illustrating a hardware configuration example of the display device 200. Hereinafter, a schematic configuration of the display system 1, particularly a hardware configuration, will be described with reference to FIGS. 1 to 5.

[Display system 1 (hardware configuration)]
(1) Overall Configuration As shown in FIG. 1, the display system 1 includes a screening system 100 and one or more display devices 200. However, the types and the number of devices constituting the display system 1 are not limited to the example shown in FIG.

  The screening system 100 is arranged in a movie theater as in the example shown in FIGS. In the example shown in FIG. 2A, the screening system 100 includes a seat 110, a screen 120, a screening device 130, and an information providing server 140.

  The seat 110 is a chair for a user who views a video such as a movie projected on the screen 120 to sit. In the movie theater example, a plurality of seats 110 are arranged side by side, and an identification number is assigned to each seat 110.

  The screen 120 is a large plane that displays the image projected from the screening device 130. In the present embodiment, as in the example shown in FIG. 2A, a front type screen that projects an image from behind the seat 110 on which the user is seated toward the screen 120 is used. However, the present invention is not limited to this, and a rear type screen that projects an image toward the back of the screen 120 may be used.

  The screening device 130 is a device that projects an image on the screen 120. The screening device 130 includes, for example, a screening server (not shown), a control device (not shown), a projector (not shown), a speaker (not shown), a storage device (not shown), and a communication device (not shown). ). Here, the screening server is a device that plays back a digital cinema. The control device is a general computer that controls the screening server. The projector projects the reproduced video on the screen 120, and the speaker outputs the reproduced audio. The storage device is a device (for example, a hard disk) that stores data in a format called a digital cinema package (DCP). The communication device is a device for communicating with the display device 200 via a wireless LAN or the like. Thus, a projector for digital cinema is used for the screening device 130 of the present embodiment.

  The information providing server 140 is an information processing device (computer) that provides real space information in the movie theater in response to an inquiry from the display device 200. The detailed configuration of the information providing server 140 and the details of the real space information will be described later.

  The display device 200 is a head mounted display (HMD) device that is used by being worn on the head of a user (for example, a movie viewer). On the display device 200, the AR image is superimposed and displayed in the actual environment (the direction in which the user is looking). For example, a subtitle AR image is superimposed on the lower wall of the screen 120 and displayed. In addition, when specifying the orientation of the user's face (ie, the direct viewing direction), it is necessary to determine a reference direction (hereinafter referred to as “reference direction”), and the reference direction is calibrated. The AR image used to do this may be displayed. Hereinafter, the AR image used for calibrating the orientation of the user's face is referred to as an AR image for calibration.

  The information providing server 140 and the display device 200 of the screening system 100 are connected to each other via a communication network so that they can communicate with each other. Here, the communication network includes a computer network such as a wireless LAN (Local Area Network).

(2) Information providing server 140 (hardware configuration)
Next, the hardware configuration of the information providing server 140 will be described.

  As shown in FIG. 3, the information providing server 140 includes a control device 141, a display 142, an input device 143, and a communication device 144, which have signal lines (such as a bus) 145 for exchanging signals. Are connected to each other.

  The control device 141 controls the entire information providing server 140. For example, the control device 141 includes a CPU (Central Processing Unit) 146, a memory 147, and a storage 148.

  The CPU 146 is a control circuit composed of a multi-core processor or the like that executes control of the above-described units and various arithmetic processes according to a program. Each function of the information providing server 140 is executed by the CPU 146 as a corresponding program. It is demonstrated by.

  The memory 147 is a high-speed accessible main storage device that temporarily stores programs and data as a work area. As the memory 147, for example, a DRAM (Dynamic Random Access Memory), an SDRAM (Synchronous Dynamic Access Memory), an SRAM (Static Random Access Memory), or the like is adopted.

  The storage 148 is a large-capacity auxiliary storage device that stores various programs including an operating system and various data. As the storage 148, for example, a hard disk, a solid state drive, a flash memory, a ROM, or the like is employed.

  The display 142 displays various data such as information necessary for operating the information providing server 140 and setting information. As the display 142, for example, a liquid crystal display, an organic EL (Electro-Luminescence), or the like is employed.

  The input device 143 inputs various data according to a user instruction. For the input device 143, for example, a pointing device such as a keyboard and a mouse is employed.

  The communication device 144 is a device for communicating with the display device 200 via a wireless LAN or the like.

(3) Display device 200 (hardware configuration)
Next, the hardware configuration of the display device 200 will be described.

  As shown in FIG. 4, the display device 200 has a structure simulating eyeglasses for correcting vision. That is, the display device 200 includes a pair of left and right temples L1 and L2, a bridge B, and a pair of left and right transparent members G1 and G2.

  Temples L1 and L2 are, for example, long bar-shaped members made of an elastic material. Each of the temples L1 and L2 is provided with an ear hook portion that is hung on the user's ear, and the transparent members G1 and G2 are fixed to the other end portion. In addition, a control unit 210 is mounted in the vicinity of the ear hook portion that is hung on one ear of the user.

  The bridge B is a short bar-shaped member for connecting the pair of left and right transparent members G1 and G2 to each other. Transparent members G1 and G2 are fixed to both ends of the bridge B. The pair of left and right transparent members G1 and G2 are held at a predetermined interval.

  The transparent members G1 and G2 are transparent materials (glass, plastic, film, etc.) that can transmit light from the outside world to the user's eyes so that the user wearing the display device 200 can observe the outside world. It is formed by.

  In the present embodiment, an imaging device 220 is provided above the transparent member G1 corresponding to the user's right eye, and a display 230 is provided so as to overlap the transparent member G1.

  The imaging device 220 images the external world in the optical axis direction of the camera lens. The imaging device 220 is fixedly held with respect to the display device 200 so that the optical axis of the camera lens substantially coincides with the direction of the user's line of sight. As a result, the imaging device 220 can take an image of the user's front visual field. For example, the imaging device 220 is a digital camera or a video camera provided with an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

  The display 230 displays various contents such as a predetermined image (video) on the transparent members G1 and G2. The display 230 may display an image (AR image) that is additionally superimposed on light from the outside. For example, the display 230 may be displayed as if an image including information such as movie subtitles exists near a specific object (for example, the screen 120) that exists in the direction of the user's line of sight. The display 230 may be a binocular type instead of the monocular type as shown in FIG.

  Although not shown in FIG. 4, an inertial sensor 240 is provided in the vicinity of the imaging device 220. The inertial sensor 240 is a sensor that outputs a signal corresponding to the movement of the user's head. For example, the inertial sensor 240 is configured by an acceleration sensor, a gyro sensor, or the like. The inertial sensor 240 may have any form, name, or structure as long as it can output a signal corresponding to the movement of the user's head.

  The control unit 210 has a configuration necessary for controlling the imaging device 220, the display 230, and the inertial sensor 240 to function as a head-mounted display device as a whole.

(Specific configuration of the control unit 210)
Next, a specific configuration of the control unit 210 will be described.

  As illustrated in FIG. 5, the control unit 210 includes a CPU 211, a memory 212, a storage 213, an input device 214, a display controller 215, and a communication device 216.

  The CPU 211 is a control circuit configured by a processor or the like that executes control of the above-described units and various arithmetic processes according to a program. Each function of the display device 200 is exhibited by the CPU 211 executing a corresponding program. The

  The memory 212 is a high-speed accessible main storage device that temporarily stores programs and data as a work area. For the memory 212, for example, DRAM, SDRAM, SRAM or the like is employed.

  The storage 213 is a large-capacity auxiliary storage device that stores various programs including an operating system and various data. For example, a flash memory or the like is employed as the storage 213.

  The input device 214 includes physical keys and buttons for inputting instructions such as turning on the power of the display device 200. An external controller (for example, a smartphone, a remote controller, etc.) may be used as the input device 214.

  The display controller 215 causes the display 230 to display an AR image including information such as movie subtitles and an AR image for calibration. For example, the display controller 215 reads an AR image from the memory 212 at a predetermined cycle, converts it into a signal for input to the display 230, and generates a pulse signal such as a horizontal synchronization signal and a vertical synchronization signal.

  The communication device 216 is a device for communicating with the screening device 130, the information providing server 140, and the like via a wireless LAN or the like.

[Display system 1 (functional configuration)]
The display system 1 having the hardware configuration as described above has the following functional configuration.

  FIG. 6 is a block diagram illustrating a functional configuration example of the screening system 100 and the display device 200.

(1) Screening device 130 (functional configuration) of screening system 100
As shown in FIG. 6, the screening device 130 includes a projection control unit 310 and a projection content storage unit 320.

  The projection control unit 310 controls the projection of the image on the screen 120. Specifically, the projection control unit 310 instructs the screening server to read out and reproduce the projection content stored in the projection content storage unit 320 at an appropriate timing. Based on this instruction, the video reproduced on the screening server is output to a projector (not shown), and the reproduced audio is output to a speaker (not shown). In addition, captions (data) reproduced on the screening server are transmitted to each display device 200 via the communication device. In this way, a digital cinema is shown in the movie hall.

  The projection content storage unit 320 stores the projection content projected on the screen 120. Here, the projection content includes data in a format called a digital cinema package (DCP), data projected on the screen 120 when the display device 200 is in the calibration mode, and the like. Here, the calibration mode refers to a state in which calibration processing is being executed in the display device 200.

  In addition, in the control device (not shown) of the screening device 130, the projection control unit 310 causes a CPU (not shown) to read a program installed in a storage (not shown) into a memory (not shown) and execute it. Realized. The projection content storage unit 320 is realized by a storage device (not shown) such as a hard disk.

(2) Information providing server 140 of screening system 100 (functional configuration)
As shown in FIG. 6, the information providing server 140 of the screening system 100 includes an information providing unit 410 and a real space information storage unit 420.

  The information providing unit 410 provides real space information 421 in the movie theater to the display device 200 in response to the inquiry. Specifically, when the information providing unit 410 receives an inquiry from the display device 200, the information providing unit 410 selects the real space information necessary for the display device 200 from all the real space information 421 stored in the real space information storage unit 420. The spatial information 421 is read and transmitted via the communication device 144.

  The real space information storage unit 420 stores real space information 421 in the movie theater. For example, the real space information 421 includes three-dimensional coordinates indicating the position of the user, three-dimensional coordinates indicating the position of the fixed specific object, and three-dimensional coordinates indicating the position of the projection content.

  Here, the user's position is specifically the position of the seat 110 where the user is seated and the estimated face position. The position (identification number or the like) of the seat 110 is held in the display device 200 as will be described later.

  Further, the position of the specific object specifically refers to the positions of the upper right corner and the lower left corner of the screen 120. The midpoint of the upper right corner and the lower left corner of the screen 120, that is, the center position of the screen 120 is hereinafter referred to as “predetermined reference position S”. Hereinafter, the projection content projected onto the screen 120 in the calibration mode is referred to as calibration projection content. Further, the predetermined reference position S is not limited to this, and may be any position as long as it is a position of an object that can be a target in the direction in which the user is looking.

  Further, the position of the projection content specifically refers to the positions of the upper right corner and the lower left corner of the calibration projection content projected on the screen 120.

(Details of real space information 421)
FIG. 7 is a diagram illustrating an example of a schematic data structure of the real space information 421. FIG. 8 is a diagram for explaining an object in a movie theater where position information is given as real space information 421.

  As shown in FIG. 7, the real space information storage unit 420 stores a plurality of real space information 421 for each identification number 422 of the seat 110 arranged in the movie theater. Each real space information 421 is associated with the name 423 of the object in the movie theater where the position information is given, and the position information 424.

  The objects in the movie hall where the position information is given include at least a user who watches the movie, the screen 120, and projection content. Information that can identify these objects is registered as a name 423.

  In the position information 424, coordinates indicating the position of the user, coordinates indicating the position of the screen 120, and coordinates indicating the position of the projection content are registered.

  Here, as shown in FIG. 8, the position information 424 corresponding to the position P of the user is the position of the seat 110 where the user is seated, and the three-dimensional coordinates (Px, Py, Pz) are registered. Note that an XYZ coordinate system having a point O (for example, a point at the lower left corner) as an origin is set in the space in the movie room. The three-dimensional coordinates (Px, Py, Pz) are values that are uniquely determined by the associated identification number 422.

  The position information 424 corresponding to the position of the screen 120 includes three-dimensional coordinates (Ax, Ay, Az) indicating the position A of the upper right corner of the screen 120 and three-dimensional coordinates indicating the position B of the lower left corner of the screen 120. (Bx, By, Bz) are registered. The three-dimensional coordinates (Ax, Ay, Az) and (Bx, By, Bz) are values common to all the real space information 421, respectively.

  As the position information 424 corresponding to the position of the projection content C, three-dimensional coordinates (Mx, My, Mz) indicating the position M of the upper right corner of the projection content C and the position N of the lower left corner of the projection content C are indicated. Three-dimensional coordinates (Nx, Ny, Nz) are registered. The three-dimensional coordinates (Mx, My, Mz) and (Nx, Ny, Nz) are values common to all the real space information 421, respectively.

  Note that the information providing unit 410 is realized in the information providing server 140 by the CPU 146 reading the program installed in the storage 148 into the memory 147 and executing it. The real space information storage unit 420 is realized by the storage 148 or the memory 147.

(3) Display device 200 (functional configuration)
As shown in FIG. 6, the control unit 210 of the display device 200 includes a display control unit 510, an input reception unit 520, an information acquisition unit 530, a direct view direction identification unit 540, an AR image generation unit 550, and an AR image storage. Part 560 and a calibration part 570.

  The display control unit 510 outputs an instruction for displaying various images on the display 230 to the display controller 215. For example, the display control unit 510 displays information such as movie subtitles based on the real space information 421 acquired from the information providing server 140 and the user's face orientation (direct viewing direction) determined by a signal output from the inertial sensor 240. The included AR image is displayed on the display 230. In the calibration mode, the display control unit 510 causes the display 230 to display an AR image for calibration.

  The input receiving unit 520 receives input from the user. For example, the input reception unit 520 specifies various instructions such as an instruction to start calibration based on a user operation performed on the input device 214.

  The information acquisition unit 530 acquires real space information 421 related to at least the position P of the user and the predetermined reference position S. Specifically, the information acquisition unit 530 transmits inquiry data to the information providing server 140 via the communication device 216, and receives the real space information 421 provided from the information providing server 140 in response thereto. To do.

  The direct viewing direction identification unit 540 identifies the orientation of the user's face (that is, the direct viewing direction) based on the signal output from the inertial sensor 240. Specifically, the direct-viewing direction specifying unit 540 detects the sensor value (registered value) of the inertial sensor 240 when the user is facing the face in a reference direction (reference direction S described later) that has been determined in advance, The sensor values of the inertial sensors 240 are compared, and the direction of the current user's face is specified from the amount of change.

  The AR image generation unit 550 generates an AR image that looks as if the AR image exists in the actual environment. For example, an AR image including information such as movie subtitles and an AR image for calibration are generated. In order to make the user feel as if the AR image exists in the actual environment, the display position and size of the AR image are determined according to the user's position P and the face direction (ie, the direct viewing direction). What is necessary is just to change a shape etc. A specific method for generating the AR image will be described later.

  The AR image storage unit 560 stores the AR image generated by the AR image generation unit 550. In addition, the AR image storage unit 560 stores subtitle data that is the original data of the AR image generated by the AR image generation unit 550. At the same time, the AR image storage unit 560 also stores information related to the AR image for calibration (hereinafter referred to as “AR image information 561”).

(Details of AR image information 561)
FIG. 9 is a diagram illustrating an example of a schematic data structure of the AR image information 561. As shown in FIG. 9, the AR image information 561 is associated with an AR image file name 562 and AR image position information 563.

  The file name 562 is data indicating the file name of the AR image stored in the AR image storage unit 560. For example, data such as “001” and “002” is registered.

  The position information 563 is data indicating the display position of the AR image, and two-dimensional coordinates indicating the position in the display 230 of the display device 200 are registered. Specifically, a two-dimensional coordinate indicating the display position of the head or center of the AR image is registered.

  Returning to FIG. 6, the calibration unit 570 calibrates the face direction (direct viewing direction) of the user determined by the signal output from the inertial sensor 240. Specifically, the calibration unit 570 corrects the above-described reference direction so that an error does not occur in the user's face orientation (direct viewing direction) determined by a signal output from the inertial sensor 240.

  Note that the display control unit 510, the input reception unit 520, the information acquisition unit 530, the direct-viewing direction identification unit 540, the AR image generation unit 550, and the calibration unit 570 are respectively installed in the storage 213 in the display device 200. This is realized by reading a program stored in the memory 212 and executing it. However, the present invention is not limited to this, and each of the functional units 510 to 550 and 570 may be realized by hardware such as an ASIC. The AR image storage unit 560 is realized by the storage 213 or the memory 212.

[Operation of display system 1]
Next, a characteristic operation of the display system 1 according to the present embodiment will be described.

(1) Projection Process FIG. 10 is a flowchart showing the procedure of the projection process executed in the screening device 130.

  The screening device 130 starts the projection process shown in FIG. 10 at the timing when the power is turned on. However, the timing of starting the projection process is not limited to this, and an operation for starting the projection process or an operation for starting the calibration mode is performed on the input device (keyboard, mouse, etc.). Sometimes, the projection process may be started.

(Step S101)
When the projection process is started, the screening device 130 functions as the projection control unit 310 and acquires the projection content for calibration from the projection content storage unit 320. Specifically, the projection control unit 310 reads the projection content for calibration from the projection content storage unit 320, transmits it to a screening server (not shown), and instructs playback.

(Step S102)
The screening device 130 reproduces the projection content for calibration and projects it onto the screen 120. Specifically, based on an instruction from the projection control unit 310, a screening server (not shown) reproduces the projection content for calibration and outputs the reproduced video from the projector.

  FIG. 11 is a diagram showing an example of the projection content C for calibration projected on the screen 120. An image including a graphic having a predetermined shape (in the example shown by a thick line in FIG. 11, an image including a rectangular graphic) is projected on the center of the screen 120 as the projection content C for calibration.

(Step S103)
The screening device 130 functions as the projection control unit 310 and determines whether or not it is time to end projection of the projection content C for calibration. For example, when a predetermined time has elapsed since the projection of the projection content C for calibration has been started, the screening device 130 determines that the projection is to end (step S103: YES), and performs projection processing. Exit. On the other hand, if the predetermined time has not elapsed since the projection of the projection content C for calibration has started, the screening device 130 determines that it is not time to end the projection (step S103: NO). ), The process returns to step S101.

  By executing the above projection processing in the screening device 130, the projection content C for calibration can be projected on the screen 120 for a predetermined time (for example, 30 minutes before the start of movie screening). While the projection content C for calibration is projected on the screen 120, the user can calibrate the reference direction of the user's face direction (direct viewing direction) using the display device 200 while sitting on the seat 110. .

(2) Calibration Process FIG. 12 is a flowchart illustrating a calibration process performed in the display device 200.

  While the projection content C for calibration is projected on the screen 120, the display device 200 starts the calibration process shown in FIG. 12 at the timing when the power is turned on. However, the timing for starting the calibration process is not limited to this, and the calibration process may be started when an operation for starting the calibration process is performed on the input device 214. In addition, before starting the movie showing on the screen 120, based on a start signal from the information providing server 140 or the like, calibration is simultaneously performed for a plurality of users who are seated on the seat 110 with the display device 200 mounted. You may make it start a process. Alternatively, the calibration process may be started periodically at a constant cycle. The calibration process is performed in a state where the user is seated on the seat 110 of the screening system 100.

(Step S201)
When the calibration process is started, the display device 200 functions as the information acquisition unit 530 and acquires the real space information 421 in the movie theater from the information providing server 140.

  Specifically, the display device 200 transmits inquiry data to the information providing server 140 via the communication device 216. Here, the inquiry data includes an identification number assigned to the seat 110 where the user wearing the display device 200 is seated. The identification number of the seat 110 may be data registered in the display device 200 by manual input when the user sits on the seat 110, or is acquired from the seat 110 by predetermined communication such as short-range wireless communication and displayed. Data registered in the apparatus 200 may be used.

  The display device 200 can acquire the real space information 421 in the movie theater by receiving the real space information 421 transmitted from the information providing server 140 according to the inquiry data as described above via the communication device 216. . The details of the information providing process in which the information providing server 140 provides the real space information 421 to the display device 200 will be described later.

(Step S202)
After acquiring the real space information 421, the display device 200 functions as the direct-viewing direction specifying unit 540, and determines the positional relationship between the position P of the user wearing the display device 200 on the head and the predetermined reference position S. Identify.

For example, the display device 200 uses the center position of the screen 120 as the predetermined reference position S, and the distance to the predetermined reference position S when viewed from the position P of the user wearing the display device 200 on the head, and Specify the direction. Specifically, the display device 200 uses, as the reference position S, the midpoint between the position A of the upper right corner and the position B of the lower left corner of the screen 120 included in the real space information 421 acquired in step S201. calculate. Here, the three-dimensional coordinates (Sx, Sy, Sz) of the reference position S are calculated by the calculation formula of (Sx, Sy, Sz) = ((Ax + Bx) / 2, (Ay + By) / 2, (Az + Bz) / 2). Calculated. Thereafter, the display device 200 calculates the distance from the user position P to the reference position S by calculating {(Sx−Px) ² + (Sy−Py) ² + (Sz−Pz) ² } ^(1/2). calculate. The display device 200 calculates a direction vector from the user position P to the reference position S by calculating (Sx−Px, Sy−Py, Sz−Pz).

(Step S203)
The display device 200 functions as the AR image generation unit 550 and generates an AR image for calibration. The AR image for calibration includes a figure having a shape corresponding to the figure of the projection content C projected on the screen 120.

  FIG. 13A is a diagram for explaining a method for generating an AR image for calibration. FIG. 13B is a diagram illustrating an example of an AR image for calibration displayed on the display 230 of the display device 200.

  As illustrated in FIG. 13A, the display device 200 assumes that the user's face is directed in the reference direction S. Here, the reference direction S refers to the direction of a predetermined reference position S viewed from the user position P. When the face of the user is directed in the reference direction S, the display device 200 starts from a figure that appears to overlap the figure of the projection content C (rectangular in the illustrated example) projected onto the screen 120 by the above-described projection processing. An AR image for calibration is generated. For this purpose, the display position, size, shape, etc. of the AR image for calibration are determined according to the distance R from the user position P to the reference position S and the inclination of the reference direction S with respect to the projection surface of the screen 120. do it. However, the inclination of the reference direction S with respect to the projection surface of the screen 120 is the inner product of the direction vector (Sx−Px, Sy−Py, Sz−Pz) calculated in step S202 and the normal vector of the plane of the screen 120, or the like. Required to use. As a result, an AR image for calibration composed of a figure having a shape (rectangle having vertices K1, K2, K3, and K4) as shown in the example of FIG. 13B is generated. Furthermore, as in the example shown in FIG. 13B, in order to prompt the user to perform an appropriate operation method, a character string such as “* Please overlay the rectangle projected on the screen” is displayed on the display 230. May be.

  After that, the display device 200 stores the information related to the calibration AR image (for example, the AR image information 561 shown in FIG. 9) in the AR image storage unit 560 together with the generated calibration AR image.

(Step S204)
The display device 200 functions as the display control unit 510, and displays the calibration AR image generated in step S203 on the display 230.

  Specifically, the display device 200 reads the AR image for calibration stored in the AR image storage unit 560 and the AR image information 561 associated therewith. Then, the display device 200 displays an AR image for calibration on the display 230 according to the content of the read AR image information 561. For example, in the example of the top record in FIG. 9, the calibration AR image with the file name “001” is read and displayed at the position (G1x, G1y) on the display 230.

(Step S205)
The display device 200 functions as the input receiving unit 520, and a graphic included in the AR image for calibration displayed on the display 230 appears to overlap with the graphic of the projection content C projected on the screen 120 from the user. Is received. For example, the display device 200 considers that the input is accepted when a predetermined button provided in the input device 214 is pressed.

  FIG. 14 is a diagram illustrating a state when the AR image for calibration cannot be seen overlapping the projection content C. As in the example illustrated in FIG. 14, when the user's actual face direction (direct viewing direction) T does not match the reference direction S, the calibration AR image and the projection content C do not overlap each other. This is because the AR image for calibration is generated in step S203 on the assumption that the user faces his face in the reference direction S.

  In this case, the user does not press a predetermined button provided on the input device 214, and continues the operation of moving the head until the AR image for calibration and the projection content C appear to overlap each other. While the predetermined button is not pressed, the display device 200 determines that the AR image for calibration and the projection content C do not overlap (step S205: NO) and stands by.

  FIG. 15 is a diagram illustrating a state when the AR image for calibration appears to overlap the projection content C. When the user's actual face direction (direct viewing direction) T matches the reference direction S as in the example shown in FIG. 15, the calibration AR image and the projected content C appear to overlap.

  In this case, the user presses a predetermined button provided on the input device 214. When the predetermined button is pressed, the display device 200 determines that the calibration AR image and the projection content C overlap (step S205: YES), and advances the processing to step S206.

(Step S206)
The display device 200 functions as the calibration unit 570, and sets (calibrates) a reference direction related to the orientation (direct viewing direction) of the user's face. Specifically, the display device 200 outputs a signal output from the inertial sensor 240 in the reference direction S when receiving an input indicating that the calibration AR image appears to overlap the projection content C in step S205. The value is registered in the memory 212 as a sensor value when the user is facing his face.

  Thereafter, the display device 200 ends the calibration process.

  By executing the above calibration processing in the display device 200, the reference direction regarding the orientation of the user's face (direct viewing direction) is determined using at least the real space information 421 regarding the user position P and the predetermined reference position S. ing. Therefore, it is possible to determine the reference direction related to the user's face orientation (direct viewing direction) without using a plurality of sensors.

  Further, since the position P of the user is the position of the seat 110 where the user is seated, the reference direction regarding the user's face direction (direct viewing direction) can be accurately determined regardless of which seat 110 the user is seated on. .

(3) Information Providing Process FIG. 16 is a flowchart showing the procedure of the information providing process executed in the information providing server 140.

  The information providing server 140 starts the information providing process shown in FIG. 16 at the timing when the power is turned on. However, the timing for starting the information providing process is not limited to this, and the information providing process may be started when an operation for starting the information providing process is performed on the input device 143.

(Step S301)
When the information providing process is started, the information providing server 140 functions as the information providing unit 410 and determines whether inquiry data from the display device 200 has been received. The inquiry data is data transmitted from the display device 200 in step S201 of the calibration process described above.

  The information providing server 140 stands by when the inquiry data from the display device 200 is not received (step S301: NO). On the other hand, when the information providing server 140 receives the inquiry data from the display device 200 via the communication device 144 (step S301: YES), the information providing server 140 advances the process to step S302.

(Step S302)
The information providing server 140 functions as the information providing unit 410, and extracts the real space information 421 having the identification number included in the inquiry data from the real space information storage unit 420 (FIG. 7). Then, the information providing server 140 transmits the extracted real space information 421 to the display device 200 via the communication device 144.

  Thereafter, the information providing server 140 ends the information providing process.

  By executing the above information provision processing in the information provision server 140, it is possible to provide real space information 421 necessary for calibration in response to an inquiry from the display device 200.

(4) Screening Process After the reference direction regarding the user's face direction (direct viewing direction) is accurately determined on the display device 200 by the above-described calibration processing, the screening device 130 projects an image such as a movie on the screen 120. To do. At the same time, the screening device 130 outputs audio from the speaker and transmits subtitles (data) via the communication device.

  While the image is projected on the screen 120, the display device 200 receives the subtitle (data) transmitted from the screening server of the screening device 130, and it appears as if the subtitle exists under the screen 120 or the like. The AR image is displayed on the display 230. For this purpose, the following processes (a) to (c) may be performed.

  (A) First, the display device 200 functions as the calibration unit 570, and the direction of the user's current face (direct viewing direction) from the signal output from the inertial sensor 240 with reference to the reference direction defined in step S206. ) Specify T. In other words, the display device 200 is based on the amount of change in the signal (sensor value) output from the inertial sensor 240 from the state in which the face is directed in the reference direction to the movement of the head in the current face direction T. The current face direction (direct viewing direction) T is specified.

  (B) The display device 200 functions as the AR image generation unit 550 and includes subtitles to be superimposed on the video projected on the screen 120 according to the current user's current face direction (direct viewing direction) T. An AR image is generated.

  FIG. 17 is a diagram for explaining a method of generating an AR image during the screening process. As shown in FIG. 17, the display device 200 generates an AR image as if the subtitles exist below the screen 120 when the current user's face orientation T is identified. Specifically, the subtitle display position, size, shape, and the like may be determined according to the distance R from the user position P to the reference position S and the current face orientation T of the user.

  (C) The display device 200 functions as the display control unit 510, and displays the generated caption AR image on the display 230 according to the determined display position, size, and shape.

  By performing the above-described screening process in the display device 200, it is possible to superimpose a caption AR image on a video such as a movie projected on the screen 120. Further, since the user's current face direction T is specified with reference to the reference direction calibrated by the calibration process, the AR image can be displayed at an accurate position as expected. And since the direction T of the user's current face is specified without using a plurality of sensors, a low-cost, lightweight, low power consumption display device and the like can be provided.

  Each processing unit in each flowchart described above is divided according to the main processing contents in order to facilitate understanding of the display system 1. The invention of the present application is not limited by the method of classification of the processing steps and the names thereof. The processing performed in the display system 1 can be divided into more processing steps. One processing step may execute more processes.

<Modification>
In addition, said embodiment intends to illustrate the summary of this invention, and does not limit this invention. Many alternatives, modifications, and variations will be apparent to those skilled in the art.

  For example, in the above embodiment, an image including a rectangular figure is projected on the screen 120 as the projection content C for calibration. However, the present invention is not limited to this, and the figure of the projection content C for calibration may include any shape figure. The projection content C for calibration may include a moving image.

  In the above-described embodiment, the calibration AR image displayed on the display 230 of the display device 200 includes a square figure corresponding to the calibration projection content C. However, the present invention is not limited to this.

  FIG. 18A is a diagram illustrating a modification of the AR image for calibration. As shown in FIG. 18A, the AR image for calibration may include a circular figure.

  FIG. 18B is a diagram illustrating a state in the user's line-of-sight direction when calibration is performed using the calibration AR image in FIG. As shown in FIG. 18B, when the circles included in the calibration AR image displayed on the display 230 appear to overlap so as to surround the rectangle of the projection content C projected on the screen 120, You may receive the input of said step S205.

  In the above embodiment, the position of the screen 120 (specifically, the center position) is set as the predetermined reference position S that can be a target in the direction in which the user is looking. However, the present invention is not limited to this, and an arbitrary position may be set as the reference position S. For example, the predetermined reference position S may be a position of an object having a predetermined shape. In this case, the AR image for calibration only needs to include a figure having a shape corresponding to the shape of the object.

  In the above embodiment, the position P of the user is the position of the seat 110 where the user is seated. However, the present invention is not limited to this, and the user position P may be specified by any method as long as the user position P is specified.

  In the above embodiment, the information providing server 140 is a server independent of the display device 200 and the screening device 130. However, the present invention is not limited to this, and the information providing server 140 may be integrated with the screening device 130. The real space information 421 may be stored in advance in the storage 213 of the display device 200 or the like. In this case, the display device 200 functions as the information acquisition unit 530 and acquires the real space information 421 from the storage 213 in step S201.

  In step S205 of the above embodiment, the user's input is accepted by pressing a predetermined button provided in the input device 214. However, the present invention is not limited to this, and the user's input may be received by allowing the user's head to stand still for a certain period of time.

  Moreover, the said embodiment demonstrated the example in which the display system 1 is used in a movie theater. However, the present invention is not limited to use in a movie theater, and can be used in any scene as long as real space information 421 can be obtained.

  In the embodiment and the modification, the display device 200 is a see-through type head mounted display device. However, the present invention is not limited to this, and the display device 200 may be a non-transmissive head mounted display device.

  In the above-described embodiment, an image that is additionally superimposed on the actual environment or the like is referred to as an “AR image”. However, the image is not limited to this name, and may be additionally superimposed on the actual environment or the like. Any image may be used as long as it is an image to be displayed. For example, an image using a name such as a CG image is also included.

  The configuration of the display system 1 described above is not limited to the above-described configuration because the main configuration has been described in describing the features of the above-described embodiments and modifications. Further, the configuration of the general display system 1 is not excluded.

  In addition, each functional configuration of the display system 1 described above is classified according to main processing contents in order to facilitate understanding of each functional configuration. The present invention is not limited by the way of classification and names of the constituent elements. Each functional configuration can be classified into more components according to the processing content. Moreover, it can also classify | categorize so that one component may perform more processes.

  The processing of each functional configuration of the display system 1 described above can also be realized by a dedicated hardware circuit. In this case, it may be executed by one hardware or a plurality of hardware.

  The program for operating the display system 1 may be provided by a computer-readable recording medium such as a USB memory, a flexible disk, or a CD-ROM, or may be provided online via a network such as the Internet. . In this case, the program recorded on the computer-readable recording medium is usually transferred to and stored in a memory or storage. Further, this program may be provided as, for example, a single application software, or may be incorporated in the software of each device as one function of the display system 1.

1 display system,
100 screening system,
110 seats,
120 screens,
130 Screening device,
140 Information providing server,
141 control device,
146 CPU,
147 memory,
148 storage,
142 display,
143 input device,
144 communication device,
145 signal line,
200 display device,
210 control unit,
211 CPU,
212 memory,
213 storage,
214 input devices,
215 display controller,
216 communication device,
220 imaging device,
230 display,
240 inertial sensor,
310 projection control unit,
320 projection content storage unit,
410 Information provider,
420 real space information storage unit,
421 Real space information,
422 identification number,
423 name,
424 location information,
510 display control unit,
520 input reception unit,
530 Information acquisition unit,
540 direct view direction specifying part,
550 AR image generator,
560 AR image storage unit,
561 AR image information,
562 file name,
563 location information,
570 Calibration unit.

Claims (13)

A display device having a display unit and being used on a user's head,
An information acquisition unit for acquiring real space information relating to at least the position of the user and a predetermined reference position;
A sensor that outputs a signal corresponding to the movement of the user's head;
A calibration unit for calibrating the direct viewing direction determined by the signal output from the sensor;
A display control unit that displays an AR image on the display unit based on the real space information acquired by the information acquisition unit and a direct viewing direction determined by a signal output from the sensor;
When executing calibration mode,
The display control unit displays the AR image for calibration at the position of the display unit corresponding to the predetermined reference position using the real space information acquired by the information acquisition unit,
When the calibration unit receives an input from the user indicating that the calibration AR image appears to overlap the predetermined reference position, a direct viewing direction determined by a signal output from the sensor is used as a reference direction. Display device to be calibrated. The predetermined reference position is a predetermined screen position;
The display device according to claim 1, wherein an image including a graphic having a predetermined shape is displayed on the screen.   The display device according to claim 2, wherein the AR image for calibration includes a figure having a shape corresponding to a figure displayed on the screen.   When the calibration unit receives an input from the user indicating that the graphic included in the calibration AR image appears to overlap the graphic displayed on the screen, the calibration unit outputs a signal based on a signal output from the sensor. The display device according to claim 3, wherein calibration is performed with a fixed direct view direction as a reference direction.   The display device according to claim 2, wherein the position of the user is a position of a seat on which the user is seated.   The display device according to claim 1, wherein the predetermined reference position is a position of an object having a predetermined shape.   The display device according to claim 6, wherein the calibration AR image includes a figure having a shape corresponding to the object.   The calibration unit has a direct viewing direction determined by a signal output from the sensor when receiving an input indicating that a graphic included in the calibration AR image appears to overlap the shape of the object from the user. The display device according to claim 7, wherein calibration is performed with reference to the reference direction.   The display device according to claim 1, wherein the information acquisition unit acquires from a server independent of the display device. A storage unit for storing the real space information;
The display device according to claim 1, wherein the information acquisition unit acquires the real space information from the storage unit. The user input received by the input receiving unit is:
The display device according to claim 1, wherein the display device is performed by pressing a predetermined button or by allowing the user's head to remain stationary for a predetermined time or more. A method for calibrating a display device having a display unit and being used on a user's head,
The display device
An information acquisition unit for acquiring real space information relating to at least the position of the user and a predetermined reference position;
A sensor that outputs a signal corresponding to the movement of the user's head;
A calibration unit for calibrating the direct viewing direction determined by the signal output from the sensor;
A display control unit that displays an AR image on the display unit based on the real space information acquired by the information acquisition unit and a direct viewing direction determined by a signal output from the sensor;
When executing calibration mode,
(A) The display control unit displays an AR image for calibration at the position of the display unit corresponding to the predetermined reference position using the real space information acquired by the information acquisition unit;
(B) When the calibration unit receives an input from the user indicating that the calibration AR image appears to overlap the predetermined reference position, a direct viewing direction determined by a signal output from the sensor is determined. Calibrating as a reference direction, and a method for calibrating a display device. A calibration program for causing a computer of a display device to be used by being mounted on a user's head, having a display unit,
The display device
An information acquisition unit for acquiring real space information relating to at least the position of the user and a predetermined reference position;
A sensor that outputs a signal corresponding to the movement of the user's head;
A calibration unit for calibrating the direct viewing direction determined by the signal output from the sensor;
A display control unit that displays an AR image on the display unit based on the real space information acquired by the information acquisition unit and a direct viewing direction determined by a signal output from the sensor;
When executing calibration mode,
(A) The display control unit displays an AR image for calibration at the position of the display unit corresponding to the predetermined reference position using the real space information acquired by the information acquisition unit;
(B) When the calibration unit receives an input from the user indicating that the calibration AR image appears to overlap the predetermined reference position, a direct viewing direction determined by a signal output from the sensor is determined. A calibration program for causing the computer to execute calibration as a reference direction.