JP6689679B2 - Display device, information display system, and program - Google Patents

Description

  The present invention relates to a display device, an information display system, and a program.

  Conventionally, there is known a system that detects the position of an object using a camera. For example, Patent Document 1 discloses a system that changes the image displayed on the head-mounted display by measuring the position and orientation of the head-mounted display using a camera.

JP, 2007-71782, A

  However, the conventional system has room for improvement, for example, from the viewpoint of improving convenience.

  An object of the present invention made in view of such circumstances is to provide a display device, an information display system, and a program with improved convenience.

A display device according to an embodiment of the present invention is a display device mounted on a user's head, and includes a light source, a sensor that detects a change in angle around the three axes of the head, and a first imaging device. a communication interface to obtain information about the position of the light source is specified based on second image of said light source imaged by the first image and the second image pickup device by it is shooting an image said light source, said light source A control unit that performs display control based on the position and the orientation of the head estimated based on the output of the sensor. As for the position of the light source, the position vector of the first image pickup device is a first vector, the position vector of the second image pickup device is a second vector, and the light source on the image plane of the first image pickup image from the first image pickup device is the light source. When the direction vector toward the light source is the third vector and the direction vector from the second imaging device toward the light source on the image plane of the second captured image is the fourth vector, the fourth vector and the second vector It is specified by using the fact that the cross product of the direction vector from the imaging device toward the light source with the vector represented by the first vector, the second vector, and the third vector is 0.

An information display system according to an embodiment of the present invention is an information processing system including a display device mounted on a user's head, a first imaging device, and a second imaging device . The display device includes a light source and a sensor for detecting an angle change of about three axes of the head, by the first image and the second image pickup apparatus of the first light source that is shooting the image by the imaging device A communication interface that obtains information about the position of the light source that is specified based on the captured second image of the light source, the position of the light source, and the orientation of the head that is estimated based on the output of the sensor, And a control unit that executes display control based on the above. As for the position of the light source, the position vector of the first image pickup device is a first vector, the position vector of the second image pickup device is a second vector, and the light source on the image plane of the first image pickup image from the first image pickup device is the light source. When the direction vector toward the light source is the third vector and the direction vector from the second imaging device toward the light source on the image plane of the second captured image is the fourth vector, the fourth vector and the second vector It is specified by using the fact that the cross product of the direction vector from the imaging device toward the light source with the vector represented by the first vector, the second vector, and the third vector is 0.

Further, a program according to an embodiment of the present invention includes a step of detecting a change in angle of the head around three axes by using a sensor included in the display device on a display device mounted on a user's head. When, identified on the basis of the second image captured by the first image and the second image pickup device of the light source that is an image shooting by the first image pickup apparatus, a position of the light source provided in the display device The step of acquiring information, the step of executing display control based on the position of the light source, and the orientation of the head estimated based on the output of the sensor are executed. As for the position of the light source, the position vector of the first image pickup device is a first vector, the position vector of the second image pickup device is a second vector, and the light source on the image plane of the first image pickup image from the first image pickup device is the light source. When the direction vector toward the light source is the third vector and the direction vector from the second imaging device toward the light source on the image plane of the second captured image is the fourth vector, the fourth vector and the second vector It is specified by using the fact that the cross product of the direction vector from the imaging device toward the light source with the vector represented by the first vector, the second vector, and the third vector is 0.

  According to the present invention, a display device, an information display system, and a program with improved convenience can be provided.

It is a functional block diagram showing a schematic structure of a position detection system concerning one embodiment of the present invention. It is a schematic diagram explaining the position detection principle by the position detection system of FIG. It is a functional block diagram which shows schematic structure of the information display system using the position detection system of FIG. FIG. 4 is an external perspective view showing an example of the head mounted display of FIG. 3. FIG. 4 is a sequence diagram showing an example of information display processing by the information display system of FIG. 3.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional block diagram showing a schematic configuration of a position detection system 100 according to an embodiment of the present invention. The position detection system 100 according to the present embodiment includes a plurality of image pickup devices and an information processing device 130 configured to communicate with each image pickup device. In the present embodiment, as shown in FIG. 1, the position detection system 100 includes two imaging devices, a first imaging device 110 and a second imaging device 120, as a plurality of imaging devices. The position detection system 100 may include three or more imaging devices.

  FIG. 2 is a schematic diagram illustrating the principle of position detection by the position detection system 100 of FIG. In the position detection system 100 according to the present embodiment, the first image capturing device 110 and the second image capturing device 120 capture an image of the target object P that is the target of position detection, and the information processing device 130 causes the first image capturing device 110 and the first image capturing device 110 to 2 The position of the object P is calculated based on the captured image acquired by the imaging device 120. The first imaging device 110 and the second imaging device 120 respectively image the target P from different directions, as shown in FIG. 2 as an example. In the present embodiment, the target P will be described as an infrared LED (Light Emitting Diode) that emits near-infrared rays of about 700 to 1500 nm, for example.

  Referring back to FIG. 1, the first imaging device 110 includes a control unit 111, a storage unit 112, a communication unit 113, and a first imaging unit 114. The first imaging device 110 images the target object P and transmits the captured image to the information processing device 130.

  The control unit 111 is a processor that controls and manages the entire first imaging device 110, including the functional blocks included in the first imaging device 110. The control unit 111 includes a processor such as a CPU (Central Processing Unit) that executes a program that defines a control procedure. The program executed by the processor is stored in, for example, the storage unit 112 or an external storage medium. The control unit 111 controls the imaging process performed by the first imaging unit 114, for example.

  The storage unit 112 has various memory devices, and stores various kinds of information, such as data necessary for the operation of the control unit 111, depending on the application. The storage unit 112 also has a device such as a RAM (Random Access Memory) that functions as a work memory. The storage unit 112 may store, for example, a captured image captured by the first image capturing unit 114 (hereinafter, also referred to as “first captured image”).

  The communication unit 113 performs various kinds of information transmission / reception by performing wired communication or wireless communication with the information processing device 130. For example, the first imaging device 110 transmits the first captured image to the information processing device 130 via the communication unit 113.

  The first imaging unit 114 images the object P. The first image capturing unit 114 includes an image sensor (image sensor) that captures an image projected on the element by converting light into an electric signal. In the present embodiment, the first imaging unit 114 is an infrared camera that detects near infrared light emitted by the object P.

  The second imaging device 120 includes a control unit 121, a storage unit 122, a communication unit 123, and a second imaging unit 124. The respective functions of the control unit 121, the storage unit 122, the communication unit 123, and the second imaging unit 124 are the same as those of the control unit 111, the storage unit 112, the communication unit 113, and the first imaging unit 114 of the first imaging device 110, respectively. Therefore, detailed description thereof is omitted here. In the present specification, the captured image captured by the second image capturing unit 124 is also referred to as “second captured image” below.

  The information processing device 130 includes a control unit 131, a storage unit 132, and a communication unit 133. The information processing device 130 calculates the position of the object P based on the captured images acquired by the first imaging device 110 and the second imaging device 120.

  The control unit 131 is a processor that controls and manages the entire information processing apparatus 130 including each functional block included in the information processing apparatus 130. The control unit 131 includes a processor such as a CPU that executes a program that defines a control procedure. The program executed by the processor is stored in, for example, the storage unit 132 or an external storage medium. The control unit 131 calculates the position of the detection target based on the first captured image acquired by the first imaging device 110 and the second captured image acquired by the second imaging device 120. Details of the process of calculating the position of the detection target by the control unit 131 will be described later.

  The storage unit 132 has various memory devices, and stores various kinds of information, such as data necessary for the operation of the control unit 131, depending on the application. The storage unit 132 also has a device such as a RAM that functions as a work memory. The storage unit 132 may store, for example, the first captured image and the second captured image acquired from the first imaging device 110 and the second imaging device 120, respectively. Further, the storage unit 132 may store, for example, a history of the calculated position of the object. The information processing apparatus 130 can calculate the displacement (trajectory) of the target object based on the history of the position of the target object.

  The communication unit 133 transmits and receives various types of information by performing wired communication or wireless communication with the first imaging device 110 and the second imaging device 120. For example, the information processing apparatus 130 acquires the first captured image and the second captured image from the first image capturing apparatus 110 and the second image capturing apparatus 120, respectively, via the communication unit 133.

  Next, the details of the process of calculating the position of the detection target by the information processing device 130 will be described with reference to FIG.

  In FIG. 2, the imaging range A of the first imaging device 110 and the imaging range B of the second imaging device 120 are virtually shown. That is, in FIG. 2, the imaging range A virtually represents the first captured image acquired by the first imaging device 110, and the imaging range B virtually represents the second captured image acquired by the second imaging device 120. Show. In the first captured image and the second captured image, the object P is reflected at predetermined positions on the image plane. In the first captured image and the second captured image, the coordinates of the position where the object P is reflected are determined, for example, with reference to the gazing points of the first imaging unit 114 and the second imaging unit 124, respectively.

In the present embodiment, the position vectors of the first imaging device 110 and the second imaging device 120 are indicated by vector c ₁ and vector c ₂ , respectively. Further, a direction vector from the first imaging device 110 to the object P is shown as a vector a, and a direction vector from the second imaging device 120 to the object P is shown as a vector b. The direction vector a is defined as, for example, a direction vector from the position of the image sensor included in the first imaging unit 114 toward the detection target P on the image plane of the first captured image. Similarly, the direction vector b is defined as a direction vector from the position of the image sensor included in the second image capturing unit 124 toward the object P on the image plane of the second captured image. The direction vector a and the direction vector b are calculated, for example, by the control unit 131 of the information processing device 130 based on the first captured image and the second captured image, respectively. In addition, in FIG. 2, the lengths of the direction vector a and the direction vector b are shown as an example. Further, the position vector of the object P is represented by the vector p.

The position vector p of the object P is represented by the following expression (1) using the position vector c ₁ and the direction vector a. The control unit 131 of the information processing device 130 sets the position vector p as in the following expression (1).

  In Expression (1), n is a real number, and is determined according to the distance from the sensor position of the first imaging unit 114 to the detection target P, for example. By calculating (determining) n, the position vector p is determined and the position of the object P is determined.

Here, the direction vector (here, vector d) from the second image pickup device 120 toward the object P is expressed by the following equation (2) using the position vector c ₂ of the second image pickup device 120. You can The control unit 131 calculates the direction vector d.

  Since the direction vector d shown in the above equation (2) is defined as the direction vector from the second imaging device 120 to the target P, the direction is equal to the direction vector b. Therefore, since the angle formed by the direction vector d and the direction vector b is 0 °, the magnitude of the outer product of the direction vector d and the direction vector b is 0. When this is expressed by an expression, it is expressed as the following expression (3).

  By solving the above equation (3) for n, the following equation (4) is derived.

  By substituting the above equation (4) into the above equation (1), the position vector p is expressed by the following equation (5).

In the above equation (5), the position vector c ₁ and the position vector c ₂ are determined by the positions where the first imaging device 110 and the second imaging device 120 are installed, respectively. Further, the direction vector a and the direction vector b can be calculated by the information processing apparatus 130 from the first captured image and the second captured image acquired by the first imaging device 110 and the second imaging device 120, respectively. Therefore, the information processing apparatus 130 can calculate the position vector p of the object P using the above equation (5). The control unit 131 calculates the position of the object P based on the direction vector d and the direction vector b as shown in Expressions (3) to (5). In particular, the control unit 131 calculates the position of the object P based on the value of the outer product of the direction vector d and the direction vector b, assuming that the value of the outer product is 0. In this way, the position detection system 100 can specify the position of the object P.

  Further, according to the position detection system 100 according to the present embodiment, the first imaging device 110 and the second imaging device 120 image the object P from different directions. Therefore, the position detection according to the present embodiment is compared with a system such as a stereo camera that calculates a distance using parallax based on an image captured from one direction using two cameras. The system 100 improves the accuracy of position detection.

  Next, an application example of the position detection system 100 described above will be described. In this specification, an example in which the position detection system 100 is applied to specify the position of a head mounted display (HMD) which is a display device mounted on the head of a user will be described. In the information display system to which the position detection system 100 is applied, the position of the HMD is specified by the position detection system 100 described above, and information according to the specified position is displayed on the display unit included in the HMD.

  FIG. 3 is a functional block diagram showing a schematic configuration of an information display system 200 to which the position detection system 100 is applied. As shown in FIG. 3, the information display system 200 includes an HMD 140 in addition to the first imaging device 110, the second imaging device 120, and the information processing device 130 included in the position detection system 100. The information processing device 130 and the HMD 140 are configured to be communicable via a wire or wirelessly.

  FIG. 4 is an external perspective view showing an example of the HMD 140. The HMD 140 includes a substantially hemispherical mounting unit 141 that the user can wear on the head, a display unit 142 that displays information to the user when the user wears the HMD 140, and a light source 143. In the present embodiment, the HMD 140 will be described as including two light sources 143a and 143b, as shown in FIG. When the two light sources 143a and 143b are not distinguished, they are collectively referred to as the light source 143.

  Referring again to FIG. 3, the HMD 140 includes a display unit 142, a light source 143, a control unit 144, a storage unit 145, a communication unit 146, and an angular velocity detection unit 147.

  The display unit 142 displays information to the user who wears the HMD 140. The display unit 142 has a display such as a liquid crystal display (LCD: Liquid Crystal Display) or an organic EL display (OELD: Organic Electroluminescence Display) for displaying information, and a housing that holds the display. FIG. 4 shows a so-called immersive display unit 142 that has a housing that shields the user's line of sight from the outside, but the display unit 142 is not limited to the immersive display unit. The display unit 142 may be, for example, a so-called transmissive type (see-through type) that allows the user to visually recognize the outside world via the display unit 142.

  The light source 143 emits light detected by the first imaging device 110 and the second imaging device 120, and corresponds to the object P described in FIG. The light source 143 is an infrared LED that emits near-infrared light, similar to that described in FIG. 2, for example. In the present embodiment, the two light sources 143a and 143b are provided in the mounting portion 141 so as to be arranged in the front-rear direction when the user wears the HMD 140, as shown as an example in FIG.

  The control unit 144 is a processor that controls and manages the entire HMD 140 including each functional block included in the HMD 140. The control unit 144 is composed of a processor such as a CPU that executes a program that defines a control procedure. The program executed by the processor is stored in, for example, the storage unit 145 or an external storage medium. The control unit 144 displays various information on the display unit 142 based on, for example, the position of the user and the orientation of the user's head. Specifically, the control unit 144 causes the display unit 142 to display a virtual space that follows the position of the user and the orientation of the head. The position of the user is determined by the position of the HMD 140 worn by the user, which is specified by the information processing device 130. The orientation of the user's head is determined by the orientation of the HMD 140 attached to the user's head.

  The storage unit 145 has various memory devices, and stores various kinds of information, such as data necessary for the operation of the control unit 144, depending on the application. The storage unit 145 also has a device such as a RAM that functions as a work memory. Further, the storage unit 145 stores, for example, various data regarding the virtual space displayed on the display unit 142.

  The communication unit 146 has a communication interface that transmits and receives various types of information by performing wired communication or wireless communication with the information processing device 130. For example, the HMD 140 acquires information regarding the position of the HMD 140 specified by the information processing apparatus 130 based on the first captured image and the second captured image from the information processing apparatus 130 via the communication unit 146.

  The angular velocity detection unit 147 is a sensor that detects a change in the angular velocity of the HMD 140, for example. The angular velocity detector 147 is composed of, for example, a gyro sensor. The angular velocity detection unit 147 is not limited to the gyro sensor. The angular velocity detection unit 147 only needs to be able to detect the angle change of the HMD 140. Therefore, the angular velocity detector 147 may be configured by, for example, an acceleration sensor, an angle sensor, another motion sensor, or a combination of these sensors. Information regarding the change in the angular velocity detected by the angular velocity detection unit 147 is transmitted to the control unit 144. The control unit 144 can estimate the orientation of the user's head based on the information regarding the change in angular velocity acquired from the angular velocity detection unit 147. In the present embodiment, the angular velocity detection unit 147 detects a change in angular velocity around three axes of a pitch axis (x axis), a yaw axis (y axis), and a roll axis (z axis), as shown in FIG. 4, for example. To do. Further, the control unit 144 estimates the angles of the user's head around the three axes, specifically, the pitch angle θ, the yaw angle ψ, and the roll angle φ, based on the output of the angular velocity detection unit 147.

  Next, a virtual space display process in the information display system 200 will be described with reference to FIG.

  First, the first imaging device 110 acquires the first captured image by capturing the image of the light source 143 (step S101).

  The first imaging device 110 transmits the acquired first captured image to the information processing device 130 (step S102).

  In addition, the second imaging device 120 acquires the second captured image by capturing the image of the light source 143 (step S103).

  The second imaging device 120 transmits the acquired second captured image to the information processing device 130 (step S104).

  The processes of steps S101 and S102 by the first imaging device 110 and the processes of steps S103 and S104 by the second imaging device 120 may be executed at the same time.

  When the information processing apparatus 130 acquires the first captured image and the second captured image from the first image capturing apparatus 110 and the second image capturing apparatus 120, respectively, the information processing apparatus 130 identifies the position of the HMD 140 based on the captured image acquired (step S105). ). Specifically, the information processing device 130 uses Expression (5) using, for example, the position of the first imaging device 110 and the position of the second imaging device 120, which are stored in advance in the storage unit 132, and the acquired captured image. Thus, the position of the HMD 140 is specified by calculating the position of the light source 143.

  The information processing device 130 transmits information on the specified position of the HMD 140 to the HMD 140 (step S106).

  The HMD 140 uses the angular velocity detection unit 147 to acquire information regarding changes in the angular velocity of the HMD 140 (step S107).

  Then, the HMD 140 estimates the orientation of the user's head based on the information regarding the change in the angular velocity acquired by the angular velocity detection unit 147 (step S108).

  The HMD 140 displays the virtual space on the display unit 142 based on the information regarding the position acquired from the information processing device 130 in step S106 and the head orientation estimated in step S108 (step S109).

  In this way, the information display system 200 displays the virtual space on the display unit 142 based on the position of the HMD 140 and the orientation of the user's head. In the information display system 200, the position of the HMD 140 is specified by applying the position detection system 100 described above, so that the position of the HMD 140 can be easily specified accurately.

  Further, in the information display system 200, since the HMD 140 estimates the orientation of the user's head, for example, the HMD 140 is notified of the orientation of the user's head estimated by the information processing apparatus 130 based on the first captured image and the second captured image. Compared with the configuration described above, the image delay with respect to the change in the orientation of the user's head is reduced. Therefore, the convenience of the information display system 200 is improved.

  It should be noted that the present invention is not limited to the above-described embodiment, and various modifications and changes can be made. For example, the functions and the like included in each component can be rearranged so as not to logically contradict, and a plurality of components can be combined into one or divided.

  For example, in the above-described embodiment, the position detection system 100 has been described as including the two imaging devices 110 and 120, but the position detection system 100 may include three or more imaging devices. When the position detection system 100 includes three or more image capturing devices, the information processing device 130 can specify the position of the target object P based on the captured images acquired by any two image capturing devices. In this case, even if the target P is not displayed in the images captured by some of the image capturing devices due to being blocked by an obstacle, for example, the information processing apparatus 130 may capture another image of the target P. The position of the object P can be specified using the image.

  Further, in the above application example, the case where the HMD 140 includes the two light sources 143a and 143b has been described, but the number of the light sources 143 included in the HMD 140 is not limited to two. The number of light sources 140 included in the HMD 140 may be one, or may be three or more.

  Further, in the above application example, the case where the HMD 140 includes the light source 143 has been described, but the application example is not limited to this. The light source 143 may be included in a device other than the HMD 140, and in this case, the information processing device 130 can identify the position of the device including the light source 143.

  The HMD 140 may also send information regarding the orientation of the user's head to an external device such as the information processing device 130. With this configuration, the convenience of the information display system 200 is further improved as described below.

  For example, the information processing device 130 that has received the information regarding the orientation of the user's head can display a predetermined image based on the position of the HMD 140 and the orientation of the user's head on, for example, an external display device. The predetermined image may include various images such as an image substantially the same as the image displayed on the HMD 140 and a map image showing the position and orientation of the user in the virtual space. Here, when the information display system 200 includes a plurality of HMDs 140, the information processing apparatus 130 causes the second HMD 140 to display an image displayed on the first HMD 140, and indicates the positions and orientations of a plurality of users. Map images can be displayed on a plurality of HMDs 140.

  In addition, the information processing apparatus 130 that has received the information regarding the orientation of the user's head can also predict the direction in which the user's position will be displaced next, based on the information regarding the orientation.

  In the above application example, the HMD 140 may be able to execute all the processes performed by the information processing device 130. At this time, the information display system 200 may include the first imaging device 110, the second imaging device 120, and the HMD 140. At this time, the HMD 140 is configured to be communicable with the first imaging device 110 and the second imaging device 120 via a wire or wirelessly.

  Further, an information processing device such as a computer or a mobile phone can be preferably used in order to function as the imaging devices 110 and 120, the information processing device 130, and the HMD 140 according to the above-described embodiments. Such an information processing apparatus stores a program describing the processing content for realizing each function of the image capturing apparatus 110, 120, the information processing apparatus 130, or the HMD 140 according to the embodiment in a storage unit of the information processing apparatus, and It can be realized by reading and executing the program by the processor of the processing device.

100 Position Detection System 110 First Imaging Device 111, 121, 131, 144 Control Unit 112, 122, 132, 145 Storage Unit 113, 123, 133, 146 Communication Unit 114 First Imaging Unit 120 Second Imaging Device 124 Second Imaging Part 130 Information processing device 140 Head mounted display (HMD)
141 mounting unit 142 display unit 143 light source 147 angular velocity detection unit 200 information display system

Claims (4)

A display device worn on a user's head,
A light source,
A sensor for detecting an angle change of the head around three axes,
A communication interface to obtain information about the position of the light source to be identified based on the second image of the light source imaged by the first image and the second image pickup device of the light source that is an image shooting by the first imaging device When,
A position of the light source, and a controller that performs display control based on the orientation of the head estimated based on the output of the sensor,
Equipped with
The position of the light source is
The position vector of the first imaging device is a first vector,
The position vector of the second imaging device is a second vector,
A third vector is a direction vector from the first imaging device toward the light source on the image plane of the first captured image,
When the direction vector from the second imaging device toward the light source on the image plane of the second captured image is a fourth vector,
It is specified by using the outer product of the fourth vector and the direction vector from the second imaging device toward the light source represented by the first vector, the second vector and the third vector being 0. It is is, the display device. The display device according to claim 1, wherein
If the position of the light source is c ₁ , the third vector is a, the real number is n, and the position vector of the light source is p as shown in the following equation (1), the position of the light source is: A display device specified by calculating the value of a real number n using the fact that the outer product is 0 .

An information processing system comprising: a display device mounted on a user's head; a first imaging device; and a second imaging device ,
The display device is
A light source,
A sensor for detecting an angle change of the head around three axes,
To obtain information about the position of the light source to be identified based on the second image of the light source imaged by the first image and the second image pickup device of the light source that is shooting the image by the first imaging device Communication interface,
A position of the light source, and a direction of the head estimated based on the output of the sensor, and a control unit that performs display control based on ,
The position of the light source is
The position vector of the first imaging device is a first vector,
The position vector of the second imaging device is a second vector,
A third vector is a direction vector from the first imaging device toward the light source on the image plane of the first captured image,
When the direction vector from the second imaging device toward the light source on the image plane of the second captured image is a fourth vector,
It is specified by using the outer product of the fourth vector and the direction vector from the second imaging device toward the light source represented by the first vector, the second vector and the third vector being 0. It is is, the information display system. On the display device worn on the user's head,
Detecting a change in angle around the three axes of the head using a sensor provided in the display device;
Wherein the light source provided in the display device are identified on the basis of the second image captured by the first image and the second image pickup apparatus which is an image shooting by the first imaging device, information about the position of the light source To get
A step of performing display control based on the position of the light source and the orientation of the head estimated based on the output of the sensor;
Was executed,
The position of the light source is
The position vector of the first imaging device is a first vector,
The position vector of the second imaging device is a second vector,
A third vector is a direction vector from the first imaging device toward the light source on the image plane of the first captured image,
When the direction vector from the second imaging device toward the light source on the image plane of the second captured image is a fourth vector,
It is specified by using the outer product of the fourth vector and the vector representing the direction vector from the second imaging device toward the light source represented by the first vector, the second vector and the third vector being 0. Be a program.

Images