JP2019004274A - Imaging display device - Google Patents

Abstract

To display a high quality image even when an imaging element having a relatively small number of pixels is used in head mounted display for an augmented reality device.SOLUTION: An imaging display device includes a first imaging unit that captures an image in a first direction, a second imaging unit that captures an image in a second direction, a first display unit that displays a first image generated on the basis of the image captured by the first imaging unit, a second display unit that displays a second image generated on the basis of the image captured by the second imaging unit, a sight line detection unit that detects the sight line of a user, and a drive unit that adjusts the directions of the first imaging unit and the second imaging unit according to the sight line detected by the sight line detection unit.SELECTED DRAWING: Figure 2

Description

  One embodiment of the present invention relates to a head-mounted imaging display device or the like for an augmented reality device.

  Various devices have been developed for head-mounted image display devices (head-mounted displays). In such an image display device, a configuration in which an image to be displayed is changed in accordance with a user's line of sight (movement of the eyeball) may be employed. Such an apparatus is disclosed in, for example, Patent Document 1 or Patent Document 2.

JP 2011-197736 A Japanese Patent Laid-Open No. 9-284676

  On the other hand, in recent years, an augmented reality device that displays a captured image in combination with a predetermined image or information in front of the user's eyes has been developed. In such an augmented reality device, a configuration in which an image to be displayed is changed according to a user's line of sight, like a conventional head-mounted image display device, can be adopted. However, when the conventional virtual reality device is applied to the augmented reality device as it is, there is a problem that it is necessary to use an image sensor and a wide-angle lens having a very large number of pixels.

  The present invention employs the following means in order to solve the above-described problems. In the following description, in order to facilitate understanding of the invention, reference numerals and the like in the drawings are appended in parentheses, but each component of the present invention is not limited to these appendices. It should be construed broadly to the extent that contractors can technically understand.

One means of the present invention is to
A first imaging unit (11) that captures an image in a first direction;
A second imaging unit (12) that captures an image in the second direction;
A first display unit (14) for displaying a first image generated based on an image captured by the first imaging unit;
A second display unit (15) for displaying a second image generated based on the image captured by the second imaging unit;
A line-of-sight detection unit (16) for detecting the line of sight of the user;
A drive unit (13) that adjusts the orientation of the first imaging unit (11) and the second imaging unit (12) according to the line of sight detected by the line-of-sight detection unit,
An imaging display device.

  In the imaging display device having the above-described configuration, for example, a configuration using a configuration of a wide-angle lens, an imaging element, or the like is used to perform imaging while adjusting the orientation of the imaging unit corresponding to the right eye and the left eye with the drive unit according to the user's line of sight. In comparison, an imaging device having a relatively small number of pixels can be used as an imaging unit. In addition, since the operation is in accordance with the actual eye movement, a natural image can be displayed.

In the imaging display device, preferably
The line-of-sight detection unit detects the line of sight by detecting the position of the user's eyeball.

  In the imaging display device having the above-described configuration, it is possible to adjust the orientation of an image to be captured and displayed while accurately detecting the user's line of sight.

In the imaging display device, preferably
The first direction and the second direction are directions corresponding to the directions of the left and right eyeballs of the user.

  In the imaging display device having the above configuration, by displaying two images with parallax, the image displayed on the display unit can be configured to make the user feel three-dimensional.

In the imaging display device, preferably
The first image is an image generated based on an image captured after 1 second by the first imaging unit,
The second image is an image generated based on an image captured after one second by the second imaging unit.

  The imaging display device having the above configuration can be configured to allow the user to experience augmented reality in substantially real time.

In the imaging display device, preferably
The driving unit reciprocates the first imaging unit and the second imaging unit in one direction.

  In the imaging display device having the above configuration, the first imaging unit and the second imaging unit are moved in only one direction, so that relatively simple control is possible and a relatively simple configuration can be achieved. For example, a configuration in which the first imaging unit and the second imaging unit are moved only in the lateral direction in which the movement range of the eyeball is relatively wide, and a configuration having an imaging unit that moves in accordance with the movement of the eyeball with a relatively simple configuration. It is possible to do.

FIG. 1 is an external perspective view of a head mounted display. FIG. 2 is a functional block diagram of the head mounted display. FIG. 3 is a flowchart showing the operation of the head mounted display.

  The image processing apparatus according to the present invention is an augmented reality device such as a head-mounted display, and has a configuration in which an imaging unit that captures a peripheral image that is a basis of an image displayed on a display unit is moved in accordance with a user's line of sight. This is one of the features.

Specific embodiments according to the present invention will be described according to the following configuration. However, the embodiment described below is merely an example of the present invention and does not limit the technical scope of the present invention. In addition, in each drawing, the same code | symbol is attached | subjected to the same component and the description may be abbreviate | omitted.
1. Embodiment 2. FIG. Modified example 2. Supplementary matter

<1. Embodiment>
Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an external perspective view of the head mounted display of the present embodiment. FIG. 2 is a functional block diagram of the head mounted display of the present embodiment.

  As shown in FIG. 1, the head mounted display of the present embodiment is configured to include a main body 1 and mounting tools 21 to 23. The wearing tools 22 and 23 are arranged so as to surround the periphery of the head of the user (head mounted display wearer), and the wearing tool 21 is arranged to extend from the user's forehead toward the back of the head through the top of the head. The As shown in FIGS. 1 and 2, the main body 1 of the head mounted display includes a control unit 10, a first imaging unit 11, a second imaging unit 12, a driving unit 13, a first display unit 14, and a second display unit. 15 and the line-of-sight detection unit 16.

<First imaging unit 11 and second imaging unit 12>
The first imaging unit 11 and the second imaging unit 12 include at least a lens and an imaging element, and are configured to be able to acquire an image of a subject. As shown in FIG. 1, the first imaging unit 11 and the second imaging unit 12 are arranged toward the outside of the main body unit 1. The 1st imaging part 11 and the 2nd imaging part 12 are arrange | positioned in the position corresponding to a user's right eye and left eye, respectively. That is, the first imaging unit 11 and the second imaging unit 12 are arranged at a predetermined distance from each other in the lateral direction so that an image captured by the first imaging unit 11 and the second imaging unit 12 is an image having parallax. The first imaging unit 11 and the second imaging unit 12 output the acquired image to the control unit 10.

<Drive unit 13>
The drive unit 13 moves the first imaging unit 11 and the second imaging unit 12 in the horizontal direction and the vertical direction under the control of the control unit 10. The drive unit 13 is configured by combining, for example, a horizontal turntable and a vertical turntable.

  More specifically, the driving unit 13 controls the orientations of the first imaging unit 11 and the second imaging unit 12 in accordance with the movement of the user's line of sight detected by the line-of-sight detection unit 16 under the control of the control unit 10. Adjust.

<First display unit 14 and second display unit 15>
The first display unit 14 and the second display unit 15 allow the user to visually recognize images generated based on images captured by the first imaging unit 11 and the second imaging unit 12, respectively, under the control of the control unit 10. Display as you can. The first display unit 14 and the second display unit 15 are disposed in the main body unit 1 in front of the user's right eye and in front of the left eye, respectively. The user can see only the first display unit 14 with the right eye, can see only the second display unit 15 with the left eye, and cannot see the other display unit. Since images having parallax are displayed on the first display unit 14 and the second display unit 15, the user feels these images displayed on the first display unit 14 and the second display unit 15 three-dimensionally. be able to.

<Gaze detection unit 16>
The line-of-sight detection unit 16 has a function of detecting a user's line of sight conventionally known, and outputs information related to the detected line of sight of the user to the control unit 10. For example, the line-of-sight detection unit 16 detects the user's line of sight by continuously detecting the position of the user's eyeball. For example, the line-of-sight detection unit 16 detects the user's line of sight by detecting the positions of the eyeballs of the user's right eye and left eye using a camera (imaging device) or a sensor disposed in front of the right eye and the left eye. To do. When the vicinity of the user's eyes is dark by wearing the head mounted display, it is preferable to use an infrared camera or the like.

  In the present specification, the position of the eyeball specifically refers to the position of the pupil or iris in the eyeball. The center position of the eyeball refers to the center of the pupil or iris in the eyeball. The line of sight indicates the normal direction of the center of the pupil or iris on the spherical surface.

<Control unit 10>
The control unit 10 is an arithmetic device such as a CPU, for example, and as illustrated in FIG. 2, the first imaging unit 11, the second imaging unit 12, the driving unit 13, the first display unit 14, and the second display unit 15. And exchanges data with the line-of-sight detection unit 16 to control the operation of these components.

  The control unit 10 controls the line-of-sight detection unit 16 to confirm the user's line of sight from the position of the user's eyeball. The control unit 10 controls the drive unit 13 based on the line-of-sight detection result by the line-of-sight detection unit 16 and adjusts the orientations of the first imaging unit 11 and the second imaging unit 12 to match the direction of the line of sight. The control unit 10 temporarily stores an image captured by the first imaging unit 11 and the second imaging unit 12 in a memory (not shown). The control unit 10 generates image information to be superimposed on the imaging results of the first imaging unit 11 and the second imaging unit 12 for display on the first display unit 14 and the second display unit 15. The control unit 10 performs processing such as superimposition of a predetermined image on the first display unit 14 and the first display unit 14 and displays images processed based on the imaging results of the first imaging unit 11 and the second imaging unit 12. 2 Display on the display unit 15.

  The flow of control of each component by the control unit 10 will be specifically described in the following operation description.

<Operation>
Next, the operation of the head mounted display of this embodiment will be described with reference to the flowchart of FIG. Note that the operations described here are only operations for adjusting the display image on the head mounted display while synchronizing with the captured image.

<S100 to S120>
When the user wears the head mounted display and operates the augmented reality function (S100), the control unit 10 acquires images of the left and right eyes of the user that are continuously imaged by the line-of-sight detection unit 16 (S110). ). The control part 10 confirms a user's eyes | visual_axis by pinpointing the center position of a user's eyeball from the acquired user's right and left eye image (S120). Note that these line-of-sight detection processes may be performed by the line-of-sight detection unit 16 instead of the control unit 10.

<S130 to S140>
Next, the control unit 10 checks whether or not the center position of the user's eyeball has changed with respect to the immediately preceding state (S130). If the center position of the user's eyeball has changed (Y in S130), the control unit 10 operates the drive unit 13 to move the first imaging unit 11 and the second imaging unit 12 in the direction of the user's line of sight. (S140). On the other hand, if the center position of the user's eyeball has not changed (N in S130), the process of S140 is performed as it is.

<S150 to S160>
Next, through the processing of S130 and S140, the control unit 10 is in a state where the first imaging unit 11 and the second imaging unit 12 are oriented in a direction that matches the user's line of sight. The imaging unit 12 is operated to image the subject (S150). Next, the control unit 10 combines the predetermined images with the images captured by the first imaging unit 11 and the second imaging unit 12 and causes the first display unit 14 and the second display unit 15 to display them (S160). The images displayed (combined) on the first display unit 14 and the second display unit 15 are, for example, two images having parallax so that an object, a person, a character, or the like is stereoscopically formed on an actual image. It is. If an image is displayed on the 1st display part 14 and the 2nd display part 15, it will repeat from the process of S120.

  Note that the processing of S110 to S160 is repeatedly performed every time corresponding to the number of still images (frames) displayed in a predetermined time (for example, 1 second). For example, when displaying 30 still images per second, the processing of S110 to S160 is repeated every 1/30 seconds.

  As described above, in the head mounted display adopting the configuration of the present embodiment, the directions of the first imaging unit 11 and the second imaging unit 12 are adjusted by the driving unit 13 in accordance with the user's line of sight, and the first imaging unit. 11 and the second imaging unit 12 are combined with a predetermined image and displayed on the first display unit 14 and the second display unit 15. Compared with the configuration in which the captured image is partially cut out and displayed in accordance with the user's line of sight, the configuration of the present embodiment is used when an image having the same number of pixels is displayed on the display unit. However, an image sensor having a small number of pixels can be used as the image pickup unit. Therefore, the configuration of the present embodiment can be a relatively inexpensive configuration.

  On the other hand, when an image sensor having the same number of pixels is used as the image sensor of the imaging unit, the configuration of the present embodiment has more effective pixels, so that a high-quality and fine image can be displayed.

  Further, the line-of-sight detection unit 16 of the present embodiment detects the line of sight by detecting the position of the user's eyeball, and adjusts the orientation of the first imaging unit 11 and the second imaging unit 12. Therefore, it is possible to adjust the orientation of an image to be captured and displayed while accurately detecting the user's line of sight.

  In the present embodiment, since the first direction and the second direction are directions corresponding to the directions of the right eye and the left eye, respectively, it is possible to display a stereoscopic image on the display unit.

  In the present embodiment, the images captured by the first display unit 14 and the second display unit 15 are processed in real time and displayed on the first display unit 14 and the second display unit 15. Therefore, the user can experience augmented reality in substantially real time. In addition, in order to maintain real-time property, it is preferable that the delay time from when the first display unit 14 and the second display unit 15 capture an image to when the image is displayed is 1 second or less. This delay time is more preferably set to 1/10 seconds or less for smooth display.

<2. Modification>
Next, a modification of the above embodiment will be described. In this modified example, the description of the deformable portion of the configuration of the above embodiment is omitted, and the description of the configuration and functions common to the above embodiment is omitted.

  In the flowchart described in the above embodiment, the control unit 10 moves the orientations of the first imaging unit 11 and the second imaging unit 12 depending on whether or not the center position of the eyeball has changed. May determine the orientation of the first imaging unit 11 and the second imaging unit 12 according to the detected center position of the eyeball, regardless of the change in the center position of the eyeball.

  Moreover, in the said embodiment, although the predetermined image was combined with the image imaged with the 1st imaging part 11 and the 2nd imaging part 12, and it displayed on the 1st display part 14 and the 2nd display part 15, a 1st display The images displayed on the unit 14 and the second display unit 15 may be images obtained by performing a predetermined filtering process on the imaging results of the first imaging unit 11 and the second imaging unit 12. That is, the images displayed on the first display unit 14 and the second display unit 15 may be images that have been subjected to arbitrary processing based on the imaging results of the first imaging unit 11 and the second imaging unit 12. .

  Moreover, in the said embodiment, although the drive part 13 moved the 1st imaging part 11 and the 2nd imaging part 12 with the horizontal turntable and the vertical turntable, the drive part 13 is horizontal. Only the turntable may be used. That is, since the human eye has a wider horizontal movement range than the vertical movement range, the drive unit 13 may be configured to follow the movement of the line of sight only in the horizontal direction. In this case, the configuration of the drive unit 13 is simplified as compared to the configuration in which the first imaging unit 11 and the second imaging unit 12 are operated in both the horizontal direction and the vertical direction. In such a configuration, a method of cutting out (extracting) and displaying only a portion included in the field of view may be adopted for the vertical eye movement.

<3. Supplementary items>
Heretofore, specific descriptions have been given of the embodiments and modifications of the present invention. The above description is merely an embodiment, and the scope of the present invention is not limited to this embodiment, but should be broadly interpreted to the extent that a person skilled in the art can grasp.

  The imaging display device of the present invention is suitably used as a head mounted display for an augmented reality device.

DESCRIPTION OF SYMBOLS 1 ... Main-body part 10 ... Control part 11 ... 1st imaging part 12 ... 2nd imaging part 13 ... Drive part 14 ... 1st display part 15 ... 2nd display part 16 ... Line-of-sight detection parts 21-23 ... Wearing tool

Claims (5)

A first imaging unit that captures an image in a first direction;
A second imaging unit that captures an image in the second direction;
A first display unit that displays a first image generated based on an image captured by the first imaging unit;
A second display unit for displaying a second image generated based on the image captured by the second imaging unit;
A line-of-sight detector that detects the line of sight of the user;
A drive unit that adjusts the orientation of the first imaging unit and the second imaging unit according to the line of sight detected by the line of sight detection unit,
Imaging display device. The line-of-sight detection unit detects the line of sight by detecting the position of the user's eyeball;
The imaging display device according to claim 1. The first direction and the second direction are directions corresponding to the directions of the left and right eyeballs of the user.
The imaging display device according to claim 1 or 2. The first image is an image generated based on an image captured after 1 second by the first imaging unit,
The second image is an image generated based on an image captured after one second by the second imaging unit.
The imaging display device according to any one of claims 1 to 3. The driving unit reciprocally moves the first imaging unit and the second imaging unit in one direction;
The imaging display device according to any one of claims 1 to 4.

Images