JP2023178093A - Display unit, control method, and program - Google Patents

Abstract

To provide a display unit that allows a user to more clearly see a virtual image such as a menu icon.SOLUTION: A display unit controls a depth position of a virtual image to be displayed, and has: first acquisition means that acquires depth information from an image; second acquisition means that acquires congestion information on both eyes of a user; and depth determination means that determines the depth position of the virtual image. The depth determination means has first determination means that determines the depth position based on the depth information acquired by the first acquisition means, and second determination means that determines the depth position based on the congestion information acquired by the second acquisition means. The display unit determines the depth position by switching between the first determination means and the second determination means according to the virtual image to be displayed.SELECTED DRAWING: Figure 1

Description

The present invention relates to a display device that displays a virtual image superimposed on real space.

In recent years, the development of technology for displaying 3D images has progressed, and head-mounted displays (hereinafter referred to as "HMD") that can display 3D images with depth have become popular, and optically transparent HMDs have also been developed. has been done. Optically transmissive HMDs use holographic elements, half mirrors, etc. to display an AR (Augmented Reality) image, which is a virtual three-dimensional image, and can also display the real space outside the HMD in a see-through manner. . At this time, the virtual image, which is the AR image, is displayed superimposed on the real space.

Various technologies have been developed to improve the stereoscopic effect of virtual images so that the virtual images, which are AR images, harmonize seamlessly with objects in real space and appear as if they exist in real space. For example, in Patent Document 1, the depth position of the displayed virtual image can be freely changed based on the acquired depth information of the real space, thereby making it possible to display the virtual image that is more in harmony with the real space. With this technology that controls the depth position of the virtual image, when the user is looking at the virtual image, the virtual image appears clear, and when the user is looking at an object in real space that has a different depth position than the virtual image, the virtual image naturally appears blurred. Therefore, the user can obtain a natural sense of depth from the virtual image, similar to that of a real object.

JP2016-24273A

However, in the conventional technology, when a user wants to view a virtual image such as a menu icon or additional information while viewing the real space, the depth position of the object in the real space that the user is viewing and the depth position of the displayed virtual image are not the same. If not, the virtual image will appear blurry. When the user focuses on the virtual image, the object in real space that the user is looking at is at a different depth from the virtual image, so it appears blurry, making it difficult to see the object in real space and the displayed virtual image at the same time. Become. When a user wants to view an object in real space and a virtual image at the same time, changing the depth position of the line of sight to match the respective depth positions places a heavy burden on the user. In addition, if you display a virtual image so that it is always clearly visible to the user, if you want to display the virtual image as if it exists in real space, the virtual image will naturally blur even if the user changes his/her line of sight. It becomes impossible to obtain a natural sense of depth. In this way, in the conventional technology, the display depth position of the virtual image is controlled based on the depth information in the real space so as to improve the three-dimensional effect of the virtual image, and the virtual image, such as a menu icon, is controlled by the object in the real space that the user is viewing. There was a problem in that it was difficult to always see clearly at the same time.

An object of the present invention is to provide a display device that allows a user to view virtual images such as menu icons more clearly.

In order to achieve the object, a display device according to one aspect of the present invention is a display device that controls the depth position of a virtual image to be displayed, and includes a first acquisition means that acquires depth information from the image, and a display device that controls the depth position of a virtual image to be displayed. It has a second acquisition means that acquires the convergence information of both eyes, and a depth determination means that determines the depth position of the virtual image, and the depth determination means uses the depth information acquired by the first acquisition means. and a second determining means that determines the depth position based on the convergence information acquired by the second acquisition means, and according to the virtual image to be displayed. , the depth position is determined by switching between the first determining means and the second determining means.

According to the display device of the present invention, a user can see virtual images such as menu icons more clearly.

1 is a system configuration diagram of a stereoscopic image display device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a method of displaying a virtual image based on real space image information according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a method of displaying a virtual image based on user's line-of-sight information according to an embodiment of the present invention. 1 is an external configuration diagram of a stereoscopic image display device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a method for controlling the depth position of a virtual image according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a method for acquiring user's line-of-sight depth information according to an embodiment of the present invention. 1 is a virtual image display sequence according to a first embodiment of the present invention. FIG. 7 is a diagram illustrating display of a virtual image based on user's line-of-sight information according to the second embodiment of the present invention. 3 is a virtual image display sequence according to a second embodiment of the present invention.

[Embodiment 1]
A first embodiment of the present invention will be described.

FIG. 4 is a diagram showing an example of the external configuration of the stereoscopic image display device 100 in this embodiment.

Reference numeral 102 denotes an imaging unit that acquires image information in real space, and is capable of simultaneously acquiring left and right image information. From the parallax information of the two acquired images, it is possible to calculate the distance between the stereoscopic image display device 100 and the object in real space. For example, from the distance (baseline length) between the two left and right imaging units 102, the focal length of the imaging unit 102, and the parallax information of the two images, the distance to the object in real space can be determined using a general triangulation method. Information can be calculated. A display unit 111 displays a virtual image superimposed on the real space image acquired by the imaging unit. The display unit 111 can display a three-dimensional image with a sense of depth by displaying a parallax image for the left eye and a parallax image for the right eye.

FIG. 2 is a diagram illustrating a virtual image display method in the case of displaying a virtual image based on depth information of an image in real space in this embodiment.

FIG. 2A is a diagram showing how a user using the stereoscopic image display device 100 views an object 200 in real space on which a virtual image is not superimposed. The stereoscopic image display device 100 sets a virtual three-dimensional orthogonal coordinate system for the real space image captured by the imaging unit 102. The stereoscopic image display device 100 recognizes subjects from images captured by the imaging unit 102 and calculates coordinates of each subject in a virtual three-dimensional orthogonal coordinate system. The stereoscopic image display device 100 calculates the depth position of the subject from the parallax information of the two images captured by the imaging unit 102, and uses the calculated depth position and the horizontal and vertical positions of the subject in the captured image to create a virtual image. The coordinate position of the subject in the three-dimensional coordinate system is calculated. In this embodiment, the stereoscopic image display device 100 recognizes the object 200 in real space, calculates the depth L1 of the object 200, and calculates the coordinates of the object 200 in a virtual three-dimensional orthogonal coordinate system.

FIG. 2B is a diagram showing how the stereoscopic image display device 100 displays a virtual image 201 superimposed on an image in real space. In this embodiment, the virtual image 201 that the user has instructed to display is displayed from among a plurality of virtual images registered in advance in the stereoscopic image display device 100. The stereoscopic image display device 100 displays the virtual image 201 at the same depth position as the calculated depth L1 of the object 200 in order to make it appear as if the virtual image 201 is placed on the object 200. In this embodiment, the stereoscopic image display device 100 detects only one object, the object 200, from the real space image, but a plurality of objects may exist. When the stereoscopic image display device 100 detects a plurality of subjects, the virtual image 201 is displayed so that the depth position of one subject selected from the plurality of subjects matches the depth position of the virtual image 201. If the stereoscopic image display device 100 does not detect any object from the real space image, the virtual image 201 is displayed at a preset depth position. Furthermore, the stereoscopic image display device 100 can classify and recognize subjects imaged by the imaging unit 102 into a plurality of types, such as people, animals, and objects. In this embodiment, for each virtual image to be displayed, it is set in advance at which position of which category of subject the virtual image is to be superimposed and displayed. For example, a virtual image of a plate is superimposed as if it were placed on a desk in real space, and a virtual image of a hat is superimposed as if it were being worn on a person's head in real space. When displaying a virtual image based on the depth information of a real space image, the stereoscopic image display device 100 displays the virtual image at the absolute coordinate position of the set virtual three-dimensional orthogonal coordinate system, and requests the user to redisplay or display the virtual image. The display coordinates of the virtual image are not changed unless a position movement instruction is given. Therefore, when the user changes the orientation of the user's face, the virtual image disappears from the user's field of view, and when the user changes the orientation of the user's face, the virtual image becomes visible again. Furthermore, when the user changes the line of sight to a depth position different from that of the virtual image, the virtual image naturally appears blurred, and when the user adjusts the line of sight to the virtual image again, the virtual image can be seen clearly. By implementing such a display method, it is possible to display a virtual image as if it existed in real space.

FIG. 3 is a diagram illustrating a virtual image display method in the case of displaying a virtual image based on user's line-of-sight information in this embodiment. FIG. 3 shows a method of displaying a virtual image 301 when a user using the stereoscopic image display device 100 wants to simultaneously view a virtual image 301 such as a menu icon or additional information while viewing a subject 300 in real space. In this embodiment, the virtual image 301 that the user has instructed to display is displayed from among a plurality of virtual images registered in advance in the stereoscopic image display device 100. The stereoscopic image display device 100 can calculate the depth information of the line of sight viewed by the user, and in FIG. 3 displays a virtual image based on the calculated line of sight depth position. In this embodiment, the virtual image to be displayed depends on whether the virtual image is displayed based on the depth information of the real space image as shown in FIG. 2, or the virtual image is displayed based on the user's line of sight information as shown in FIG. It is set in advance for each. For example, a virtual image that is desired to be displayed as if it exists in real space is displayed based on the depth information of the real space image, and virtual images such as menu icons and additional information are displayed based on the user's line of sight depth information. The stereoscopic image display device 100 displays the virtual image 301 at the same depth position as the calculated user's line-of-sight depth L2. The stereoscopic image display device 100 constantly calculates the user's line-of-sight depth information, and when the user's line-of-sight depth position changes, the display position of the virtual image 301 is changed so that the depth position of the virtual image 301 always matches the line-of-sight depth position. do. Therefore, even if the user changes the object they want to see in real space, the virtual image 301 is always displayed at the same depth position as the user's line of sight, so the user can always clearly see the virtual image 301 and the object they want to see in real space at the same time. It can be seen in

FIG. 1 is a system configuration diagram of a stereoscopic image display device 100 in this embodiment.

101 is a CPU that controls the entire stereoscopic image display device 100. Reference numeral 102 denotes an imaging unit that acquires image information in real space, and includes an imaging optical unit that performs focus adjustment and exposure adjustment, an imaging device that converts an optical image into an electrical signal, and the like. Reference numeral 103 denotes an image processing unit that performs various image processing such as noise reduction processing on captured image data. 106 is a subject detection unit that detects a subject from a captured image. A DRAM 104 serves as a memory for temporarily storing image data. 105 is a ROM that serves as a memory for storing virtual image information to be displayed and various setting values. Reference numeral 107 denotes an image three-dimensional coordinate calculation unit that calculates three-dimensional coordinates from captured image data, and an image depth calculation unit 108 calculates the depth position of a detected object. A line-of-sight three-dimensional coordinate detection unit 109 calculates the three-dimensional coordinates of the user's line of sight, and a line-of-sight depth calculation unit 110 calculates the line-of-sight depth position of the user. 112 is a virtual image three-dimensional coordinate determination unit that determines the display position of the virtual image that the user has instructed to display; virtual image depth determining means for determining the depth position of the virtual image. Reference numeral 111 denotes a display unit that displays a virtual image, which the user has instructed to display, at a display position determined by the virtual image three-dimensional coordinate determining unit 112.

FIG. 5 is a diagram illustrating a method of controlling the depth position of a displayed virtual image in this embodiment.

A Z-axis is defined in the viewing direction of the viewpoint 500, and the convex lens 501 is arranged so that the optical axis of the convex lens 501 and the Z-axis coincide with each other on the Z-axis. The focal length of the convex lens 501 is F. 502 and 503 are transmissive display elements such as transmissive organic EL, and distances A1 and A2 from the convex lens 501 can be changed by a motor (not shown), respectively. Although only two transmissive display elements 502 and 503 are shown in FIG. 5, three or more transmissive display elements may be provided. Transmissive display elements 502 and 503 can independently display different images 504 and 505, respectively. Transmissive display elements 502 and 503 are members that can transmit visible light. Images displayed by the transmissive display elements 502 and 503 appear to be superimposed when viewed from the viewpoint 500. The transmissive display elements 502 and 503 are arranged inside the focal point of the convex lens 501, and when viewing the display image 504 from the viewpoint 500, the display image 504 is observed as a virtual image 506 at a distance B1 from the convex lens 501. Ru. At this time, the relationship between the distances A1, B1, and the focal length F is defined by the known lens formula shown in equation (1) below.
1/A1 - 1/B1 = 1/F... Formula (1)

Further, the ratio of the size Q of the virtual image 506 to the size P1 of the display image 504 is expressed by the following equation (2).
Q/P1 = B1/A1... Formula (2)

Further, by transforming the equation (1), the distance A1 can be expressed as a function of the distance B1 as shown in the following equation (3).
A1 = FB1/(F+B1) = F/(1+F/B1) ... Formula (3)

Using equation (3), in order to display the virtual image 506 at a position at a distance B1 from the convex lens 501, a position A1 at which the transmissive display element 502 that displays the image 504 should be placed can be determined.

Furthermore, by substituting equation (1) into equation (2) and transforming it, it can be expressed as equation (4) below.
P1 = QF/(B1+F)... Formula (4)

Using equation (4), it is possible to determine the size P1 to be displayed as the display image 504 in order to display the virtual image 506 of size Q at the position of distance B1. Next, a control method when it is desired to display the virtual image 507 having the same size Q as the virtual image 506 at a distance B2 different from the distance B1 from the convex lens 501 will be described. The distance A2 between the convex lens 501 and the transmissive display element 503 can be expressed by the following equation (5) from equation (3).
A2 = F/(1+F/B2)... Formula (5)

Using equation (5), in order to display the virtual image 507 at a position at a distance B2 from the convex lens 501, a position A2 where the transmissive display element 503 that displays the image 505 should be placed can be determined.

Further, the size P2 of the image 505 to be displayed can be expressed by the following equation (6) from equation (4).
P2 = QF/(B2+F) ... Formula (6)

Using equation (6), it is possible to determine the size P2 that should be displayed as the display image 505 in order to display the virtual image 507 of size Q at the position of distance B2. In this way, by changing the distances A1 and A2 of the transmissive display elements 502 and 503 from the convex lens 501 and the display sizes P1 and P2, the depth position Z1 of the displayed virtual images 506 and 507 from the viewpoint 500, Z2 and size Q can be controlled.

FIG. 6 is a diagram illustrating a method for acquiring user's line-of-sight depth information. As explained in FIG. 5, 500 is a viewpoint, 501 is a convex lens, and 502 and 503 are transmissive display elements. A line-of-sight direction detector 600 can detect the angle at which the line-of-sight of the user's left and right eyes intersect, that is, the convergence angle θ, as convergence information. The line-of-sight direction detection method of the line-of-sight direction detector 600 can be implemented using various general methods. For example, for both the left and right eyes, near-infrared light is irradiated onto the eyeballs, the reflected light is detected by photodetectors placed on both sides of the black eye, and changes in the amount of light are detected at the boundary between the black eye and the white of the eye. There are ways to detect direction or eye movement. Another method is to detect eye movement by shining a spot light onto the eyeball and capturing the reflected light with a two-dimensional optical position detector, or by obtaining positional information of the melanogaster using a microscopic television camera. I can do it. I in FIG. 6 is the interpupillary distance. From the convergence angle θ and the interpupillary interval I, the convergence position where the line of sight intersects from the viewpoint 500, that is, the depth position Z of the line of sight, can be expressed by the following equation (7). (The unit of vergence angle θ is radian)
Z = I/θ... Formula (7)

Since the depth position of the user's line of sight can be calculated in this way, a virtual image can be displayed at the user's line of sight depth position by the control described with reference to FIG.

FIG. 7 is a virtual image display sequence in this embodiment.

When the CPU 101 receives an instruction from the user to display a virtual image (S701), the CPU 101 determines whether the virtual image that the user has instructed to display is a virtual image to be displayed at a fixed coordinate position calculated from the captured image depth information, and whether it is displayed at the user's line-of-sight depth position. Determine whether it is a virtual image. In this embodiment, the depth position calculated by which method should be displayed is preset for each virtual image to be displayed. If the virtual image is to be displayed at a fixed coordinate position calculated from the captured image depth information (S702), the depth position of the subject detected from the captured image is calculated (S703). In the case of a virtual image to be displayed at the user's line-of-sight depth position, the user's line-of-sight depth position is calculated (S704). The virtual image depth determination unit 113 determines the depth position of the virtual image to be displayed using the calculated depth position of the subject and the calculated user's line-of-sight depth position (S705). The display unit 111 displays the virtual image at the determined depth position (S706). If the user does not instruct to stop the virtual image display, the CPU 101 repeats the virtual image display sequence, and if the user instructs to stop the virtual image display (S707), the CPU 101 stops the virtual image display (S708).

As described above, with the control of this embodiment described above, the depth position of the virtual image is controlled as if it existed in real space, and the virtual image such as a menu icon is always aligned with the object in real space that the user is looking at. You can see clearly at the same time.

Note that in this embodiment, the depth position of the virtual image to be displayed is determined using either only the depth information of the captured image or only the user's line of sight depth information, but the depth position of the virtual image to be displayed is determined based on both pieces of information. The depth position may also be determined. For example, when multiple objects are detected from a captured image and one object is selected from the detected objects, the object at the depth position closest to the user's line of sight depth position is selected, and the object at the same depth as the selected object is selected. A virtual image may be displayed at the position.

In addition, in this embodiment, the virtual image is displayed at the same depth position as the depth of the captured image and the user's line of sight depth, but the virtual image is displayed at a depth position within a preset distance from the depth of the captured image and the user's line of sight depth. A virtual image may be displayed.

Furthermore, in this embodiment, when displaying a virtual image based on the depth position of a captured image, the virtual image is displayed at a fixed depth position. good.

[Embodiment 2]
A second embodiment of the present invention will be described.

This embodiment differs from the first embodiment in the virtual image display method and virtual image display sequence based on the user's line-of-sight information. The other configurations are the same as those in the first embodiment, so their explanation will be omitted.

FIG. 8 is a diagram illustrating a virtual image display method based on user's line-of-sight information in this embodiment. FIG. 8 shows a method of displaying the virtual image 301 when a user using the stereoscopic image display device 100 wants to view the virtual image 301 such as a menu icon or additional information while viewing the object 300 in real space. In this embodiment, the virtual image 301 that the user has instructed to display is displayed from among a plurality of virtual images registered in advance in the stereoscopic image display device 100. In this embodiment, the virtual image to be displayed depends on whether the virtual image is displayed based on the depth information of the real space image as shown in FIG. 2, or the virtual image is displayed based on the user's line of sight information as shown in FIG. It is set in advance for each. The stereoscopic image display device 100 displays the virtual image 301 at the same depth position as the calculated user's line-of-sight depth L2. The stereoscopic image display device 100 has set a virtual three-dimensional orthogonal coordinate system, and calculates the coordinates of the user's line of sight in the set three-dimensional coordinate system. In this embodiment, the virtual image 301 is displayed not only at the depth L2 of the user's line of sight but also at a position close to the user's line of sight coordinates on the calculated three-dimensional coordinates. This is because if the virtual image is displayed at the same depth L2 as the user's line of sight, the object in real space and the virtual image can be seen clearly at the same time, but if the object and the virtual image are far apart, the user needs to move his or her line of sight. This is because there is a possibility that the subject and the virtual image cannot be viewed at the same time. However, if the virtual image is displayed at the same coordinate position as the subject 300 that the user is looking at, the user will not be able to see the subject 300, so the virtual image is displayed at a different position from the subject 300 that the user is looking at. . In this embodiment, the subject detection unit 106 detects whether a subject exists in the captured image. If the subject does not exist at the coordinates of the user's line of sight, a virtual image is displayed at the coordinate position of the user's line of sight on the three-dimensional coordinates. When a subject exists at the user's line-of-sight coordinates, a virtual image 301 is displayed at a coordinate position within a predetermined distance range from the line-of-sight coordinates on the three-dimensional coordinates and at a coordinate position where there are no other subjects that the user is not looking at. The value of this predetermined distance range is set in advance, and if there is no area within the predetermined distance range where the subject does not exist, the predetermined distance range is expanded and a virtual image is displayed in the area where the subject does not exist. If there is no area where the subject does not exist even after enlarging the predetermined distance range, a virtual image is displayed at the same position as the subject that the user is not looking at. Furthermore, in this embodiment, even if the user moves his or her line of sight within a predetermined distance range, the display position of the virtual image 301 does not change. This is because when the user moves his line of sight from the subject 300 to the virtual image 301 in order to view the virtual image 301, the virtual image 301 also moves, making it impossible for the user to view the virtual image 301. When the user's line of sight position moves beyond a predetermined distance range, the display position of the virtual image 301 is changed so that the virtual image 301 is displayed within a predetermined distance range from the line of sight destination. In this embodiment, the display position of the virtual image 301 is not changed while the user's line of sight is moving, and the display position of the virtual image 301 is changed after a predetermined time has elapsed after the user's line of sight stops moving. Change the display position. This is because, if the virtual image 301 also changes its display position rapidly while the user is moving his or her line of sight rapidly, the display of the virtual image will look bothersome to the user.

By performing the above control, the user can always clearly view the object he or she wants to see in real space and the virtual image at the same time without significantly moving the line of sight.

FIG. 9 is a virtual image display sequence in this embodiment.

When the CPU 101 receives an instruction from the user to display a virtual image (S901), the CPU 101 determines whether the instructed virtual image is a virtual image to be displayed at a fixed coordinate position calculated from the three-dimensional coordinates of the captured image, or whether the instructed virtual image is a virtual image to be displayed at a fixed coordinate position calculated from the three-dimensional coordinates of the user's line of sight. Determine whether it is a virtual image to be displayed at the location. In this embodiment, the coordinate position calculated by which method should be displayed is set in advance for each virtual image to be displayed. If the virtual image is to be displayed at a fixed coordinate position calculated from the three-dimensional coordinates of the captured image (S902), the three-dimensional coordinate position of the subject detected from the captured image is calculated (S903). The virtual image three-dimensional coordinate determining unit 112 determines the three-dimensional coordinate position of the virtual image to be displayed using the calculated three-dimensional coordinate position of the subject (S904). The display unit 111 displays the virtual image at the determined three-dimensional coordinate position (S920). If the virtual image is to be displayed at a coordinate position calculated from the three-dimensional coordinates of the user's line of sight, the three-dimensional coordinate position of the user's line of sight is calculated (S910). The image three-dimensional coordinate calculation unit 107 calculates the coordinate position of the subject detected by the subject detection unit 106 (S911). The virtual image three-dimensional coordinate determining unit 112 determines whether the virtual image was displayed at a position based on the three-dimensional coordinates of the user's line of sight in the previous frame (S912). If the virtual image is not displayed in the previous frame, the virtual image three-dimensional coordinate determination unit 112 calculates an area within a predetermined distance from the line-of-sight coordinates and in which no subject exists as the display coordinate position of the virtual image (S915). If the previous frame was displayed, the virtual image three-dimensional coordinate determination unit 112 determines whether the current user's line-of-sight coordinates exist within the predetermined distance range calculated in the previous frame (S913). If the current user's line-of-sight coordinate position is outside the predetermined distance range calculated in the previous frame, calculate the area within the predetermined distance from the current line-of-sight coordinate and where no subject exists as the display coordinate position of the virtual image. (S915). If the current coordinate position of the user's line of sight is within the predetermined distance range calculated in the previous frame, it is determined whether a subject exists at the virtual image display position (S914). If a subject exists at the virtual image display position, an area within a predetermined distance from the current line-of-sight coordinates and where the subject does not exist is calculated as the display coordinate position of the virtual image (S915). If the subject does not exist at the virtual image display position, the coordinate position of the virtual image to be displayed is not changed from the virtual image position displayed in the previous frame (S916). The display unit 111 displays the virtual image at the three-dimensional coordinate position determined by the virtual image three-dimensional coordinate determination unit 112 (S920). If the user does not instruct to stop the virtual image display, the CPU 101 repeats the virtual image display sequence, and if the user instructs to stop the virtual image display (S921), the CPU 101 stops the virtual image display (S922).

As described above, with the control of the present embodiment described above, the depth position of the virtual image is controlled as if it existed in real space, and virtual images such as menu icons can be viewed without the user significantly moving the line of sight. You can always clearly see the object you want to see in real space and the virtual image at the same time.

Note that when displaying a virtual image based on three-dimensional information of a captured image, the display position of the virtual image may also be determined using user's line-of-sight information. For example, when multiple objects are detected from a captured image and one object is selected from the detected objects, the object whose coordinates are closest to the user's line of sight coordinate position is selected, and the coordinate position of the selected object is A virtual image may be displayed based on this.

Further, in this embodiment, the virtual image is displayed at the same depth position as the depth of the captured image, but the virtual image may be displayed at a depth position within a preset distance from the depth of the captured image.

Furthermore, in this embodiment, when displaying a virtual image based on the depth position of a captured image, the virtual image is displayed at a fixed depth position. good.

[Other embodiments]
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

100 Stereoscopic image display device 101 CPU
102 Imaging unit 103 Image processing unit 104 DRAM
105 ROM
106 Subject detection section 107 Image three-dimensional coordinate calculation section 108 Image depth calculation section 109 Line-of-sight three-dimensional coordinate calculation section 110 Line-of-sight depth calculation section 111 Display section 112 Virtual image three-dimensional coordinate determination section 113 Virtual image depth determination means

Claims (10)

A display device that controls the depth position of a displayed virtual image,
a first acquisition means for acquiring depth information from an image;
a second acquisition means for acquiring convergence information of the user's eyes;
depth determining means for determining the depth position of the virtual image,
The depth determining means includes:
a first determining means for determining a depth position based on the depth information acquired by the first acquiring means;
a second determining means for determining a depth position based on the congestion information acquired by the second acquiring means;
A display device characterized in that a depth position is determined by switching between the first determining means and the second determining means depending on a virtual image to be displayed. It has an imaging means for capturing an image,
The display device according to claim 1, wherein the first acquisition unit acquires depth information based on an image captured by the imaging unit. comprising a subject detection means for detecting a subject from an image taken by the imaging means,
The display device according to claim 2, wherein the first acquisition means acquires depth information of the subject. The display device according to claim 1, wherein the first determining means sets a position within a predetermined distance range from the depth position acquired by the first acquisition means as the depth position of the virtual image. The display device according to claim 1, wherein the second determining means sets a position within a predetermined distance range from the convergence position acquired by the second acquiring means as the depth position of the virtual image. The first acquisition means acquires three-dimensional coordinates of the depth of the image,
The second acquisition means acquires three-dimensional coordinates of the convergence position of both eyes,
The display device according to claim 1, wherein the depth determining means determines three-dimensional coordinates of the virtual image. The display according to claim 6, wherein the second determining means sets coordinates within a predetermined distance range from the three-dimensional coordinates of the convergence position acquired by the second acquiring means as the coordinates of the virtual image. Device. The display device according to claim 6, wherein the depth determining unit determines coordinates different from the three-dimensional coordinates of the depth acquired by the first acquiring unit as the three-dimensional coordinates of the virtual image. A control method for controlling the depth position of a virtual image displayed on a display device, comprising:
a first acquisition step of acquiring depth information from the image;
a second acquisition step of acquiring convergence information of the user's eyes;
a depth determining step of determining a depth position of the virtual image,
The depth determining step includes:
a first determining step of determining a depth position based on the depth information acquired by the first acquisition means;
a second determining step of determining a depth position based on the congestion information acquired by the second acquisition means,
A control method characterized in that the depth position is determined by switching between the first determining step and the second determining step depending on the virtual image to be displayed. A program for causing a computer to execute each step of the control method according to claim 9.

Images