KR20200120466A - Head mounted display apparatus and operating method for the same - Google Patents

Abstract

The present invention relates to a head mounted display device, which comprises: a gaze tracking sensor which acquires gaze information of both eyes of a user; a depth sensor which acquires depth information on one or more objects; and a processor which acquires information on a gaze point based on the acquired gaze information and determines a measurement parameter of the depth sensor based on the information on the gaze point.

Description

Head mounted display apparatus and operating method for the same

Various embodiments relate to a head mounted display device for determining a gaze point in an actual space and acquiring depth information by using a parameter optimized for the gaze point, and an operating method thereof.

The real space in which we live is composed of three-dimensional coordinates. Humans perceive a three-dimensional space with a three-dimensional effect by combining visual information seen by both eyes. However, a picture or video taken with a general digital device is a technology that expresses three-dimensional coordinates as two-dimensional coordinates, and does not include information on space. In order to express such a sense of space, 3D cameras or display products are emerging that use two cameras together to shoot and show three-dimensional images.

On the other hand, in order to express a sense of space, depth information on a real space must be sensed. In the conventional depth information sensing, depth sensing has been performed on the entire range of spaces that can be measured by the depth sensor without considering the user's region of interest. In particular, in the case of a depth sensor that performs depth sensing by projecting light, the IR LED is driven to project light (for example, infrared light) over a full range of space, and power according to the driving of the IR LED Consumption increases. In addition, the amount of calculation increases in order to obtain depth information for a full range of spaces, thereby increasing power consumption. As power consumption of the depth sensor increases, there is a problem in that it is difficult to mount the depth sensor in a small device.

In addition, conventional depth sensing methods have a problem in that the accuracy of depth sensing is low due to the weakness of each of the depth sensors.

The head mounted display device according to an embodiment determines a gaze point, and obtains depth information about a preset region of interest based on the gaze point, using a parameter optimized for the gaze point, and an operation thereof Can provide a way.

The head mounted display device according to an embodiment includes a gaze tracking sensor that acquires gaze information of both eyes of a user, a depth sensor that acquires depth information for one or more objects, and a gaze point based on the acquired gaze information. It may include a processor that obtains information about the gaze and determines a measurement parameter of the depth sensor based on the information about the gaze point.

The processor according to an embodiment may acquire 2D location information of the gaze point based on the gaze information.

The processor according to an embodiment may obtain information on the estimated depth of the gaze point based on the gaze information.

The depth sensor according to an exemplary embodiment may obtain depth information on a region of interest set in advance based on the gaze point according to the determined measurement parameter.

The measurement parameter of the depth sensor according to an embodiment may include at least one of a parameter for a target area, a parameter for emission light output, and a parameter for reflected light sensing.

The processor according to an embodiment determines the measurement parameter of the depth sensor again based on the obtained depth information on the ROI, and the depth sensor determines the measurement parameter for the ROI. The depth information can be obtained again.

The depth sensor according to an embodiment may obtain depth information on the region of interest by at least one of a Time Of Flight (TOF) method, a Structured Light (SL) method, and a Stereo Image (SI) method.

When the depth sensor according to an embodiment includes a depth sensor of a TOF (Time Of Flight) method, the processor, based on the two-dimensional position information of the gaze point, among the light sources included in the depth sensor The measurement parameter may be determined so that some light sources corresponding to the gaze point are driven, and the depth sensor may drive the some light sources to obtain depth information on the ROI.

The head mounted display device according to an embodiment further includes a display displaying an actual space including the region of interest, and the processor further includes at least one virtual display device on the region of interest based on depth information on the region of interest. The display can be controlled so that the object of is displayed.

The processor according to an embodiment sets the measurement parameter of the depth sensor as a first parameter, and obtains total depth information on a space that can be sensed by the depth sensor while including the region of interest based on the first parameter. The depth sensor is controlled so that the depth sensor is controlled, based on the total depth information, first depth information for the ROI is obtained, and a measurement parameter of the depth sensor is set as a second parameter based on the first depth information. And, based on the second parameter, the depth sensor may be controlled to obtain second depth information on the ROI.

An operation method of an electronic device according to an embodiment includes: acquiring gaze information of both eyes of a user, acquiring information on a gaze point based on the acquired gaze information, and based on information on the gaze point Thus, it may include determining a measurement parameter of the depth sensor.

A computer-readable recording medium according to an embodiment includes: acquiring gaze information of both eyes of a user, acquiring information on a gaze point based on the acquired gaze information, and information on the gaze point Based on, a program for performing the step of determining a measurement parameter of the depth sensor may be stored.

The electronic device according to an embodiment acquires depth information based on a gaze point, so that efficiency of depth sensing may increase and power consumption may decrease.

Since the electronic device according to an embodiment acquires depth information by using a parameter optimized for a gaze point, accuracy of the depth information may be improved.

1 is a diagram illustrating an electronic device according to an exemplary embodiment.
2 to 3D are diagrams referred to for describing a method of tracking a user's gaze by an electronic device according to an exemplary embodiment.
4A and 4B are diagrams for describing a method of obtaining, by an electronic device, depth information on a gaze point, according to an exemplary embodiment.
5A and 5B are diagrams for explaining a method of determining measurement parameters of a depth sensor based on a user's gaze point, according to an exemplary embodiment.
6 to 7B are diagrams referenced for explaining a method of determining measurement parameters by an electronic device according to an exemplary embodiment when a depth sensor according to an exemplary embodiment is a TOF method.
8 and 9 are diagrams referenced for explaining a method of determining measurement parameters by an electronic device according to an exemplary embodiment when a depth sensor according to an exemplary embodiment is a structured light (SL) method.
FIG. 10 is a diagram referenced for explaining a method of determining measurement parameters by an electronic device according to an exemplary embodiment when the depth sensor according to an exemplary embodiment is a stereo image (SI) method.
11 is a diagram for describing a method of displaying a virtual object by an electronic device according to an exemplary embodiment.
12 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.
13 is a diagram for describing a method of obtaining depth information by an electronic device according to an exemplary embodiment.
14 is a flowchart illustrating a method of obtaining depth information by an electronic device according to an exemplary embodiment.
15 is a diagram illustrating an example in which an electronic device repeatedly performs depth information acquisition operations of FIG. 14 according to an exemplary embodiment.
16 is a diagram illustrating an example in which an electronic device provides a virtual object in an augmented reality (AR) method according to an exemplary embodiment.
17 is a diagram illustrating an example in which an electronic device recognizes a human face using depth information according to an exemplary embodiment.
18 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
19 is a block diagram illustrating a configuration of an electronic device according to another embodiment.
20 and 21 are diagrams for describing a method of automatically adjusting focus by an electronic device according to an exemplary embodiment.
22 is a diagram for describing a method of performing a gaze-based spatial modeling by an electronic device according to an exemplary embodiment.

The terms used in the present specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected from general terms that are currently widely used while considering functions in the present invention, but this may vary depending on the intention or precedent of a technician working in the field, the emergence of new technologies, and the like. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall contents of the present invention, not a simple name of the term.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated. In addition, terms such as "... unit" and "module" described in the specification mean units that process at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. .

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are assigned to similar parts throughout the specification.

1 is a diagram illustrating an electronic device according to an exemplary embodiment.

Referring to FIG. 1, the electronic device 100 according to an embodiment may be a glasses-type wearable device. The glasses-type wearable device may include a head mounted display (HMD, hereinafter referred to as “HMD”) that can be mounted on the head. For example, the head mounted display HMD may include a device having a shape such as glasses, a helmet, and a hat, but is not limited thereto.

In addition, the electronic device 100 according to an embodiment includes a mobile phone, a smart phone, a laptop computer, a desktop computer, a tablet PC, an e-book terminal, a digital broadcasting terminal, and a personal digital assistant (PDA). , Portable Multimedia Player (PMP), navigation, MP3 player, camcorder, Internet Protocol Television (IPTV), Digital Television (DTV), and a wearable device. However, it is not limited thereto.

In the embodiments of the present specification, the term "user" refers to a person who controls functions or operations of the electronic device 100, and may include an administrator or an installer.

The electronic device 100 according to an embodiment is a device that provides at least one virtual object in the form of augmented reality (AR), mixed reality (MR), or virtual reality (VR). Can be

When a virtual object is provided in the form of augmented reality or mixed reality, the electronic device 100 displays the virtual object on the display so that the virtual object matches the shape, arrangement, distance, and depth of the real object in the real world. Can be displayed. For example, the electronic device 100 may overlap and display an image of a virtual object on a real image of the real world. However, it is not limited thereto.

The electronic device 100 according to an embodiment may include a depth sensor 150.

The depth sensor 150 may acquire depth information on one or more objects included in the real world. The depth information may correspond to a distance from the depth sensor 150 to a specific object. The depth value may increase as the distance from the depth sensor 150 according to an exemplary embodiment to a specific object increases.

As shown in FIG. 1, in a three-dimensional space, the X-axis is a reference axis that passes right and left through the electronic device 100, the Y-axis is a reference axis that passes vertically through the electronic device 100, and the Z-axis is the front and back of the electronic device 100. It may be a reference axis passing through. In addition, the X-axis, Y-axis, and Z-axis may be perpendicular to each other.

Accordingly, depth information according to an embodiment may mean a distance on the Z axis from the depth sensor 150 to a specific object.

The depth sensor 150 according to an embodiment may obtain depth information of an object in various ways. For example, the depth sensor 150 may acquire depth information by using at least one of a time of flight (TOF) method, a structured light method, and a stereo image method. . Each method will be described in detail later.

The depth sensor 150 according to an embodiment may include at least one camera (image sensor). The depth sensor 150 may acquire depth information on an actual space included in a field of veiw (FOV) of the camera. Hereinafter, an actual space within a range that can be sensed by the depth sensor 150 will be referred to as a “total space”. In general, the'total space' corresponds to the entire area of the image sensor.

On the other hand, when the user of the electronic device 100 gazes at a partial space of the entire space, depth information on the remaining spaces excluding the partial space is less important than the space at which the user gazes. For example, a user of the electronic device 100 is gazing at the space around the desk 50 among the room spaces (total spaces, 10) shown in FIG. 1. Using the depth sensor 150, the entire space 10 In the case of acquiring depth information about ), depth information about the remaining spaces excluding the space around the desk 50 may be information of lower importance than depth information about the space around the desk 50. Among the entire space, a point where the user gazes and a region around the point where the user gazes are referred to as an ROI. The region of interest according to an embodiment may be a preset region centered on a point at which the user gazes.

In addition, when the electronic device 100 acquires depth information on the entire space, since the amount of calculation increases, power consumption increases, and the response speed of the electronic device 100 decreases. In addition, when depth information is obtained by targeting not only a partial space (for example, the gaze point of the electronic device user) but targeting the entire space, the gaze point of the user of the electronic device 100 The accuracy of the depth information for may be lowered.

Accordingly, the electronic device 100 according to an embodiment may determine a point at which the user gazes out of the entire space. For example, the electronic device 100 may track the gazes of both eyes of the user, obtain gaze information, and determine a point at which the user gazes (a gaze point) based on the gaze information.

The electronic device 100 according to an embodiment may reduce power consumption required for obtaining depth information and improve accuracy for depth information by adjusting a measurement parameter of the depth sensor based on the determined gaze point.

The measurement parameter of the depth sensor is a variable required to obtain depth information when depth information on at least one object is obtained using the depth sensor, and may be numerical information that must be set in advance in the depth sensor. For example, the measurement parameters of the depth sensor include a parameter for a target area for emitting light, a parameter for a reflected light sensing area for sensing reflected light, a parameter for an output pattern of emitted light, a parameter for an output size of emitted light, It may include a sensing speed for sensing reflected light, a sensing period, or a parameter for sensing sensitivity. Hereinafter, it will be described in detail with reference to the drawings.

2 to 3D are diagrams referred to for describing a method of tracking a user's gaze by an electronic device according to an exemplary embodiment.

Referring to FIG. 2, the electronic device 100 may track a user's gaze. In general,'gaze' refers to the viewing direction, and'eye tracking' refers to a process of measuring the user's gaze (for example, the point at which the user gazes 210). By tracking the movement, it can be done.

The electronic device 100 according to an embodiment may include a gaze tracking sensor 160 to track a user's gaze. The gaze tracking sensor 160 according to an embodiment may include a first gaze tracking sensor 161 for tracking the gaze of the user's left eye and a second gaze tracking sensor 162 for tracking gaze of the right eye. I can. The first gaze tracking sensor 161 and the second gaze tracking sensor 162 have the same structure and operate in the same manner.

3A is a diagram for describing a method of tracking a user's gaze based on an amount of reflected light reflected from the user's eyes.

Since the first gaze tracking sensor 161 and the second gaze tracking sensor 162 according to an embodiment have the same structure and operate in the same manner, in FIG. 3A, the description is made based on the first gaze tracking sensor 161 I will do it.

Referring to FIG. 3A, the first gaze tracking sensor 161 according to an embodiment may include an illumination unit 301 providing light to a user's eyes and a sensing unit 302 detecting light. The illumination unit 301 may include a light source for providing light and a scanning mirror for controlling a direction of light provided from the light source. The scanning mirror may control a direction so that light provided from a light source is directed toward the user's eyes 320 (eg, the cornea 310). The scanning mirror includes a structure capable of mechanically changing a reflection angle so as to reflect light provided from a light source to face the user's eyes 320, and the cornea 310 using light provided from the light source according to the changed reflection angle. ) Can be scanned.

The sensing unit 302 may detect light reflected from the user's eyes 320 and measure the amount of the detected light. For example, when light is reflected from the center of the user's cornea 310, the amount of light detected by the sensing unit 302 may be maximum. Accordingly, when the amount of light detected by the detection unit 302 is the maximum, the first gaze tracking sensor 161 is based on the point 330 at which light is incident and reflected to the user's eyes. The gaze direction 340 may be determined. For example, when the amount of light is at the maximum, the first gaze tracking sensor 161 determines the direction 340 in which light is incident on the user's eyes and connects the reflected point 330 to the center point of the user's eyes. It may be determined as the gaze direction of the eye (for example, the user's left eye). However, it is not limited thereto.

In addition, the second gaze tracking sensor 162 may also determine the gaze direction of the user's eye (eg, the right eye) in the same manner as described in FIG. 3A.

3B is a diagram illustrating a method of tracking a user's gaze based on a position of reflected light reflected from the user's eyes.

Since the first gaze tracking sensor 161 and the second gaze tracking sensor 162 according to an embodiment have the same structure and operate in the same manner, in FIG. 3B, the description is made based on the first gaze tracking sensor 161 I will do it. The first gaze tracking sensor 161 may include an illumination unit 351 and a photographing unit 352. The lighting unit 351 may include an infrared light emitting diode (IR LED) or the like. As shown in FIG. 3B, the lighting unit 351 may include a plurality of light emitting diodes disposed at different positions. The lighting unit 351 may provide light (eg, infrared light) to the user's eyes when photographing the user's eyes. As light is provided to the user's eye, reflected light may be generated in the user's eye.

In addition, the photographing unit 352 may include at least one camera, and in this case, at least one camera may include an infrared camera (IR). The electronic device 100 may track the gaze of the user's eye (eg, the user's left eye) by using the user's eye image captured by the photographing unit 352. For example, the first gaze tracking sensor 161 may track the user's gaze by detecting pupils and reflected light in the user's eye image. The first gaze tracking sensor 161 may detect the location of the pupil and the reflected light in the user's eye image, and determine the gaze direction of the user's eye based on a relationship between the location of the pupil and the location of the reflected light.

For example, the first gaze tracking sensor 161 detects the pupil 370 and the reflected light 381 from the photographed first eye image 361, and the position of the pupil 370 and the reflected light 381 Based on the relationship between the positions, the gaze direction 391 of the user's eyes may be determined. In the same way, the pupil 370 and the reflected light 382, 383, 384, 385 are detected in each of the second to fifth eye images 362, 363, 364, 365, and between the position of the pupil and the position of the reflected light Based on the relationship of, the user's eye gaze directions 392, 393, 394, and 395 may be determined.

In addition, the second gaze tracking sensor 162 may also determine the gaze direction of the user's eye (eg, the right eye) in the same manner as described in FIG. 3B.

3C is a diagram illustrating a three-dimensional eyeball model for a user's gaze.

2 and 3C, the electronic device 100 determines the gaze direction of the user's left eye using the first gaze tracking sensor 161, and uses the second gaze tracking sensor 162, It is possible to determine the gaze direction of the user's right eye. For example, the electronic device 100 may determine a gaze direction based on an average eyeball model of a person. The eyeball model may be modeled by assuming that the human eyeball 3100 has a spherical shape and that the eyeball ideally rotates according to the gaze direction. In addition, the eye model may be mathematically expressed as Equations 1 and 2 below.

In Equation 1, d denotes the distance between the center of the user's eye 3150 and the virtual screen 3200, and α denotes a case where the user's eyes stare directly at the virtual screen 3200, Represents the angle at which the user's eyes rotate in the x-axis direction, and β denotes the angle in which the user's eyes rotate in the y-axis direction based on the case where the user's eyes stare directly at the virtual screen 3200. In addition, in Equation 2, r represents the radius of the sphere, assuming that the user's eyes are spheres.

The first gaze tracking sensor 161 according to an embodiment uses the method described in FIGS. 3A and 3B to determine the degree of rotation (eg, α and β) of the user's eye (eg, left eye). Measurement may be performed, and the electronic device 100 may calculate a two-dimensional coordinate of the user's eye line of sight on the virtual screen using the rotational degrees α and β of the user's eye.

3D is a diagram referenced to explain a method of calibrating an eye tracking sensor according to an exemplary embodiment.

As another example, when the user first uses the electronic device 100, the first eye tracking sensor 161 and the second eye tracking sensor 162 are calibrated to accurately measure the gaze of the user's left and right eyes. Calibration) can be performed. The electronic device 100 displays virtual images VI1, VI2, VI3 having different depths (eg, d1, d2, d3) in which a plurality of points (generally 9) for inducing the user's gaze are displayed. ) Can be output to induce the user to gaze at each of the plurality of points.

When the user gazes at each point included in the virtual images VI1, VI2, VI3, the electronic device 100 stores information (gaze information) output from the gaze tracking sensor 160 in a table format in advance. I can.

As described in Fig. 3A, in the method using the amount of light reflected from the user's cornea, the reflection angle and light amount information of the scanning mirror for each point can be stored in advance in the form of a table as gaze information, and the user's eyes are photographed using infrared light. In this method, an image including the user's eyes and reflected light photographed at each point may be linked and stored in advance as gaze information.

The electronic device 100 may determine a gaze direction of a user's eyes by comparing gaze information stored in advance with gaze information output from a gaze tracking sensor. The electronic device 100 uses the gaze information output from the first gaze tracking sensor 161 to determine the gaze direction of the user's left eye, and uses gaze information output from the second gaze tracking sensor 162 Thus, the gaze direction of the user's right eye can be determined.

The electronic device 100 uses the gaze direction of the user's left eye, the gaze direction of the right eye, and the distance between both eyes, as shown in FIG. 2, at a point 210 at which the user gazes in the entire space 10. You can estimate the coordinates of

For example, the electronic device 100 uses coordinate mapping, etc., in the entire space 10 described in FIG. 1, the point 210 at which the user gazes is 2D coordinate information (eg, x coordinate value). , y-coordinate value), or can be saved in a table format in advance.

4A and 4B are diagrams for describing a method of obtaining, by an electronic device, depth information on a gaze point, according to an exemplary embodiment.

Referring to FIG. 4A, the electronic device 100 uses the convergence of the gaze direction of the right eye and the gaze direction of the left eye (where two virtual straight lines indicating the gaze direction intersect), It is possible to estimate depth information about the gaze point.

For example, as shown in FIG. 4B, based on the first gaze direction 410 corresponding to the left eye, the second gaze direction 420 corresponding to the right eye, and the distance between both eyes, the gaze point 430 , It is possible to calculate the distance value (Z1) to the point where the eyes of both eyes converge). The electronic device 100 can obtain depth information on the gaze point by using the gaze information of both eyes measured using the gaze tracking sensor 160 and Equation 3 below according to the geometric arrangement shown in FIG. 4B. have.

In Equation 3, Δx represents the difference between the x-coordinate (x1) of the left eye and the x-coordinate (x2) of the right eye on the virtual screen 450. In this case, it can be calculated on the assumption that the y-coordinate of the left eye and the y-coordinate of the right eye are the same. In addition, in Equation 3, a represents the distance between both eyes of the user, and a preset value (eg, 7 cm) may be used. Further, D represents the distance between the user's eyes and the virtual screen 450.

The electronic device 100 may obtain a distance value Z1 to a point where both eyes of the user converge, as a sum of the z value and the D value.

Alternatively, the electronic device 100 may estimate depth information (eg, Z1) for the gaze point 430 based on the angle formed by the first gaze direction 410 and the second gaze direction 420. have. The smaller the angle between the first gaze direction 410 and the second gaze direction 420 is, the longer the distance to the point of interest 430 is, and the angle formed by the first gaze direction 410 and the second gaze direction 420 The larger is, the closer the distance to the point of interest is.

5A and 5B are diagrams for explaining a method of determining measurement parameters of a depth sensor based on a user's gaze point, according to an exemplary embodiment.

The measurement parameter of the depth sensor is a variable required to obtain depth information when depth information on at least one object is obtained using the depth sensor, and may be numerical information that must be set in advance in the depth sensor. For example, the measurement parameters of the depth sensor include a parameter for a target area for emitting light, a parameter for a reflected light sensing area for sensing reflected light, a parameter for an output pattern of emitted light, a parameter for an output size of emitted light, It may include a sensing speed for sensing reflected light, a sensing period, or a parameter for sensing sensitivity.

5A and 5B, an electronic device according to an embodiment may include a gaze tracking sensor 160, a processor 120, and a depth sensor 150.

Referring to FIG. 5A, the gaze tracking sensor 160 according to an embodiment may obtain gaze information of a user. This has been described in detail in FIGS. 2 to 3D, and thus a detailed description will be omitted.

The gaze tracking sensor 160 may transmit gaze information of the user to the processor 120, and the processor 120 may obtain information on the gaze point based on the gaze information of the user.

For example, based on the gaze direction for the user's left eye (first gaze direction) and the gaze direction for the right eye (second gaze direction) included in the gaze information, the processor 120 Dimensional location information can be obtained. The processor 120 uses a first gaze direction and a second gaze direction to provide two-dimensional coordinates (eg, x coordinate values and y coordinate values) in the entire space for a gaze point at which the user gazes. Can be obtained.

The processor 120 may determine a measurement parameter of the depth sensor using the 2D location information of the gaze point. For example, the processor 120 may determine a parameter for the target area using 2D location information of the gaze point. The processor 120 may determine a predetermined region of interest based on the gaze point as the target region.

The depth sensor 150 may acquire depth information based on the determined parameter for the target area. For example, when the light emission unit 170 emits light, the light emission area is limited to a target area (a region corresponding to the gaze point), or when the sensor unit 180 senses the reflected light, the light sensing area May be limited to a target region (a region corresponding to the gaze point). Accordingly, the depth sensor 150 may obtain depth information on objects included in the target area (the gaze point and the area surrounding the gaze point), not the entire space.

Referring to FIG. 5B, the gaze tracking sensor 160 according to an embodiment may obtain gaze information of a user and transmit gaze information to the processor 120. The processor 120 may obtain information on the estimated depth of the gaze point based on the gaze information of the user. For example, the processor 120 may obtain estimated depth information (eg, z coordinate value) of the gaze point based on the user's first gaze direction and second gaze direction included in the gaze information.

The processor 120 according to an embodiment determines a parameter for the emitted light output (eg, the emitted light output pattern, the size of the emitted light output) of the depth sensor 150 using the estimated depth information of the gaze point. I can.

For example, when the estimated depth is small (when the distance is close), the processor 120 shortens the output pattern (width of the optical pulse) of light emitted from the light emitting unit 170, and when the estimated depth is large (distance Is far away), a parameter of the depth sensor 150 may be determined to lengthen the output pattern (width of the light pulse) of light emitted from the light emitting unit 170. In addition, when the sensor unit 180 senses the reflected light, when the estimated depth is small, the sensing speed or the sensing period is accelerated, and when the estimated depth is large, the parameter of the depth sensor 150 is to slow the sensing speed or the sensing period. Can be determined. Accordingly, the depth sensor 150 may obtain depth information on objects included in the ROI based on the determined output pattern (width of the light pulse) and a sensing speed or a parameter for a sensing period. In this case, the region of interest may be a preset region centered on the gaze point.

In addition, when the estimated depth of the gaze point is small, the processor 120 reduces the output of the light emitted from the light emitting unit 170, and when the estimated depth is large, the output of the light emitted from the light emitting unit 170 is increased. The parameters of the depth sensor 150 may be determined so as to be performed. In addition, when sensing the reflected light from the sensor unit 180, the electronic device 100 determines a parameter of the depth sensor 150 to increase the sensitivity when the estimated depth is small, and when the estimated depth is large, I can. Accordingly, the depth sensor 150 may acquire depth information on objects included in the ROI in consideration of the determined output size and sensing sensitivity. In this case, the region of interest may be a preset region centered on the gaze point.

Hereinafter, examples in which measurement parameters of the depth sensor are determined based on information on the gaze point (two-dimensional position information of the gaze point, and estimated depth information of the gaze point) will be described in detail according to the type of the depth sensor. do.

6 to 7B are diagrams referenced for explaining a method of determining measurement parameters by an electronic device according to an exemplary embodiment when a depth sensor according to an exemplary embodiment is a TOF method.

The depth sensor 150 according to an embodiment may include a TOF type depth sensor 610.

The TOF method is a method of measuring the distance d to the object 620 by analyzing the time when light is reflected off the object and returned. The TOF type depth sensor 610 may include a light emitting unit 630 (emitter) and a sensor unit 640.

The light emitting unit 630 may be disposed to surround the sensor unit 640, but is not limited thereto, and the sensor unit 640 may be disposed to surround the light emitting unit 630.

The light emitting unit 630 may include a light source that generates light having a predetermined wavelength, and the light source is an infrared light emitting diode (IR LED) capable of emitting light having an infrared wavelength invisible to the human eye Alternatively, it may be a laser diode (LD). However, the present invention is not limited thereto, and the wavelength band and the type of light source may be variously configured. The light emitting unit 630 may emit light to an object by driving a light source according to a control signal. For example, by repeating on/off (0ff) of the light source, light may be emitted to the object 620.

The sensor unit 640 may sense reflected light reflected from the object 620 and returned. For example, the sensor unit 640 may include a light sensing device such as a pinned photo diode (PPD), a photogate, and a charge coupled device (CCD). The sensor unit 640 may include a plurality of sensors arranged in an array, and one cell 650 included in the array includes an in phase receptor 651 and an out phase receptor 652 ) Can be configured as a pair. At this time, the in-phase receptor 651 is activated only when it is in the in-phase state (while the light emitter is emitting light) to detect the light, and the out-phase receptor 652 is in the out phase state (the light is emitted from the light emitter). It is activated only when it is not) and can detect light.

7A is a graph showing light emitted from a light emitting unit, reflected light, light received from an in-phase receptor, and light received from an out-phase receptor.

The light emitted from the light emitting unit 630 is reflected by an object 620 that is a certain distance (d) away from the depth sensor 610 and returned, and the reflected light 720 is a certain distance from the emitted light 710 There is a time delay as much as possible. For example, light cannot be received during a certain period of the operation period of the in-phase receptor, which is activated only when the light emitter 630 emits light, and on the contrary, light reflected during a certain period of the operation period of the out-phase receptor. Can be received. In addition, the in phase receptor and the out phase receptor can measure the amount of light received. For example, the in-phase receptor and out-phase receptor receive light reflected from an object, generate electrons, and accumulate them, thereby measuring the amount of accumulated electrons (charge amount). The distance d to the object 620 may be calculated as in Equation 4 below.

At this time, c denotes the speed of light, t denotes the length of the light pulse 710, q1 denotes the accumulated charge amount (the electric charge measured by the in-phase receptor), and q2 denotes the light When not released, it refers to the amount of accumulated charge (the amount of charge measured at the out-phase receptor). That is, as the distance increases, the time point at which the reflected light starts to be received is delayed, and accordingly, q2 is relatively increased compared to q1. Accordingly, the TOF type depth sensor 610 may calculate depth information using Equation 4.

Meanwhile, the electronic device 100 according to an embodiment may determine a gaze point among the entire space by using gaze information of the user. The electronic device 100 may obtain gaze information of the user using the gaze tracking sensor 160 and acquire 2D location information of the gaze point based on the gaze information of the user.

The electronic device 100 according to an embodiment may control the light emitting unit 630 and the sensor unit 640 based on 2D location information of a gaze point.

Referring again to FIG. 6, according to an exemplary embodiment, the light emitting unit 630 may be divided into a plurality of regions, and the electronic device 100 may drive light sources for each region. The electronic device 100 may drive light sources included in an area corresponding to the 2D location information of a gaze point among a plurality of areas. For example, when the light emitting unit 630 is divided into four areas A1, A2, A3, and A4, the electronic device 100 may have a first area A1 corresponding to the 2D location information of the gaze point. The light sources included in) may be driven and the light sources included in the remaining second to fourth areas A2, A3, and A4 may not be driven.

Accordingly, power consumption due to driving of the light source can be reduced. In addition, the sensor unit 640 may be divided into a plurality of areas, and the electronic device 100 may drive only sensors included in an area corresponding to the 2D location information of the gaze point among the plurality of areas. . For example, when the sensor unit 640 is divided into four areas B1, B2, B3, and B4, the electronic device 100 may have a first area B1 corresponding to the 2D location information of the gaze point. By driving only the sensors included in the signal can be sensed. Accordingly, it is possible to reduce power consumption due to calculation of depth information.

The electronic device 100 according to an embodiment drives only a light source in an area corresponding to the 2D position information of the gaze point in the light emitting unit 630 included in the depth sensor, based on the 2D position information of the gaze point. , The sensor unit 640 may sense a signal by driving only sensors included in an area corresponding to the 2D location information of the gaze point, and calculate depth information.

In addition, the electronic device 100 according to an embodiment may determine the length and size of the emitted light pulse 710 as optimized parameters based on the estimated depth information of the gaze point. For example, the electronic device 100 may determine the length and size (output of light) of the optical pulse 710 based on the estimated depth information of the gaze point. For example, when the estimated depth is small (the distance is close), the electronic device 100 shortens the length (wavelength of light) of the light pulse 710 projected from the light emitting unit 630, and the light pulse ( 710) can be reduced. In addition, when the estimated depth is large (if the distance is long), the length (wavelength of the light) of the light pulse 710 projected from the light emitting unit 630 may be lengthened, and the output of the light pulse 710 may be increased. .

In addition, when the sensor unit 830 senses the reflected light pulse, if the estimated depth is small, the sensing speed may be increased and the sensing sensitivity may be decreased. In addition, when the estimated depth is large, the sensing speed may be slowed, and the sensing sensitivity may be increased. As such, as the measurement parameters of the depth sensor are determined as parameters optimized for the gaze point, the accuracy of depth information for the gaze point may be improved.

Also, the electronic device 100 may not calculate depth information for the entire space, but may calculate depth information only for the ROI. Accordingly, the speed of calculating the depth information can be increased and power consumption can be reduced.

In addition, the electronic device 100 according to an embodiment may re-optimize the measurement parameters of the depth sensor based on depth information (actual depth information) calculated by the depth sensor, and use the optimized measurement parameters again, You can perform depth sensing. Accordingly, the accuracy of depth information may be further improved.

7B is a diagram referenced to explain a method of determining measurement parameters of a depth sensor by using a matching table by an electronic device according to an exemplary embodiment.

The matching table 750 illustrated in FIG. 7B is a table that correlates the user's gaze information and a measurement parameter of the depth sensor, and may be pre-stored in the electronic device 100 according to an exemplary embodiment. The electronic device 100 according to an embodiment may control the light emitting unit 630 and the sensor unit 640 of FIG. 6 using the matching table 750.

The electronic device 100 acquires two-dimensional position information (x coordinates, y coordinates) and estimated depth information (z coordinates) of the gaze point based on the user's gaze information acquired from the gaze tracking sensor, and the matching table 750 ) May be used to determine a measurement parameter corresponding to the obtained 2D position information and estimated depth information.

For example, when the two-dimensional position information of the gaze point is (3, 4) and the estimated depth information z value is 3, the electronic device 100 is included in the first area A1 of the light emitting unit 630 The light sources may be driven, and light sources included in the remaining second to fourth areas A2, A3, and A4 may not be driven. Further, the electronic device 100 may sense a signal by driving only sensors included in the first area B1 of the sensor unit 640. In addition, the electronic device 100 sets the size of the light pulse 710 output from the light emitting unit 630 to 2, the cycle of the light pulse 710 to 2 ms, and the duty cycle of the light pulse 710 to 10. The light emission unit 630 may be controlled to be %.

Alternatively, when the 2D location information of the gaze point is (-2, -5) and the estimated depth information z value is 5, the electronic device 100 is included in the third area A3 of the light emitting unit 630 The light sources may be driven, and light sources included in the remaining first, second, and fourth areas A1, A2, and A4 may not be driven. Further, the electronic device 100 may sense a signal by driving only sensors included in the third area B3 of the sensor unit 640. In addition, the electronic device 100 sets the optical pulse 710 output from the light emitting unit 630 to 2, the optical pulse 710 period to 2 ms, and the optical pulse 710 duty cycle to 20%. The emission unit 630 can be controlled.

According to the electronic device 100 according to the embodiment, based on the user's gaze information (two-dimensional position information and estimated depth information of the gaze point) acquired from the gaze tracking sensor 160, the measurement parameter of the depth sensor (eg, For example, a parameter for a target area, a parameter for emission light output, a parameter for reflected light sensing, etc.) can be calculated. In this case, the electronic device 100 may calculate a measurement parameter of the depth sensor using a preset equation or algorithm.

The electronic device 100 may control the light emitting unit 630 and the sensor unit 640 of the depth sensor using the measurement parameter calculated in real time.

Meanwhile, the electronic device 100 according to an embodiment may perform wired or wireless communication (eg, Wi-Fi, Bluetooth, Zigbee, infrared, etc.) with an external device. The electronic device 100 may transmit the user's gaze information (two-dimensional position information and estimated depth information of the gaze point) acquired from the gaze tracking sensor 160 to an external device.

For example, the matching table 750 of FIG. 7B may be stored in an external device connected to the electronic device 100 through wired or wireless communication. The external device is based on the gaze information received from the electronic device 100 and the matching table 750, based on the measurement parameters of the depth sensor (for example, a parameter for a target area, a parameter for emission light output, a parameter for reflected light sensing). Parameters, etc.) can be determined.

Alternatively, the external device may measure parameters of the depth sensor in real time based on the gaze information received from the electronic device 100 (e.g., a parameter for a target area, a parameter for emission light output, a parameter for reflected light sensing, etc.) ) Can be calculated. In this case, the external device may calculate the measurement parameter using a preset equation or algorithm.

The external device may transmit the measurement parameter of the depth sensor to the electronic device 100, and the electronic device 100 may use the received measurement parameter of the depth sensor to provide the light emitting unit 630 and the sensor unit 640 of the depth sensor. ) Can be controlled. However, it is not limited thereto.

8 and 9 are diagrams referenced for explaining a method of determining measurement parameters by an electronic device according to an exemplary embodiment when a depth sensor according to an exemplary embodiment is a structured light (SL) method.

The depth sensor 150 according to an embodiment may include a structured light (SL) type depth sensor. The structured light method is a method of measuring the distance (depth information) to the object 830 by shining light with a pattern onto the object 830 and analyzing the shape and position of the pattern formed on the surface of the object 830 . In general, the SL type depth sensor projects light having a linear pattern or dot pattern onto the object 830, and the pattern or pattern of light formed on the object is changed according to the curvature of the object 830. The structured optical method may include an optical projector 810 and a camera 820, and one of the two cameras used in the depth sensor of the stereo image method may be replaced with the optical projector 810. . In general, a film having a fixed pattern or a liquid crystal film capable of changing a pattern shape is disposed on a path of light projected from the light projector 810, and the pattern or pattern of light is changed by passing the light through the film. For example, the SL-type depth sensor may calculate depth information by analyzing the shape and position of a pattern generated by light projected by the light projector 810 on the surface of the object 830 using an algorithm.

Meanwhile, the electronic device 100 according to an embodiment may determine a gaze point among the entire space by using gaze information of the user. The electronic device 100 may obtain gaze information of the user using the gaze tracking sensor and acquire 2D location information of the gaze point based on the gaze information of the user. The electronic device 100 according to an embodiment may control the optical projector 810 and the camera 820 based on 2D location information of a gaze point.

The electronic device 100 according to an embodiment may control the light projector 810 to project light only to a region corresponding to the 2D location information of a gaze point. For example, the light projector 810 projects light to pass through the second region C2 corresponding to the two-dimensional position information of the gaze point by changing the pattern of the liquid crystal film, and the first region, the third region, and Light passing through the fourth regions C1, C3, and C4 may not be projected. Accordingly, power consumption due to driving of the light source can be reduced.

In addition, the camera 820 may also be divided into a plurality of areas D1, D2, D3, and D4, and the electronic device 100 may be divided into two areas D1, D2, D3, and D4. Depth information may be calculated using only image signals acquired in an area corresponding to the dimensional location information.

For example, the first image 910 illustrated in FIG. 9 is an image indicating that light having a fixed pattern as a whole is projected in an actual space. The camera 820 may photograph a pattern generated by condensing the projected light on the surface of the object. For example, the second image 920 shown in FIG. 9 is an image obtained by photographing a pattern generated by light projected over an actual space. The electronic device 100 may calculate depth information on an actual space by analyzing the second image 920.

The electronic device 100 according to an embodiment may determine a gaze point in the entire space by using gaze information of both eyes of the user. The electronic device 100 may acquire gaze information of both eyes of the user using the gaze tracking sensor, and acquire 2D location information of the gaze point based on the gaze information of both eyes of the user.

The electronic device 100 according to an embodiment, when 2D location information of a gaze point is obtained based on gaze information of both eyes of the user, based on the 2D location information of the gaze point, transmits light having a fixed pattern. You can only project on the area of interest. The electronic device 100 may determine a predetermined area centered on a gaze point to which the user's both eyes are directed as the ROI. For example, as shown in FIG. 9, the light projector projects light having a fixed pattern only to the rear part 915 (region of interest) of the vehicle, and the camera shoots the region of interest 915 on which the light is projected. 3 An image 930 may be acquired.

Accordingly, the electronic device 100 does not calculate depth information for the entire space, but calculates depth information only for the ROI, thereby increasing the speed of calculating depth information and reducing power consumption.

FIG. 10 is a diagram referenced for explaining a method of determining measurement parameters by an electronic device according to an exemplary embodiment when the depth sensor according to an exemplary embodiment is a stereo image (SI) method.

The depth sensor 150 according to an embodiment may include a stereo image (SI) type depth sensor. The stereo image method refers to a method of photographing a three-dimensional effect of an object using two cameras. In this case, the depth sensor may include two cameras. The depth sensor may calculate depth information (distance) for a specific object based on the principle of triangulation, using difference information between images viewed by each camera. A person feels a three-dimensional effect through the difference between the image entering the left eye and the right eye, and the depth sensor measures the distance in a manner similar to the principle that the human eye feels the three-dimensional effect. For example, if the depth is small (closer distance), the difference between the images captured by two cameras is large, and if the depth is large (farther distance), the difference between images captured by two cameras is small.

In the case of the stereo image method, since two images must be simultaneously processed in real time, a fast processing power of the processor is required, and hardware processing is required. Therefore, it is difficult to perform real-time processing of a stereo image method with only a processor of a small device.

The SI type depth sensor according to an embodiment may include a first camera and a second camera, and in this case, the first camera and the second camera may photograph an actual space in different directions from different locations. . For example, a first camera may capture a real space from a first location in a first direction to obtain a first image 1010, and a second camera may take a real space from a second location in a second direction. Thus, the second image 1020 may be obtained. In this case, when the difference image between the first image 1010 and the second image 1020 is used, depth information for the entire actual space may be obtained.

Meanwhile, the electronic device 100 according to an embodiment may determine a region of interest among the entire space by using gaze information of both eyes of the user. The electronic device 100 may acquire gaze information of both eyes of the user using the gaze tracking sensor, and acquire 2D location information of the gaze point based on the gaze information of both eyes of the user. For example, when the gaze point of the electronic device user is the first point, the electronic device 100 may determine a predetermined area centered on the first point as the ROI. A first area 1015 of FIG. 10 represents an area of the first image 1010 corresponding to an ROI, and a second area 1025 represents an area of the second image corresponding to an ROI. Accordingly, the electronic device 100 may calculate depth information on the ROI by calculating only the difference image between the image for the first region 1015 and the images 1030 and 1040 for the second region 1025. .

According to an embodiment, when the 2D location information of the gaze point is obtained based on gaze information of both eyes of the user, the electronic device 100 according to an embodiment 2 It is possible to determine a region of interest in an image and calculate depth information for only the region of interest. Alternatively, the electronic device 100 may magnify the region of interest by using a zoom function, obtain images 1030 and 1040 for the region of interest, and calculate depth information only for the region of interest. Accordingly, the calculation speed of depth information can be increased, and power consumption can be reduced.

11 is a diagram for describing a method of displaying a virtual object by an electronic device according to an exemplary embodiment.

Referring to FIG. 11, the electronic device 100 according to an embodiment may display a virtual object on a display based on the acquired depth information of a gaze point. For example, the electronic device 100 may display a virtual object in the form of an augmented reality (AR). When displaying a virtual object in the form of augmented reality, the electronic device 100 displays the virtual object on the display so that the virtual object overlaps with the real space (1110, two-dimensional or three-dimensional space of the real world) observed through the display. can do.

For example, the electronic device 100 according to an embodiment acquires depth information of an area around a gaze point (eg, an area around a desk), and the depth acquired on the virtual object 1120 (eg, a vase) It can be displayed on the display so that the user perceives it as if it is located in the area around the desk by giving a depth similar to the information.

12 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

Referring to FIG. 12, the electronic device 100 according to an embodiment may acquire gaze information of both eyes of a user (S1210).

The electronic device 100 according to an embodiment may provide light to the user's eyes (left eye, right eye) and detect the amount of light reflected from the user's eye using a gaze tracking sensor. The electronic device 100 may determine the gaze directions of both eyes of the user based on the detected amount of light.

Alternatively, the electronic device 100 may provide light to the user's eyes and photograph the user's eyes using a gaze tracking sensor. Also, the electronic device 100 may determine the gaze directions of both eyes of the user based on the captured user's eye image.

The electronic device 100 according to an embodiment may acquire a gaze point based on gaze information (S1220).

The electronic device 100 may obtain 2D coordinate information (x-coordinate value and y-coordinate value) for a point gazed by the user based on the gaze direction of the user's right eye and the gaze direction of the left eye. In addition, a distance (z coordinate value) to a point where the user gazes may be estimated based on the gaze direction of the user's right eye and the gaze direction of the left eye. Accordingly, the electronic device 100 may obtain 2D location information and estimated depth information of the gaze point.

The electronic device 100 according to an embodiment may determine a measurement parameter of the depth sensor based on the information on the gaze point (S1230).

The measurement parameter of the depth sensor may include at least one of a parameter for a target area, a parameter for an emission light output (emission light output pattern, an emission light output size), and a parameter for reflected light sensing. For example, the electronic device 100 may determine a parameter for a target area using 2D location information of a gaze point, and output light output (emit light output pattern, Emitted light output size) and reflected light sensing parameters may be determined. This has been described in detail with reference to FIGS. 5A to 10, so a detailed description will be omitted.

The electronic device 100 according to an embodiment may acquire depth information on at least one object included in a preset ROI based on a gaze point based on the determined measurement parameter.

13 is a diagram for describing a method of obtaining depth information by an electronic device according to an exemplary embodiment.

The depth sensor according to an exemplary embodiment may include at least one camera, and may obtain depth information on an actual space 1310 included in a field of veiw (FOV) of a camera included in the depth sensor. Hereinafter, a space within a range that can be sensed by the depth sensor will be referred to as a “total space”.

According to an embodiment, as illustrated in FIG. 13, the electronic device 100 may determine an ROI 1320 of the total space 1310.

For example, as described in FIGS. 2 to 3D, the electronic device 100 acquires gaze information of both eyes of the user using gaze tracking sensors, and based on the gaze information of both eyes of the user, the gaze point of the user Can be obtained. In addition, the region of interest may be determined in a predetermined region based on the gaze point based on the obtained gaze point. Alternatively, the electronic device 100 acquires an image of the entire space 1310 and recognizes a major object (eg, a person, a face, a hand, etc.) within the determined region of interest using object recognition technology, It is possible to determine the recognized main object as the region of interest rather than a preset region centering on.

When the ROI 1320 is determined, the electronic device 100 may obtain depth information for the ROI 1320 and the remaining spaces excluding the ROI by using different measurement parameters.

For example, the electronic device 100 may obtain depth information 1330 for the ROI by setting the measurement parameter of the depth sensor as the first parameter. In this case, the electronic device 100 may set the first parameter based on the information on the gaze point. For example, the electronic device 100 sets a preset area centered on the gaze point as a region of interest based on 2D location information of the gaze point, and senses a region emitting light to correspond to the set region of interest, and light. You can set the area to be used. In addition, the electronic device 100 may set a pattern of emitted light or a light output based on the estimated depth information of the gaze point.

For example, when the depth sensor is a TOF type depth sensor, the sensor unit senses signals corresponding to the region of interest by increasing the sampling rate or decreasing the sampling period, and does not sense signals corresponding to the remaining regions. By not doing so, it is possible to obtain depth information on the region of interest in high resolution.

In addition, the electronic device 100 may obtain depth information 1340 for the remaining areas excluding the ROI by setting the measurement parameter of the depth sensor as the second parameter. For example, when the depth sensor is a TOF-type depth sensor, the sensor unit senses signals corresponding to the remaining areas with a low sampling rate or increasing the sampling period, and does not sense signals corresponding to the ROI. By not doing so, it is possible to obtain depth information for the rest of the region at a low resolution.

Alternatively, depth information on the entire space including the region of interest may be obtained at a low resolution.

Accordingly, the electronic device 100 obtains high-accuracy depth information (high-resolution depth information) for the region of interest, and also obtains approximate depth information (low-resolution depth information) for the remaining regions (regions around the gaze point). can do.

14 is a flowchart illustrating a method of obtaining depth information by an electronic device according to an exemplary embodiment.

Referring to FIG. 14, the electronic device according to an embodiment may determine an ROI from among the entire space (S1410 ). Since the method of determining the region of interest has been described in detail in FIG. 13, a detailed description will be omitted.

The electronic device 100 may obtain depth information on the entire space by setting the measurement parameter of the depth sensor as the first parameter set (S1420).

For example, the electronic device 100 may obtain low-resolution depth information for the entire space.

The electronic device 100 may obtain first depth information on the ROI based on depth information on the entire space (S1430).

For example, depth information on an ROI, included in depth information on the entire space, may be determined as the first depth information.

The electronic device 100 may determine a second parameter set based on the first depth information (S1440). For example, the electronic device 100 may determine a parameter for emission light output (emission light output pattern, emission light output size) and reflected light sensing of the depth sensor based on the first depth information of the region of interest. have.

The electronic device 100 may obtain second depth information on the ROI by using a depth sensor in which the measurement parameter is set as the second parameter set (S1450). In this case, the second depth information may be high-resolution depth information, and may be depth information having improved accuracy than the low-resolution depth information obtained in step 1420 (S1420).

15 is a diagram illustrating an example in which an electronic device repeatedly performs depth information acquisition operations of FIG. 14 according to an exemplary embodiment.

Referring to FIG. 15, the electronic device 100 according to an embodiment obtains depth information 1510 (low-resolution depth information) for the entire space (S1420) and second depth information 1520 for a region of interest (high resolution). The operation of obtaining depth information) (S1450) may be repeatedly performed with a certain period.

For example, a first period T1 for obtaining depth information 1510 for the entire space and a second period T2 for obtaining second depth information 1520 may be set. In this case, the electronic device 100 may adjust the first period T1 according to the movement of the electronic device 100 to adjust the update period of depth information of the remaining areas except for the ROI. That is, the first period T1 may be adjusted according to the size of the amount of change in which the remaining areas other than the ROI in the image generated by the movement of the electronic device 100 move. For example, when the movement of the electronic device 100 is small (when the electronic device 100 is static), the first period T1 may be increased, and when the movement of the electronic device 100 is large (electronic device 100 When the device 100 is dynamic), the first period T2 may be reduced.

Also, the electronic device 100 may adjust the second period T2 according to the minimum time required for an interaction between the electronic device user and the virtual object displayed at the gaze point. For example, the electronic device 100 may set the second period T2 to be equal to or shorter than a minimum time required for updating depth information on a hand for an interaction such as hand gesture recognition. However, it is not limited thereto.

16 is a diagram illustrating an example in which an electronic device provides a virtual object in an augmented reality (AR) method according to an exemplary embodiment.

Referring to FIG. 16, the electronic device 100 according to an embodiment may include at least one camera (image sensor). For example, at least one camera may be a depth camera included in the depth sensor or a camera separately provided from the depth sensor.

At least one camera may acquire an image 1610 corresponding to a space included in the FOV of the camera. The electronic device may detect a main object within the acquired image 1610. For example, the electronic device may display the virtual object 1630 in an augmented reality (AR) method so that the user recognizes as if the virtual object 1630 is positioned around the real object 1620. In addition, when an electronic device user interacts with the virtual object 1630 using a hand, the main object may be the user's hand 1640.

The electronic device 100 may detect a region of the hand 1640 in the acquired image, determine the region of the hand as the region of interest, and determine the remaining region excluding the region of the hand 1640 as a background region.

The electronic device 100 may obtain high-resolution depth information for an ROI and low-resolution depth information for a background region. For example, for the area of the hand 1640, the electronic device 100 may obtain high-resolution depth information by setting a measurement parameter of the depth sensor so as to obtain depth information with high accuracy. On the other hand, for a background region, low-resolution depth information may be obtained by setting a measurement parameter of a depth sensor to obtain depth information with low accuracy.

The electronic device 100 may estimate a hand pose or recognize a hand gesture by using high-resolution depth information on a region of the hand 1640. On the other hand, camera pose tracking or background modeling may be performed using low-resolution depth information on the background area. However, it is not limited thereto.

17 is a diagram illustrating an example in which an electronic device recognizes a human face using depth information according to an exemplary embodiment.

Referring to FIG. 17, the electronic device 100 according to an embodiment may include at least one camera. For example, at least one camera may be a depth camera included in the depth sensor or a camera separately provided from the depth sensor. At least one camera may acquire an image including a face.

The electronic device 100 may detect a main feature area within the acquired face image. For example, in a person's face, the eyes, nose, and mouth areas may be important areas to distinguish from other people, and the electronic device 100 may detect the eyes, nose, and mouth areas within the face image. have. The electronic device 100 according to an exemplary embodiment may acquire depth information with high resolution for the eye, nose, and mouth regions 1540 of the face and low resolution for the remaining regions.

The first depth image 1710 of FIG. 17 represents depth information acquired with low resolution for the entire face area, and the second depth image 1720 represents depth information acquired with high resolution over the entire face area. In addition, the third depth image 1730 includes depth information obtained by the electronic device 100 in high resolution for the eye, nose, and mouth regions 1740 of the face, and low resolution for the remaining regions. Show.

When performing face recognition (identity recognition) using the third depth image 1730 according to an embodiment, recognition performance (recognition accuracy) is better than when performing face recognition using the first depth image 1710 This may be improved, and a recognition speed may be faster than when face recognition is performed using the second depth image 1720. In addition, in the case of recognizing a face of a distant person, the resolution of the main feature portion is increased to obtain a depth image, thereby improving recognition performance.

18 is a block diagram illustrating a configuration of an electronic device according to an exemplary embodiment.

Referring to FIG. 18, the electronic device 100 according to an embodiment may include a gaze tracking sensor 160, a depth sensor 150, and a processor 120.

The gaze tracking sensor 160 according to an embodiment may include an illumination unit that provides light to a user's eyes and a detection unit that detects light. The illumination unit may include a light source for providing light and a scanning mirror for controlling a direction of light provided from the light source. The scanning mirror may control a direction so that light provided from a light source is directed toward the user's eyes (eg, cornea). The sensing unit may detect light reflected from the user's eyes, and measure the amount of light to be detected. The gaze tracking sensor 160 may track the gaze of both eyes of the user based on the measured amount of light.

Alternatively, the gaze tracking sensor 160 according to an embodiment may include an illumination unit and a photographing unit. The lighting unit may include an infrared light-emitting diode (IR LED), and may provide light (eg, infrared light) to the user's eye when photographing the user's eye. As light is provided to the user's eye, reflected light may be generated in the user's eye. In addition, the photographing unit may include at least one camera, and in this case, at least one camera may include an infrared camera (IR). The photographing unit may photograph the user's eyes. The gaze tracking sensor 160 may track the gaze of both eyes of the user based on the user's eye image.

The depth sensor 150 according to an embodiment may obtain depth information on one or more objects included in the real world. The depth information may correspond to a distance from the depth sensor 150 to a specific object, and a depth value may increase as the distance from the depth sensor 150 to a specific object increases. The depth sensor 150 according to an embodiment may acquire depth information of an object in various ways, for example, a TOF (Time of Flight) method, a stereo image method, a structured light ) Depth information may be obtained by using at least one of the methods.

The depth sensor 150 according to an embodiment may include at least one camera, and may obtain depth information on an actual space included in a field of view (FOV) of a camera included in the depth sensor 150. have.

The processor 120 according to an embodiment may generally control the electronic device 100. The processor 120 according to an embodiment may execute one or more programs stored in a memory.

The memory according to an embodiment may store various data, programs, or applications for driving and controlling the electronic device 100. A program stored in the memory may include one or more instructions. A program (one or more instructions) or an application stored in the memory may be executed by the processor 120.

The processor 120 according to an embodiment may obtain information on a gaze point based on the user's gaze information obtained from the gaze tracking sensor 160. For example, the processor 120 may obtain two-dimensional coordinate information (x-coordinate value and y-coordinate value) for a point at which the user gazes, based on the gaze direction of the user's right eye and the gaze direction of the left eye. I can. Also, a distance (z coordinate value) to a point at which the user gazes may be obtained based on the gaze direction of the user's right eye and the gaze direction of the left eye. Accordingly, the electronic device 100 may obtain 2D location information and estimated depth information of the gaze point.

The processor 120 according to an embodiment may set a predetermined area centered on the gaze point as the area of interest based on the information on the gaze point and obtain depth information on at least one object included in the area of interest. have. For example, the processor 120 may determine measurement parameters of the depth sensor 150 based on the information on the gaze point. Measurement parameters of the depth sensor 150 may include a parameter for a target area, a parameter for an emission light pattern, a parameter for an emission light output, and the like. For example, the processor 120 may determine a parameter for the target area by using the 2D position information of the gaze point, and use the estimated depth information of the gaze point, the parameter for the emission light pattern and the emission light output. You can determine the parameters for.

This has been described in detail with reference to FIGS. 5 to 10, so a detailed description will be omitted. The processor 120 may obtain depth information on at least one object included in the ROI based on the determined measurement parameters.

The processor 120 according to an embodiment may set the measurement parameter of the depth sensor as the first parameter set to obtain low-resolution depth information for the entire space. In addition, the processor 120 may determine a second parameter set based on depth information on the ROI included in the low-resolution depth information. The processor 120 may obtain high-resolution depth information on the ROI by setting the measurement parameter of the depth sensor 150 as the second parameter set.

19 is a block diagram illustrating a configuration of an electronic device according to another embodiment. The electronic device 1900 of FIG. 19 may be an embodiment of the electronic device 100 of FIG. 18.

Referring to FIG. 19, an electronic device 1900 according to an embodiment includes a sensing unit 1910, a memory 1960, a control unit 1930, an output unit 1920, a user input unit 1940, and a communication unit 1950. It may include.

Since the control unit 1930 and the memory 1960 of FIG. 19 correspond to the processor 120 and the memory 130 of FIG. 18, respectively, the same description will be omitted.

The sensing unit 1910 may detect a state of the electronic device 1900 or a state around the electronic device 1900, and transmit the sensed information to the controller 1930.

The sensing unit 1910 includes an image sensor 1911, a depth sensor 1912, a gaze tracking sensor 1913, an acceleration sensor 1914, a position sensor 1915 (eg, GPS), and temperature/humidity. It may include at least one of a sensor 1916, a geomagnetic sensor 1917, a gyroscope sensor 1918, and a microphone 1919, but is not limited thereto.

The image sensor 1911 according to an embodiment may acquire an image frame such as a still image or a moving image. For example, the image sensor 1911 may capture an image outside the electronic device 1900. In this case, the image captured by the image sensor 1911 may be processed by the controller 1930 or a separate image processor (not shown).

Since the depth sensor 1912 and the gaze tracking sensor 1913 of FIG. 19 correspond to the depth sensor 150 and the gaze tracking sensor 160 of FIG. 18, the same description will be omitted.

The microphone 1919 receives an external sound signal and processes it into electrical voice data. For example, the microphone 1919 may receive an acoustic signal from an external device or a speaker. The microphone 1919 may use various noise removal algorithms to remove noise generated in the process of receiving an external sound signal.

The functions of the acceleration sensor 1914, the position sensor 1915, the temperature/humidity sensor 1916, the geomagnetic sensor 1917, and the gyroscope sensor 1918 can be intuitively inferred by those skilled in the art from their names. , Detailed description will be omitted.

The output unit 1920 is for outputting an audio signal, a video signal, or a vibration signal, and may include a display 1921, an audio output unit 1922, a vibration motor 1923, and the like.

The display 1921 may display and output information processed by the electronic device 1900. For example, the display 1921 may display a virtual object.

According to an embodiment of the present invention, the display 1921 may be a transparent display or an opaque display. The transparent display refers to an information display device in which the back side of a screen displaying information is reflected. The transparent display is composed of a transparent element, and the transparency can be adjusted by adjusting the light transmittance of the transparent element or the RGB value of each pixel.

The sound output unit 1922 outputs audio data received from the communication unit 1950 or stored in the memory 1960. In addition, the sound output unit 1922 outputs sound signals related to functions (eg, a call signal reception sound, a message reception sound, etc.) performed by the electronic device 1900. The sound output unit 1922 may include a speaker, a buzzer, and the like.

According to an embodiment of the present invention, when an input is generated through a virtual input interface, the sound output unit 1922 may output an audio signal corresponding to the generated input.

The vibration motor 1923 may output a vibration signal. For example, the vibration motor 1923 may output a vibration signal corresponding to the output of audio data or video data (eg, a call signal reception sound, a message reception sound, etc.). In addition, the vibration motor 1923 may output a vibration signal when an input through a virtual object is generated.

The user input unit 1940 refers to a means for a user to input data for controlling the electronic device 1900. For example, the user input unit 1940 includes a key pad, a dome switch, and a touch pad (contact type capacitive type, pressure type resistive film type, infrared sensing method, surface ultrasonic conduction method, integral type). Tension measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like, but are not limited thereto. According to an embodiment of the present invention, the user input unit 1940 may include a virtual input interface.

The communication unit 1950 may include one or more components that enable communication between the electronic device 1900 and an external device or between the electronic device 1900 and a server. For example, the communication unit 1950 may include a short-range communication unit 1951, a mobile communication unit 1952, and a broadcast reception unit 1953.

The short-range communication unit 1951 includes a Bluetooth communication unit, a short-range wireless communication unit (NFC/RFID unit), a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared data association (IrDA) communication unit, an ultra wideband (UWB) communication unit, and Ant+. It may include a communication unit, etc., but is not limited thereto.

For example, the communication unit 1950 may transmit the user's gaze information (two-dimensional position information and estimated depth information of the gaze point) obtained from the gaze tracking sensor 1913 to an external device. A measurement parameter of the depth sensor corresponding to the gaze information may be received.

The mobile communication unit 1952 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server over a mobile communication network. Here, the wireless signal may include a voice call signal, a video call signal, or various types of data according to transmission/reception of text/multimedia messages.

The broadcast receiving unit 1953 receives a broadcast signal and/or broadcast-related information from outside through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. Depending on implementation examples, the electronic device 1900 may not include the broadcast receiving unit 1953.

The memory 1960 may store a program for processing and control of the controller 1930, and input/output data (e.g., gesture information corresponding to an input mode, a virtual input interface, a virtual input interface). It can also store input data, sensing information measured by a sensor, content, etc.).

The memory 1960 according to an embodiment may store the matching table 750 illustrated in FIG. 7B. Alternatively, the memory 1960 may store an equation or an algorithm for calculating a measurement parameter of a depth sensor based on gaze information according to an embodiment.

For example, the matching table 750 may be stored in a ROM. When driving the depth sensor, the controller 1930 may load the matching table 750 stored in the ROM into RAM, and use the loaded matching table 750 to match gaze information. It is possible to determine the measurement parameters to be used.

The controller 1930 may control the depth sensor 1912 to obtain depth information by using the determined measurement parameter by outputting the determined measurement parameter to the depth sensor 1912.

The memory 1960 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk And at least one type of storage medium among optical disks. Also, the electronic device 1900 may operate a web storage or a cloud server that performs a storage function of the memory 1960 over the Internet.

Programs stored in the memory 1960 may be classified into a plurality of modules according to their functions, for example, a UI module 1961, a notification module 1962, and the like.

The UI module 1961 may provide a specialized UI, GUI, etc. that are interlocked with the electronic device 1900 for each application. In addition, according to an embodiment of the present invention, the UI module 1961 may select and provide a virtual input interface suitable for a situation.

The notification module 1962 may generate a signal for notifying the occurrence of an event of the electronic device 1900. Examples of events occurring in the electronic device 1900 include call signal reception, message reception, key signal input through a virtual input interface, schedule notification, and the like. The notification module 1962 may output a notification signal in the form of a video signal through the display 1921, may output a notification signal in the form of an audio signal through the sound output unit 1922, or the vibration motor 1923. It is also possible to output a notification signal in the form of a vibration signal. Also, the notification module 1962 may output a haptic signal using an external device.

Meanwhile, the block diagrams of the electronic devices 100 and 1900 shown in FIGS. 18 and 19 are block diagrams for an embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the electronic devices 100 and 1900 that are actually implemented. That is, if necessary, two or more components may be combined into a single component, or one component may be subdivided into two or more components and configured. In addition, functions performed by each block are for explaining embodiments, and specific operations or devices thereof do not limit the scope of the present invention.

20 and 21 are diagrams for describing a method of automatically adjusting focus by an electronic device according to an exemplary embodiment.

Referring to FIG. 20, when an electronic device according to an embodiment displays a virtual object 2020 as if a virtual object is located around a real object 2010, a user may have a convergence-accommodation conflict. Experience. For example, when the distance from the electronic device 100 to the real object 2010 is d1, the electronic device 100 may display the virtual object 2020 as if it is located at a distance d1. At this time, since the user sees the virtual object 2020 as if it is located at a distance d1, the convergence distance of both eyes of the user becomes d1. On the other hand, since the virtual object 2020 is actually displayed on the display of the electronic device 100, the focal distance of both eyes of the user becomes the distance d2 from the user's eyes to the display. In this case, the convergence distance and the focal distance disagree, and when the electronic device 100 is used for a long time, the user feels dizziness, dizziness, motion sickness, and the like.

Accordingly, in order to alleviate a convergence-accommodation conflict, the electronic device 100 according to an embodiment may adjust the focal length.

Referring to FIG. 21, according to an embodiment, the electronic device 100 may include a focus adjustment lens 2110. The focus adjustment lens 2110 may mean an optical element capable of adjusting optical properties such as a focal length or an optical axis position, but is not limited thereto. For example, the effective refractive index of the focusing lens 2110 may be locally varied according to the applied voltage. In general, a liquid crystal may be used for the focus adjustment lens 2110, but is not limited thereto.

The electronic device 100 according to an embodiment may acquire the user's gaze information using the gaze tracking sensor, and based on the user's gaze information, the gaze point (for example, the real object 2010) Information on the gaze may be obtained, and depth information on the gaze point may be obtained based on the information on the gaze point. This will be omitted since it has been described in detail in FIGS. 1 to 19.

Also, the electronic device 100 may display the virtual object 2020 based on depth information on the real object 2010. For example, the electronic device 100 may display the virtual object 2020 on the display so that the user recognizes that a virtual object is positioned around the real object 2010 observed through the display.

The electronic device 100 according to an embodiment may adjust a focal length based on depth information (or depth information of a virtual object) on the real object 2010. For example, when the distance to the actual object 2010 is d1, the electronic device 100 may adjust the focal length of the user's eye to d1 using the focus adjustment lens 2110. In this case, the electronic device 100 acquires information on the first area 2121 and the second area 2122 through which the user's gaze passes among the entire areas of the focus adjustment lens 2110 based on the user's gaze information. can do. The electronic device 100 may change the refractive index so that the focal length of the first region 2121 and the second region 2122 becomes d1 by adjusting the voltage applied to the focus adjustment lens 2110. Accordingly, the convergence distance and the focal distance can be matched, and a convergence control inconsistency phenomenon can be prevented.

22 is a diagram for describing a method of performing a gaze-based spatial modeling by an electronic device according to an exemplary embodiment.

Referring to FIG. 22, the total space 2010 shown in FIG. 22 represents a space within a range that can be sensed by a depth sensor included in the electronic device. According to an embodiment, the electronic device 100 may acquire gaze information of a user using a gaze tracking sensor. For example, as described with reference to FIGS. 2 to 4B, 2D location information and depth information on a point or space at which the electronic device user gazes may be obtained. Based on the acquired information, the electronic device 100 may acquire depth information on a point or space (gaze point) at which the user gazes. This has been described in detail with reference to FIGS. 1 to 19, so a detailed description will be omitted. For example, when the user is staring at the chair 2220 among the entire space 2210, depth information on the chair 2220 may be obtained. The electronic device 100 may overlap and display a virtual object around the chair, based on depth information on the chair.

In addition, when the user's gaze moves, the electronic device 100 provides depth information on the space in which the user's gaze moves based on information on the space in which the user's gaze moves (2D location information and depth information). Can be obtained.

According to an embodiment, the electronic device 100 acquires gaze information of the user in real time, acquires a gaze point in the entire space based on the gaze information of the user, and selects a preset area around the obtained gaze point. By determining the region of interest, depth information may be obtained only for the determined region of interest. The electronic device 100 may perform modeling on the region of interest based on the acquired depth information.

Accordingly, the electronic device 100 may increase modeling speed and reduce power consumption by performing only modeling for a required space.

The method of operating an electronic device according to an exemplary embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

In addition, an electronic device and a method of operating the electronic device according to the disclosed embodiments may be included in a computer program product and provided. Computer program products can be traded between sellers and buyers as commodities.

The computer program product may include a S/W program and a computer-readable storage medium storing the S/W program. For example, the computer program product may include a product (e.g., downloadable app) electronically distributed through an electronic device manufacturer or an electronic market (e.g., Google Play Store, App Store). have. For electronic distribution, at least a part of the S/W program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing an SW program.

The computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device. Alternatively, when there is a third device (eg, a smart phone) communicating with a server or a client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a S/W program itself transmitted from a server to a client device or a third device, or transmitted from a third device to a client device.

In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of a server, a client device, and a third device may execute a computer program product to distribute and implement the method according to the disclosed embodiments.

For example, a server (eg, a cloud server or an artificial intelligence server) may execute a computer program product stored in the server to control a client device communicating with the server to perform the method according to the disclosed embodiments.

Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also the scope of the present invention. Belongs to.

Claims (21)

A gaze tracking sensor that acquires gaze information of both eyes of the user;
A depth sensor that obtains depth information on one or more objects; And
And a processor configured to obtain information on a gaze point based on the acquired gaze information and to determine a measurement parameter of the depth sensor based on the information on the gaze point. The method of claim 1,
The processor,
Based on the gaze information, the head-mounted display device for obtaining 2D position information of the gaze point. The method of claim 2,
The processor,
Based on the gaze information, the head mounted display device for obtaining information on the estimated depth of the gaze point. The method of claim 1,
The depth sensor,
According to the determined measurement parameter, the head-mounted display device for acquiring depth information on a predetermined region of interest based on the gaze point. The method of claim 1,
The measurement parameter of the depth sensor,
A head mounted display device comprising at least one of a parameter for a target area, a parameter for emission light output, and a parameter for reflected light sensing. The method of claim 4,
The processor,
Based on the obtained depth information on the region of interest, determining a measurement parameter of the depth sensor again,
The depth sensor,
Re-acquiring depth information on the region of interest according to the measurement parameter determined again. The method of claim 4,
The depth sensor,
A head-mounted display device for acquiring depth information on the ROI by at least one of a Time Of Flight (TOF) method, a Structured Light (SL) method, and a Stereo Image (SI) method. The method of claim 7,
When the depth sensor includes a TOF (Time Of Flight) depth sensor, the processor corresponds to the gaze point among light sources included in the depth sensor based on the two-dimensional position information of the gaze point. Determining the measurement parameter so that some light sources are driven,
The depth sensor, by driving the some light sources, to obtain depth information on the region of interest, the head mounted display device. The method of claim 4,
The head mounted display device,
Further comprising a display for displaying the actual space including the region of interest,
The processor, based on depth information on the region of interest, controls the display to display at least one virtual object in the region of interest. The method of claim 4,
The processor,
Setting the measurement parameter of the depth sensor as a first parameter,
Based on the first parameter, controlling the depth sensor to obtain total depth information on a space that can be sensed by the depth sensor while including the region of interest,
Acquiring first depth information on the ROI based on the total depth information,
Based on the first depth information, setting a measurement parameter of the depth sensor as a second parameter,
Based on the second parameter, controlling the depth sensor to obtain second depth information on the ROI. In the method of operating an electronic device,
Acquiring gaze information of both eyes of the user;
Obtaining information on a gaze point based on the obtained gaze information; And
And determining a measurement parameter of a depth sensor based on the information on the gaze point. The method of claim 11,
Obtaining information on the gaze point,
And acquiring two-dimensional position information of the gaze point based on the gaze information. The method of claim 12,
Obtaining information on the gaze point,
The method of operating a head-mounted display device further comprising obtaining information on the estimated depth of the gaze point based on the gaze information. The method of claim 11,
The operation method,
According to the determined measurement parameter, further comprising the step of acquiring depth information on a predetermined region of interest centering on the gaze point, the operating method of the head mounted display device. The method of claim 11,
The step of determining the measurement parameter of the depth sensor,
A method of operating a head mounted display device comprising determining at least one of a parameter for a target area, a parameter for emission light output, and a parameter for reflected light sensing. The method of claim 14,
The operation method,
Re-determining a measurement parameter of the depth sensor based on the obtained depth information on the ROI; And
Further comprising the step of re-acquiring depth information on the ROI according to the determined measurement parameter again. The method of claim 14,
The step of obtaining depth information on the region of interest,
A method of operating a head-mounted display device comprising obtaining depth information on the ROI by at least one of a Time Of Flight (TOF) method, a Structured Light (SL) method, and a Stereo Image (SI) method. The method of claim 17,
The step of determining the measurement parameter of the depth sensor,
When obtaining depth information by the TOF (Time Of Flight) method, based on the two-dimensional position information of the gaze point, some light sources corresponding to the gaze point among the light sources included in the depth sensor are driven. Determining a measurement parameter,
The step of obtaining depth information on the region of interest,
And acquiring depth information on the region of interest by driving the some light sources. The method of claim 14,
The operation method,
Displaying an actual space including the region of interest; And
The method of operating a head-mounted display apparatus further comprising displaying at least one virtual object in the region of interest based on depth information on the region of interest. The method of claim 14,
The step of obtaining depth information on the region of interest,
Setting a measurement parameter of the depth sensor as a first parameter, and acquiring total depth information on a space that can be sensed by the depth sensor based on the first parameter;
Obtaining first depth information on the ROI based on the total depth information; And
Setting a measurement parameter of the depth sensor as a second parameter based on the first depth information, and acquiring second depth information for the ROI based on the second parameter, head mount How to operate the display device. Acquiring gaze information of both eyes of the user;
Obtaining information on a gaze point based on the obtained gaze information; And
At least one computer-readable recording medium storing a program for performing the step of determining a measurement parameter of a depth sensor based on the information on the gaze point.

Images