KR102224930B1 - Method of displaying menu based on depth information and space gesture of user - Google Patents

Abstract

Based on depth information corresponding to the measured distance from the screen of the user terminal to the user's hand, at least one of the plurality of layers is identified, and a menu page corresponding to the identified at least one layer is displayed. A method of displaying a menu may be provided based on depth information for applying a graphic effect to at least one of the layers of.

Description

METHOD OF DISPLAYING MENU BASED ON DEPTH INFORMATION AND SPACE GESTURE OF USER

The following embodiments relate to a method of displaying a menu based on depth information and a user's spatial gesture.

In a 2D display based on a touch interface, a user directly touches a menu icon located in the 2D display to execute a desired application. This is a method of conveying the user's intention to the device by touching the user's hand on the screen, thereby implementing a highly reliable user interface (UI).

However, since the user must directly touch the display of the device, the user's intention cannot be communicated to the device when the hand is covered with liquid or gloves, or when the distance between the display and the user is long. In addition, in a 2D display based on a touch interface, a desired page must be touched to select a menu icon, or a page must be changed by dragging to move to the page where the menu icon is located.

According to an embodiment, a method of displaying a menu based on depth information includes: measuring depth information corresponding to a distance from a screen of a user terminal to a user's hand; Identifying at least one of a plurality of layers based on the depth information; And applying a graphic effect to at least one of the plurality of layers so that a menu page corresponding to the identified at least one layer is displayed.

The method further comprises detecting a movement from the first layer to the second layer based on the depth information, and in the step of applying the graphic effect, in response to the depth information, a menu page corresponding to the first layer is The method may include applying a graphic effect to at least one of the plurality of layers such that the menu page gradually disappears and the menu page corresponding to the second layer appears gradually.

The step of applying a graphic effect to at least one of the plurality of layers so that a menu page corresponding to the identified at least one layer appears, improves visibility of a menu page corresponding to the at least one identified layer. And reducing the visibility of the menu page corresponding to the other at least one layer.

The method further includes detecting a movement from the first layer to the second layer based on the depth information, and the applying of the graphic effect includes comparing a distance from the screen to a user's hand with a preset distance; And applying a graphic effect to at least one of the plurality of layers so that the menu page corresponding to the second layer appears discontinuously from the menu page corresponding to the first layer using the comparison result. have.

Measuring depth information from the screen of the user terminal to the user's hand includes measuring depth information from the screen of the user terminal to the user's hand using at least one Dynamic Vision Sensor (DVS). I can.

The step of measuring the depth information from the screen of the user terminal to the user's hand is performed by using the difference between the position of the hand in the image output from the first DVS and the position of the hand in the image output from the second DVS. It may include measuring depth information from the screen of the terminal to the user's hand.

The measuring of the depth information from the screen of the user terminal to the user's hand includes measuring the difference between the position of the hand measured by two or more image sensors or the depth information of the user's hand using a TOF sensor. Can include.

A menu page corresponding to the plurality of layers may include a 2D screen.

The method may further include hierarchically displaying a menu page corresponding to the plurality of layers.

One of the menu pages corresponding to the plurality of layers includes icons corresponding to a plurality of applications, the other of the menu pages includes a plurality of emoticons, or another one of the menu pages May include a virtual keyboard.

According to an embodiment, a method of displaying a menu based on a user's spatial gesture includes: displaying at least one polyhedron including a plurality of faces; Recognizing a user's spatial gesture for manipulating the at least one polyhedron; Identifying a surface corresponding to the spatial gesture among the plurality of surfaces included in the at least one polyhedron; And displaying a menu corresponding to the identified surface.

The step of identifying a surface corresponding to the spatial gesture among the plurality of surfaces included in the at least one polyhedron includes identifying a surface corresponding to the changed spatial gesture as the spatial gesture changes, and the Displaying the menu corresponding to the identified surface includes displaying the menu corresponding to the second surface corresponding to the changed spatial gesture continuously from the menu corresponding to the first surface corresponding to the spatial gesture before the change. It may include steps.

A plurality of faces included in the at least one polyhedron may include different menus.

Recognizing the user's spatial gesture for manipulating the at least one polyhedron may include, when the user's spatial gesture is a motion of a predefined form, the motion is performed by selecting another polyhedron inside or outside the at least one polyhedron. It may include the step of recognizing a user's spatial gesture for manipulation.

The step of identifying a plane corresponding to the spatial gesture among the plurality of planes included in the at least one polyhedron may include a plurality of planes included in the other polyhedron when the spatial gesture is a pinch zoom-type motion. It may include the step of identifying a surface corresponding to the position of the pinch zoom.

Recognizing a user's spatial gesture for manipulating the at least one polyhedron may include comparing a distance from the screen to the user's hand with a preset distance; And recognizing the user's spatial gesture as a user's spatial gesture for manipulating another polyhedron inside or outside the at least one polyhedron based on the comparison result.

It may further include hierarchically displaying the at least one menu polyhedron including a plurality of faces.

One of the menus corresponding to the plurality of sides includes icons corresponding to a plurality of applications, the other of the menus includes a plurality of emoticons, or another one of the menus uses a virtual keyboard. Can include.

Recognizing a user's spatial gesture for manipulating the at least one polyhedron includes recognizing a user's spatial gesture for manipulating the at least one polyhedron using at least one Dynamic Vision Sensor (DVS). I can.

Recognizing the user's spatial gesture for manipulating the at least one polyhedron may be performed using a coordinate difference between an image output from the first DVS and an image output from the second DVS with respect to the user's gesture performed in a predetermined space. And recognizing a user's spatial gesture for manipulating the at least one polyhedron.

Recognizing a user's spatial gesture for manipulating the at least one polyhedron may include measuring the position, movement speed, and angular velocity of the user's hand on the screen of the user terminal; And recognizing the user's spatial gesture using the measurement result.

Menus corresponding to a plurality of faces of the at least one polyhedron may have similar properties.

Menus corresponding to a plurality of faces of the at least one polyhedron may include a 2D screen.

1 is a flowchart illustrating a method of displaying a menu based on depth information according to an exemplary embodiment.
FIG. 2 is a diagram illustrating a configuration of a user terminal for measuring depth information and a method for measuring depth information according to the method of FIG. 1.
3 is a diagram illustrating a screen of a user terminal in which a menu page corresponding to at least one layer identified by the method of FIG. 1 is displayed.
4 is a flowchart illustrating a method of displaying a menu based on a user's spatial gesture according to an exemplary embodiment.
5 is a flowchart illustrating a method of differently recognizing and processing a user's spatial gesture according to a distance from a screen to a user's hand in the method of FIG. 4.
6 is a diagram illustrating a configuration of a user terminal for recognizing a user's spatial gesture and a method for the user terminal to recognize a spatial gesture according to the method of FIG. 4.
FIG. 7 is a diagram illustrating that a menu corresponding to an identified surface among a plurality of surfaces included in at least one polyhedron according to the method of FIG. 4 is displayed on a screen of a user terminal.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the exemplary embodiments. In addition, the same reference numerals shown in each drawing indicate the same member.

1 is a flowchart illustrating a method of displaying a menu based on depth information according to an exemplary embodiment.

Referring to FIG. 1, a user terminal according to an embodiment may measure depth information corresponding to a distance from a screen of the user terminal to a user's hand (110).

At 110, the user terminal uses a plurality of image sensors on the screen of the user terminal, that is, the difference in the position of the user's hand detected within a certain distance from the screen of the user terminal or by using a time of flight (TOF) sensor. You can measure the depth information of your hand. Here, the TOF (Time of Flight) sensor is a sensor that detects depth information of a target by immediately measuring a real-time distance.

The user terminal may identify at least one layer among the plurality of layers based on the depth information measured at 110 (130). In this case, the plurality of layers are virtual layers, and different layers corresponding to each of the plurality of layers may exist according to depth information.

The user terminal may apply a graphic effect to at least one of the plurality of layers such that a menu page corresponding to the at least one layer identified in 130 is displayed (150).

One of the menu pages corresponding to a plurality of layers includes icons corresponding to a plurality of applications, the other of the menu pages includes a plurality of emoticons, and another of the menu pages is a virtual keyboard It may include. Here, the virtual keyboard can be understood in the form of an input keyboard including, for example, Korean, English, numbers, and special characters.

A menu page corresponding to a plurality of layers may be displayed on a user terminal as a two-dimensional screen.

The user terminal may hierarchically display all menu pages corresponding to a plurality of layers, for example, as in the screen of FIG. 3.

In this case, the user terminal may improve visibility of a menu page corresponding to at least one layer identified in 130 and reduce visibility of a menu page corresponding to at least one other layer.

Also, the user terminal may detect movement from the first layer to the second layer based on the depth information measured at 110. In this case, the user terminal may apply a graphic effect to at least one of the plurality of layers so that the menu page corresponding to the first layer gradually disappears and the menu page corresponding to the second layer gradually appears in response to the depth information. have.

According to an embodiment, the user terminal compares the distance from the screen to the user's hand with a preset distance, and using the comparison result, a menu page corresponding to the second layer appears discontinuously from the menu page corresponding to the first layer. A graphic effect may be applied to at least one of a plurality of layers so as to be possible.

FIG. 2 is a diagram illustrating a configuration of a user terminal for measuring depth information and a method for measuring depth information according to the method of FIG. 1.

Referring to FIG. 2, a user terminal in which a method for displaying a menu is performed based on depth information according to an embodiment is equipped with a touch sensor panel (TSP) 210 and a DVS (Dynamic Vision) for spatial recognition. Sensor) 230 may be included. In this case, the touch sensor panel (TSP) 210 may have high sensitivity capacitance.

The DVS 230 is an example of a sensor for measuring depth information corresponding to a distance from a screen of a user terminal to a user's hand.

In an embodiment, depth information corresponding to a distance from the screen of the user terminal (eg, the touch sensor panel (TSP) 210) to the user's hand (tip) may be measured using the DVS 230.

At this time, in addition to the DVS (Dynamic Vision Sensor) 230, various sensors capable of performing spatial recognition (for example, an image sensor, a Time of Flight (ToF) sensor, an infrared sensor combined with an active IR (Infra Red) light source, An armored strain sensor, an acceleration sensor, a gyro sensor, etc.) can be used to measure depth information.

In an embodiment, a finger may be detected at a location (eg, a preset threshold distance) from a screen of a user terminal in a non-contact manner by the DVS (Dynamic Vision Sensor) 230. Thereafter, the user terminal may identify a layer corresponding to the location and select and display a menu page corresponding to the identified layer.

In this case, if the two DVS 230 are used, the position of the hand (x, y, z) in the image output from each DVS 230 is different. The closer the position of the DVS 230 and the hand is, the greater the difference between these positions. This principle is the principle of binocular disparity as a method by which two eyes of a person perceive the distance to an object. Therefore, it is possible to measure the depth information to the hand by using two DVS 230 or other image sensors and using the principle of binocular parallax.

In addition, the TOF sensor has a light source in the sensor, so the light is scattered on an object and then the reflected light is measured with a light receiver. The longer the distance between the object and the sensor is, the longer the time to measure the reflected light is, so the depth information of the hand can be recognized using this. Depth information can be measured by performing 3D movement trajectory, velocity change, symbol recognition, etc. through measurement of movement velocity (vx, vy, vz), and angular velocity using other acceleration and gyro sensors.

3 is a diagram illustrating a screen of a user terminal in which a menu page corresponding to at least one layer identified by the method of FIG. 1 is displayed.

Referring to FIG. 3, the user terminal according to an embodiment applies a graphic effect to the layer so that a menu page corresponding to the layer identified based on depth information is displayed, thereby providing a two-dimensional space recognition user interface (UI). It can be expressed on the display.

In this case, the user terminal may include, for example, a device such as a smart phone, a PC, or a TV, and depth information corresponding to the distance from the user's hand motion and the user terminal's screen (that is, the display) to the user's hand is stored. It can be measured (or recognized).

These three-dimensional space recognition include a passive optical image sensor that recognizes three-dimensional space, a time of flight (TOF) sensor, an infrared sensor combined with an active Infra Red (IR) light source, an armored strain sensor worn on the hand, and acceleration. It can be implemented as a sensor, a gyro sensor, or the like.

Accordingly, in one embodiment, when the user terminal recognizes the hand motion in space or the distance between the display and the fingertip, a menu page appears on the screen of the user terminal by applying a graphic effect to at least one of a plurality of layers as shown in FIG. 3. A new concept of 2D user interface (UI) can be provided.

The user thinks that a menu page corresponding to each layer exists in space, and can move or select a menu page according to depth information on a 2D screen.

Here, the'menu page' corresponding to each layer is included in not only different menu pages including icons corresponding to various applications, but also a sub-menu or sub-menu corresponding to a selected menu (or icon) among one menu page. It can be understood as including all of the content, etc.

In addition, a 2D screen can be configured so that a desired menu page moves when the distance between the user terminal and the hand is close. In other words, the user may think that a virtual menu page layer exists in the space and adjust the menu page displayed on the screen of the user terminal.

In the user terminal, as the distance between the user's hand and the surface 305 of the display gets closer, the menu page 310 corresponding to the first layer before the change fades and gradually disappears, and at the same time, the second layer disappears. A graphic effect may be applied to gradually appear while the corresponding menu page 320 becomes clearer.

For example, in FIG. 3, each of the layers may be implemented as a 3D stereoscopic image through the display of the user terminal, and the first layer may have a low 3D depth so that the first layer is at the top and feels close to the user. At this time, as the layer goes through the second layer to the third layer and the fourth layer, the arrangement height of the menu page corresponding to the layer gradually decreases, and the user terminal may have a deep 3D depth so that the screen of the user terminal feels away from the user.

In other words, the menu page 310 corresponding to the first layer (eg, the main menu page) may be disposed on the top, and then, the menu page 320 corresponding to the second layer (eg, the main menu page) The main page corresponding to the icon of the application e-book selected on the menu page), the menu page 330 corresponding to the third layer (for example, a sub page or product page provided by the application e-BOOK), It may be expressed as having a 3D depth that deepens in the order of the menu pages 340 (eg, payment pages of the selected product) corresponding to the 4 layers.

According to an embodiment, when the user terminal detects movement from the first layer to the second layer based on depth information, it may compare a distance from the screen to the user's hand with a preset distance. And, as a result of the comparison, when a predetermined distance is reached, the menu page corresponding to the second layer may be discontinuously (or suddenly) appear from the menu page corresponding to the first layer.

In addition, according to an embodiment, when the user's hand moves a certain distance horizontally on a space separated from the screen of the user terminal, the user terminal detects such movement and changes the current menu page to the next menu page and displays it. You may.

Therefore, according to an embodiment, it solves the hassle of a touch-based user interface in which a user must manually touch and drag pages for menu selection, and intuitive interaction between the user and the user terminal. ) Can be implemented.

In addition, the user terminal may transparently and hierarchically display all menu pages 310, 320, 330, and 340 corresponding to a plurality of layers on the screen. At this time, the user terminal improves the visibility of the menu page (eg, 310) corresponding to at least one layer identified among them, and the remaining at least one layer (eg, 320, 330, 340). By reducing the visibility of the menu page corresponding to, it may indicate that the menu page corresponding to the identified at least one layer (eg, the current first layer) is selected.

In FIG. 3, the page layer on which the menu is located can be naturally moved using depth information in space, but various applications are possible by extending this principle.

For example, an input keyboard (that is, a virtual keyboard) such as Korean, English, numbers, and special characters used by the user when entering characters is implemented using the above-described virtual space layer, or various sets of emoticons are implemented in a virtual space. It can be implemented using layers.

In other words, one of the menu pages corresponding to a plurality of layers includes icons corresponding to a plurality of applications, the other of the menu pages includes a plurality of emoticons, and another one of the menu pages May include a virtual keyboard.

As described above, in FIG. 3, a user interface is configured by displaying and moving a menu page by setting a virtual spatial layer. Hereinafter, a case of configuring a user interface by displaying a menu using a polyhedron will be described.

4 is a flowchart illustrating a method of displaying a menu based on a user's spatial gesture according to an exemplary embodiment.

Referring to FIG. 4, the user terminal according to an embodiment may display at least one polyhedron including a plurality of faces (410 ).

The user terminal may recognize a user's spatial gesture for manipulating at least one polyhedron (420).

At 420, the user terminal may measure the position, movement speed, and angular velocity of the user's hand on the screen of the user terminal, and recognize the user's spatial gesture using the measurement result. A method for the user terminal to recognize a user's spatial gesture will be described with reference to FIG. 6.

In 420, when the user's spatial gesture is a motion of a predefined type, the user terminal may recognize the motion as a user's spatial gesture for manipulating another polyhedron inside or outside of at least one polyhedron. Here, the predefined type of motion may be, for example, a pinch zoom type of motion, a movement motion of moving a hand left or right, or a motion of clenching and opening a fist.

'Pinch zoom-type motion' may be understood as an action in which a user takes a motion as if holding a space with a thumb, index or middle finger, or increases or decreases the size of the motion.

In addition, the user terminal may recognize the spatial gesture differently based on a comparison result between the distance from the screen to the user's hand and the preset distance. A method in which the user terminal recognizes and processes spatial gestures differently will be described with reference to FIG. 5.

The user terminal may identify a surface corresponding to the spatial gesture among a plurality of surfaces included in the at least one polyhedron (430).

In this case, a plurality of faces included in at least one polyhedron may include different menus. For example, one of the menus corresponding to a plurality of sides includes icons corresponding to a plurality of applications, the other of the menus includes a plurality of emoticons, and another of the menus is a virtual keyboard It may include.

When the spatial gesture recognized at 420 is a pinch zoom-type motion, the user terminal may identify a plane corresponding to the position (or size) of the pinch zoom among a plurality of planes included in another polyhedron at 430. . At this time, the surface corresponding to the pinch zoom position corresponds to the center or the top of the position (or size) of the user's two finger ranges formed in a certain space on the screen of the user terminal by, for example, a pinch zoom type. It can be understood as cotton or the like.

The user terminal may display a menu corresponding to the surface identified at 430 (440).

As the spatial gesture changes at 430, the user terminal may identify a surface corresponding to the changed spatial gesture. In this case, the user terminal may display a menu corresponding to the second surface corresponding to the changed spatial gesture to continuously appear from the menu corresponding to the first surface corresponding to the spatial gesture before the change.

The user terminal may hierarchically display at least one menu polyhedron including a plurality of faces. In this case, menus corresponding to a plurality of faces of at least one polyhedron may be displayed as a two-dimensional screen.

5 is a flowchart illustrating a method of differently recognizing and processing a user's spatial gesture according to a distance from a screen to a user's hand in the method of FIG. 4.

Referring to FIG. 5, the user terminal according to an embodiment may measure a distance (or depth information corresponding to the distance) from a screen to a user's hand (510). In addition, the user terminal may compare the distance from the screen to the user's hand and a preset distance.

The user terminal may determine whether the distance from the screen to the user's hand is greater than a preset distance (520).

As a result of the determination of 520, if the distance from the screen to the user's hand is greater than the preset distance, the user terminal may recognize the user's spatial gesture as a user's spatial gesture for manipulating another polyhedron outside at least one polyhedron. There is (530).

In addition, the user terminal may identify a surface corresponding to the spatial gesture among a plurality of surfaces included in another polyhedron outside (540 ).

Thereafter, the user terminal may display a menu corresponding to the identified surface (550).

On the other hand, as a result of the determination of 520, if the distance from the screen to the user's hand is less than or equal to the preset distance, the user terminal uses the user's space gesture to manipulate another polyhedron inside at least one polyhedron. It can be recognized as a gesture (560).

In this case, the user terminal may identify a surface corresponding to a spatial gesture among a plurality of surfaces included in another polyhedron inside (570) and display a menu corresponding to the identified surface (550 ).

6 is a diagram illustrating a configuration of a user terminal for recognizing a user's spatial gesture and a method for the user terminal to recognize a spatial gesture according to the method of FIG. 4.

Referring to FIG. 6, a user terminal in which a method of displaying a menu is performed based on a user's spatial gesture according to an embodiment is a DVS for spatial recognition together with a touch sensor panel (TSP) 610 ( Dynamic Vision Sensor) 630 may be included. In this case, the touch sensor panel (TSP) 610 may have high sensitivity capacitance.

Here, the DVS 630 is an example of a sensor for recognizing a user's spatial gesture for manipulating at least one polyhedron or measuring depth information corresponding to a distance from the screen of the user terminal to the user's hand.

In one embodiment, the user terminal is a user's space for manipulating at least one polyhedron using a coordinate difference between the image output from the first DVS and the image output from the second DVS for the user's gesture performed in a certain space. Gestures can be recognized.

In addition, in one embodiment, the user terminal uses an acceleration, a gyro sensor, etc., on the screen of the user terminal (touch sensor panel (TSP) 610) Movement speed (vx, vy, vz), and angular speed can be measured. In addition, the user terminal may recognize a user's spatial gesture by identifying a 3D movement trajectory and a change in speed using the measurement result or by performing symbol recognition. In the same way, using two DVS or image sensors or TOF sensors, the user terminal can measure depth information corresponding to the distance from the screen (touch sensor panel (TSP) 610) to the user's finger (tip). have.

FIG. 7 is a diagram illustrating that a menu corresponding to an identified surface among a plurality of surfaces included in at least one polyhedron according to the method of FIG. 4 is displayed on a screen of a user terminal.

The user terminal according to an embodiment may recognize a user's spatial gesture performed in a predetermined space on the screen of the user terminal. In this case, the user's spatial gesture may be for manipulating a polyhedron displayed as a two-dimensional screen on the user terminal.

The polyhedron may be, for example, a cube of a hexahedron, or other three-dimensional or more hexahedral. Further, menus having similar properties may appear on each of a plurality of surfaces of the polyhedron.

For example, a cube in which different menus are displayed on each side may be displayed on the screen of the user terminal. At this time, when the user makes a gesture of holding and turning a virtual cube in a certain space on the screen of the user terminal, the cube displayed on the screen of the user terminal may also rotate together. In addition, when the user stops the movement and maintains a predetermined time so that the desired surface is on the top, a menu corresponding to the surface identified as the top surface in the virtual cube by the user's spatial gesture may be displayed on the screen of the user terminal.

In addition, according to an embodiment, different cubes having different sizes may exist inside and outside one cube. Therefore, the user can open the thumb and forefinger holding the virtual cube in a closed state in a certain space on the screen of the user terminal, or take a pinch zoom-type motion that is closed in an open state to the outside of the cube. Alternatively, you can select another cube inside and select a menu that another cube contains.

In addition, depending on the embodiment, it is possible to select another cube (layer) in one cube or outside the cube by using the distance from the screen to the user's hand (or depth information corresponding to the distance) as shown in FIG. 5.

According to an embodiment, it is possible to select a desired menu without touching and dragging each page by a screen display method through a 3D space user interface on the 2D screen as described above.

In addition, according to an embodiment, an intuitive interaction between a user and a user terminal may be implemented, and a rich and natural user experience may be provided.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

210: Touch Sensor Panel (TSP)
230: Dynamic Vision Sensor (DVS)

Claims (24)

Measuring depth information corresponding to the distance from the screen of the user terminal to the user's hand using at least one DVS (Dynamic Vision Sensor);
Based on the comparison result between the depth information and a preset distance, whether the user's spatial gesture is a first type of spatial gesture for identifying a layer corresponding to a menu page, or a face of a polyhedron including different menus Determining whether it is a second type of spatial gesture to make;
When it is determined that the spatial gesture is the first type of spatial gesture,
Identifying a first layer among a plurality of layers based on the depth information;
A first graphic on the identified first layer among the plurality of layers such that a first menu page corresponding to the identified first layer-the first menu page including first menu items-appears on the screen. Applying the effect;
Sensing movement of the hand from the identified first layer to a second layer of the plurality of layers based on the depth information; And
In response to detecting the movement of the hand from the first layer to the second layer, the first menu page gradually disappears from the screen without movement, and the second menu page corresponding to the second layer disappears from the screen without movement. The first layer and the identified first layer and the identified so that the first menu page disappears into the second menu page including second menu items without movement of the first menu page and the second menu page. Applying a second graphic effect to the second layer
Including,
When it is determined that the spatial gesture is the second type of spatial gesture,
Identifying one of the plurality of faces of the polyhedron corresponding to the second type of spatial gesture; And
Displaying a menu corresponding to the identified surface
Containing,
How to display the menu based on the depth information. delete The method of claim 1,
The step of applying the second graphic effect
Improving visibility of the second menu page; And
Reducing visibility of a menu page corresponding to the first layer
A method of displaying a menu based on depth information comprising a. The method of claim 1,
The step of applying the second graphic effect
Comparing a distance from the screen to a user's hand and a preset distance; And
Using the comparison result, applying the second graphic effect to the second layer such that a second menu page corresponding to the second layer appears discontinuously from the first menu page corresponding to the first layer.
A method of displaying a menu based on depth information comprising a. The method of claim 1,
Measuring depth information from the screen of the user terminal to the user's hand
Measuring depth information from the screen of the user terminal to the user's hand using at least one DVS (Dynamic Vision Sensor)
A method of displaying a menu based on depth information comprising a. The method of claim 5,
Measuring depth information from the screen of the user terminal to the user's hand
Measuring depth information from the screen of the user terminal to the user's hand using the difference between the first position of the hand in the image output from the first DVS and the second position of the hand in the image output from the second DVS step
A method of displaying a menu based on depth information comprising a. The method of claim 1,
Measuring depth information from the screen of the user terminal to the user's hand
Measuring the depth information of the user's hand using the difference in the position of the hand measured by two image sensors or a Time Of Flight (TOF) sensor
A method of displaying a menu based on depth information comprising a. The method of claim 1,
The menu page corresponding to the plurality of layers
A method of displaying a menu based on depth information including a two-dimensional screen. The method of claim 1,
Hierarchically displaying a menu page corresponding to the plurality of layers
A method of displaying a menu based on depth information further comprising a. The method of claim 1,
Of the menu pages corresponding to the plurality of layers, a first menu page includes icons corresponding to a plurality of applications, or
The second menu page of the menu pages includes a plurality of emoticons, or
A third menu page among the menu pages includes a virtual keyboard,
How to display a menu based on depth information. delete delete delete delete delete delete delete delete delete delete delete delete delete A computer-readable recording medium on which a program for performing the method of any one of claims 1, 3 to 10 is recorded.

Images