KR20230073869A - Method of providing user interface supporting interaction and elecronic apparatus performing the same - Google Patents

Abstract

According to an embodiment disclosed in this document, an electronic device that provides a user interface supporting interaction acquires a plurality of images taken from a plurality of cameras, and displays a hand among the obtained plurality of images. identify images that contain Based on the identified image and preset hand feature parameters, 3D pose information of the hand is obtained, and based on an application running on the electronic device, the acquired 3D pose information of the hand is applied. Corresponding interaction information may be provided.

Description

A method for providing a user interface supporting interaction and an electronic device performing the same

Embodiments disclosed in this document relate to a method for providing a user interface supporting interaction and an electronic device performing the same.

Various technologies are being developed to enable users to experience virtual reality (VR)/augmented reality (AR). Virtual reality is a system that builds a specific environment, situation, or virtual scenario of real reality through computer modeling and helps users interact in this virtual environment. In addition, augmented reality is a system that provides virtual information about space and situations by superimposing virtual objects on the real world and outputting them. In augmented reality, digitally recreated images, or parts thereof, can be presented to the user in such a way that they appear to be real, or can be perceived as real.

In order to allow the user to experience virtual reality/augmented reality more realistically, it is important to properly provide interaction with the user's input as in real life. To this end, as accurately recognizing a user's input in virtual reality/augmented reality corresponds to an important part, various technologies for this are being developed.

The present disclosure relates to a method for providing a user interface capable of accurately recognizing a user input and appropriately performing an interaction therewith, and an electronic device performing the same.

A method for providing a user interface supporting interaction by an electronic device according to an embodiment of the present disclosure includes obtaining a plurality of images captured by a plurality of cameras; identifying an image including a hand among a plurality of acquired images; obtaining 3D pose information of the hand based on the identified image and preset hand characteristic parameters; and providing interaction information corresponding to the obtained 3D pose information of the hand, based on an application executed on the electronic device.

An electronic device providing a user interface supporting interaction according to an embodiment of the present disclosure includes a sensing unit including a plurality of cameras; output unit; and at least one processor, wherein the at least one processor acquires a plurality of images taken from a plurality of cameras, identifies an image including a hand among the plurality of acquired images, and identifies the identified image and a preset hand. 3D pose information of the hand may be acquired based on the characteristic parameter, and interaction information corresponding to the acquired 3D pose information of the hand may be provided based on an application executed in the electronic device.

A computer program product including a recording medium storing a program for causing an electronic device to perform a method of providing a user interface supporting interaction according to an embodiment of the present disclosure includes obtaining a plurality of images taken from a plurality of cameras. An operation of identifying an image including a hand among a plurality of acquired images, an operation of obtaining 3D pose information of the hand based on the identified image and a preset hand characteristic parameter, and an application running on the electronic device. Based on this, a program for performing an operation of providing interaction information corresponding to the obtained 3D pose information of the hand may be stored.

1 is a conceptual diagram illustrating a method of providing a user interface supporting interactions by an electronic device according to an exemplary embodiment.
2 is a diagram for explaining a field of view (FoV) of a plurality of cameras.
3 is a flowchart illustrating a method of providing a user interface supporting interaction by an electronic device according to an exemplary embodiment.
4 is a diagram for describing hand characteristic parameters according to an exemplary embodiment.
5 is a flowchart illustrating a method of obtaining, by an electronic device, 3D pose information of a hand based on whether a plurality of images including the hand are identified, according to an exemplary embodiment.
6 is a diagram for explaining a method of acquiring 3D information of a hand according to hand motion by an electronic device according to an exemplary embodiment.
7 is a flowchart illustrating a method of acquiring a hand characteristic parameter by an electronic device according to an exemplary embodiment.
8 is a diagram for explaining a method of calibrating 3D pose information of a hand by an electronic device according to an exemplary embodiment.
9 and 10 are block diagrams of an electronic device according to an exemplary embodiment.
11 is a block diagram illustrating a camera module, in accordance with various embodiments.

Hereinafter, embodiments will be described in detail so that those skilled in the art can easily practice the invention with reference to the accompanying drawings. However, the disclosure may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

In the present disclosure, a “gesture” is a physical or vocal action or expression representing a user input, such as a tap (press), drag (movement while pressing), pinch (spreading with two fingers). /narrow), press (long press), and flick (quickly scroll).

In the present disclosure, a “pose” is a gesture form at a fixed viewpoint, and consecutive poses may constitute one gesture.

In the present disclosure, a “user profile” is information representing a user's identity and settings related to the user, and may include, for example, information about the user's age, gender, height, weight, or region.

Terms used in the specification may be used to describe various components, but the components should not be limited by the terms. Terms are only used to distinguish one component from another.

Hereinafter, the disclosure will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram illustrating a method of providing a user interface supporting interactions by an electronic device according to an exemplary embodiment.

Referring to FIG. 1 , the electronic device 100 may overlap and output at least one image to the real world so that the user 5 may experience an augmented reality (AR)/virtual reality (VR) environment. Also, the electronic device 100 may sense a user input for the output image. For example, when the image 10 including the icons 12, 14, and 16 of a plurality of applications is overlapped and output on the real world, the electronic device 100 displays the icons 12, 14, and 16 of the plurality of applications. ) may sense a user input selecting one of them. The user input may include a hand gesture, etc., but this is only an example, and the user input is not limited to the hand gesture.

As the user input is sensed, the electronic device 100 may provide interaction information corresponding to the sensed user input. For example, as at least one image is acquired through a camera including an image sensor, the electronic device 100 may detect a hand in at least one acquired image. The electronic device 100 may identify a hand gesture based on the detected 3D pose information of the hand. When the identified hand gesture is a “tap gesture”, the electronic device 100 may provide an execution screen of an application icon (eg, 12) corresponding to the position of the finger as interaction information.

Meanwhile, in order to appropriately provide interaction information corresponding to the user input by reflecting the user's intention, the electronic device 100 needs to accurately identify the sensed user input. For example, as in the above-described embodiment, when the user input is a tap gesture, the electronic device 100 needs to accurately identify the position of the user's finger based on the 3D pose information of the hand.

3D pose information of the hand may be generally obtained through a stereoscopic method. The stereoscopic method imitates the way a human recognizes an object using both eyes, and the electronic device 100 detects the shape and size of the hand in each of the images acquired through two cameras having different FoVs. and 3D pose information of the hand may be obtained based on the depth information.

When 3D pose information of the hand is acquired using the stereoscopic method, the absolute size and shape of the hand can be accurately identified, but this is limitedly applied when the hand is detected in an area where the FoVs of the two cameras overlap. can For example, if the hand of the user 5 is detected in an area where the FoVs of the two cameras do not overlap, that is, if the hand is detected only in an image captured by one camera, the hand's 3 It is difficult to obtain dimensional pose information. Embodiments of the present disclosure are intended to provide a method for obtaining 3D pose information of a hand using a hand characteristic parameter even in an area where FoVs of two cameras do not overlap.

Meanwhile, in order to provide a more realistic AR/VR environment, the number of cameras included in electronic devices tends to increase. In this case, as the electronic device needs to process images of overlapping regions for each camera, the amount of calculations required to obtain 3D pose information of the hand may increase. Embodiments of the present disclosure may provide 3D pose information of a hand from a single image using a hand characteristic parameter, thereby preventing an increase in computational complexity.

Also, the electronic device according to embodiments of the present disclosure may maintain the accuracy of the 3D pose information of the hand at the level of accuracy when obtaining the 3D pose information of the hand from a plurality of images. To this end, when a hand is detected at a point where the FoVs of a plurality of cameras overlap, the electronic device may update the hand characteristic parameter in consideration of criteria such as the position of the hand or the sharpness of fingers of the hand detected from the image. . A criterion for updating the hand characteristic parameter by the electronic device will be described later in detail with reference to FIG. 8 .

2 is a diagram for explaining a field of view (FoV) of a plurality of cameras.

The FoV of a camera is an area corresponding to a photographed image when an image is captured through a lens provided in the camera, and the FoV value may be determined according to the size of the sensor and the magnification of the lens. In addition, the area where the FoV is formed may be different depending on the position where the camera is provided in the electronic device 100, and the FoV of the plurality of cameras provided in the electronic device 100 may be different.

Referring to FIG. 2 , the electronic device 100 may include two cameras, and among the two cameras, a FoV 22 of a first camera and a FoV 24 of a second camera may be different from each other. An overlapping area may exist between the FoV 22 of the first camera and the FoV 24 of the second camera. In the present disclosure, a region in which FoVs of different cameras overlap is described as a stereo region 26 .

When the hand is located in the stereo area 26, the electronic device 100 may obtain accurate 3D pose information of the hand by applying a stereoscopic method to images obtained through each camera. However, this method may be applied only to the stereo region 26 . In addition, even if the hand is located in the stereo region 26, when the 3D pose information of the hand is obtained by continuously applying the stereoscopic method, the amount of computation of the electronic device 100 may increase. In particular, when it is necessary to acquire 3D pose information of a hand moving in real time, the amount of computation of the electronic device 100 may rapidly increase.

The electronic device 100 according to an embodiment of the present disclosure may set a hand characteristic parameter and acquire 3D pose information of the hand from one image including the hand by using the set hand characteristic parameter. The electronic device 100 according to an embodiment may determine a hand characteristic parameter by applying a stereoscopic method only when the hand is located in the stereo area and satisfies a preset criterion. Through this, the electronic device can prevent the accuracy of the obtained 3D pose information of the hand from deteriorating.

3 is a flowchart illustrating a method of providing a user interface supporting interaction by an electronic device according to an exemplary embodiment.

In step S310, the electronic device may acquire a plurality of images captured by a plurality of cameras. An electronic device may include a plurality of cameras, and FoVs of the plurality of cameras may be different from each other.

As an application requesting information on an object around the electronic device is executed, the electronic device may photograph the surroundings of the electronic device using a plurality of cameras. Applications requesting information on objects around the electronic device may include a photo taking application, an application providing an AR/VR environment, and the like. However, this is only an example, and an application requesting information about an object around the electronic device is not limited to the above example. As a result of the electronic device capturing the surroundings of the electronic device using a plurality of cameras, images in which an area corresponding to the FoV of each camera is photographed may be obtained from the plurality of cameras.

In step S320, the electronic device may identify an image including a hand among a plurality of acquired images.

When an application being executed supports a hand gesture, the electronic device may identify an image including a hand among a plurality of acquired images. However, this is only an example, and the electronic device may identify an image including a hand among a plurality of acquired images even when the user's hand characteristic parameter needs to be set.

An electronic device according to an embodiment may determine an order of performing a hand detection operation for identifying whether a hand is included in each of a plurality of images. The electronic device may perform a hand detection operation on a plurality of images based on the determined order, and may stop the hand detection operation when an image including a preset number of hands is acquired. For example, the electronic device may determine an order in which a hand detection operation is performed on a plurality of images, based on information about a hand detection probability from each of the plurality of images. Hand detection probability information from each of the plurality of images may be determined by estimating hand motion based on hand tracking information acquired by the electronic device.

In step S330, the electronic device may acquire 3D pose information of the hand based on the identified image and preset hand characteristic parameters.

When an image including a hand is identified, the electronic device may obtain information about the shape and size of the detected hand from the identified image. In addition, the electronic device may obtain 3D pose information of the hand from information obtained using a preset hand characteristic parameter. As it is difficult to obtain accurate 3D pose information of the hand with one image, the electronic device may calibrate the shape and size of the hand detected from one image according to preset hand characteristic parameters.

Hand characteristic parameters according to an embodiment may be set based on a user's profile. For example, when a user first attempts an operation of inputting a hand gesture in an electronic device, the electronic device may set a hand characteristic parameter as an initial value using the user's profile. The user's profile may include information about the user's age, gender, height, weight, region, etc., but this is only an example, and the user's profile is not limited to the above example. Since the size and shape of the user's hand may vary according to gender, age, height, weight, region, etc., the electronic device may set the hand characteristic parameter as an initial value based on the user's profile. Hand characteristic parameter values for each gender, age, height, weight, or region may be previously stored in the electronic device according to an embodiment. According to another embodiment, the electronic device may obtain an initial value of a hand characteristic parameter corresponding to a user profile from an external server.

On the other hand, since the hand characteristic parameter set based on the user's profile is determined based on average hand characteristics for each gender, age, height, weight, and region, there may be parts that do not match the actual user's hand characteristics. there is. Accordingly, when the hand is located in the stereo region and satisfies a predetermined criterion, the electronic device updates hand characteristic parameters by applying a stereoscopic method to a plurality of images obtained as a result of photographing the hand located in the stereo region from different cameras. can do.

In step S340, the electronic device may provide interaction information corresponding to the obtained 3D pose information of the hand based on the executed application.

The electronic device may identify the user's hand gesture based on the 3D pose information of the hand. In each of the applications supporting hand gestures in the electronic device, information to be provided by the application or commands to be executed are preset for each hand gesture.

For example, if a map application is running on the electronic device and the hand gesture identified based on the 3D pose information of the hand is a drag gesture, the electronic device moves from the starting point to the ending point of the drag gesture. Information on a route may be provided as interaction information. According to another example, if a menu application is being executed on the electronic device and the hand gesture identified based on the 3D pose information of the hand is a tap gesture, the electronic device may send an application corresponding to a position where the tap gesture is sensed in the menu application. An execution screen of an icon may be provided as interaction information.

4 is a diagram for describing hand characteristic parameters according to an exemplary embodiment.

A hand characteristic parameter according to the present disclosure may include size information for each part of the hand determined based on at least some points constituting the hand.

Referring to FIG. 4 , the electronic device may specify some of a plurality of points constituting the hand in order to define characteristics of the hand. For example, the electronic device relates to the length from the wrist 405 to the tip of the thumb 418, the tip of the index finger 428, the tip of the middle finger 438, the tip of the ring finger 448, and the tip of the little finger 458. Information can be set as hand characteristic parameters.

Also, according to another example, the electronic device may specify detailed points for each finger. For example, in the case of a thumb, a thumb start point 412, a thumb joint a 414, a thumb joint b 416, and the like may be specified. In addition, in the case of the little finger, the little finger starting point 452, the little finger joint a 454, the little finger joint b 456, and the like may be specified.

The electronic device may set length information between specified points as a hand characteristic parameter. However, this is only an example, and points specified for setting hand characteristic parameters are not limited to the above example.

5 is a flowchart illustrating a method of obtaining, by an electronic device, 3D pose information of a hand based on whether a plurality of images including the hand are identified, according to an exemplary embodiment.

In step S510, the electronic device may perform a hand detection operation on at least one of the plurality of images.

The electronic device may perform a hand detection operation to identify whether at least one of the plurality of images includes a hand. An electronic device according to an embodiment may determine an order of images in which a hand detection operation is performed based on hand detection probability information. The electronic device may perform a hand detection operation on a plurality of images based on the determined order, and may stop the hand detection operation when an image including a preset number of hands is acquired.

As described above, probability information of a hand being detected from each of the plurality of images may be determined based on hand tracking information. For example, after sensing the movement direction and speed of the hand, the electronic device may acquire tracking information representing the position of the hand over time based on the sensed movement direction and speed. When the electronic device determines that the hand whose movement started at point a has stopped at point b located in the middle of points a and c (for example, speed value: less than y), hand detection for the image in which point c is captured Probability can be set to 0. Also, based on the acquired hand tracking information, the electronic device may stop capturing of a camera having an FoV of an area in which a probability of detecting a hand is less than a preset value among a plurality of cameras having different FoVs. For example, the electronic device may stop photographing by a camera including point c as the FoV in order to reduce power consumption.

In step S520, the electronic device may determine whether a plurality of images including hands are identified.

In step S530, the electronic device may obtain hand depth information and hand size information based on the identified plurality of images.

For example, the electronic device may acquire a plurality of images including the hands when the hands are located in a stereo area where FoVs of different cameras overlap, and the hands located in the stereo area are captured by different cameras. The electronic device may acquire accurate hand depth information and hand size information by applying a stereoscopic method to a plurality of images.

In step S540, the electronic device may update hand characteristic parameters based on the acquired hand depth information and the acquired hand size information.

The electronic device according to an embodiment obtains, when sharpness of fingers of a hand detected in a plurality of identified images is equal to or greater than a preset standard, or when the position of the detected hand is included within a preset distance range from the center of the identified image. Based on the obtained hand depth information and the acquired hand size information, hand characteristic parameters may be updated. The electronic device may determine that the possibility of obtaining an accurate hand characteristic parameter is relatively high when the sharpness of the finger is equal to or greater than a preset standard or when the detected hand position is within a preset distance range from the center of the identified image. . However, this is only an example, and the condition for updating the hand characteristic parameter is not limited to the above example.

In step S550, as one image including the hand is identified after the update, the electronic device may acquire 3D pose information of the hand based on the identified one image and the updated hand characteristic parameters. For example, the electronic device may acquire 3D pose information of the hand by performing calibration on the shape or size of the hand identified from one image according to the updated hand characteristic parameters.

In step S560, the electronic device may acquire 3D pose information of the hand based on the identified image and preset hand characteristic parameters. When an image including a hand is identified, the electronic device may obtain information about the shape and size of the detected hand from the identified image. In addition, the electronic device may obtain 3D pose information of the hand from information obtained using a preset hand characteristic parameter. As it is difficult to obtain accurate 3D pose information of the hand with one image, the electronic device may correct the shape and size of the hand detected from one image according to preset hand characteristic parameters.

6 is a diagram for explaining a method of acquiring 3D information of a hand according to hand motion by an electronic device according to an exemplary embodiment.

시점에 다른 카메라의 FoV와 중첩하지 않은 제 1 카메라의 FoV 영역인, 제 1 모노(mono) 영역(612)에 위치할 수 있다. 전자 장치는 제 1 카메라에서 촬영된 하나의 영상으로부터 사용자의 손(605)의 형태 및 크기 등을 식별할 수 있다. 또한, 전자 장치는 식별된 사용자의 손(605)의 형태 및 크기 등을 기 설정된 손 특성 파라미터를 이용하여 교정할 수 있다. 이 때, 전자 장치가 사용자의 손 제스쳐를 최초로 인식하는 경우에는, 손 특성 파라미터는 사용자의 프로파일에 기초하여 결정될 수 있다. Referring to Figure 6, the user's hand 605

It may be located in a first mono region 612, which is an FoV region of the first camera that does not overlap with FoVs of other cameras at the viewpoint. The electronic device may identify the shape and size of the user's hand 605 from one image captured by the first camera. In addition, the electronic device may calibrate the shape and size of the identified user's hand 605 using preset hand characteristic parameters. In this case, when the electronic device recognizes the user's hand gesture for the first time, hand characteristic parameters may be determined based on the user's profile.

시점에 제 1 카메라의 FoV 영역과 제 2 카메라의 FoV 영역이 중첩되는 스테레오 영역(614)에 위치할 수 있다. 일 실시예에 따른 전자 장치는 사용자의 손(605)이 스테레오 영역(614)에 위치한 경우, 기 설정된 손 특성 파라미터의 정확도 또는 기 설정된 손 특성 파라미터의 개수 중 적어도 하나에 기초하여, 스테레오스코픽 방식에 따라 손의 3차원 포즈 정보를 획득할지 여부를 결정할 수 있다. In addition, the user's hand 605

At the viewpoint, the FoV area of the first camera and the FoV area of the second camera may be located in the stereo area 614 overlapping. When the user's hand 605 is located in the stereo area 614, the electronic device according to an embodiment uses a stereoscopic method based on at least one of the accuracy of preset hand characteristic parameters or the number of preset hand characteristic parameters. Accordingly, it may be determined whether or not to acquire 3D pose information of the hand.

For example, when the number of preset hand characteristic parameters is greater than or equal to a threshold value, the electronic device obtains 3D pose information of the hand using one image captured by either the first camera or the second camera and the hand characteristic parameters. can be obtained In addition, when the number of preset hand characteristic parameters is less than a threshold value, the electronic device may obtain 3D pose information of the hand according to a stereoscopic method based on images captured by the first camera and the second camera. Since the electronic device can obtain absolute information about the size and shape of the hand when using the stereoscopic method, it can update hand characteristic parameters based on this.

According to another example, the electronic device may determine whether accuracy of interaction information provided in response to a previously input hand gesture is equal to or less than a preset standard. When an event in which the same hand gesture is re-entered as a user's feedback after providing the interaction information or an event in which a user input indicating that the interaction information is not appropriate occurs more than a certain number of times, the electronic device determines the accuracy of the interaction information based on a preset standard. It can be judged to be below. When the electronic device determines that the accuracy of the interaction information is less than or equal to a predetermined criterion, the electronic device may also determine that the accuracy of the hand characteristic parameter is low. When the accuracy of the hand characteristic parameter does not satisfy a predetermined criterion, the electronic device may acquire 3D pose information of the hand in a stereoscopic manner based on images captured by the first camera and the second camera. Since the electronic device can obtain absolute information about the size and shape of the hand when using the stereoscopic method, it can update hand characteristic parameters based on this.

시점에 다른 카메라의 FoV와 중첩하지 않은 제 2 카메라의 FoV 영역인, 제 2 모노 영역(616)에 위치할 수 있다. 전자 장치는 제 2 카메라에서 촬영된 하나의 영상으로부터 사용자의 손의 형태 및 크기 등을 식별할 수 있다. 또한, 전자 장치는 식별된 사용자의 손의 형태 및 크기 등을 기 설정된 손 특성 파라미터를 이용하여 교정할 수 있다. 이 때,

시점에 스테레오 영역(614)에서 손 특성 파라미터가 업데이트된 경우, 전자 장치는 업데이트된 손 특성 파라미터 및 제 2 카메라에서 촬영된 하나의 영상을 이용하여 손의 3차원 포즈 정보를 결정할 수 있다. In addition, the user's hand 605

It may be located in the second mono area 616, which is the FoV area of the second camera that does not overlap with the FoV of the other camera at the viewpoint. The electronic device may identify the shape and size of the user's hand from one image captured by the second camera. In addition, the electronic device may calibrate the identified user's hand shape and size using preset hand characteristic parameters. At this time,

When the hand characteristic parameter is updated in the stereo area 614 at the viewpoint, the electronic device may determine 3D pose information of the hand using the updated hand characteristic parameter and one image captured by the second camera.

7 is a flowchart illustrating a method of acquiring a hand characteristic parameter by an electronic device according to an exemplary embodiment.

In step S710, the electronic device may identify a plurality of images including hands. For example, when the hand is located in the stereo area, the electronic device may acquire a plurality of images including the hand, which are photographed by different cameras.

In step S720, the electronic device determines whether the sharpness of the fingers of the hand detected in the identified plurality of images is greater than or equal to a predetermined standard or the position of the detected hand is included within a predetermined distance range from the center of the identified image. Hand characteristic parameters can be obtained from the image of . The electronic device according to an embodiment may set the sharpness of the hand or the location of the hand when photographing the hand to determine whether the hand characteristic parameters obtained from the plurality of images appropriately represent the hand characteristic.

In step S730, the electronic device may identify whether the number of hand characteristic parameters set in the electronic device is greater than or equal to a threshold value.

In step S740, when the number of set hand characteristic parameters is greater than or equal to the threshold value, the electronic device may determine the set hand characteristic parameters as the final hand characteristic parameters.

In step S750, the electronic device may store the acquired hand characteristic parameters. When the number of set hand characteristic parameters is less than a threshold value, the electronic device according to an embodiment may determine that additional hand characteristic parameters need to be set. Accordingly, the electronic device may store the hand characteristic parameter acquired in step S720 described above.

8 is a diagram for explaining a method of calibrating 3D pose information of a hand by an electronic device according to an exemplary embodiment.

Referring to FIG. 8 , the electronic device may output a keyboard image 810 superimposed on the real world to provide an AR environment. A user may generate a message 820 by inputting desired text in the keyboard image 810 through a tap gesture. The electronic device according to an embodiment may obtain 3D pose information of the hand using one image obtained from one of a plurality of cameras and a hand characteristic parameter. In this case, if the text provided as the interaction information for the 3D pose information of the hand is different from the next intended by the user, the electronic device may determine that the accuracy of the set hand characteristic parameter is low. Accordingly, the electronic device may update hand characteristic parameters and correct 3D hand pose information.

For example, the electronic device may use two or more cameras among a plurality of cameras to photograph the user's hand, and apply a stereoscopic method to the plurality of photographed images to update the user's hand characteristic parameter.

The electronic device may newly acquire 3D pose information of the hand by using the updated hand characteristic parameter. The electronic device may generate a new message 825 based on the newly acquired 3D pose information of the hand.

9 and 10 are block diagrams of an electronic device according to an exemplary embodiment.

As shown in FIG. 9 , the electronic device 100 according to an embodiment may include a sensing unit 110 , a processor 130 and an output unit 150 . However, not all illustrated components are essential components. The electronic device 100 may be implemented with more components than those illustrated, or the electronic device 100 may be implemented with fewer components.

For example, as shown in FIG. 10 , the electronic device 100 according to an embodiment of the present disclosure includes a sensing unit 110, a processor 130, and an output unit 150 as well as a communication unit 160 and a memory. (170) may be further included.

Hereinafter, the above components are examined in turn.

The sensing unit 110 may detect a state of the electronic device 100 or a state around the electronic device 100 and transmit the sensed information to the processor 130 . For example, the sensing unit 110 may capture a user's hand and transmit the captured image to the processor 130 .

The sensing unit 110 includes an image sensor 111, a motion sensor 112, an acceleration sensor 113, a position sensor 114, an eye tracking sensor 115, a temperature/humidity sensor 116, and a geomagnetic sensor 117. ), a gyroscope sensor 118, a microphone 119, and a wearing detection sensor 120 may be included, but is not limited thereto. Since a person skilled in the art can intuitively infer the function of each sensor from its name, a detailed description thereof will be omitted.

The processor 130 typically controls overall operations of the electronic device 100 . For example, the processor 130 may overall control the sensing unit 110, the output unit 150, the communication unit 160, the memory 170, etc. by executing programs stored in the memory 170. .

The processor 130 may perform the operation of the electronic device described above with reference to FIGS. 1 to 8 . The processor 130 according to an embodiment may acquire a plurality of images captured by a plurality of cameras. Also, the processor 130 may identify an image including a hand among a plurality of acquired images. The processor 130 may obtain 3D pose information of the hand based on the identified image and preset hand feature parameters. The processor 130 may provide interaction information corresponding to the obtained 3D pose information of the hand, based on an application executed in the electronic device 100 .

The output unit 150 is for outputting an audio signal, a video signal, or a vibration signal, and may include a display unit 151 and a sound output unit 152.

The display unit 151 may output information processed by the electronic device 100 to be displayed. Meanwhile, when the display unit 151 and the touch pad form a layer structure to form a touch screen, the display unit 151 may be used as an input device as well as an output device.

The audio output unit 152 outputs audio data received from the communication unit 160 or stored in the memory 170 . Also, the sound output unit 152 outputs sound signals related to functions performed by the electronic device 100 (eg, a call signal reception sound, a message reception sound, and a notification sound). The sound output unit 152 may include a speaker, a buzzer, and the like.

The communication unit 160 may include one or more components that enable communication between the electronic device 100 and an external device or between the electronic device 100 and a server. For example, the communication unit 160 may include a short-distance communication unit 161, a mobile communication unit 162, and a broadcast reception unit 163.

The short-range wireless communication unit 161 includes a Bluetooth communication unit, a Bluetooth Low Energy (BLE) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, and an infrared ( It may include an infrared data association (IrDA) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, etc., but is not limited thereto.

The mobile communication unit 162 transmits and receives radio signals with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the radio signal may include a voice call signal, a video call signal, or various types of data according to text/multimedia message transmission/reception.

The broadcast receiver 163 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. Broadcast channels may include satellite channels and terrestrial channels. According to an implementation example, the electronic device 100 may not include the broadcast reception unit 163.

The memory 170 may store programs for processing and control of the processor 130 or may store input/output data.

The memory 170 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , an optical disk, and at least one type of storage medium.

Programs stored in the memory 170 may be classified into a plurality of modules according to their functions, for example, a UI module 171, a notification module 172, and the like.

The UI module 171 may provide a specialized UI, GUI, or the like that works with the electronic device 100 for each application.

The notification module 172 may generate a signal for notifying occurrence of an event of the electronic device 100 . Examples of events generated by the electronic device 100 include reception of a call signal, reception of a message, input of a key signal, and notification of a schedule.

11 is a block diagram illustrating a camera module, in accordance with various embodiments.

The camera module 1100 according to an embodiment may correspond to the camera or a plurality of cameras described above in FIGS. 1 to 8 .

Referring to FIG. 11 , a camera module 1100 includes a lens assembly 1110, a flash 1120, an image sensor 1130, an image stabilizer 1140, a memory 1150 (eg, a buffer memory), or an image signal processor. (1160). The lens assembly 1110 may collect light emitted from a subject that is a photographing target. The lens assembly 1110 may include one or more lenses. According to one embodiment, the camera module 1100 may include a plurality of lens assemblies 1110 . In this case, the camera module 1100 may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies 210 may have the same lens properties (eg, angle of view, focal length, auto focus, f number, or optical zoom), or at least one lens assembly may have the same lens properties as other lens assemblies. may have one or more lens properties different from the lens properties of . The lens assembly 1110 may include, for example, a wide-angle lens or a telephoto lens.

The flash 1120 may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash 1120 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.

The image sensor 1130 may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly 1110 into an electrical signal. According to an embodiment, the image sensor 1130 may be, for example, an image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, It may include a plurality of image sensors having a property, or a plurality of image sensors having other properties. Each image sensor included in the image sensor 1130 may be implemented using, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

The image stabilizer 1140 may move at least one lens or image sensor 1130 included in the lens assembly 1110 in a specific direction in response to movement of the camera module or the electronic device 100 including the camera module, or the image sensor ( Operation characteristics of 1130) may be controlled (eg, read-out timing is adjusted, etc.). This makes it possible to compensate at least part of the negative effect of the movement on the image being taken.

The memory 1150 may at least temporarily store at least a part of an image acquired through the image sensor 1130 for a next image processing task. For example, when image acquisition is delayed according to the shutter, or when a plurality of images are acquired at high speed, the acquired original image (eg, a Bayer-patterned image or a high-resolution image) is stored in the memory 1150 and , a copy image (eg, a low resolution image) corresponding thereto may be previewed through the display 151 . Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a part of the original image stored in the memory 1150 may be acquired and processed by, for example, the image signal processor 1160 . According to one embodiment, the memory 1150 may be configured as at least a part of the memory 170 or as a separate memory operated independently of the memory 170 .

The image signal processor 1160 may perform one or more image processes on an image acquired through the image sensor 1130 or an image stored in the memory 1150 . The one or more image processes, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring ( blurring, sharpening, or softening According to one embodiment, the image signal processor 1160 is configured as at least a part of the processor 130 or operates independently of the processor 130. It can be configured as a separate processor.

According to an embodiment, the electronic device 100 may include a plurality of camera modules each having different properties or functions. In this case, for example, at least one of the plurality of camera modules may be a wide-angle camera and at least the other may be a telephoto camera. Similarly, at least one of the plurality of camera modules may be a front camera, and at least another one may be a rear camera.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document may be implemented as software (eg, a program) including one or more instructions stored in a storage medium readable by a machine (eg, the electronic device 100). For example, a processor (eg, the processor 130 ) of a device (eg, the electronic device 100 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary storage medium' only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium and temporary It does not discriminate if it is saved as . For example, a 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store or between two user devices (eg smartphones). It can be distributed (e.g., downloaded or uploaded) directly or online. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored on a device-readable storage medium such as a memory of a manufacturer's server, an application store server, or a relay server. It can be temporarily stored or created temporarily.

Claims (21)

A method for providing a user interface supporting interaction by an electronic device, comprising:
Acquiring a plurality of images taken from a plurality of cameras;
identifying an image including a hand from among the plurality of acquired images;
obtaining 3D pose information of the hand based on the identified image and preset hand feature parameters; and
and providing interaction information corresponding to the obtained 3D pose information of the hand, based on an application executed in the electronic device. According to claim 1,
obtaining user profile information including at least one of the user's gender, age, height, weight, and region; and
The method further comprises setting a value of the preset hand characteristic parameter to a value corresponding to the acquired user profile information. According to claim 1,
Among the obtained plurality of images, as a plurality of images obtained as a result of photographing a hand located in a stereo area, which is an area where fields of view (FoV) of the plurality of cameras overlap, are identified, the identified plurality obtaining depth information of the hand and size information of the hand based on an image of;
Based on the obtained hand depth information and the obtained hand size information, updating hand characteristic parameters; and
As one image including the hand is identified after the update, obtaining 3D pose information of the hand based on the identified one image and the updated hand characteristic parameter, the method further comprising: . The method of claim 3, wherein updating the hand characteristic parameter comprises:
When the sharpness of the fingers of the hand detected in the plurality of identified images is equal to or greater than a predetermined standard, or the position of the detected hand is within a predetermined distance range from the center of the identified image, the obtained depth of the hand and updating the hand characteristic parameter based on information and the obtained hand size information. According to claim 3,
When the number of hand characteristic parameters set in the electronic device is greater than or equal to a threshold, updating of the hand characteristic parameters is not performed. According to claim 3,
Further comprising identifying accuracy of the provided interaction information based on user input information received after providing the interaction information;
In the step of updating the hand characteristic parameters,
If the accuracy of the provided interaction information does not satisfy a preset criterion, updating the hand characteristic parameter based on the obtained hand depth information and the obtained hand size information. The method of claim 1, wherein the hand characteristic parameter,
and size information for each part of the hand determined based on at least some points constituting the hand. According to claim 1,
obtaining tracking information of the hand; and
Acquiring probability information for a hand to be detected from the plurality of images based on the acquired hand tracking information;
Identifying the image including the hand,
and performing a hand detection operation on the plurality of images according to an order determined based on the obtained probability information. The method of claim 7, wherein the hand detection operation,
The method of claim 1 , wherein the hand detection operation according to the determined order is stopped as images including a predetermined number of hands are acquired. According to claim 7,
Based on the acquired hand tracking information, stopping shooting of a camera having an FoV of an area in which a probability of detecting the hand is less than a predetermined value among the plurality of cameras having different FoVs, . An electronic device providing a user interface supporting interaction,
a sensing unit including a plurality of cameras;
output unit; and
including at least one processor, wherein the at least one processor comprises:
Acquiring a plurality of images taken from the plurality of cameras, identifying an image including a hand among the acquired plurality of images, and based on the identified images and preset hand feature parameters to obtain 3D pose information of the hand, and provide interaction information corresponding to the acquired 3D pose information of the hand based on an application running in the electronic device. The method of claim 11, wherein the at least one processor,
Acquiring user profile information including at least one of the user's gender, age, height, weight, and region;
The electronic device sets a value of the preset hand characteristic parameter to a value corresponding to the acquired user profile information. The method of claim 11, wherein the at least one processor,
Among the obtained plurality of images, as a plurality of images obtained as a result of photographing a hand located in a stereo area, which is an area where fields of view (FoV) of the plurality of cameras overlap, are identified, the identified plurality Obtaining depth information of the hand and size information of the hand based on an image of
Based on the obtained hand depth information and the obtained hand size information, update the hand characteristic parameters;
When one image including the hand is identified after the update, 3D pose information of the hand is obtained based on the identified one image and the updated hand characteristic parameter. The method of claim 13, wherein the at least one processor,
When the sharpness of the fingers of the hand detected in the plurality of identified images is equal to or greater than a predetermined standard, or the position of the detected hand is within a predetermined distance range from the center of the identified image, the obtained depth of the hand and updating the hand characteristic parameter based on information and the obtained hand size information. According to claim 13,
When the number of hand characteristic parameters set in the electronic device is greater than or equal to a threshold, updating of the hand characteristic parameters is not performed. The method of claim 13, wherein the at least one processor,
Identifying accuracy of the provided interaction information based on user input information received after providing the interaction information;
and updating the hand characteristic parameter based on the obtained hand depth information and the obtained hand size information when accuracy of the provided interaction information does not satisfy a predetermined criterion. The method of claim 11, wherein the hand characteristic parameter,
and size information for each part of the hand determined based on at least some points constituting the hand. The method of claim 11, wherein the at least one processor,
Obtain tracking information of the hand;
Based on the acquired hand tracking information, obtaining hand detection probability information from the plurality of images;
identify an image containing the hand;
The electronic device performs a hand detection operation on the plurality of images according to an order determined based on the obtained probability information. The method of claim 18, wherein the hand detection operation,
While the hand detection operation is performed according to the determined order, the electronic device is stopped when an image including a predetermined number of hands is acquired. The method of claim 18, wherein the at least one processor,
Based on the acquired hand tracking information, among the plurality of cameras having different FoVs, a camera having a FoV of an area in which a probability of detecting the hand is less than a preset value is stopped from being photographed. A computer program product including a recording medium storing a program that causes an electronic device to perform a method of providing a user interface supporting interaction,
Obtaining a plurality of images taken from a plurality of cameras;
An operation of identifying an image including a hand among the plurality of acquired images;
Obtaining 3D pose information of the hand based on the identified image and preset hand feature parameters; and
A recording medium storing a program for performing an operation of providing interaction information corresponding to the acquired 3D pose information of the hand, based on an application executed in the electronic device.

Images