TW202213040A - Gesture control method based on image and electronic apparatus using the same - Google Patents

Abstract

A gesture control method based on image and an electronic apparatus using the same are provided. An image is displayed via the display screen. A hand image of a user hand is captured through the image capturing device. A gesture performed by the user hand in 3D space is detected by using the hand image, and whether the gesture is matched a predetermined control gesture is determined by using the hand image. If yes, keypoint detection is performed on the hand image to obtain at least one keypoint coordinate of the user hand. The keypoint coordinate is mapped to at least one 2D screen coordinate on the display screen. An operation is performed to a image object in the image according to the 2D screen coordinate.

Description

Image-based gesture control method and electronic device using the same

The present invention relates to an electronic device, and more particularly, to an image-based gesture control method and an electronic device using the same.

In the conventional user input interface, the electronic device is usually controlled by using keys, a keyboard or a mouse. With the advancement of technology, a new generation of user interfaces has become more and more user-friendly and more convenient, of which the touch interface is a successful example, which allows users to intuitively select objects on the screen and achieve the effect of control. For today's touch-sensitive electronic products, a user can control the electronic product through a stylus or a finger, so that the electronic product can perform various functions in response to the touch operation. However, as electronic products have more and more functions, the touch operation method of directly touching the screen has gradually been unable to meet the user's operational needs. For example, touch technology requires the user to touch or approach the touch screen to be effective, which directly limits the actual distance between the user and the electronic product. On the other hand, when the screen of the electronic product does not have the touch function, the user must use an additional input device to control the electronic device, which is usually less intuitive and inconvenient. In addition, corresponding to different user operation situations, there are more suitable user input methods.

In view of this, the present invention provides an image-based gesture control method and an electronic device using the method, which can improve the intuition and convenience of using the electronic device.

An embodiment of the present invention provides an image-based gesture control method, which is suitable for an electronic device including an image capture device and a display screen, and includes the following steps. Display the image through the display screen. A hand image is captured from the user's hand through the image capturing device. The hand image is used to detect a gesture performed by the user's hand in the three-dimensional space, and the hand image is used to determine whether the gesture conforms to the predetermined control gesture. If so, perform key point detection on the hand image to obtain at least one key point coordinate of the user's hand. At least one key point coordinate is mapped to at least one two-dimensional screen coordinate on the display screen. An operation is performed on the image object in the image according to at least one two-dimensional screen coordinate.

An embodiment of the present invention provides an electronic device including an image capture device, a storage device, and a processor. The processor is coupled to the image capture device and the storage device, and is configured to perform the following steps. Display the image through the display screen. A hand image is captured from the user's hand through the image capturing device. The hand image is used to detect a gesture performed by the user's hand in the three-dimensional space, and the hand image is used to determine whether the gesture conforms to the predetermined control gesture. If so, perform key point detection on the hand image to obtain at least one key point coordinate of the user's hand. At least one key point coordinate is mapped to at least one two-dimensional screen coordinate on the display screen. An operation is performed on the image object in the image according to at least one two-dimensional screen coordinate.

Based on the above, in the embodiment of the present invention, the user can perform an operation on an image object in the image displayed on the display screen through the floating gesture, so that the user can experience a more intuitive and intuitive experience when using the image design software. Convenient operation effect.

Some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Element symbols quoted in the following description will be regarded as the same or similar elements when the same element symbols appear in different drawings.

FIG. 1 is a functional block diagram of an electronic device according to an embodiment of the present invention. Referring to FIG. 1 , the electronic device 10 includes a display screen 110 , a storage device 120 , an image capture device 130 , and a processor 140 . The electronic device 10 may be a notebook computer, a desktop computer, a smart phone, a tablet computer, a game console, or other electronic devices having a display function, and the type of the electronic device 10 is not limited herein.

The display screen 110 may be a liquid crystal display screen (LCD), a light-emitting diode (Light Emitting Diode, LED) display screen, an organic light-emitting diode (Organic Light Emitting Diode, OLED) and other types of display screens. The present invention is not limited to this.

The storage device 120 is used to store data such as files, images, instructions, code, software components, etc., which can be, for example, any type of fixed or removable random access memory (random access memory, RAM), read-only memory body (read-only memory, ROM), flash memory (flash memory), hard disk or other similar devices, integrated circuits and combinations thereof.

The image capturing device 130 may include an image sensor having a Charge Coupled Device (CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) device for capturing an image in front of the display screen 110 , so as to detect the position and type of gesture operation performed by the user in front of the display screen 110 in the three-dimensional space. For example, the image capturing device 130 may be an RGB color camera device, but the invention is not limited thereto.

The processor 140 is coupled to the storage device 120 , the image capture device 130 and the display screen 110 for controlling the overall operation of the electronic device 10 , such as a central processing unit (CPU) or other programmable General-purpose or special-purpose microprocessors (Microprocessors), digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (Programmable Logic Device, PLD), or other similar devices or combinations of these devices. The processor 140 can execute the program codes, software modules, instructions, etc. recorded in the storage device 120 to implement the gesture control method of the embodiment of the present invention.

FIG. 2 is a flowchart of a gesture control method according to an embodiment of the present invention. Referring to FIG. 2 , the method of this embodiment is applicable to the electronic device 10 in the above-mentioned embodiment. The following describes the detailed steps of this embodiment in conjunction with various elements in the electronic device 10 .

In step S201 , an image is displayed through the display screen 110 . In one embodiment, the electronic device 10 can display photos or other image files through the display screen 110 when the user operates the image design software or the image playback software. The present invention does not limit the image file format. In step S202 , the processor 140 captures a hand image of the user's hand through the image capture device 130 .

In step S203, the processor 140 uses the hand image to detect gestures performed by the user's hand in the three-dimensional space, and acquires at least one key point coordinate of the user's hand. The user can compare various gestures by moving the finger. In one embodiment, using skin color detection, edge detection, machine learning model or other related computer vision technologies, the processor 140 can identify the type of the user's hand gestures through the hand image. In one embodiment, the processor 140 can determine, according to the hand image, that the user's gesture is a clenched fist gesture, a single-finger gesture, a two-finger gesture, or other multi-finger gestures. For example, the processor 140 may determine whether the user's gesture is a single-finger gesture in which the index finger is extended according to the hand image. Alternatively, the processor 140 may determine whether the user's gesture is a two-finger gesture of extending the index finger and extending the thumb according to the hand image. In this embodiment, the processor 140 can first determine whether the gesture conforms to the predetermined control gesture, and if so, the processor 140 performs key point detection on the hand image to obtain at least one key point coordinate of the user's hand.

In addition, in one embodiment, the processor 140 may perform Keypoint Triangulation on the user's hand image through the machine learning model, so as to detect a plurality of hand key points of the user's hand, and Get the 3D keypoint coordinates of these hand keypoints. As shown in FIG. 3 , FIG. 3 is a schematic diagram of the coordinates of key points of a user's hand according to an embodiment of the present invention. The processor 140 can calculate the 3D key point coordinates of the 21 user's hands from the single hand image Img_f through the machine learning model. As shown in FIG. 3 , the processor 140 may acquire a plurality of key point coordinates (for example, the key point coordinates KP1 of the fingertips of two index fingers and the key point coordinates KP2 of the thumb tip).

In step S204 , the processor 140 maps at least one key point coordinate to at least one two-dimensional screen coordinate on the display screen 110 . In order to achieve the purpose of manipulating the image objects on the display screen 110 , the processor 140 maps the coordinates of the key points of the user's hand into two-dimensional screen coordinates, so as to perform subsequent operations according to the two-dimensional screen coordinates. Specifically, the processor 140 may first convert the 3D key point coordinates into 2D virtual coordinates on a 2D plane, and then normalize the 2D virtual coordinates into 2D screen coordinates conforming to the screen coordinate system. In one embodiment, the processor 140 projects at least one key point coordinate onto a virtual plane between the user's hand and the image capturing device 130 to acquire at least one two-dimensional virtual coordinate on the virtual plane. Next, the processor 140 normalizes at least one two-dimensional virtual coordinate to obtain at least one two-dimensional screen coordinate on the display screen 110 according to the resolution of the display screen and the screen selection range.

In detail, FIG. 4 is a schematic diagram of generating two-dimensional screen coordinates according to an embodiment of the present invention. Referring to FIG. 4 , the three-dimensional key point coordinate KP1 of the index finger tip is (X, Y, Z). The processor 140 can project the key point coordinate KP1 on the virtual plane 41 between the user's hand and the camera position C1 of the image capture device 130 to obtain a two-dimensional virtual coordinate PV1 on the virtual plane 41, which is expressed as ( x, y). Next, the processor 140 can normalize the two-dimensional virtual coordinate PV1 according to the screen resolution to generate a two-dimensional screen coordinate PS1 on the screen coordinate system, which is represented as (x _cur , y _cur ).

In one embodiment, based on the similar triangle principle, the processor 140 can convert the three-dimensional key point coordinate KP1 into a two-dimensional virtual coordinate PV1 according to a depth ratio. The processor 140 multiplies the first coordinate component of the at least one key point coordinate KP1 by a depth ratio f/Z to obtain the first coordinate component of the at least one two-dimensional virtual coordinate PV1, ie x=X*f/Z. The processor 140 multiplies the second coordinate component of the at least one key point coordinate KP1 by a depth ratio f/Z to obtain the second coordinate component of the at least one two-dimensional virtual coordinate PV1, ie y=Y*f/Z. The depth ratio f/Z is the predetermined distance f between the virtual plane 41 and the image capturing device 130 and the third coordinate component Z of the at least one key point coordinate KP1 over the depth f.

式(1)

式(2)

式(3)

式(4) 藉此，處理器140可將使用者手部的一或多個關鍵點座標轉換為顯示螢幕110上螢幕選定範圍內的一或多個二維螢幕座標。 Then, referring to FIG. 4, the processor 140 can determine the two-dimensional screen coordinate PS1 according to the following equations (1) to (4) and the two-dimensional virtual coordinate PV1. Here, the coordinates of the upper left corner of the screen selection range are (x _min , y _min ), and the coordinates of the lower right corner of the screen selection range are (x _max , y _max ). The size and position of the selected range of the screen can be set according to actual needs, which is not limited in the present invention. In one embodiment, when the screen selection range is the full screen, (x _min , y _min ) can be represented as (0, 0), then (x _max , y _max ) can be represented as (S _width -1, S _height -1), and the resolution of the display screen 110 is S _width *S _height .

Formula 1)

Formula (2)

Formula (3)

Equation (4) Thereby, the processor 140 can convert one or more key point coordinates of the user's hand into one or more two-dimensional screen coordinates within a selected range of the screen on the display screen 110 .

Finally, in step S205, the processor 140 performs an operation on the image object in the image according to at least one two-dimensional screen coordinate. Specifically, the processor 140 can extract image objects from the image through various image analysis techniques, such as people, animals and plants, vehicles, daily necessities or other recognizable image objects in the image. In addition, the above operations may include selection operations, drag operations, zoom operations, or other image editing operations performed on the image object, which are not limited in the present invention. In one embodiment, the processor 140 can identify the image object selected by the user according to the two-dimensional screen coordinates associated with the user's hand. In one embodiment, the processor 140 can drag the image object from the first position to the second position according to the two-dimensional screen coordinates associated with the user's hand. In one embodiment, the processor 140 can zoom in or zoom out the image object according to the two-dimensional screen coordinates associated with the user's hand. In one embodiment, the processor 140 may perform color processing or other image retouching processing on the image object according to the two-dimensional screen coordinates associated with the user's hand. In this way, the user can perform various operations on the image object through a very intuitive operation method, which greatly improves the operation fluency and convenience of the image design software. The user is also not limited by the distance limit for performing touch operations, and can perform related operations from a position farther away from the electronic device 10 .

FIG. 5 is a schematic diagram of an application scenario of a gesture control method according to an embodiment of the present invention. Referring to FIG. 5 , the user U1 can select the image object obj_1 in the image Img_1 through the gesture G1 . Specifically, by mapping the key point coordinate KP1 of the user's hand to the two-dimensional screen coordinate PS1 on the display screen 110 , the electronic device 10 can determine that the user U1 selects the image object obj_1 . After the electronic device 10 determines that the user selects the image object obj_1, the user U1 can drag the image object obj_1 in the image Img_1 to a folder through the gesture G2 to save the image object obj_1 to the folder selected by the user. Alternatively, in other embodiments, the user can drag the image object obj_1 in the image Img_1 to another image through the gesture G2, so that the image object obj_1 is synthesized on the other image.

In order to explain the present invention more clearly, the following will take the selection operation and drag operation of the image object as an example for description. FIG. 6 is a flowchart of a gesture control method according to an embodiment of the present invention. Referring to FIG. 6 , the method of this embodiment is applicable to the electronic device 10 in the above-mentioned embodiment, and the detailed steps of this embodiment are described below in conjunction with various elements in the electronic device 10 .

In step S601 , an image is displayed through the display screen 110 . In step S602, the processor 140 performs a semantic segmentation (Semantic Segmentation) operation on the image to obtain object boundaries of the image objects in the image. In detail, the processor 140 can classify each pixel in the image as one or more image objects or image backgrounds through semantic segmentation operations. FIG. 7 is a schematic diagram of performing a semantic segmentation operation on an image according to an embodiment of the present invention. Referring to FIG. 7 , in one embodiment, the processor 140 may first perform object detection on the image Img_2 to detect the image objects in the image Img_2. For example, the processor 140 may perform object detection on the image Img_2 through a machine learning model (eg, a CNN model, etc.), so as to identify the image objects in the image Img_2 and their corresponding object types. After the object detection is performed, the processor 140 may acquire the bounding box and the object type corresponding to each image object, such as the bounding boxes B1 to B5 shown in FIG. 7 . Then, the processor 140 can perform semantic segmentation processing on the image blocks framed by the object frames B1-B5, and classify each pixel in the image Img_2 into a background and a plurality of image objects, so as to obtain the corresponding images of the image objects. Object boundaries M1 to M5.

In step S603 , the processor 140 captures a hand image of the user's hand through the image capture device 130 . In step S604, the processor 140 uses the hand image to determine whether the gesture conforms to the predetermined control gesture. In this embodiment, the predetermined control gesture includes a specific one-finger gesture and a specific two-finger gesture. In this embodiment, when the gesture does not conform to the specific one-finger gesture or the specific two-finger gesture, the processor 140 will not perform any operation on the image object. On the other hand, if the determination in step S604 is yes, in step S605, the processor 140 performs key point detection on the hand image to obtain at least one key point coordinate of the user's hand. In step S606 , the processor 140 maps at least one key point coordinate to at least one two-dimensional screen coordinate on the display screen 110 . For the operations of steps S604 to S605, reference may be made to the description of the foregoing embodiments.

When the gesture conforms to a specific one-finger gesture, in step S607, the processor 140 determines whether the at least one two-dimensional screen coordinate corresponding to the at least one key point coordinate is within the object boundary. If the determination in step S607 is yes, in step S608, the processor 140 performs a selection operation on the image object. Conversely, if the 2D screen coordinates of the key point coordinates are not within the object boundary, the processor 140 may mark a cursor on the display screen 110 for prompting the user according to the 2D screen coordinates of the key point coordinates.

For example, FIG. 8 is a schematic diagram of performing a selection operation on an image object according to an embodiment of the present invention. Referring to FIG. 8 , it is assumed that the image Img_3 is displayed on the display screen 110 , and the image objects Obj_1 to Obj_4 can be acquired by the image Img_3 through the semantic segmentation operation. When the key point coordinate KP1_1 of the index finger tip is (X _i , Y _i , Z _i ), the processor 140 can map the key point coordinate KP1_1 to the two-dimensional screen coordinate PS1_1 . The processor 140 can determine that the two-dimensional screen coordinate PS1_1 is not within the object boundary of the image objects Obj_1 to Obj_4, so the processor 140 can control the display screen 110 to display a cursor at the two-dimensional screen coordinate PS1_1. Afterwards, when the user's hand item moves right, the key point coordinate KP1_2 of the index finger tip is (X _f , Y _f , Z _f ), and the processor 140 can map the key point coordinate KP1_2 to the two-dimensional screen coordinate PS1_2 . The processor 140 can determine that the two-dimensional screen coordinate PS1_2 is within the object boundary of the image object Obj_3, so the processor 140 can perform a selection operation on the image object Obj_3 to perform other operations on the image object Obj_3 according to other subsequent gestures. Alternatively, in one embodiment, when the user has selected to execute a specific image editing function, the processor 140 may directly implement the aforementioned image editing function on the image object Obj_3 according to the above-mentioned selection operation. In one embodiment, the processor 140 can control the display screen 110 to display a thick border effect, a magnified image object Obj_3 or other visual effects around the image object Obj_3 to remind the user that the image object Obj_3 has been selected.

On the other hand, when the gesture conforms to a specific two-finger gesture, in step S609, the processor 140 determines whether the distance between the coordinates of the first key point and the coordinates of the second key point is less than a threshold value. If the determination in step S609 is yes, in step S610, the processor 140 starts to perform a drag operation on the image object. In step S611, in response to the distance between the coordinates of the first key point and the coordinates of the second key point being greater than another threshold value, the processor 140 ends the drag operation on the image object.

式(5) FIG. 9 is a schematic diagram of calculating the distance between the first key point and the second key point according to an embodiment of the present invention. Referring to FIG. 9 , when the gesture conforms to a specific two-finger gesture, the processor 140 may determine the key point coordinate KP1 (ie the first key point coordinate) of the index finger tip and the key point coordinate KP2 (ie the second key point) of the thumb tip coordinates) to determine whether the user attempts to perform a drag operation on the image object and the drag path corresponding to the drag operation. As shown in FIG. 9 , the processor 140 can obtain the key point coordinate KP1 and the key point coordinate by calculating the Euclidean distance between the coordinates (X ₁ , Y ₁ , Z ₁ ) and the coordinates (X ₂ , Y ₂ , Z ₂ ) The distance d between KP2 is represented by the following formula (5).

Formula (5)

FIG. 10 is a schematic diagram of performing a drag operation on an image object according to an embodiment of the present invention. It is assumed that the user has selected the image object obj_10. Referring to FIG. 10 , when the user's index finger and thumb are close enough, the distance between the key point coordinate KP1_1 and the key point coordinate KP2_1 will be smaller than the threshold value. At the same time, in response to the distance between the key point coordinate KP1_1 and the key point coordinate KP2_1 being smaller than the threshold value, the processor 140 may start to perform a drag operation on the image object obj_10 in the folder F1. The user can then move the user's hand position without changing the gesture. After dragging the image object obj_10 to the target position (eg, the screen display position of the folder F2 ), the user can pull the index finger and thumb apart to release. When the user's index finger and thumb are pulled apart to a sufficient extent, the distance between the key point coordinates KP1_2 and the key point coordinates KP2_2 will be greater than another threshold value. In response to the fact that the distance between the key point coordinate KP1_2 and the key point coordinate KP2_2 is greater than another threshold value, the processor 140 may end performing a drag operation on the image object obj_10 . In one embodiment, in response to the distance between the key point coordinate KP1_2 and the key point coordinate KP2_2 being greater than another threshold value, the processor 140 may decide to drag according to the key point coordinate KP1_2 or the two-dimensional screen coordinate corresponding to the key point coordinate KP2_2 The drag end point of the operation. Thereby, the image object obj_10 can be copied or moved to the folder F2.

To sum up, in the embodiment of the present invention, the user can perform an operation on an image object in the image displayed on the display screen through the floating gesture. Users can perform various operations on image objects through a very intuitive operation method, which greatly improves the operation fluency and convenience of the image design software. The user is also not limited by the distance limit for performing touch operations, and can perform related operations from a position farther away from the electronic device.

Although the present invention has been disclosed above by the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the scope of the appended patent application.

10: Electronics 110: Display screen 120: Storage Device 130: Line of sight detection device 140: Processor Img_f: hand image KP1, KP2, KP1_1, KP1_2, KP2_1, KP2_2: key point coordinates 41: Virtual Plane PV1: 2D virtual coordinates PS1, PS1_1, PS1_2: 2D screen coordinates C1: Camera position G1~G2: Gestures Img_1, Img_2, Img_3: Image B1～B5: Object frame M1～M5: object boundary F1, F2: folder obj_1～obj_4, obj_10: image objects S201～S205, S601～S611: Steps

FIG. 1 is a functional block diagram of an electronic device according to an embodiment of the present invention. FIG. 2 is a flowchart of a gesture control method according to an embodiment of the present invention. FIG. 3 is a schematic diagram of key point coordinates of a user's hand according to an embodiment of the present invention. FIG. 4 is a schematic diagram of generating two-dimensional screen coordinates according to an embodiment of the present invention. FIG. 5 is a schematic diagram of an application scenario of a gesture control method according to an embodiment of the present invention. FIG. 6 is a flowchart of a gesture control method according to an embodiment of the present invention. FIG. 7 is a schematic diagram of performing a semantic segmentation operation on an image according to an embodiment of the present invention. FIG. 8 is a schematic diagram of performing a selection operation on an image object according to an embodiment of the present invention. FIG. 9 is a schematic diagram of calculating the distance between the first key point and the second key point according to an embodiment of the present invention. FIG. 10 is a schematic diagram of performing a drag operation on an image object according to an embodiment of the present invention.

S201~S205: Steps

Claims (10)

An image-based gesture control method suitable for an electronic device including an image capture device and a display screen, the method comprising: display an image through the display screen; capturing a hand image of the user's hand through the image capturing device; Use the hand image to detect a gesture performed by the user's hand in three-dimensional space, use the hand image to determine whether the gesture conforms to a predetermined control gesture, and if so, perform key point detection on the hand image to obtain at least one key point coordinate of the user's hand; mapping the at least one keypoint coordinate to at least one two-dimensional screen coordinate on the display screen; and An operation is performed on the image object in the image according to the at least one two-dimensional screen coordinate. The image-based gesture control method of claim 1, wherein the step of mapping the at least one key point coordinate to the at least one two-dimensional screen coordinate on the display screen comprises: projecting the at least one key point coordinate onto a virtual plane between the user's hand and the image capturing device to obtain at least one two-dimensional virtual coordinate on the virtual plane; and The at least one two-dimensional virtual coordinate is normalized according to the resolution of the display screen and a screen selection range to obtain the at least one two-dimensional screen coordinate on the display screen. The image-based gesture control method as claimed in claim 1, wherein the at least one key point coordinate is projected onto the virtual plane between the user's hand and the image capture device to obtain the The steps of the at least one two-dimensional virtual coordinate include: multiplying the first coordinate component of the at least one key point coordinate by a depth ratio to obtain the first coordinate component of the at least one two-dimensional virtual coordinate; and Multiplying the second coordinate component of the at least one key point coordinate by the depth ratio to obtain the second coordinate component of the at least one two-dimensional virtual coordinate, wherein the depth ratio is the difference between the virtual plane and the image capture device The predetermined distance to depth ratio is the third coordinate component of the at least one key point coordinate. The image-based gesture control method of claim 1, wherein the step of performing the operation on the image object in the image according to the at least one two-dimensional screen coordinate comprises: performing a semantic segmentation operation on the image to obtain the object boundary of the image object in the image; When the gesture conforms to a specific one-finger gesture, determining whether the at least one two-dimensional screen coordinate corresponding to the at least one key point coordinate is within the bounds of the object; and If so, perform a selection operation on the image object. The image-based gesture control method of claim 1, wherein the at least one key point coordinate includes a first key point coordinate and a second key point coordinate, and the image object in the image is determined according to the at least one two-dimensional screen coordinate The steps to do this include: performing a semantic segmentation operation on the image to obtain the object boundary of the image object in the image; When the gesture conforms to a specific two-finger gesture, determine whether the distance between the coordinates of the first key point and the coordinates of the second key point is less than a threshold value; If so, start performing a drag operation on the image object; and In response to the distance between the coordinates of the first key point and the coordinates of the second key point being greater than another threshold value, the drag operation is ended. An electronic device, comprising: an image capture device; a display screen; a storage device recording a plurality of instructions; and a processor, coupled to the display screen, the image capture device and the storage device, configured to: an image through the display screen; capturing a hand image of the user's hand through the image capturing device; Use the hand image to detect a gesture performed by the user's hand in three-dimensional space, use the hand image to determine whether the gesture conforms to a predetermined control gesture, and if so, perform key point detection on the hand image to obtain at least one key point coordinate of the user's hand; mapping the at least one keypoint coordinate to at least one two-dimensional screen coordinate on the display screen; and An operation is performed on an image object in the image according to the at least one two-dimensional screen coordinate. The electronic device of claim 6, wherein the processor is further configured to: projecting the at least one key point coordinate onto a virtual plane between the user's hand and the image capturing device to obtain at least one two-dimensional virtual coordinate on the virtual plane; and The at least one two-dimensional virtual coordinate is normalized according to the resolution of the display screen and a screen selection range to obtain the at least one two-dimensional screen coordinate on the display screen. The electronic device of claim 6, wherein the processor is further configured to: multiplying the first coordinate component of the at least one key point coordinate by a depth ratio to obtain the first coordinate component of the at least one two-dimensional virtual coordinate; and Multiplying the second coordinate component of the at least one key point coordinate by the depth ratio to obtain the second coordinate component of the at least one two-dimensional virtual coordinate, wherein the depth ratio is the difference between the virtual plane and the image capture device The predetermined distance to depth ratio is the third coordinate component of the at least one key point coordinate. The electronic device of claim 6, wherein the processor is further configured to: performing a semantic segmentation operation on the image to obtain the object boundary of the image object in the image; When the gesture conforms to a single-finger gesture, determining whether the at least one two-dimensional screen coordinate corresponding to the at least one key point coordinate is within the object boundary; and If so, perform a selection operation on the image object. The electronic device of claim 6, wherein the at least one keypoint coordinate includes a first keypoint coordinate and a second keypoint coordinate, and the processor is further configured to: performing a semantic segmentation operation on the image to obtain the object boundary of the image object in the image; When the gesture conforms to the two-finger gesture, determine whether the distance between the coordinates of the first key point and the coordinates of the second key point is less than a threshold value; If so, perform a drag operation on the image object; and In response to the distance between the coordinates of the first key point and the coordinates of the second key point being greater than another threshold value, the drag operation is ended.

Images