WO2018076720A1

WO2018076720A1 - One-hand operation method and control system

Info

Publication number: WO2018076720A1
Application number: PCT/CN2017/089027
Authority: WO
Inventors: 黄源浩; 刘龙; 肖振中; 许星
Original assignee: 深圳奥比中光科技有限公司
Priority date: 2016-10-25
Filing date: 2017-06-19
Publication date: 2018-05-03
Also published as: CN106569716B; CN106569716A

Abstract

The present invention discloses a one-hand operation method and an operation system, the one-hand operation method comprising the steps of: S1: Acquiring an operation surface and a depth image of an operation object on the operation surface; S2: Obtaining a first position of the operation object on the operation surface by means of the depth image; S3: Positioning a second position on a display screen according to the first position of the operation object on the operation surface; S4: According to a predetermined operation action determined by a shape and an action of the operation object, identifying and translating the predetermined operation action into a touch instruction which needs to be executed; S5: Performing a touch operation at a second position according to the touch instruction. Through the operation surface, the present invention can well realize the one-hand operation of a large-screen electronic device and avoid a touch control conflict.

Description

One-handed control method and control system

Technical field

The present invention relates to the field of electronic technology, and in particular to a one-handed control method and a control system.

Background technique

With the popularity of the mobile Internet and the increasing demand for functions of smart devices such as mobile phones, the functions of communication, Internet access and video playback are the most basic configurations of electronic devices. At present, the size of devices such as mobile phones has almost entered the era of big screens, and one-handed operation is out of reach. However, taking into account larger screens and one-handed operation is still the current user demand for mobile phones.

Taking a mobile phone as an example, adding a function key to the back of the mobile phone is a solution for one-handed operation, but it will undoubtedly affect the beauty of the back of the mobile phone, and thus this solution has not been accepted by the user. Another option is to add an additional touch screen to the back of the phone. This solution allows control of the area of the phone's screen that cannot be operated with one hand by manipulating the finger on the back. However, this solution is costly and cannot be a mainstream one-hand operation.

At the same time, the touch operation is performed by using the depth image, and the display screen outside the area cannot be touched between the control object and the display screen, and the position of the manipulation object on the display screen directly maps to the pixel on the acquired depth image. The coordinates, that is, the position of the manipulation object on the display screen is the pixel coordinate mapping of the manipulation object in the depth image. This method of directly obtaining the position on the display screen through the pixel coordinate mapping cannot touch the manipulation object. The area is touch-operated, so only some functions such as page turning can be implemented simply, and the one-hand control problem of the large screen cannot be solved well.

Moreover, the existing scheme of using the depth image for touching is to set the position of the manipulation object on the display screen by setting the depth camera on the display screen side, and the best way is to control the object. When the display is touched, this method is suitable for a display without a touch function. For a touch screen with a touch function, the finger touch display screen is determined to generate a touch on the screen. The location information touches other areas of the display. Since two touch commands are generated, the touch conflict is caused, so that other areas of the display can no longer be touched, and other areas are touched. When controlling, only the touch object is not required to touch the display, which makes The mapping relationship between the position of the manipulation object on the display screen through the depth image acquisition is complicated, and the experience is greatly weakened.

The above disclosure of the present invention is only for assisting in understanding the inventive concept and technical solution of the present invention, and it does not necessarily belong to the prior art of the present patent application, and there is no clear evidence that the above content has been disclosed on the filing date of the present patent application. In the event that the above background art should not be used to evaluate the novelty and inventiveness of the present application.

Summary of the invention

The object of the present invention is to provide a one-handed control method and a control system, which can solve the problem that the above-mentioned prior art can not easily achieve one-hand control, and the touch screen with touch function is easy to generate touch conflict. problem.

To this end, the present invention provides a one-handed control method in which the display screen and the control surface for the touch operation are on different planes, including the following steps: S1: acquiring the control surface and the depth of the manipulation object on the control surface Image; S2: obtaining a first position of the manipulation object on the manipulation surface by the depth image; S3: positioning the second position on the display screen according to the first position of the manipulation object on the manipulation surface; S4: according to the shape and shape of the manipulation object The predetermined control action determined by the action is recognized and converted into a touch command to be executed; S5: performing a touch operation at the second position according to the touch command.

Preferably, the manipulation method of the present invention may further have the following technical features:

The control surface comprises at least one control area, the at least one of the control areas being automatically delimited by the acquired depth image and the obtained position of the manipulation object on the control surface.

The size of the automatically delineated control area is easily accessible on the control surface when the manipulation object is operated by one hand.

The display screen includes a near touch area for controlling an object to be easily touched, and a far touch area that is not easily touched except the near touch area, and the manipulation object utilizes the display screen itself in the near touch area. The touch function performs one-touch operation, and the touch is performed in the second position obtained by positioning in the far touch area.

The near touch area and the far touch area are automatically delimited according to the position of the manipulation object on the display screen.

The step of acquiring the depth image includes: S11: acquiring a first depth image including a control surface and not including the manipulation object; S12: acquiring a second depth image including the manipulation surface and the manipulation object; S13: passing the second depth image And obtaining a third depth image of the manipulation object with the first depth image.

Obtaining the first location includes the following steps: S21: determining, according to the second depth image, whether the manipulation object is in contact with the control surface, and if not, performing the next step; S22: obtaining the manipulation object according to the third depth image In the spatial position information of the coordinate system in which the display screen is located, the coordinates of the vertices of the manipulation object are taken as the first position.

The obtaining of the second location includes the following steps: S31: establishing a mapping relationship between the control plane and the display screen; S32: obtaining a second location of the display screen by the mapping relationship and the first location.

A linear mapping relationship is established according to the control surface and the horizontal and vertical dimensions of the display screen.

In addition, the present invention also provides a one-hand control system for performing the above touch method, including an image acquisition unit, a processor, a control surface, and a display screen, wherein the control surface and the display screen are on different planes. ;

The image acquisition unit is configured to acquire a control surface and a depth image of the manipulation object and depth information of the manipulation object;

The processor includes an identification module, a positioning module, a conversion module, and an execution module, and the identification module is configured to acquire a first position of the manipulation object on the manipulation surface according to the depth image, and identify a predetermined manipulation action of the manipulation object The positioning module is configured to locate a second position on the display screen that needs to be manipulated according to the first position; the conversion module converts and generates a corresponding touch instruction according to a predefined control action; the execution module is used to Performing a touch command at the second position completes a touch operation on the display screen.

Advantageous effects of the present invention compared to the prior art include:

The present invention utilizes a depth image to implement a touch operation and is a one-handed control method in the control method. The control surface for the control and the display screen are arranged on different planes, the control object completes the touch operation on the control surface, and the depth image is used to obtain the first position of the manipulation object on the manipulation surface, and the display screen is further obtained through the first position. The second position is combined with a predetermined manipulation action to perform a touch operation at the second position, so that when the user touches the electronic device with the large screen, when the touch screen is touched The control can be completed on the control surface, and the position of the control object on the display screen is completed, thereby avoiding the problem of touch conflict between the existing control and the touch.

At the same time, for the hard-to-touch area on the display screen, simply press the predetermined control action on the control surface to touch the display of the other plane. On the control surface, the object can be manipulated at any time. It is in contact with the control surface to facilitate accurate and quick access to the position of the object on the control surface.

Compared with the prior art, the present invention can not only realize simple gesture operations such as page turning and rewinding for a display screen without a touch function, but also realize one-hand touch well for a single-hand touch. The object can be touched through the control surface and can achieve precise touch.

In a preferred embodiment, the control surface includes at least one control area to solve the acquisition and manipulation of the depth image under different touch habits of the manipulation object, such as the left and right hands, and the manipulation area is automatically delineated according to the depth image. Generally, the manipulation area is A preset area on the control surface is automatically delineated by the position and depth image of the manipulated object on the manipulated object. It is not necessary to control the object to be manipulated in a certain area to perform the touch operation, thereby improving the manipulation. Experience, in particular, the size of the control area is easily achieved on the control surface when the manipulation object is operated by one hand, and the shape defined by the optimization control area is not limited to a rectangle, and may be the most on the control surface according to the manipulation object. The shape of the area that is easy to touch is determined, generally an irregular fan shape.

Based on the above-mentioned control method, for the touch function of the display screen itself, a mode that uses both its own touch function and a directional control method for hybrid control is implemented, such as: The object is touched by the touch function of the display screen in the near touch area for one-hand touch, and the touch is performed in the second position obtained by positioning in the far touch area. This hybrid control method can compensate for the low precision caused by the randomness of the manipulation of the moving object, and provides a better experience for the user. The above-mentioned near touch area and far touch area are also automatically delineated according to the user's touch habits, to adapt to the user's habit of manipulation and to optimize the shape of the near touch area and the far touch area.

Obtaining a third depth image of the manipulation object according to the first depth image and the second depth image, and then acquiring the first position according to the third depth image, so that the obtained third depth image only includes the manipulation object The depth image, so when the first position is acquired, the calculation amount of the processing process can be reduced, the calculation speed can be improved, and the response speed of the system can be improved.

By establishing a mapping relationship between the control planes and the display screens on different planes, the second position on the display screen can be quickly obtained through the first position, and the linear mapping relationship can be used to control the control surface or the control area as a rectangle. The shape quickly establishes a mapping relationship based on the horizontal and vertical dimensions.

DRAWINGS

1 is a schematic structural view of a control system according to a first embodiment of the present invention;

2 is a schematic structural diagram of a processor according to a first embodiment of the present invention.

3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.

4 is a schematic rear view of a single hand operated mobile phone according to the third and fourth embodiments of the present invention.

Fig. 5 is a side elevational view showing the one hand control of the mobile phone according to the third and fourth embodiments of the present invention.

Figure 6 is a schematic illustration of the hybrid manipulation of embodiments 2 and 5 of the present invention.

FIG. 7 is a flowchart 1 of the operation of the fourth embodiment of the present invention.

FIG. 8 is a second flowchart of the operation of the fourth embodiment of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the specific embodiments and with reference to the accompanying drawings. It is to be understood that the following description is only illustrative, and is not intended to limit the scope of the invention.

Non-limiting and non-exclusive embodiments will be described with reference to the drawings, wherein like reference numerals refer to the

Embodiment 1:

This embodiment provides a one-handed control system, as shown in Figure 1, including an image acquisition unit 1, a processor 2, a control surface 3 and a display screen 4;

The image acquisition unit 1 is configured to acquire the control surface 3 and the depth image of the manipulation object and the depth information of the manipulation object, and the control surface 3 and the display screen 4 are on different planes;

As shown in FIG. 2, the processor 2 includes an identification module 21, a positioning module 22, a conversion module 23, and an execution module 24. The identification module 21 is configured to acquire a control object on the control surface 3 according to the depth image. a position, and a predetermined manipulation action for identifying the manipulation object; the positioning module 22 is configured to position a second position on the display screen 4 to be manipulated according to the first position; the conversion module 23 is configured to be based on a predefined The control action is generated to generate a corresponding touch command; the execution module 24 is configured to perform a touch operation on the display screen 4 by executing a touch command at the second position;

The system can acquire the depth image of the control surface 3 through the image acquisition unit, and also acquire the depth image of the manipulation object that is in contact with the display screen 4, according to which the position and motion of the manipulation object on the control surface 3 can be recognized, thereby It is converted into a corresponding position on the display screen 4 and the instruction, so that the manipulation of the device can be realized. In this embodiment, for example, the display screen of the e-book reader is considered to be on the display screen. Touch manipulation on the display will affect the display effect. For example, when the page is constantly flipped with a finger on the e-book, the hand will inevitably block part of the screen, thus affecting the reading experience. Using the manipulation object to control on the control surface will alleviate this problem and enhance the user experience.

The image acquisition unit 1 herein is a depth camera based on the structured light or TOF principle, and generally includes a receiving unit and a transmitting unit.

Using the control surface can effectively avoid the problem of touch conflicts. At the same time, for the hard-to-touch area on the display, simply press the predetermined control action on the control surface to touch the display of the other plane. On the control surface, the object can be manipulated at any time. Face contact to facilitate accurate and fast access to the position of the object on the control surface.

Embodiment 2:

Compared with the first embodiment, the difference is that the display screen of the embodiment is combined with the control surface 3 and the display screen 4 to complete the touch operation, and the display screen 4 is a touch screen. As shown in FIG. 6 , the touch screen includes a touch panel 1 and a far touch region 2 , and the control object 16 performs touch control in the near touch region 1 according to the touch function of the touch screen itself. The far touch area 2 is directed by the control surface 16 to complete the touch operation.

Specifically, the near touch area 1 that can be reached by one hand and the far touch area 2 that cannot be reached by one hand, the touch control is still used in the near touch area 1 to ensure better precision, and in the far touch area. 2, the first position of the control surface 3 is used and the operation is performed according to the shape and motion of the touch object. This embodiment is particularly suitable for a touch-enabled device such as a mobile phone that is currently unable to be completely controlled by a single screen due to a large screen.

Embodiment 3:

This embodiment provides an electronic device, which may be a mobile phone or a tablet, and includes a bus 5 for data transmission. As shown in FIG. 3, the bus 4 is divided into a data bus, an address bus, and a control bus, respectively. To transfer data, data addresses and control signals. Connected to the bus 4 are a CPU 6, a display 7, an IMU 8 (inertial measurement device), a memory 9, a camera 11, a sound device 12, a network interface 13, and a mouse/keyboard 14. For some electronic devices with touch functions, The mouse and keyboard are replaced by the display 7, and the IMU device is used for positioning, tracking and other functions. The memory 9 is used to store an operating system, an application, etc., and can also be used to store temporary data during operation.

The image acquisition unit 1 in the manipulation system provided on the electronic device corresponds to the camera 11, and processes The device 2 corresponds to the CPU 8, and may also be a separate processor. The display screen 4 corresponds to the display 7. The control surface is disposed on the back of the mobile phone, and is in a different plane from the display screen.

As shown in Figure 4, the camera 11 is typically secured to an electronic device. In this embodiment, the imaging direction is greatly different from that of the camera of the existing device. Existing cameras are typically front or rear, and such a configuration cannot capture an image of the device itself. In this embodiment, the configuration of the camera 11 can be various. One is to rotate the camera by rotating the camera 90 degrees after rotating, and the other is to take the camera externally. The camera as a whole is connected to the device through a certain fixed measure and via an interface such as USB. Those skilled in the art can select the form of the camera on the electronic device according to the actual situation, without limitation.

At this time, with the existing camera, the camera 11 of the present embodiment is a depth camera for acquiring a depth image of the target area.

Figure 4 shows the back side of the phone with one hand. A camera 11 is arranged on the top of the mobile phone, and the camera 11 collects the direction from the top to the bottom of the mobile phone, so that the control surface on the back of the mobile phone and the image of the finger (control object) can be obtained, and FIG. 5 shows the operation of the mobile phone with one hand. Side view, 17 is the first position on the control surface, 16 is the manipulation object.

The display 7 of the electronic device may be of a touch function or a touch function. When the touch function is not provided, it can be controlled only by the control surface, and when the touch function is provided, the display can be touched by the control object. The area is manipulated to perform control plane manipulation on areas that are not touched.

The camera 11 in this embodiment may also be a normal RGB camera for capturing RGB images; the camera 11 may also be an infrared camera for capturing infrared images; or may be a depth camera, such as based on the principle of structured light or based on the TOF principle. Depth camera, etc.

Depth images acquired with depth cameras are not affected by dark light, and can be measured even in the dark. In addition, positioning and motion recognition using depth images is more accurate than RGB images. Therefore, in the following description, the depth camera and the depth image will be explained. However, the invention should not be limited to depth cameras.

Embodiment 4:

A one-handed control method, as shown in Figures 4-5 and 7, includes the following steps:

S1: acquiring a depth image of the control surface 3 and the manipulation object 16 on the control surface 3;

S2: obtaining a first position 17 of the manipulation object 16 on the manipulation surface 3 by the depth image;

S3: positioning the second position on the display screen 4 according to the first position 17 of the manipulation object 16 on the control surface 3;

S4: performing a predetermined manipulation action determined according to the shape and motion of the manipulation object 16, and identifying and converting into a touch instruction that needs to be executed;

S5: Perform a touch operation at the second position according to the touch instruction.

The control surface 3 comprises at least one control region 15 , the at least one of which is automatically delimited by the acquired depth image and the position of the controlled object 16 on the control surface 3 . The size of the automatically delineated control area 15 is easily accessible on the control surface 3 when the handling object 16 is operated with one hand. In the process of automatically delimiting, the contact point when the control object is in contact with the control surface 3 is determined. When the touch operation is performed, all the points in contact with the touch surface in this operation are determined as the manipulation area 15.

The control surface 3 includes at least one control area 15 to solve the acquisition and manipulation of the depth image under different touch habits of the manipulation object 16, such as left and right hands, and the manipulation area 15 is automatically delineated according to the depth image. Generally, the manipulation area 15 For a certain area on the control surface 3, the position and depth image of the manipulation object 16 on the manipulation object 16 are automatically demarcated, and the manipulation object 16 is required to be controlled in a certain area to perform the touch operation. In particular, the size of the manipulation area 15 is easily achieved on the control surface 3 when the manipulation object 16 is operated by one hand, and the shape delimited by the manipulation area 15 is not limited to a rectangle. It can be determined according to the shape of the area of the manipulation object 16 that is most easily touched on the control surface 3, generally an irregular fan shape.

The acquired depth image includes other unrelated parts in addition to the control surface 3 and the finger. In this case, the measurement range of the depth camera can be limited, that is, a certain threshold is set, and the depth information exceeding the threshold is removed. The depth image acquired in this way will only contain the information on the back of the phone and the finger, which can reduce the amount of calculation of the recognition. In order to further improve the system running speed and reduce the amount of calculation, the image segmentation method is used to obtain the depth image, and one method includes the following steps:

S11: acquiring a first depth image including the control surface 3 and not including the manipulation object 16;

S12: acquiring a second depth image including the control surface 3 and the manipulation object 16;

S13: obtaining, by the second depth image and the first depth image, the first object of the manipulation object 16 Three depth images.

The manipulation object 16 of the present embodiment is a finger, and the depth image of the front end portion of the finger obtained by the background segmentation method in the above step can reduce the calculation amount at the time of modeling and increase the calculation speed.

As shown in FIG. 8, acquiring the first location 17 includes the following steps:

S21: determining, according to the second depth image, whether the manipulation object 16 is in contact with the control surface 3, and if contacting, performing the next step;

S22: Obtain spatial position information of the manipulation object 16 in the coordinate system of the display screen 4 according to the third depth image, and take the coordinates of the apex of the manipulation object 16 as the first position 17.

The obtaining of the second location comprises the following steps:

S31: Establish a mapping relationship between the control surface 3 and the display screen 4;

S32: obtaining the second position of the display screen 4 by the mapping relationship and the first position 17.

A linear mapping relationship is established according to the horizontal and vertical dimensions of the control surface 3 and the display screen 4.

By establishing a mapping relationship between the control surface 3 and the display screen 4 on different planes, the second position on the display screen 4 can be quickly obtained through the first position 17, and the control plane 3 or the manipulation area can be used by the linear mapping relationship. 15 is a relatively regular shape such as a rectangle, and a mapping relationship is quickly established according to the horizontal and vertical dimensions.

In the existing touch technology, positioning and touch are completed at the same time, but this is not feasible on the invisible back. Therefore, the positioning needs to be performed first, and the manipulation action may be a change of the shape of the finger, such as a change in the angle between the finger and the display screen 4, or an action of the finger, such as a click action. The shape and motion recognition of the finger are prior art and will not be described in detail herein.

Of course, some operations that do not require positioning, such as page turning, rollback, etc., are not excluded. For this type of operation, it is possible to recognize only the shape or motion of the finger.

In this embodiment, when the finger touches the back side and reduces the tilt angle of the finger, a click action is completed. The shape of the finger and the manipulation command corresponding to the action need to be preset. When a finger shape and motion is recognized, the processor converts it into a corresponding manipulation command.

Embodiment 5:

Considering the randomness of finger movement, its accuracy is difficult to reach the degree of touch. Therefore, for some one-handed control devices with touch-enabled mobile phones, the touch operation is still selected in the area that can be reached by one hand, and the non-contact control in the area that cannot be reached by one hand is a better experience. So this embodiment The display screen 4 is a touch screen with a touch function. As shown in FIG. 6 , the display screen 4 includes a near touch area 1 for controlling the object 16 to be easily touched, and is not easy to touch except for the near touch area. In the far touch area, the control object 16 performs one-hand touch using the touch function of the display screen 4 in the near touch area, and touches the second position obtained by positioning in the far touch area.

Since the hand size, left and right hand habits, etc. of different users are different, the distinction between the touch area 1 and the pointing area 2 can be automatically recognized and delineated by the system. That is, when the finger on the display 4 side is in contact with the touch display screen 4, the processor processes the signal fed back by the touch display screen 4 and executes the corresponding touch command, while the finger on the display screen 4 side and the touch display screen 4 When not in contact, the depth camera recognizes the non-contact motion and feeds back to the processor. The processor processes the depth image acquired by the depth camera and performs a touch operation through the second position.

Based on the above-mentioned control method, for the touch screen function of the display screen 4 itself, a mode in which the touch control function is utilized and the control method of the pointing type is used for the hybrid control is implemented, such as: The control object 16 performs one-hand touch using the touch function of the display screen 4 in the near touch area, and performs touch control in the second position obtained by positioning in the far touch area. This hybrid control method can compensate for the low precision caused by the randomness of the manipulation of the object 16 and provide a better experience for the user. The above-mentioned near touch area and far touch area are also automatically delineated according to the user's touch habits, to adapt to the user's habit of manipulation and to optimize the shape of the near touch area and the far touch area.

Those skilled in the art will recognize that many variations are possible in the above description, and thus the embodiments are only used to describe one or more specific embodiments.

While the invention has been described and described with reference to the embodiments of the embodiments In addition, many modifications may be made to adapt a particular situation to the teachings of the invention, without departing from the inventive concept. Therefore, the invention is not limited to the specific embodiments disclosed herein, but the invention may also include all embodiments and equivalents thereof.

Claims

A one-handed control method is characterized in that the display screen and the control surface for the touch operation are on different planes, including the following steps:

S1: acquiring a depth image of the control surface and the manipulation object on the manipulation surface;

S2: acquiring a first position of the manipulation object on the manipulation surface by using the depth image;

S3: positioning the second position on the display screen according to the first position of the manipulation object on the manipulation surface;

S4: performing a predetermined manipulation action determined according to the shape and motion of the manipulation object, and performing recognition and converting into a touch instruction that needs to be executed;

S5: Perform a touch operation at the second position according to the touch instruction.
The manipulation method according to claim 1, wherein the control surface comprises at least one manipulation area, and at least one of the manipulation regions passes the acquired depth image and the obtained manipulation object on the manipulation surface The location is automatically delimited.
The manipulation method according to claim 2, wherein the automatically delineated size of the manipulation area is easily accessible on the manipulation surface when the manipulation object is operated by one hand.
The control method of claim 1 , wherein the display screen comprises a near touch area for controlling an object to be easily touched, and a far touch area that is not easily touched except for the near touch area. The control object performs one-hand touch using the touch function of the display screen in the near touch area, and performs touch control in the far touch area by using the second position obtained by the positioning.
The control method according to claim 4, wherein the near touch area and the far touch area are automatically delimited according to the position of the manipulation object on the display screen.
The manipulation method according to claim 1, wherein the step of acquiring the depth image comprises:

S11: acquiring a first depth image that includes a control surface and does not include a manipulation object;

S12: acquiring a second depth image including a control surface and a manipulation object;

S13: Obtain a third depth image of the manipulation object by using the second depth image and the first depth image.
The manipulation method according to claim 6, wherein the acquiring the first location comprises the following steps:

S21: determining, according to the second depth image, whether the manipulation object is in contact with the control surface, and if contact, Then proceed to the next step;

S22: Obtain spatial position information of the manipulation object in a coordinate system of the display screen according to the third depth image, and take the coordinates of the control object vertex as the first position.
The manipulation method according to claim 1, wherein the obtaining of the second position comprises the following steps:

S31: Establish a mapping relationship between the control plane and the display screen.

S32: Obtain a second position of the display screen by the mapping relationship and the first location.
The manipulation method according to claim 8, wherein a linear mapping relationship is established according to the horizontal and vertical dimensions of the control surface and the display screen.
A one-handed control system, comprising: the touch method according to any one of claims 1-9, comprising an image acquisition unit, a processor, a control surface and a display screen, the control surface and the display screen On different planes;

The image acquisition unit is configured to acquire a control surface and a depth image of the manipulation object and depth information of the manipulation object;

The processor includes an identification module, a positioning module, a conversion module, and an execution module, and the identification module is configured to acquire a first position of the manipulation object on the manipulation surface according to the depth image, and identify a predetermined manipulation action of the manipulation object The positioning module is configured to locate a second position on the display screen that needs to be manipulated according to the first position; the conversion module converts and generates a corresponding touch instruction according to a predefined control action; the execution module is used to Performing a touch command at the second position completes a touch operation on the display screen.