CN112363629B - A new non-contact human-computer interaction method and system - Google Patents

Abstract

The present invention is applicable to the technical field of human-computer interaction, and provides a new non-contact human-computer interaction method and system. The method includes the steps of: S100. In the two-dimensional plane image of the depth camera, automatically detect the Three vertices A, B, C and target point F; S200, combine the internal parameters of the depth camera, convert the plane pixel coordinates of the vertices A, B, C and the target point F into three-dimensional coordinates in the depth camera coordinate system; S300, calculate the target The three-dimensional coordinates of the projection point F' of the point F on the display screen in the depth camera coordinate system; S400, calculating the plane pixel coordinates of the projection point F' on the display screen; S500, recognizing the action of the target point F, and calling the relevant system mouse The interface triggers mouse events. The invention solves the problem of using detectable target objects such as fingertips to control the screen content in a mouseless environment, does not need any calibration, and has the advantages of small calculation amount and simple hardware equipment.

Description

A new non-contact human-computer interaction method and system

technical field

The invention belongs to the technical field of human-computer interaction, and in particular relates to a new non-contact human-computer interaction method and system.

Background technique

At present, non-contact human-computer interaction technology has a wide range of applications in interactive games, interactive museums, interactive tourist museums, and VR/AR (virtual reality/augmented reality). Various non-contact gestures can be used to interact with electronic products. For example, gestures can be used to adjust the volume. This human-computer interaction technology without mouse operation has great market value and economic value.

In the patent application document with the publication number CN102841679A, the invention title is a non-contact human-computer interaction method and device, a non-contact human-computer interaction method and device are disclosed, mainly using the camera calibration principle, which requires To obtain the position information and direction information of the camera to calculate the calibration results, it is also necessary to introduce a gravity sensing module and a sliding resistance measurement module, and the hardware design is relatively complicated.

With the emergence of depth cameras and the development of deep learning technology in the field of computer vision, there are more and more human-computer interaction systems based on depth cameras. The difference between a depth camera and an ordinary camera is that, in addition to obtaining plane image information, it can also obtain depth information of the shooting object, that is, three-dimensional position and size information, so that the entire computing system can obtain three-dimensional data of the environment and objects. The depth camera has the ability of three-dimensional sensing and recognition. After further processing, it can also complete applications such as three-dimensional modeling.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a new non-contact human-computer interaction method, which aims to solve the problems in the prior art that the non-contact human-computer interaction using the camera calibration principle is complicated in calculation process and hardware.

The embodiments of the present invention are implemented in this way, and provide a new non-contact human-computer interaction method, which includes the following steps:

S100, in the two-dimensional plane image of the depth camera, use the deep learning system to automatically detect the three vertices A, B, C and the target point F of the display screen, and obtain the plane pixel coordinates of each point;

S200, combining the internal parameters of the depth camera, convert the plane pixel coordinates of the vertices A, B, C and the target point F into three-dimensional coordinates in the depth camera coordinate system;

S300, calculate the three-dimensional coordinates of the projection point F' of the target point F on the display screen under the depth camera coordinate system;

S400, calculate the plane pixel coordinates of the projection point F' on the display screen;

S500. Identify the action of the target point F, and call the relevant system mouse interface to trigger mouse events.

Further, the step S100 includes the following sub-steps:

S110, using the target detection algorithm based on deep learning to automatically detect the vertices A, B, and C of the display screen, and obtain the plane pixel coordinates of each point;

S120. Establish a two-stage target detection deep neural network structure to automatically detect the target point F, including:

S121, establishing a target object detection deep neural network for detecting the target object in the two-dimensional plane image, expanding the detected target object area, and locating the target object according to the expanded area;

S122 , establishing a target point detection deep neural network for detecting the target point of the target object, locating the target point, and obtaining the plane pixel coordinates of the target point F.

Further, the step S120 further includes: establishing a neural network with a dual-channel attention mechanism to improve the positioning accuracy of the detected target point.

Further, the step S300 includes:

According to the vertices A, B, C, calculate the normal vector n of the screen plane passing through the vertex A, and calculate the three-dimensional coordinates of the projection point F' according to the line segment FF' parallel to the normal vector n and the line segment AF' perpendicular to the normal vector n.

Further, the step S400 includes:

Calculate the ratio of the abscissa u to the ordinate v in the plane pixel coordinates of the projection point F', and then combine with the screen resolution of the display screen to obtain the plane pixel coordinates of the projection point F'.

Further, the step S400 includes:

Calculate the distance D _FAB from the projection point F' to the line segment AB and the distance D _FAC from the projection point F' to the line segment AC respectively, and obtain the ratio of the abscissa u to the ordinate v in the plane pixel coordinates of the projection point F', and then combine with the display screen According to u=(D _FAB /AC)*screen horizontal pixel, v=(D _FAC /AB)*screen vertical pixel, the plane pixel coordinate F'(u,v) of the projection point F' is obtained.

Another object of the embodiments of the present invention is to provide a new non-contact human-computer interaction system, including a computer, a depth camera, and a display screen. The computer is respectively connected to the depth camera and the display screen. The system adopts the above-mentioned new contactless human-computer interaction method.

Another object of the embodiments of the present invention is to provide a computer-readable storage medium, which stores a program for electronic data exchange, and the program is used to execute the above-mentioned new contactless human-computer interaction method.

Compared with the prior art, the beneficial effect of a new non-contact human-computer interaction method and system provided by the present invention is that it solves the problem of using detectable target objects such as fingertips to control screen content in a mouse-free environment. This new non-contact human-computer interaction method does not require any calibration, and has the advantages of small computational complexity and simple hardware equipment. Based on deep learning, the invention obtains two-dimensional plane information and three-dimensional depth information through a depth camera, calculates the projected coordinates of the target object on the display screen, triggers mouse events, and realizes non-contact human-computer interaction.

Description of drawings

In order to explain the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings required in the technical description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a flowchart of a new non-contact human-computer interaction method provided by an embodiment of the present invention.

FIG. 2 is a two-dimensional plane image captured by a depth camera in an embodiment of the present invention.

FIG. 3 is a schematic diagram of a two-stage target detection deep neural network structure in an embodiment of the present invention.

FIG. 4 is a schematic diagram of the positional relationship between a depth camera, a fingertip, and a display screen in an embodiment of the present invention.

Fig. 5 is a flow chart of calculating the three-dimensional coordinates of the projection point F' in the depth camera coordinate system in the embodiment of the present invention.

6 is a schematic diagram of the relationship between the current position of the fingertip and the original position when a mouse double-click event is triggered in an embodiment of the present invention.

FIG. 7 is a flowchart of triggering a mouse double-click event in an embodiment of the present invention.

FIG. 8 is a schematic structural diagram of a new non-contact human-computer interaction system provided by an embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Referring to FIG. 1, FIG. 1 is a flowchart of a new non-contact human-computer interaction method provided by an embodiment of the present invention, and the method includes the following steps:

S100, in the two-dimensional plane image of the depth camera, use the deep learning system to automatically detect the three vertices A, B, C and the target point F of the display screen, and obtain the plane pixel coordinates A(ua,va) of each point, B( ub, vb), C (uc, vc), F (u0, v0);

S200. Combine the internal parameters of the depth camera, convert the plane pixel coordinates of each point into three-dimensional coordinates in the depth camera coordinate system, and obtain the three-dimensional coordinates A(x ₁ , y ₁ , z ₁ ), B(x ₂ , y ₂ of each point) , z ₂ ), C(x ₃ , y ₃ , z ₃ ), F(x ₀ , y ₀ , z ₀ );

S300, calculate the three-dimensional coordinates F' (x', y', z') of the projection point F' of the target point F on the display screen in the depth camera coordinate system;

S400. Calculate the plane pixel coordinates F'(u, v) of the projection point F' on the display screen, including:

Calculate the distance D _FAB from the projection point F' to the line segment AB and the distance D _FAC from the projection point F' to the line segment AC respectively, and obtain the ratio of the abscissa u to the ordinate v in the plane pixel coordinates of the projection point F', and then combine with the display screen The screen resolution (screen horizontal pixels * screen vertical pixels), according to u = (D _FAB /AC) * screen horizontal pixels, v = (D _FAC /AB) * screen vertical pixels, get the plane pixel coordinates of the projection point F'F'(u,v);

S500. Identify the action of the target point F, display a mouse mark at the position of the projection point F', call the relevant system mouse interface to trigger a mouse event, and manipulate the display screen.

FIG. 2 is a two-dimensional plane image captured by a depth camera in an embodiment of the present invention, and the image shows a rectangular display screen and a hand with a gesture of extending an index finger. In the embodiment of the present invention, the fingertip is used as the target point for description. In other embodiments, other detectable target objects may also be selected as the target point.

Referring to FIG. 2, the above step S100 includes the following sub-steps:

S110. Use a deep learning-based target detection algorithm (such as the YOLO algorithm or the SSD algorithm) to automatically detect the three vertices A, B, and C of the display screen, and select the three vertices A, B, and C of the display screen to form a screen plane, And take vertex A as the origin of the pixel coordinate system of the screen plane, and obtain the plane pixel coordinates A(ua,va), B(ub,vb), C(uc,vc) of each point;

S120, referring to FIG. 3, establish a two-stage target detection deep neural network structure to automatically detect the fingertip F, including:

S121, establishing a finger detection deep neural network for detecting fingers in a two-dimensional plane image, expanding the detected finger region, and locating the finger according to the expanded region;

S122 , establishing a deep neural network for fingertip detection, which is used to detect the fingertip of the finger, locate the fingertip, and obtain the plane pixel coordinates F(u0, v0) of the fingertip F of the finger.

In order to improve the positioning accuracy of detecting fingertips, the embodiment of the present invention also establishes a neural network with a dual-channel attention mechanism. The attention mechanism is a model that simulates the attention of the human brain. It uses limited attention to quickly screen out important information, thereby improving the efficiency and accuracy of the human brain in processing visual information.

Referring to FIG. 4 , FIG. 4 is a schematic diagram of the positions of the depth camera, the fingertip, and the display screen in the embodiment of the present invention. The figure shows the projection point F' of the fingertip F on the display screen.

Referring to FIG. 5, the above step S300 includes the following sub-steps:

S310, according to the vertices A, B, C, calculate the normal vector n=(a, b, c) of the screen plane passing through the vertex A, wherein:

a=y ₁ (z ₂ -z ₃ )+y ₂ (z ₃ -z ₁ )+y ₃ (z ₁ -z ₂ ),

b=z ₁ (x ₂ -x ₃ )+z ₂ (x ₃ -x ₁ )+z ₃ (x ₁ -x ₂ ),

c=x ₁ (y ₂ -y ₃ )+x ₂ (y ₃ -y ₁ )+x ₃ (y ₁ -y ₂ );

S320. Let the three-dimensional coordinates of the projection point F' in the depth camera coordinate system be F'(x', y', z'), and according to the line segment FF' being parallel to the normal vector n, a system of equations is obtained:

S330. According to the line segment AF' being perpendicular to the normal vector n, the equation is obtained:

a(x'-x ₁ )+b(y'-y ₁ )+c(z'-z ₁ )=0(**);

S340. Simultaneously combine the equation system (*) and the equation (**) to obtain t, and substitute t into the equation system (*) to obtain the three-dimensional coordinates F'(x', y', z') of the projection point F'.

Specifically, the action of identifying the target point F in the above step S500 includes:

When the other vertices A, B, and C are all fixed, the user moves the finger, and uses the above steps S100 to S200 to recalculate the three-dimensional coordinates of the fingertip F of the finger. The actions of the target point F include clicking, double-clicking, keeping the pressed state, and releasing after being pressed. The event triggered when the mouse button is released (mousedown), and the event triggered when the mouse button is released (mouseup).

The following is an example of triggering a mouse double-click event.

6 is a schematic diagram of the relationship between the current position of the fingertip and the original position of the fingertip when the mouse double-click event is triggered in the embodiment of the present invention, in the figure D1 and D2 are the distances from the current position of the fingertip to the original position, D1> D2. Referring to FIG. 6 , in the embodiment of the present invention, a double-click action is defined as: the fingertip leaves the original position and then returns to the vicinity of the original position. The definition of the fingertip leaving the original position is: the difference between the position of the fingertip and the original position. When the distance is greater than D1, the definition of returning the fingertip to the vicinity of the original position is that the distance between the position of the fingertip and the original position is less than D2.

FIG. 7 is a flowchart of triggering a mouse double-click event in an embodiment of the present invention. Referring to FIG. 7 , since a mouse double-click event generally ends within 2 seconds, a click action occurs within a 50-frame image sequence. In the embodiment of the present invention, the coordinates of the fingertip are detected by every 5 frames of images, and the mark FAR=TRUE is set to indicate that the distance between the fingertip coordinates of the tnth frame image and the original position is greater than D1, and the mark NEAR=TRUE is set to be the tmth frame ( 0<tn<tm<50) The distance between the fingertip coordinates of the image and the original position is less than D2; when FAR=TRUE and NEAR=TRUE, the mouse double-click event is triggered to control the display screen. The specific process is as follows:

First, read the image, and judge whether there is no NEAR=TRUE after 50 frames of images from FAR=TRUE, if so, set FAR=FALSE and NEAR=FALSE, that is, if the fingertip of the tnth frame image leaves the original position, Until the fingertip of the (tn+50)th frame image does not return to the original position, the fingertip of the tnth frame image is returned to the original position; if not, the coordinates of the fingertip are detected every 5 frames of images, When the detected coordinates of the fingertip satisfy FAR=TRUE, continue to detect until the coordinates of the fingertip also satisfy NEAR=TRUE, trigger the mouse double-click event, and set FAR=FALSE and NEAR=FALSE, and repeat the above steps.

Let the three-dimensional coordinates of the fingertip of the original position be (x ₀ , y ₀ , z ₀ ), the coordinates of the fingertip of the tn-th frame image are (x _tn , y _tn , z _tn ), and the finger-tip coordinates of the tm-th frame image is (x _tm , y _tm , z _tm ), then when both:

(x _tn -x ₀ ) ² +(y _tn -y ₀ ) ² +(z _tn -z ₀ ) ² >D1 ² and

When (x _tm -x ₀ ) ² +(y _tm -y ₀ ) ² +(z _tm -z ₀ ) ² < D2 ² , a mouse double-click event is triggered.

The embodiment of the present invention provides a non-contact human-computer interaction method, which is used to solve the problem of manipulating screen content with detectable target objects such as fingertips in a mouse-free environment. This new non-contact human-computer interaction method does not require any calibration, but is based on deep learning, obtains two-dimensional plane information and three-dimensional depth information through a depth camera, calculates the projected coordinates of the target object on the display screen, and triggers mouse events. , to achieve non-contact human-computer interaction. It has the advantages of small calculation amount and simple hardware device.

Referring to FIG. 8 , an embodiment of the present invention further provides a new non-contact human-computer interaction system, including: a computer, a depth camera, and a display screen, and the computer is respectively connected to the depth camera and the display screen. The system adopts the method of the above embodiment to perform human-computer interaction. The depth camera collects the position information of the vertex of the display screen and the position information of the target point. The computer is used to calculate the position information of the projection of the target point on the display screen according to the position information collected by the depth camera, and call the relevant system mouse interface for mouse events. Trigger to control the display screen.

The embodiment of the present invention also provides a computer-readable storage medium, which stores a program for electronic data exchange, and the program is used to execute the novel non-contact human-computer interaction method of the present invention.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the principles of the present invention shall be included in the protection scope of the present invention. Inside.

Claims (8)

1. A novel non-contact man-machine interaction method comprises the following steps:

s100, automatically detecting three vertexes A, B, C and a target point F of a display screen by using a deep learning system in a two-dimensional plane image of a depth camera to obtain plane pixel coordinates of each point;

s200, combining the internal reference of the depth camera, and converting the plane pixel coordinates of the top point A, B, C and the target point F into three-dimensional coordinates under a depth camera coordinate system;

s300, calculating a three-dimensional coordinate of a projection point F' of the target point F on the display screen in a depth camera coordinate system;

s400, calculating the plane pixel coordinate of the projection point F' on the display screen;

and S500, identifying the action of the target point F, and calling a related system mouse interface to trigger a mouse event.

2. A new contactless human-computer interaction method according to claim 1, characterized in that said step S100 comprises the following sub-steps:

s110, automatically detecting a vertex A, B, C of the display screen by using a target detection algorithm based on deep learning to obtain plane pixel coordinates of each point;

s120, establishing a two-stage target detection deep neural network structure, and automatically detecting a target point F, wherein the two-stage target detection deep neural network structure comprises the following steps:

s121, establishing a target object detection depth neural network for detecting a target object in the two-dimensional plane image, expanding the detected target object area and positioning the target object according to the expanded area;

and S122, establishing a target point detection depth neural network for detecting a target point of the target object and positioning the target point to obtain a plane pixel coordinate of the target point F.

3. The new contactless human-computer interaction method according to claim 2, wherein the step S120 further comprises: and establishing a neural network with a dual-channel attention mechanism for improving the positioning precision of the detected target point.

4. The new contactless human-computer interaction method according to claim 1, wherein the step S300 includes:

from the vertex A, B, C, a normal vector n of the screen plane passing through the vertex a is calculated, and from the line segment FF ' parallel to the normal vector n and the line segment AF ' perpendicular to the normal vector n, the three-dimensional coordinates of the projected point F ' are calculated.

5. The new contactless human-computer interaction method according to claim 1, wherein the step S400 comprises:

and calculating the ratio of the abscissa u to the ordinate v in the plane pixel coordinate of the projection point F ', and then combining the screen resolution of the display screen to obtain the plane pixel coordinate of the projection point F'.

6. The new contactless human-computer interaction method according to claim 1, wherein the step S400 comprises:

respectively calculating the distance D from the projection point F' to the line segment AB_FABAnd the distance D from the projection point F' to the line segment AC_FACObtaining the ratio of the abscissa u to the ordinate v in the plane pixel coordinate of the projection point F', and then combining the screen resolution of the display screen according to the u ═ D_FABV ═ screen horizontal pixels, (/ AC) ((D))_FAC(AB) screen vertical pixels, resulting in the flatness of the projected point FThe plane pixel coordinates F' (u, v).

7. A novel non-contact human-computer interaction system, which comprises a computer, a depth camera and a display screen, wherein the computer is respectively connected with the depth camera and the display screen, and the system adopts the novel non-contact human-computer interaction method as claimed in any one of claims 1-6 for human-computer interaction.

8. A computer-readable storage medium storing a program for electronic data exchange, the program being for executing the new contactless human-computer interaction method according to any one of claims 1-6.

Images