CN115793923A - Human-computer interface motion track identification method, system, equipment and medium - Google Patents
Human-computer interface motion track identification method, system, equipment and medium Download PDFInfo
- Publication number
- CN115793923A CN115793923A CN202310086079.3A CN202310086079A CN115793923A CN 115793923 A CN115793923 A CN 115793923A CN 202310086079 A CN202310086079 A CN 202310086079A CN 115793923 A CN115793923 A CN 115793923A
- Authority
- CN
- China
- Prior art keywords
- track
- motion
- trail
- human
- matching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses a method, a system, equipment and a medium for identifying a motion trail of a human-computer interface, which relate to the field of identifying the action of input equipment in computer application and solve the problem of low identification rate of the motion trail of the human-computer interface, and the technical scheme has the key points that: capturing a motion track formed by a user on a current page of a human-computer interface; inputting the motion track into a trained track recognition model for track recognition; when the icon of the motion trail is identified, executing an operation instruction corresponding to the icon to enter a next page, and storing the motion trail into a data sample set; or when the icon of the motion track is not identified, inputting the motion track into a track matching model for track matching, and storing the motion track corresponding to successful track matching or failed track matching into a data sample set; wherein, a standard motion track is preset in the track matching model. The invention improves the identification accuracy and the identification probability of the motion trail.
Description
Technical Field
The invention relates to the field of input device action recognition in computer application, in particular to a human-computer interface motion track recognition method, a human-computer interface motion track recognition system, human-computer interface motion track recognition equipment and a human-computer interface motion track recognition medium.
Background
The traditional man-machine interface of a computer is generally divided into different sub-windows, such as a menu, a toolbar and the like, and different function pages can be accessed by clicking the sub-windows. There are some obvious problems with this method of operation, which requires clicking a control such as an exit button when exiting the function sub-window. Menus and toolbars occupy limited display space and the real useful display area becomes smaller. Menus and toolbars have their own organization structure and are usually located at the edge of the screen, and under a graphical man-machine interface, people instinctively slide the input device when operating a software interface, and the sight of an operator is habitually focused around the position of the input device cursor to move. Therefore, the layout and the operation mode are not very convenient and do not accord with the operation instinct of people. Since the operator's point of attention is usually not at the edge of the screen. Therefore, if the menu and the toolbar are to be operated, the input device needs to slide for a long distance to click the corresponding function button in the menu located on the edge of the screen (usually on the top, the left side, the right side and the bottom), which is difficult for a not-familiar operator to accurately position the button, the sight line can be greatly switched along with the position movement of the cursor of the input device, the time is relatively long, and the operation efficiency is not high. Especially when the menu and toolbar organization tree structure is relatively complex, the navigation between pages is relatively difficult, and the corresponding relation between the layout and the functions of the menu items is relatively difficult to remember, so that an operator can form some patterns, symbols and the like by utilizing the sliding motion track of the cursor of the input device, and a computer system can identify the symbols and execute corresponding functions according to the symbols, thereby replacing part of the menu functions, and bringing greater convenience to the operation of a human-computer interface.
At present, in the related art, the motion track of the input device is used to replace operations such as clicking a menu and the like to complete software interface operations such as browsing, canceling, reserving and the like, but the adopted identification module is quite simple in comparison algorithm, for example, extra information of track point time needs to be recorded; in addition, the traditional method cannot greatly tolerate action differences among different operators, for example, differences caused by inconsistent sizes of symbols and patterns drawn by the operators, for example, the movement track has more burrs, the movement track to be closed is not completely closed, and the like, so that the problems of poor movement track identification capability and low identification rate of the input device are caused.
Therefore, how to solve the problems that when different operators operate the input device, the recognition probability rate of the motion trail is low and the recognition accuracy is urgently needed to be solved due to the factors such as the size, straightness and smoothness of the drawn motion trail.
Disclosure of Invention
The invention provides a human-computer interface motion track identification method, a human-computer interface motion track identification system, human-computer interface motion track identification equipment and a human-computer interface motion track identification medium, aiming at solving the problem of low human-computer interface motion track identification rate in the related art.
The technical purpose of the invention is realized by the following technical scheme:
in a first aspect of the present application, a method for identifying a motion trajectory of a human-computer interface is provided, where the method includes:
capturing a motion track formed by a user on a current page of a human-computer interface;
inputting the motion track into a trained track recognition model for track recognition;
when the icon of the motion trail is identified, executing an operation instruction corresponding to the icon to enter a next page, and storing the motion trail into a data sample set; alternatively, the first and second liquid crystal display panels may be,
when the icon of the motion track is not identified, inputting the motion track into a track matching model for track matching, and storing the motion track corresponding to successful track matching or failed track matching into a data sample set; wherein, a standard motion track is preset in the track matching model.
In some possible embodiments, capturing a motion trajectory formed by a user on a current page of a human-computer interface specifically includes:
and carrying out data normalization processing on coordinate data corresponding to the captured input track of the user on the current page of the human-computer interface to obtain a motion track.
In some possible embodiments, the motion trajectory is input into a trained trajectory recognition model for trajectory recognition, specifically:
and identifying the identification probability of the icon corresponding to the motion track, and outputting the icon corresponding to the motion track when the identification probability is greater than a probability threshold, wherein the icon comprises a pattern, a symbol or a character.
In some possible embodiments, the trajectory recognition model is obtained by learning a training data set through a machine learning algorithm, wherein the training data set is used for capturing a historical motion trajectory formed by a user on a human-computer interface.
In some possible embodiments, training time is preset, when the training time is up, a machine learning algorithm is used for learning the motion trail in the data sample set and the historical motion trail in the training data set, and network parameters of the trail identification model are updated to obtain an updated trail identification model;
and identifying the motion track formed by the user on the current page of the human-computer interface captured in real time by using the updated track identification model.
In some possible embodiments, when the motion trajectory is identified, executing an operation instruction corresponding to the motion trajectory to enter a next page, and storing the motion trajectory in the data sample set, specifically:
presetting standard icon information, wherein the standard icon information carries a corresponding operator;
determining icon information of an icon corresponding to the motion track in the standard icon information;
and executing an operation instruction based on the operator corresponding to the icon information to enter a next page, and storing the motion trail into a data sample set.
In some possible embodiments, when the motion trajectory is not identified, the motion trajectory is input into a trajectory matching model for trajectory matching, and the motion trajectories corresponding to successful trajectory matching or failed trajectory matching are stored in a data sample set; the track matching model is preset with a standard motion track, and the track matching model specifically comprises the following steps:
calculating the similarity between the motion track and a preset standard motion track in the track matching model;
when the similarity is larger than or equal to the similarity threshold value, successfully matching the tracks, executing an operation instruction corresponding to the standard motion track to enter a next page, verifying the matched standard motion track based on the retention time of a user on the next page, attaching a first label of the identified icon to the corresponding motion track when the verification is passed, and storing the motion track and the first label in a data sample set; the standard motion track corresponds to corresponding standard icon information, and the standard icon information carries corresponding operational characters;
or when the similarity is smaller than the similarity threshold value and the track matching fails, attaching a second label which cannot be identified to the corresponding motion track, and storing the motion track and the second label in the data sample set.
In a second aspect of the present application, a human-computer interface motion trajectory recognition system is provided, including:
the track capturing module is used for capturing a motion track formed by a user on a current page of the human-computer interface;
the track recognition module is used for inputting the motion track into a trained track recognition model for track recognition;
the track processing module is used for executing an operation instruction corresponding to the icon to enter a next page when the icon of the motion track is identified, and storing the motion track into the data sample set; alternatively, the first and second liquid crystal display panels may be,
the track matching module is used for inputting the motion track into the track matching model for track matching when the icon of the motion track is not identified, and storing the motion track corresponding to successful track matching or failed track matching into the data sample set; wherein, a standard motion track is preset in the track matching model.
In a third aspect of the present application, an electronic device is provided, which includes a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the human-machine interface motion trajectory identification method according to any one of the first aspect of the present application.
In a fourth aspect of the present application, a computer-readable storage medium is provided, where a computer program is stored on the computer-readable storage medium, where the computer program, when executed by a processor, implements the steps of the human-machine interface motion trajectory identification method according to any one of the first aspect of the present application.
Compared with the prior art, the invention has the following beneficial effects:
1. the method firstly captures the motion trail formed by the user on the current page of the human-computer interface, inputs the motion trail into the trained trail recognition model for trail recognition, and improves the tolerance of the difference of the motion trail formed by using the input equipment by different people based on the trail recognition model, thereby improving the recognition accuracy of the icon meaning corresponding to the motion trail formed by using the input equipment on the computer interface. And when the motion trail is not identified by the trail identification model, the motion trail is input into the trail matching model for trail matching, the trail matching is a trail guessing process, and corresponding motion trails are stored into the data sample set based on the matching result, so that the data richness of the motion trail in the data sample set is improved, the tolerance of the difference of the motion trails formed by different operators by using input equipment is further improved, and the identification probability of the motion trail is improved.
2. The invention also dynamically collects the recognizable movement track and the unrecognizable movement track, realizes the continuous training and updating of the track recognition model, and improves the cognition degree of the track which cannot be recognized in the past based on the updated track recognition model, thereby ensuring the recognition accuracy and the recognition probability of the movement track.
3. The invention also considers that for the unidentifiable motion track, the closest standard motion track and the retention time of the next page entered by the user based on the standard motion track are determined according to the matching degree of the motion track and the standard motion track, whether the successfully matched standard motion track is correct is determined, the successfully matched motion track and the unmatched motion track are labeled, and then the labels and the motion tracks are stored in a data sample set to help a machine learning algorithm to improve the difference tolerance of different operators.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
fig. 1 is a schematic flowchart of a human-machine interface motion trajectory identification method according to an embodiment of the present disclosure;
FIG. 2 is a schematic flow chart illustrating a combination of trajectory recognition and trajectory recognition models provided by embodiments of the present application;
fig. 3 is a block diagram of a structure of a human-machine interface motion trajectory recognition system according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to examples and accompanying drawings, and the exemplary embodiments and descriptions thereof are only used for explaining the present invention and are not meant to limit the present invention.
It is to be understood that the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
The operator can use the movement track of the input device cursor to form some patterns, symbols and the like, and the computer system can recognize the symbols and execute corresponding functions according to the symbols, so that the operation of the human-computer interface is greatly facilitated if partial menu functions are replaced.
In the related art, the motion track of the input device is used for replacing operations such as clicking a menu and the like to complete software interface operations such as browsing, canceling, reserving and the like, but the adopted identification module comparison algorithm is quite simple, for example, extra information of track point time needs to be recorded; in addition, the traditional method cannot greatly tolerate action differences among different operators, for example, differences caused by inconsistent sizes of symbols and patterns drawn by the operators, for example, the movement track has more burrs, the movement track to be closed is not completely closed, and the like, so that the problems of poor movement track identification capability and low identification rate of the input device are caused.
Based on the problems in the related art, the embodiment of the application provides a method, a system, equipment and a medium for identifying a motion trajectory of a human-computer interface, so that the difference of the motion trajectories caused by different operation habits of operators is overcome, and the identification accuracy and the identification probability of the motion trajectory are improved.
In the embodiment of the present application, the method for identifying a motion trajectory of a human-computer interface is applicable to electronic devices such as a computer and a computer, such as a terminal device or a server, wherein an operating system of the electronic device may include, but is not limited to, an Android operating system, an IOS operating system, a Synbian operating system, a Black Berry operating system, a Windows Phone8 operating system, and the like. In this embodiment, the electronic device may be provided with a User Interface (UI), an Interface module, and a Central Processing Unit (CPU).
First, the recognition method of the present embodiment can be applied to a default human-machine interface under any function page, and the interface may not have a special menu bar or tool bar. The operator performs a definite starting action on the input device, and it is understood that the input device includes hardware devices such as a mouse and a stylus pen, for example, a left button of the mouse is pressed and kept in a pressed state, and when the mouse is moved, a trace formed by a trace of the mouse and a trace formed by the movement of the mouse are displayed on the interface, and the traces may form a figure, such as a rectangle, a circle, a zigzag, a hook, or a character. Corresponding patterns, symbols and the like can be drawn on the touch screen in a touch-based mode.
Referring to fig. 1, fig. 1 is a schematic flow chart of a human-computer interface motion trajectory identification method provided in an embodiment of the present application, and as shown in fig. 1, the method includes the following steps:
s110, capturing a motion track formed by a user on the current page of the human-computer interface.
In this embodiment, an operator uses an input method such as a mouse, a finger, or a sensing pen to stroke and form a certain track on a display screen of a computer (including any similar devices, such as a mobile phone, a tablet, a notebook, and the like). The operator draws a movement track to form a certain figure and a symbol with preset meanings. Before forming the motion trail, the operator needs to make a special starting action, such as pressing a certain key and keeping the pressed state when using a mouse, and when using a touch screen, quickly clicking twice on the screen by a finger or an induction pen continuously, and the like; for an operator using a mouse, the track ending action may be releasing the mouse; it may be that for a user using a touch screen, the finger and the sensing pen are off the screen for a period of time.
And S120, inputting the motion track into a trained track recognition model for track recognition.
In this embodiment, the trained trajectory recognition model may recognize the estimated specific operation meaning, and may recognize the input trajectory of the operator. As will be understood by those skilled in the art, the trajectory recognition model is fully trained by using different input data of different types of operators when the trajectory recognition model is learned in advance, so that the trained trajectory recognition model has great tolerance to input differences, can tolerate considerable input differences and maintain high accuracy, such as the size of an input trajectory, whether the input trajectory is closed, whether the proportion is harmonious, whether lines are smooth and the like.
And S130, when the icon of the motion trail is identified, executing an operation instruction corresponding to the icon to enter a next page, and storing the motion trail in a data sample set.
In this embodiment, it should be understood by those skilled in the art that it is common knowledge of those skilled in the art to preset a corresponding icon to execute a corresponding operation instruction, for example, in the prior art, a technique applied to a three-finger downslide screenshot of an android mobile phone is equivalent to capturing three downward lines generated on a touch screen of the mobile phone by a user, and for example, when the mobile phone is turned off, the screen may be woken up by drawing an "O" on the screen.
Therefore, a left mouse button (or any mouse button) is pressed down on any page of the human-computer interface, a rectangle is drawn by using a mouse track, and an operation instruction corresponding to the rectangle is preset, so that the track recognition model automatically recognizes the rectangle drawn by an operator, and the user wants to use a menu, namely, a menu and an operation item related to the current page appear at a position near a mouse cursor in a screen, and thus, the menu does not need to be positioned at the edge so as to facilitate the operation of the operator; moreover, the cursor is generally in the sight focus area of the operator, so that the operator can complete click operation in the presented menu bar without moving the sight to enter the functional page which the operator wants to use.
Likewise, the menu bar may disappear when the operator enters a new function page, without affecting the operator's line of sight. A similar approach may be used if the operator has triggered a menu with a cursor track on the interface, but wants to cancel the menu. Such as: and pressing a left mouse button outside the menu area to draw a motion track by using a cursor, then drawing the motion track into the menu range without loosening the left button, and then performing continuous smearing actions so as to identify a corresponding smeared icon and close the menu.
Similarly, the operator may wish to operate some functions on the new page, and may press the mouse button again to draw a rectangle to enter the menu bar of the page. When the operator wants to exit the new page, the operator can select to press the mouse to draw an action symbol with the cursor track to end the page browsing without making a click operation by making the cursor enter the scope of the toolbar, for example, pressing the left button of the mouse to draw a zigzag with the cursor to close the page.
If the operator wants to keep the page for browsing back and forth, the operator can choose to draw a circle to save the page. The operator can use mouse cursor to draw left triangle and right triangle to turn page back and forth on any page, and make the man-machine interface enter into corresponding function interface, or complete some function of the page.
It should be understood that, the above embodiments only list some combinations of simple icons and operation commands, and may also preset operation instructions corresponding to other icons, where this embodiment is not limited, and then identify a motion trajectory input by an operator based on the trajectory identification model, so as to execute a corresponding instruction or enter into the trajectory matching model based on the identified result.
Or S140, when the icon of the motion track is not identified, inputting the motion track into the track matching model for track matching, and storing the motion track corresponding to successful track matching or failed track matching into the data sample set; wherein, a standard motion track is preset in the track matching model.
In this embodiment, if the trajectory recognition model cannot recognize the trajectory, it means that the style and motion training of the motion trajectory are insufficient, in this case, inputting the motion trajectory into the trajectory matching model may perform a guess motion once, and because the standard motion trajectory is preset in the trajectory matching model, a guess result may be output in the trajectory matching model based on the similarity between the captured motion trajectory and the standard motion trajectory, for example, based on whether the calculated similarity satisfies a preset similarity threshold, where it is understood that the trajectory recognition model cannot recognize the motion trajectory, so that the factors describing the size, the closure degree, the coordination degree, the line smoothness degree of the motion trajectory are extremely irregular, and thus, the setting of the similarity threshold may be relatively low, so as to expand the sample types in the data sample set, thereby further improving the richness of the training data set of the trajectory recognition model, and improving the tolerance of the difference trajectories of motion trajectories formed by different operators using input devices.
And executing corresponding operation instructions based on the matched category, and labeling the motion trail data which cannot be matched according to the execution result of the function. For example, when jumping to the next page of guess, the operator immediately cancels the operation, at this time, it indicates that the guessed standard motion trajectory is wrong with a high probability, at this time, the data may be labeled with an unidentifiable label, and the motion trajectory data and the label are also stored in the data sample set. And if the operator continues browsing after jumping to the corresponding next page, wherein the guess is correct approximately, the system gives a guess label corresponding to the unidentified motion trajectory data, and the motion trajectory data and the corresponding label are stored in the data sample set in a similar way.
In summary, according to the method for identifying the motion trail of the human-computer interface provided by this embodiment, the motion trail formed by the user on the current page of the human-computer interface is captured first, the motion trail is input into the trained trail identification model for carrying out trail identification, and tolerance of differences of the motion trails formed by different people using the input device is improved based on the trail identification model, so that accuracy of identifying the icon meaning corresponding to the motion trail formed by the input device on the computer interface is improved. And when the motion trail is not identified by the trail identification model, the motion trail is input into the trail matching model for trail matching, the trail matching is a trail guessing process, and corresponding motion trails are stored into the data sample set based on the matching result, so that the richness of the motion trail in the data sample set is improved, the tolerance of the difference of the motion trails formed by different operators by using input equipment is further improved, and the identification probability of the motion trail is improved.
In one embodiment, capturing a motion trajectory formed by a user on a current page of a human-computer interface specifically includes:
and carrying out data normalization processing on coordinate data corresponding to the captured input track of the user on the current page of the human-computer interface to obtain a motion track. Specifically, as a person skilled in the art should understand, performing data normalization processing according to coordinate data corresponding to an input trajectory of an input device on a current page of a human-computer interface to obtain a motion trajectory is a conventional technical means of the person skilled in the art, and therefore details on how to obtain the motion trajectory are not described here.
In one embodiment, the motion trajectory is input into a trained trajectory recognition model for trajectory recognition, specifically:
and identifying the identification probability of the icon corresponding to the motion track, and outputting the icon corresponding to the motion track when the identification probability is greater than a probability threshold, wherein the icon comprises a pattern, a symbol or a character.
In this embodiment, the track recognition model is an artificial intelligence recognition model, and the artificial intelligence models are all classified according to recognition probability, for example, the probability of recognizing an icon corresponding to a motion track by the track recognition model is 85%, and assuming that the probability threshold is set to 75%, a corresponding function or operator may be determined based on the icon, as a person skilled in the art should understand, the function corresponding to an electronic device such as a mobile phone, a tablet, and a computer is executed based on a shortcut, and in consideration of input of a user, the icon may be a pattern, a symbol or a character, and may be a triangle, a rectangle, and the like with respect to the pattern, the symbol may be a "hook", "fork", and the like, the character may be an english character, an arabic character, and the like, the english character may be an "s", "z", and the like, the arabic character may be a "1", "2", and the strokes of the icons are simple, and are convenient for quick input by the user. It is to be understood that the icon may be a regular and symmetrical icon, or may be an irregular and asymmetrical icon, and the embodiment is not limited thereto.
In one embodiment, the trajectory recognition model is obtained by learning a training data set through a machine learning algorithm, wherein the training data set is used for capturing a historical motion trajectory formed by a user on a human-computer interface.
Specifically, as those skilled in the art will understand, the trajectory recognition model is obtained by learning a training data set through a machine learning algorithm, and the training data set is a conventional technical means of those skilled in the art for capturing a historical motion trajectory formed by a user on a human-computer interface, so details on how to obtain a trained trajectory recognition model are not described here.
Referring to fig. 2, fig. 2 is a schematic diagram of a flow of combining a trajectory recognition model and a trajectory recognition model provided in an embodiment of the present application, and as shown in fig. 2, a training time is preset, when the training time is reached, a machine learning algorithm is used to learn a motion trajectory in a data sample set and a historical motion trajectory in a training data set, and a network parameter of the trajectory recognition model is updated to obtain an updated trajectory recognition model;
and identifying the motion track formed by the user on the current page of the human-computer interface captured in real time by using the updated track identification model.
In this embodiment, the machine learning algorithm is a unit working in the background or another electronic device, and the algorithm extracts newly generated motion trajectory data and corresponding labels from a data sample set when a training time arrives, and trains and learns the trajectory recognition model, so that input habits and styles of different operators can be continuously learned by using the newly generated data, and the mouse trajectory recognition model to be deployed is continuously optimized, so that mouse trajectories capable of being recognized and mouse trajectories incapable of being recognized are dynamically collected for continuous training of the machine learning training model. The optimized mouse track recognition model can be regularly updated to the original track recognition model according to a strategy. Therefore, as shown in fig. 2, in a preset training time, learning training is performed on the recognizable motion trajectory and the unrecognizable motion trajectory in the data sample set.
The updated track recognition model is used for recognizing and processing the motion track formed by the user on the current page of the human-computer interface in real time, so that the tolerance of habits and track differences of different operators is further improved, and the recognition accuracy of the icon meaning formed by the motion track drawn by the input equipment on the human-computer interface of the computer is improved.
In one embodiment, when the motion trajectory is identified, executing an operation instruction corresponding to the motion trajectory to enter a next page, and storing the motion trajectory in a data sample set, specifically:
presetting standard icon information, wherein the standard icon information carries a corresponding operator;
determining icon information of an icon corresponding to the motion trail in the standard icon information;
and executing an operation instruction based on the operator corresponding to the icon information to enter the next page, and storing the motion trail in a data sample set.
As will be understood by those skilled in the art, the operation instruction of the icon identified by the trajectory identification model can be executed only when the computer presets the corresponding standard icon information and the corresponding operator in advance, and the user cannot identify the icon if drawing a motion trajectory which does not correspond to the standard icon information at all on the human-computer interface, so in the present embodiment, the standard icon information is preset, wherein the standard icon information carries the corresponding operator; determining icon information of an icon corresponding to the motion trail in the standard icon information; and executing an operation instruction based on the operator corresponding to the icon information to enter a next page, and storing the motion trail into a data sample set.
In one embodiment, when the motion track is not identified, the motion track is input into a track matching model for track matching, and the motion tracks corresponding to successful track matching or failed track matching are stored in a data sample set; wherein, the standard motion track has been preset in the track matching model, specifically includes:
calculating the similarity between the motion track and a preset standard motion track in the track matching model;
when the similarity is larger than or equal to the similarity threshold value, successfully matching the track, executing an operation instruction corresponding to the standard motion track to enter a next page, verifying the matched standard motion track based on the retention time of a user on the next page, attaching a first label of the identified icon to the corresponding motion track when the verification is passed, and storing the motion track and the first label in a data sample set; the standard movement track corresponds to corresponding standard icon information, and the standard icon information carries a corresponding operator.
As will be appreciated by those skilled in the art, there are generally two ways to train the model, one being label-free training and the other being label training.
For the labeled training, it is to be understood that, except for the similarity between the calculated motion trajectory and the standard motion trajectory, the matching degree between the motion trajectory and the standard motion trajectory may also be calculated, in this embodiment, the similarity is taken as an example, when the similarity is greater than or equal to the similarity threshold, a motion trajectory that cannot be identified in the trajectory identification model may be designated, a standard motion trajectory similar to the motion trajectory is matched in the trajectory matching model, and since the matching result may be a result of considering a guess, it is necessary to verify whether the guess is correct, so that the next page is entered based on the operation instruction corresponding to the user executing the standard motion trajectory, and the matched standard motion trajectory is verified based on the staying time of the user on the next page, for example, the staying time of the user on the next page meets the requirement, and it may be considered that the guess result is correct with a high probability.
Of course, it is also wrong with a small probability based on the staying time of the user on the next page, for example, the user stays on the next page due to other factors, so that the staying time meets the requirement, but the verification is still considered to be passed, the first label of the identified icon is attached to the corresponding motion track, and the motion track and the first label are both stored in the data sample set, which may cause that the data is learned during the dynamic training of the subsequent track identification model, resulting in the reduction of the identification accuracy of the final track identification model.
Therefore, further, the browsing trace of the user on the next page of the human-computer interface is captured, when the browsing trace exists and the retention time reaches a preset retention time threshold value, a first label of the identified icon is attached to the corresponding motion track, and the motion track and the first label are stored in the data sample set. Therefore, influence factors in the actual process can be considered, the accuracy of the concentrated motion trail of the data sample is ensured, and the training effect of the trajectory recognition model in the training process is improved.
As another embodiment, for label-free training, when the similarity is smaller than the similarity threshold and the track matching fails, attaching a second label that cannot be identified to the corresponding motion track, and storing both the motion track and the second label in the data sample set.
According to the situation that the standard motion track is not matched in the track matching model, the fact that the icon drawn on the human-computer interface by the user through the input device is completely different from the icon which is determined in the computer in advance can be shown, at the moment, a second label which cannot be identified is attached to the corresponding motion track, the motion track and the second label are stored in the data sample set, when the track identification model is optimized through track data in the data sample set in the follow-up process, the machine learning algorithm can learn the completely unidentifiable motion track, in the follow-up identification process, initial identification can be omitted for the motion track, the motion track can be directly sent into the track matching model, the operation efficiency of the motion track identification method can be improved, the overlong response time after the user inputs the track is avoided, and the experience feeling of the user is reduced.
Referring to fig. 3, fig. 3 is a block diagram of a structure of a human-machine interface motion trajectory recognition system according to an embodiment of the present disclosure, and as shown in fig. 3, the system includes:
the track capturing module 310 is configured to capture a motion track formed by a user on a current page of the human-computer interface;
the track recognition module 320 is used for inputting the motion track into a trained track recognition model for track recognition;
the track processing module 330 is configured to, when an icon of the motion track is identified, execute an operation instruction corresponding to the icon to enter a next page, and store the motion track in the data sample set; alternatively, the first and second electrodes may be,
the trajectory matching module 340 is configured to, when the icon of the motion trajectory is not identified, input the motion trajectory into the trajectory matching model for trajectory matching, and store the motion trajectories corresponding to successful trajectory matching or failed trajectory matching into the data sample set; wherein, a standard motion track is preset in the track matching model.
Therefore, the human-computer interface motion trajectory recognition system provided by the embodiment has the following beneficial effects: the method comprises the steps of capturing a motion track formed by a user on a current page of a human-computer interface, inputting the motion track into a trained track recognition model for track recognition, and improving tolerance of difference of the motion track formed by using input equipment by different people based on the track recognition model, so that the recognition accuracy of icon meanings corresponding to the motion track formed by using the input equipment on the computer interface is improved. And when the motion trail is not identified by the trail identification model, the motion trail is input into the trail matching model for trail matching, the trail matching is a trail guessing process, and corresponding motion trails are stored into the data sample set based on the matching result, so that the data richness of the motion trail in the data sample set is improved, the tolerance of the difference of the motion trails formed by different operators by using input equipment is further improved, and the identification probability of the motion trail is improved.
In yet another embodiment of the present invention, an electronic device is provided that includes one or more processors; a memory coupled to the processor for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the human-machine interface motion trajectory identification method described in the above embodiments. The processor may be a Central Processing Unit (CPU), and may also be other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), ready-made programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are a computing core and a control core of the terminal, and are specifically adapted to load and execute one or more instructions in a computer storage medium so as to implement a corresponding method flow or a corresponding function; the processor provided by the embodiment of the invention can be used for executing the operation of the human-computer interface motion trajectory identification method.
In yet another embodiment of the present invention, the present invention further provides a readable storage medium, specifically a computer readable storage medium (Memory), which is a Memory device in a computer device for storing programs and data. It is understood that the computer readable storage medium herein can include both built-in storage medium in the computer device and, of course, extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space storing an operating system of the terminal. Also, one or more instructions, which may be one or more computer programs (including program code), are stored in the memory space and are adapted to be loaded and executed by the processor. It should be noted that the computer-readable storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory. One or more instructions stored in the computer-readable storage medium may be loaded and executed by the processor to implement the corresponding steps of the method for identifying a motion trajectory of a human-computer interface in the above embodiments. As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above-mentioned embodiments, objects, technical solutions and advantages of the present invention are further described in detail, it should be understood that the above-mentioned embodiments are only examples of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A human-computer interface motion track identification method is characterized by comprising the following steps:
capturing a motion track formed by a user on a current page of a human-computer interface;
inputting the motion track into a trained track recognition model for track recognition;
when the icon of the motion trail is identified, executing an operation instruction corresponding to the icon to enter a next page, and storing the motion trail into a data sample set; alternatively, the first and second liquid crystal display panels may be,
when the icon of the motion track is not identified, inputting the motion track into a track matching model for track matching, and storing the motion track corresponding to successful track matching or failed track matching into a data sample set; wherein, a standard motion track is preset in the track matching model.
2. The human-computer interface motion trail identification method according to claim 1, characterized in that the motion trail formed by the user on the current page of the human-computer interface is captured, specifically:
and carrying out data normalization processing on coordinate data corresponding to the captured input track of the user on the current page of the human-computer interface to obtain a motion track.
3. The human-computer interface motion trail recognition method according to claim 1, wherein the motion trail is input into a trained trail recognition model for trail recognition, specifically:
and identifying the identification probability of the icon corresponding to the motion track, and outputting the icon corresponding to the motion track when the identification probability is greater than a probability threshold, wherein the icon comprises a pattern, a symbol or a character.
4. The method as claimed in claim 1, wherein the trajectory recognition model is obtained by learning a training data set through a machine learning algorithm, and the training data set is used for capturing a historical motion trajectory formed by the user on the human-computer interface.
5. The human-computer interface motion trail recognition method according to claim 4, characterized in that training time is preset, when the training time is up, a machine learning algorithm is used for learning the motion trail in the data sample set and the historical motion trail in the training data set, and network parameters of the trail recognition model are updated to obtain an updated trail recognition model;
and identifying the motion track formed by the user on the current page of the human-computer interface captured in real time by using the updated track identification model.
6. The method for identifying the motion trail of the human-computer interface according to claim 1, wherein when the motion trail is identified, an operation instruction corresponding to the motion trail is executed to enter a next page, and the motion trail is stored in a data sample set, specifically:
presetting standard icon information, wherein the standard icon information carries a corresponding operator;
determining icon information of an icon corresponding to the motion trail in the standard icon information;
and executing an operation instruction based on the operator corresponding to the icon information to enter a next page, and storing the motion trail into a data sample set.
7. The human-computer interface motion trail identification method according to claim 1, wherein when the motion trail is not identified, the motion trail is input into a trail matching model for trail matching, and the motion trail corresponding to the successful trail matching or the failed trail matching is stored in a data sample set; the track matching model is preset with a standard motion track, and the track matching model specifically comprises the following steps:
calculating the similarity between the motion track and a standard motion track preset in the track matching model;
when the similarity is larger than or equal to the similarity threshold value, successfully matching the tracks, executing an operation instruction corresponding to the standard motion track to enter a next page, verifying the matched standard motion track based on the retention time of a user on the next page, attaching a first label of the identified icon to the corresponding motion track when the verification is passed, and storing the motion track and the first label in a data sample set; the standard motion track corresponds to corresponding standard icon information, and the standard icon information carries corresponding operational characters;
or when the similarity is smaller than the similarity threshold value and the track matching fails, attaching a second label which cannot be identified to the corresponding motion track, and storing the motion track and the second label in the data sample set.
8. A human-computer interface motion trajectory recognition system, comprising:
the track capturing module is used for capturing a motion track formed by a user on a current page of the human-computer interface;
the track recognition module is used for inputting the motion track into a trained track recognition model for track recognition;
the track processing module is used for executing an operation instruction corresponding to the icon to enter a next page when the icon of the motion track is identified, and storing the motion track into the data sample set; alternatively, the first and second electrodes may be,
the track matching module is used for inputting the motion track into the track matching model for track matching when the icon of the motion track is not identified, and storing the motion track corresponding to successful track matching or failed track matching into the data sample set; wherein, a standard motion track is preset in the track matching model.
9. An electronic device, characterized in that the electronic device comprises a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the human-machine interface motion trajectory recognition method according to any one of claims 1 to 7.
10. A computer-readable storage medium, having a computer program stored thereon, wherein the computer program, when being executed by a processor, implements the steps of the human-machine interface motion trajectory recognition method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310086079.3A CN115793923B (en) | 2023-02-09 | 2023-02-09 | Human-computer interface motion track identification method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310086079.3A CN115793923B (en) | 2023-02-09 | 2023-02-09 | Human-computer interface motion track identification method, system, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115793923A true CN115793923A (en) | 2023-03-14 |
| CN115793923B CN115793923B (en) | 2023-04-18 |
Family
ID=85430623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310086079.3A Active CN115793923B (en) | 2023-02-09 | 2023-02-09 | Human-computer interface motion track identification method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115793923B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102854982A (en) * | 2012-08-01 | 2013-01-02 | 华平信息技术（南昌）有限公司 | Method for recognizing customized gesture tracks |
| CN104035698A (en) * | 2013-03-04 | 2014-09-10 | 联想(北京)有限公司 | Terminal equipment and state switchover method for same |
| CN104679285A (en) * | 2013-12-02 | 2015-06-03 | 艾酷电子科技有限公司 | System and method for achieving quick response application by identifying moving track |
| CN110427737A (en) * | 2019-06-20 | 2019-11-08 | 平安科技（深圳）有限公司 | Man-machine recognition methods, device and the computer equipment of operation behavior |
| CN111353364A (en) * | 2019-08-19 | 2020-06-30 | 深圳市鸿合创新信息技术有限责任公司 | Dynamic face identification method and device and electronic equipment |
| CN111652912A (en) * | 2020-06-10 | 2020-09-11 | 北京嘀嘀无限科技发展有限公司 | Vehicle counting method and system, data processing equipment and intelligent shooting equipment |
| CN112149410A (en) * | 2020-08-10 | 2020-12-29 | 招联消费金融有限公司 | Semantic recognition method and device, computer equipment and storage medium |
| CN112308110A (en) * | 2020-09-21 | 2021-02-02 | 哈尔滨工业大学（深圳） | Hand motion recognition method and system capable of realizing man-machine interaction |
| CN112783321A (en) * | 2019-11-08 | 2021-05-11 | 苹果公司 | Machine learning based gesture recognition using multiple sensors |
| CN113672696A (en) * | 2021-07-08 | 2021-11-19 | 浙江一山智慧医疗研究有限公司 | Intention recognition method, device, computer equipment and computer readable storage medium |
| CN113918019A (en) * | 2021-10-19 | 2022-01-11 | 亿慧云智能科技（深圳）股份有限公司 | Gesture recognition control method and device for terminal equipment, terminal equipment and medium |
| CN115116130A (en) * | 2022-06-13 | 2022-09-27 | 北京百度网讯科技有限公司 | Call action recognition method, device, equipment and storage medium |
| CN115171223A (en) * | 2022-07-12 | 2022-10-11 | 重庆傲雄在线信息技术有限公司 | Online signature identification method and system based on unconstrained dynamic time programming |
| CN115576475A (en) * | 2022-09-09 | 2023-01-06 | 天翼视讯传媒有限公司 | Matching method based on touch point track |
| US20230027040A1 (en) * | 2021-01-26 | 2023-01-26 | Boe Technology Group Co., Ltd. | Control Method, Electronic Device, and Storage Medium |
-
2023
- 2023-02-09 CN CN202310086079.3A patent/CN115793923B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102854982A (en) * | 2012-08-01 | 2013-01-02 | 华平信息技术（南昌）有限公司 | Method for recognizing customized gesture tracks |
| CN104035698A (en) * | 2013-03-04 | 2014-09-10 | 联想(北京)有限公司 | Terminal equipment and state switchover method for same |
| CN104679285A (en) * | 2013-12-02 | 2015-06-03 | 艾酷电子科技有限公司 | System and method for achieving quick response application by identifying moving track |
| CN110427737A (en) * | 2019-06-20 | 2019-11-08 | 平安科技（深圳）有限公司 | Man-machine recognition methods, device and the computer equipment of operation behavior |
| CN111353364A (en) * | 2019-08-19 | 2020-06-30 | 深圳市鸿合创新信息技术有限责任公司 | Dynamic face identification method and device and electronic equipment |
| CN112783321A (en) * | 2019-11-08 | 2021-05-11 | 苹果公司 | Machine learning based gesture recognition using multiple sensors |
| CN111652912A (en) * | 2020-06-10 | 2020-09-11 | 北京嘀嘀无限科技发展有限公司 | Vehicle counting method and system, data processing equipment and intelligent shooting equipment |
| CN112149410A (en) * | 2020-08-10 | 2020-12-29 | 招联消费金融有限公司 | Semantic recognition method and device, computer equipment and storage medium |
| CN112308110A (en) * | 2020-09-21 | 2021-02-02 | 哈尔滨工业大学（深圳） | Hand motion recognition method and system capable of realizing man-machine interaction |
| US20230027040A1 (en) * | 2021-01-26 | 2023-01-26 | Boe Technology Group Co., Ltd. | Control Method, Electronic Device, and Storage Medium |
| CN113672696A (en) * | 2021-07-08 | 2021-11-19 | 浙江一山智慧医疗研究有限公司 | Intention recognition method, device, computer equipment and computer readable storage medium |
| CN113918019A (en) * | 2021-10-19 | 2022-01-11 | 亿慧云智能科技（深圳）股份有限公司 | Gesture recognition control method and device for terminal equipment, terminal equipment and medium |
| CN115116130A (en) * | 2022-06-13 | 2022-09-27 | 北京百度网讯科技有限公司 | Call action recognition method, device, equipment and storage medium |
| CN115171223A (en) * | 2022-07-12 | 2022-10-11 | 重庆傲雄在线信息技术有限公司 | Online signature identification method and system based on unconstrained dynamic time programming |
| CN115576475A (en) * | 2022-09-09 | 2023-01-06 | 天翼视讯传媒有限公司 | Matching method based on touch point track |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115793923B (en) | 2023-04-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5542774B2 (en) | Flick gesture | |
| KR102460737B1 (en) | Method, apparatus, apparatus and computer readable storage medium for public handwriting recognition | |
| EP3384372B1 (en) | Apparatus and method for note taking with gestures | |
| US5481278A (en) | Information processing apparatus | |
| US8570294B2 (en) | Techniques for recognizing temporal tapping patterns input to a touch panel interface | |
| US8159457B2 (en) | Zero-click activation of an application | |
| CN103270483B (en) | There is the system of gesture identification unit | |
| US20140143721A1 (en) | Information processing device, information processing method, and computer program product | |
| EP2650766A1 (en) | Multi-character continuous handwriting input method | |
| JP2019516189A (en) | Touch screen track recognition method and apparatus | |
| WO2014147716A1 (en) | Electronic device and handwritten document processing method | |
| JP2001005599A (en) | Information processor and information processing method an d recording medium recording information processing program | |
| CN114365075B (en) | Method for selecting a graphical object and corresponding device | |
| KR20180119647A (en) | Method for inserting characters into a string and corresponding digital device | |
| US11482001B2 (en) | Image processing device, image processing method, and image processing system | |
| WO2017148833A1 (en) | Method and system for character insertion in a character string | |
| CN107861684A (en) | Write recognition methods, device, storage medium and computer equipment | |
| CN115793923B (en) | Human-computer interface motion track identification method, system, equipment and medium | |
| CN115357177A (en) | Device control method, device, storage medium and electronic device | |
| Tsai et al. | Reverse time ordered stroke context for air-writing recognition | |
| CN112118491A (en) | Bullet screen generation method and device and computer readable storage medium | |
| JP2004272377A (en) | Device of character editing, character input/display device, method of character editing, program of character editing, and storage medium | |
| WO2022246834A1 (en) | Interaction method, electronic device, and computer storage medium | |
| Petit et al. | Unifying gestures and direct manipulation in touchscreen interfaces | |
| CN118020054A (en) | Information editing method, device, equipment and storage medium for interactive tablet |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |