CN116737051A

CN116737051A - Visual touch combination interaction method, device and equipment based on touch screen and readable medium

Info

Publication number: CN116737051A
Application number: CN202311029107.4A
Authority: CN
Inventors: 陆峰
Original assignee: Beihang University
Current assignee: Beihang University
Priority date: 2023-08-16
Filing date: 2023-08-16
Publication date: 2023-09-12
Anticipated expiration: 2043-08-16
Also published as: CN116737051B

Abstract

The embodiment of the disclosure discloses a touch screen-based visual touch combined interaction method, a device, equipment and a readable medium. One embodiment of the method comprises the following steps: acquiring face information through an eye movement assembly; determining the position information of the fixation point according to the face information; generating a gaze point, and displaying the gaze point on the touch display assembly according to the gaze point position information; in response to detecting a first user operation for a preset area, performing the following interaction steps: determining a first contact and a second contact; determining a moving contact and an instruction contact according to the first contact and the second contact; receiving first instruction information generated by an instruction contact; according to the first instruction information and the position of the fixation point, executing interactive operation; and in response to detecting that the interaction step is completed, acquiring the face information again through the eye movement assembly. This embodiment avoids performing erroneous interface interactions.

Description

Visual touch combination interaction method, device and equipment based on touch screen and readable medium

Technical Field

Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a touch-based visual touch interaction method, device, apparatus, and readable medium.

Background

The line of sight is an important cue revealing the way a person understands the external environment, and the attention and intention of the user are obtained by tracking the eye movements and gaze point of the user. In terms of human-computer interaction, the sight line estimation can be used for improving interface interaction experience of a user and providing a more visual and natural interaction mode. Currently, when using gaze estimation for interface interactions, the following methods are generally adopted: and determining the gaze point position of the user, determining the sight line residence time, determining the interaction intention and performing interface interaction through a pre-trained neural network model.

However, when the above manner is adopted to perform interface interaction by using line-of-sight estimation, the following technical problems often exist:

first, determining the intent of the interaction through the line-of-sight dwell time may result in excessive or too short time due to the inability of the line-of-sight dwell time to be controlled, resulting in erroneous interface interactions being performed.

Secondly, when correcting the neural network model, usually setting a fixed correction time for correction, and possibly failing to accurately identify the gaze point position and performing interface interaction because the neural network model has a problem and the correction time is not reached; when the gaze environment is changed, the pre-trained model cannot adapt to the change of the environment, so that the gaze point cannot be identified, and interface interaction cannot be performed.

Thirdly, by correcting parameters of the neural network model, and when the user does not finish interface interaction, the parameters of the model need to be corrected in a longer time, so that the response of interface interaction needs to be performed in a longer time, and the user experience is affected.

Fourth, when the neural network model is corrected online by using the samples in the queue, the difference of time and head gestures of sample storage is not considered, so that the gaze point identified by the corrected neural network model may have deviation, and the gaze point cannot be identified, so that interface interaction cannot be performed.

The above information disclosed in this background section is only for enhancement of understanding of the background of the inventive concept and, therefore, may contain information that does not form the prior art that is already known to those of ordinary skill in the art in this country.

Disclosure of Invention

The disclosure is in part intended to introduce concepts in a simplified form that are further described below in the detailed description. The disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Some embodiments of the present disclosure propose touch-based touch-screen interaction methods, apparatuses, electronic devices, and computer-readable media to solve one or more of the technical problems mentioned in the background section above.

In a first aspect, some embodiments of the present disclosure provide a touch-based touch-and-touch interaction method, including: acquiring face information through the eye movement assembly; determining the position information of the gazing point according to the face information; generating a gaze point, and displaying the gaze point on the touch display assembly according to the gaze point position information; in response to detecting a first user operation for a preset area, performing the following interaction steps: determining a first contact and a second contact in response to the first user operation characterizing a two-finger operation; determining a moving contact and a command contact according to the first contact and the second contact; receiving first instruction information generated by the instruction contact in response to detecting the movement operation of the movement contact on the gaze point and detecting that the movement is completed; according to the first instruction information and the position of the gaze point, executing interactive operation; and in response to the completion of the execution of the interaction step, acquiring the face information again through the eye movement assembly.

In a second aspect, some embodiments of the present disclosure provide a touch screen-based visual touch combined interaction device, the device including: a first acquisition unit configured to acquire face information through the eye movement assembly; a determining unit configured to determine gaze point position information based on the face information; a generating unit configured to generate a gaze point and display the gaze point on the touch display module according to the gaze point position information; an execution unit configured to execute, in response to detection of a first user operation for a preset area, the following interaction steps: determining a first contact and a second contact in response to the first user operation characterizing a two-finger operation; determining a moving contact and a command contact according to the first contact and the second contact; receiving first instruction information generated by the instruction contact in response to detecting the movement operation of the movement contact on the gaze point and detecting that the movement is completed; according to the first instruction information and the position of the gaze point, executing interactive operation; and a second acquisition unit configured to acquire face information again through the eye movement assembly in response to detection of completion of the execution of the interaction step.

In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors causes the one or more processors to implement the method described in any of the implementations of the first aspect above.

In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

The above embodiments of the present disclosure have the following advantages: by the touch screen-based visual touch combination interaction method of some embodiments of the present disclosure, execution of erroneous interface interaction operations may be avoided. Specifically, the cause of the interface interaction causing the execution error is that: determining the intent of the interaction by the line-of-sight dwell time may result in excessive or too short of a time due to the inability of the line-of-sight dwell time to be controlled, resulting in erroneous interface interactions being performed. Based on this, in the touch-based visual touch interaction method based on the touch screen according to some embodiments of the present disclosure, first, face information is obtained through the above-mentioned eye movement component. Thus, the gaze point position of the human eye can be determined by recognizing the face information. And secondly, determining the gaze point position information according to the face information. Thus, the gaze point can be generated and displayed at the determined gaze point location. Then, generating a gaze point, and displaying the gaze point on the touch display assembly according to the gaze point position information; in response to detecting a first user operation for a preset area, performing the following interaction steps: the first contact and the second contact are determined in response to the first user operation characterizing a two-finger operation. Thereby, the contact position of the two-finger operation can be determined. And secondly, determining a moving contact and a command contact according to the first contact and the second contact. Thus, the contact point of the moving point of gaze and the contact point at which the instruction is input can be distinguished. Third, in response to detecting a movement operation of the moving contact point on the gaze point, and detecting that the movement is completed, first instruction information generated by the instruction contact point is received. Thus, the interactive operation required to be performed can be determined according to the first instruction information. Fourth, according to the first instruction information and the position of the gaze point, interactive operation is executed. Thereby, the double-finger interaction step is completed. Therefore, when the user operation acting on the touch control area (preset area) is detected, corresponding interface interaction is performed according to the generated gaze point, so that interface interaction can be performed in a mode that the touch control component is combined with the vision line, interaction is not performed by using the vision line residence time, and wrong interface interaction operation caused by overlong or too short vision line residence time is avoided. Finally, in response to detecting that the interaction step is completed, the face information is acquired again through the eye movement assembly. Therefore, after the completion of the previous interface interaction is detected, the face information can be acquired again to carry out the next interface interaction. Wrong interface interaction operations are avoided.

Drawings

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.

FIG. 1 is a flow chart of some embodiments of a touch screen based visual touch-coupled interaction method according to the present disclosure;

FIG. 2 is a schematic illustration of single touch screen operation of some embodiments of a touch screen based visual touch combined interaction method according to the present disclosure;

FIG. 3 is a schematic structural view of some embodiments of a touch screen-based visual touch-coupled interaction device according to the present disclosure;

fig. 4 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.

It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.

It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.

It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.

The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 1 illustrates a flow 100 of some embodiments of a touch screen-based visual touch-coupled interaction method according to the present disclosure. The visual touch combination interaction method based on the touch screen comprises the following steps of:

Step 101, obtaining face information through an eye movement assembly.

In some embodiments, the execution subject of the touch-based interaction method (e.g., a mobile device with a touch screen) may obtain the face information through the eye movement component. The touch screen may be a display screen capable of performing touch control. The touch screen may include an eye movement assembly and a touch display assembly. The above-described eye movement component may be a component for detecting face information. As an example, in response to the touch screen being a front-of-handset touch screen, the eye-movement component may be a front-of-handset camera. As another example, the above-described eye movement assembly may also be an eye movement instrument associated with the performing body. The face information may be face image information. The touch display assembly may be an assembly for a display area and an interaction area. The interactive area may be an area of the display screen that is scaled down in equal proportion. For example, the ratio of the display screen area to the interaction area may be 5:1.

Step 102, determining the gaze point position information according to the face information.

In some embodiments, the executing body may determine gaze point location information according to the face information. Wherein the gaze point location information may characterize the gaze point location. For example, the gaze point position information may be coordinate information of the gaze point in an interaction area displayed by the touch display module. In practice, the execution subject may determine the gaze point position information in various ways. For example, the execution subject may determine the gaze point location information by means of human eye model modeling, face image regression detection, or the like.

In some optional implementations of some embodiments, the executing body may input the face information into a pre-trained gaze point location model to obtain gaze point location information. The gaze point location model may be a neural network model that is trained in advance to determine the gaze point location information. For example, the gaze point location model may be a convolutional neural network model or other deep learning model.

And step 103, generating a gaze point, and displaying the gaze point on the touch display component according to the gaze point position information.

In some embodiments, the executing body may generate a gaze point, and display the gaze point on the touch display assembly according to the gaze point position information. In practice, first, the execution subject may generate a light bright spot. Second, the bright point may be displayed at a position indicated by the gaze point position information.

In some alternative implementations of some embodiments, the gaze point may be, but is not limited to, one of: cursor point, regional special effect, display content highlighting.

As an example, in response to the gaze point being a display content highlight, content displayed at a location characterized by the gaze point location information in the interaction region may be highlighted to generate a highlight content, and the highlight content displayed at the location characterized by the gaze point location information.

Step 104, in response to detecting the first user operation for the preset area, performing the following interaction steps:

step 1041, determining a first contact and a second contact in response to the first user operation characterizing a two-finger operation.

In some embodiments, the executing body may characterize the two-finger operation in response to the first user operation described above, determining the first contact and the second contact. The double-finger operation characterization uses double fingers to perform touch operation. The preset area may be a preset touch area included in the touch display assembly.

In some optional implementations of some embodiments, the touch display component may be one of: a positive touch screen, a negative touch screen and a touch pad. The front touch screen and the back touch screen may be a front touch screen and a back touch screen of a dual-screen mobile device.

Step 1042, determining a moving contact and a command contact based on the first contact and the second contact.

In some embodiments, the execution body may determine the moving contact and the command contact according to the first contact and the second contact. In practice, the above-described execution bodies may determine the moving contact and the command contact in various ways. As an example, in response to the display area and the movement area being set to the same area (e.g., the touch display assembly is a single touch screen and the display area and the movement area are both the entire touch screen display area), the executing entity may determine the first generated contact as the movement contact and the second generated contact as the command contact. As yet another example, in response to the display area and the movement area being set to different areas (e.g., the touch display assembly is a dual touch screen (a folded touch screen or a front-rear touch screen), and the display area and the movement area are respectively different touch screens), the execution subject may determine a contact of the display area included in the touch display assembly as an instruction contact and a contact of the touch area included in the touch display assembly as a movement contact.

In step 1043, first instruction information generated by the instruction contact is received in response to detecting a movement operation of the movement contact on the gaze point and detecting that the movement is completed.

In some embodiments, the executing body may receive the first instruction information generated by the instruction contact in response to detecting a movement operation of the moving contact on the gaze point and detecting that the movement is completed. The first instruction information may be instruction information generated by the instruction contact and used for characterizing a certain interaction operation. The first instruction information may include, but is not limited to: touch instructions and voice instructions. In practice, the completion of the movement may be indicative of the moving contact being held down but no longer moving in response to the first user operation being indicative of a two-finger operation. Responsive to the first user operation characterizing a single finger operation, the completion of the movement may characterize releasing the moving contact. As an example, in response to the command contact triggering a voice input control, an associated voice input module may be controlled to record voice information and identify the recorded voice information to generate a voice command. In practice, the condition that the command contact triggers the voice input control may be that the command contact is held down and not moved.

As an example, in response to the first instruction information being a voice instruction, the first instruction information may be obtained by triggering the AI voice function by the instruction contact, receiving an input voice, and recognizing the received voice.

Step 1044, executing an interactive operation according to the first instruction information and the position of the gaze point.

In some embodiments, the executing body may execute the interaction operation according to the first instruction information and the location of the gaze point. In practice, the executing body may execute the interactive operation on the position of the gaze point according to the interactive operation represented by the first instruction information.

Optionally, after step 1044, the following interaction steps are further included:

and a first step of responding to detection of a first user operation for a preset area, wherein the first user operation characterizes movement of the fixation point and confirms whether the movement is completed.

And a second step of generating post-movement gaze point position information according to the movement position of the gaze point in response to completion of movement.

And thirdly, acquiring second face information through the eye movement assembly, and determining the second face information and the moving point-of-gaze position information as point-of-gaze movement information. The moving gaze point position information may represent a position of the moving gaze point in the interaction area.

And step four, adding the gaze point movement information into a preset movement information sequence. The preset movement information sequence may be a preset sequence for storing gaze point movement information. The preset movement information sequence is initially empty.

Optionally, after the third step, in response to the current time being a preset correction time, or in response to the preset movement information sequence meeting a first preset condition, performing the following correction step:

and a first correction step of correcting the gaze point positioning model according to the preset movement information sequence to generate a corrected gaze point positioning model.

In some embodiments, the execution body may correct the gaze point positioning model according to the preset movement information sequence to generate a corrected gaze point positioning model. The preset correction time may be a preset time required to perform parameter correction on the gaze point positioning model. The first preset condition is that the number of preset moving information in the preset moving information sequence is greater than or equal to the preset information number. The preset information amount may be an amount of preset movement information included in a preset movement information sequence. Here, the error values may be propagated forward from the last layer of the model using back propagation, random gradient descent, etc. to adjust the parameters of each layer. Of course, a network freezing method can be adopted as required, network parameters of some layers are kept unchanged and are not adjusted, and the method is not limited.

And a second correction step of obtaining third face information through the eye movement assembly, and inputting the third face information into the corrected gaze point positioning model to obtain gaze point position information.

In some embodiments, the executing body may acquire third face information through the eye movement assembly, and input the third face information into the corrected gaze point location model to obtain gaze point location information.

Optionally, before the performing the following correction step in response to the current time being a preset correction time or in response to the preset movement information sequence satisfying the first preset condition, the performing body may further perform the foregoing correction step by:

in the first step, in response to detecting that the distance between the position represented by the gaze point position information and the position represented by the moved position information is greater than or equal to a preset distance value, the face information and the moved gaze point position information are added to the preset movement information sequence as gaze point movement information, and the correction step is performed according to the preset movement information sequence.

And a second step of determining, in response to detection of a second user operation acting on the touch area and in response to detection of a movement operation acting on the gaze point, post-movement position information of the gaze point after movement and the face information as gaze point movement information, inputting the post-movement position information and the face information into the preset movement information sequence, and performing the correction step in accordance with the preset movement information sequence. The second user operation may be a preset operation for characterizing correction of the gaze point positioning model. For example, the second user operation may be a two-finger sliding or a one-finger sliding out of a predetermined pattern.

And thirdly, responding to the detection of the change of the fixation environment through the eye movement assembly, acquiring an environment image, and carrying out replacement processing on the fixation point positioning model according to the environment image. Wherein the gaze environment includes, but is not limited to, at least one of: light intensity, background environment, head pose.

In practice, the execution subject may perform a replacement process on the gaze point positioning model by:

and a first sub-step of inputting the environment image into a pre-trained environment positioning model to obtain an environment recognition result. The environmental positioning model may be a pre-trained classification model for identifying environmental images. For example, the environmental localization model may be a convolutional neural network model.

And a second sub-step of selecting a preset gaze point positioning model corresponding to the environmental recognition result from a set of preset gaze point positioning models, and replacing the preset gaze point positioning model with a gaze point positioning model. The preset gaze point positioning models in the preset gaze point positioning model set may be different gaze point positioning models stored in advance, and are used for adapting to different recognition environments.

The related content in the first step to the third step is taken as an invention point of the disclosure, so that the second technical problem mentioned in the background art is solved, when the neural network model is corrected, a fixed correction time is usually set for correction, and the correction time possibly does not reach due to the occurrence of problems of the neural network model, so that the fixation point position cannot be accurately identified, and interface interaction cannot be performed; when the gaze environment is changed, the pre-trained model cannot adapt to the change of the environment, so that the gaze point cannot be identified, and interface interaction cannot be performed. Factors that cause failure to interface are often as follows: when the neural network model is corrected, fixed correction time is usually set for correction, and the correction time possibly does not reach due to the occurrence of problems of the neural network model, so that the fixation point position cannot be accurately identified, and interface interaction cannot be performed; when the gaze environment is changed, the pre-trained model cannot adapt to the change of the environment, so that the gaze point cannot be identified, and interface interaction cannot be performed. If the above factors are solved, the effect of avoiding the situation that interface interaction cannot be performed can be achieved. To achieve this, first, in response to detecting that a distance between a position represented by the gaze point position information and a position represented by the post-movement position information is equal to or greater than a preset distance value, the face information and the post-movement gaze point position information are input as gaze point movement information into the preset movement information sequence, and the correction step is performed in accordance with the preset movement information sequence. Therefore, the correction operation can be automatically performed when the error value of the gaze point position recognition result is larger than or equal to the preset threshold value, so that the model can be automatically corrected when the model is in a problem, and the situation that interface interaction cannot be performed due to the fact that correction time is not reached is avoided. Second, in response to detecting a second user operation acting on the touch area, and in response to detecting a movement operation acting on the gaze point, determining post-movement position information of the moved gaze point and the face information as gaze point movement information, inputting the gaze point movement information and the face information into the preset movement information sequence, and performing the correction step according to the preset movement information sequence. Therefore, after the user operation for correcting the characterization is detected, the model is corrected, and the situation that interface interaction cannot be performed due to the fact that correction time is not reached is further avoided. Thirdly, in response to detecting that the fixation environment changes, an environment image is acquired, and the fixation point positioning model is replaced according to the environment image. Therefore, when the change of the gaze point environment is detected, the gaze point positioning model is replaced, so that the gaze point positioning model can adapt to the current gaze point environment, the situation that the gaze point cannot be identified due to the fact that the pre-trained model cannot adapt to the change of the environment is avoided, and the situation that interface interaction cannot be performed is further avoided.

Optionally, the step of interacting further includes the steps of:

and fifthly, responding to the first user operation to characterize the single-finger operation, and detecting the movement operation of the single-finger contact point to the fixation point to determine whether the movement is completed.

In some embodiments, the executing body may characterize a single-finger operation in response to the first user operation, and detect a movement operation of the single-finger contact point to the gaze point, and determine whether the movement is completed. The single-finger operation characterization uses a single finger to perform touch operation.

And sixthly, receiving second instruction information generated by the single-finger contact point in a preset time period in response to the completion of the movement, and executing interactive operation according to the second instruction information and the position of the fixation point.

In some embodiments, the executing body may receive the second instruction information generated by the single-finger contact point in a preset duration in response to determining that the movement is completed, and execute the interactive operation according to the second instruction information and the location of the gaze point. The preset duration may be a duration after the movement is completed. For example, the predetermined time period may be 0.5 seconds.

And seventhly, performing clicking interaction operation on the position of the gaze point in response to the fact that second instruction information generated by the single-finger contact is not received within a preset time.

In some embodiments, the executing body may perform a click interaction operation on the location of the gaze point in response to not receiving the second instruction information generated by the single-finger contact point within a preset period of time.

Optionally, after step 104, the above-mentioned execution body may further execute the following online correction steps:

a first online correction step, obtaining a predefined loss function. In practice, the predefined loss function may be obtained from an associated database by means of a wired connection or a wireless connection. The obtained predefined loss function is as follows:。

wherein, the liquid crystal display device comprises a liquid crystal display device,representing a predefined loss function. />Representing custom constants. />Representation->An augmented matrix of dimensions. />Representation->An augmented matrix of dimensions. />Representing the transpose of the matrix. />Representing the number of preset movement information included in the preset movement information sequence.

Here, the correction formula may be fitted by a least square method. Wherein (1)>Indicating gaze point location information. />Indicating post-movement gaze point location information. />And- >Respectively, representing different constants. Can let->Then->。

And a second online correction step, determining a custom constant according to the predefined loss function. In practice, the predefined loss function comprisesObtaining a bias derivative of 0 to obtain a custom constant: />。

And a third online correction step, in response to receiving new gaze point position information output by the gaze point positioning model, mapping the new gaze point position information into moved gaze point position information as gaze point position information according to the custom constant and a least square fitting correction formula.

And a fourth online correction step of generating a gaze point and displaying the gaze point on the touch display assembly according to the gaze point position information.

The related content in the first online correction step-the fourth online correction step is taken as an invention point of the disclosure, so that the technical problem mentioned in the background art is solved, and when the user does not complete the interface interaction, the parameter of the model needs to be corrected in a long time by correcting the parameter of the neural network model, and thus the response of the interface interaction needs to be performed in a long time, and the user experience is affected. The factors that affect the user experience often are as follows: by correcting the parameters of the neural network model, and when the user does not finish the interface interaction, the parameters of the model need to be corrected in a longer time, so that the response of the interface interaction needs to be carried out in a longer time, and the user experience is affected. If the factors are solved, the effects of reducing the response time of interface interaction and improving the user experience can be achieved. To achieve this effect, first, a predefined loss function is obtained. Thus, the representation formula of the custom constant can be determined by a predefined loss function. Secondly, determining a custom constant according to the predefined loss function; and in response to receiving new gaze point position information output by the gaze point positioning model, mapping the new gaze point position information into moved gaze point position information as gaze point position information according to the custom constant and a least squares fitting correction formula. Therefore, the loss relation between the gaze point position information identified by the model and the gaze point actual position information can be represented by a linear function in a fitting mode, so that the gaze point position information output by the gaze point positioning model can be directly mapped into the gaze point actual position information in the next interface interaction, the interface interaction can be completed without correcting the parameters of the gaze point positioning model, the response time for carrying out the interface interaction can be reduced, and the user experience is improved. And finally, generating a gaze point, and displaying the gaze point on the touch display component according to the gaze point position information. Thereby, determination of the gaze point position is completed. The response time for interface interaction can be reduced, and the user experience is improved.

Optionally, the following online correction step may also be performed:

and a fifth correction step of determining a first correction value of the sample in response to the head pose correlation value being greater than a preset correlation value threshold. The head pose correlation value may be determined according to a difference between the head pose of the current sample and a standard head pose, or may be determined according to a difference between the head poses of other samples in the queue. In practice, the execution subject may determine the first correction value of the sample by the following formula:。

wherein, the liquid crystal display device comprises a liquid crystal display device,a first correction value representing the sample. />Representation and sample->Adding the queue time-dependent weights.Representation sample->Is a head pose position-related weight. />Indicate->Samples. />Representing the information obtained according to the fitting method (e.g. least squares method) about +.>The correction value contribution for the current time of the sample.

Here, theThe determination can be made by the following formula: />。

Wherein, the liquid crystal display device comprises a liquid crystal display device,and->Representing a preset weight. Here, for +.>And->The setting of (2) is not limited, and may be a weight obtained by experiment. />Is about sample->Time-related values added to the queue, which may be based on the time stamp or +. >Position determination in the queue. Therefore, the weight of the sample newly put in the queue can be increased, and the reliability of the sample data is improved.

Here the number of the elements is the number,the determination can be made by the following formula: />。

Wherein, the liquid crystal display device comprises a liquid crystal display device,and->Representing a preset weight. Here, for +.>And->The setting of (2) is not limited, and may be a weight obtained by experiment. />Representation about sample->Is a head pose correlation value of (a). The head pose correlation value may be determined based on the difference in head pose of the sample relative to a standard head pose or relative to other corrected data sample points in the queue. Therefore, the weight of the samples with larger head posture deviation in the queue can be increased, and the reliability of the sample data is improved.

And a sixth online correction step, wherein a second correction value of the sample is determined in response to the head posture related value being greater than a preset related value threshold. In practice, the second correction value for the sample may be determined by the following formula:

。

wherein, the liquid crystal display device comprises a liquid crystal display device,representing the second correction value. />Is a fixed weight, here, for +.>The setting of (2) is not limited, and may be a fixed weight obtained by experiment. />Representing the sequence number in the queue characterized by the moment of change of head pose.

The related content in the fifth online correction step to the sixth online correction step is taken as an invention point of the present disclosure, which solves the fourth technical problem mentioned in the background art, that when the neural network model is online corrected by using the samples in the queue, the difference of the time and the head gesture of the sample storage is not considered, so that the point of gaze identified by the corrected neural network model may have a deviation, and the point of gaze cannot be identified, so that interface interaction cannot be performed. Factors that cause failure to identify the gaze point and thus failure to perform interface interactions are often as follows: when the neural network model is corrected online by using the samples in the queue, the difference of time and head gestures stored by the samples is not considered, so that the corrected neural network model may have deviation of the identified gaze point, and the gaze point cannot be identified, so that interface interaction cannot be performed. If the factors are solved, the situation that the gaze point cannot be identified can be avoided, and the effect that interface interaction cannot be performed is avoided. To achieve this effect, by establishing a time-dependent weight and a head posture-dependent weight, increasing the weight of a sample after the time and increasing the weight of a sample after the head posture is changed, it is possible to reduce the deviation due to the head posture change and the time lapse. Meanwhile, when the head posture is changed greatly, the weight before the head posture is changed is reduced, so that the condition that a sample is not applicable due to the change of the head posture can be avoided, the condition that the gaze point cannot be identified can be avoided, and interface interaction cannot be performed.

Step 105, in response to detecting that the interaction step is completed, acquiring the face information again through the eye movement assembly.

In some embodiments, the executing body may acquire the face information again through the eye movement assembly in response to detecting that the executing of the interaction step is completed.

Fig. 2 illustrates a single touch screen operation diagram 200 of some embodiments of touch screen-based visual-touch combined interaction methods according to the present disclosure. As can be seen from fig. 2, in the single touch screen shown in fig. 2, the touch screen is divided into a first touch area, a second touch area and a display area. An eye movement assembly is arranged above the touch screen. The eye movement assembly is a front camera.

As shown in fig. 2, when the first user operation is a double-finger operation and the gaze point position determined by the gaze point positioning model is inaccurate, the moving contact point may be moved through the sliding operation in the first touch area, and the gaze point may be guided to an actual position (a real gaze point position), and at this time, the gaze point in the display area may be moved to the actual position according to a movement ratio. The moving proportion is the proportion of the first touch area to the touch screen. Here, the custom movement proportion may also be set. When the movement is completed, first instruction information generated by the instruction contact point in the second touch area can be received. As shown in fig. 2, the first instruction information characterizes a part of the area in the selection display area, and the instruction contact point can be moved in the second touch area according to the movement proportion, so as to control the gaze point after movement to select the area to be selected, so as to generate a selected area, and perform interactive operation according to the selected area.

With further reference to fig. 3, as an implementation of the method shown in the foregoing drawings, the present disclosure provides some embodiments of a touch screen-based visual touch-combined interaction apparatus, which corresponds to those method embodiments shown in fig. 1, and is particularly applicable to various electronic devices.

As shown in fig. 3, the touch screen-based visual-touch combined interaction device 300 of some embodiments includes: a first acquisition unit 301, a determination unit 302, a generation unit 303, an execution unit 304, and a second acquisition unit 305. Wherein the first obtaining unit 301 is configured to obtain face information through the eye movement assembly; the determining unit 302 is configured to determine gaze point position information based on the face information; the generating unit 303 is configured to generate a gaze point, and display the gaze point on the touch display component according to the gaze point position information; the execution unit 304 is configured to perform the following interaction steps in response to detecting a first user operation for a preset area: responding to the first user operation to represent the region selection operation, and determining the position of the gaze point as a region selection starting point; in response to detecting an area selection start point movement operation acting on the preset area, moving the gaze point in equal proportion according to a movement path of the area selection start point movement operation; determining the position of the moved fixation point as a region selection end point; generating a rectangular selected area according to the area selection starting point and the area selection ending point; performing interactive operation according to the rectangular selected area; the second acquiring unit 305 is configured to acquire face information again through the above-described eye movement component in response to detecting that the above-described interactive step execution is completed.

It will be appreciated that the elements described in the touch screen based visual contact interface 300 correspond to the various steps in the method described with reference to fig. 1. Thus, the operations, features and the beneficial effects described above for the method are equally applicable to the touch screen-based visual touch combined interaction device 300 and the units contained therein, and are not described herein again.

Referring now to fig. 4, a schematic diagram of an electronic device 400 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), car terminals (e.g., car navigation terminals), and the like, as well as stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 4 is merely an example and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.

As shown in fig. 4, the electronic device 400 may include a processing means 401 (e.g., a central processing unit, a graphics processor, etc.) that may perform various suitable actions and processes in accordance with a program stored in a Read Only Memory (ROM) 402 or a program loaded from a storage means 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

In general, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 408 including, for example, magnetic tape, hard disk, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. While fig. 3 shows an electronic device 400 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 4 may represent one device or a plurality of devices as needed.

In particular, according to some embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via communications device 409, or from storage 408, or from ROM 402. The above-described functions defined in the methods of some embodiments of the present disclosure are performed when the computer program is executed by the processing device 401.

It should be noted that, the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: and acquiring the face information through the eye movement assembly. And determining the gaze point position information according to the face information. And generating a gaze point, and displaying the gaze point on the touch display assembly according to the gaze point position information. In response to detecting a first user operation for a preset area, performing the following interaction steps: responding to the first user operation to represent the region selection operation, and determining the position of the gaze point as a region selection starting point; in response to detecting an area selection start point movement operation acting on the preset area, moving the gaze point in equal proportion according to a movement path of the area selection start point movement operation; determining the position of the moved fixation point as a region selection end point; generating a rectangular selected area according to the area selection starting point and the area selection ending point; and performing interactive operation according to the rectangular selected area. And in response to the completion of the execution of the interaction step, acquiring the face information again through the eye movement assembly.

Computer program code for carrying out operations for some embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes a first acquisition unit, a determination unit, a generation unit, an execution unit, and a second acquisition unit. The names of these units do not constitute a limitation on the unit itself in some cases, and for example, the first acquisition unit may also be described as "a unit that acquires face information through the above-described eye movement component".

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A view-touch combined interaction method based on a touch screen, wherein the touch screen comprises an eye movement assembly and a touch display assembly, and the method comprises the following steps:

acquiring face information through the eye movement assembly;

determining the gaze point position information according to the face information;

generating a gaze point, and displaying the gaze point on the touch display assembly according to the gaze point position information;

in response to detecting a first user operation for a preset area, performing the following interaction steps:

determining a first contact and a second contact in response to the first user operation characterizing a two-finger operation;

determining a moving contact and an instruction contact according to the first contact and the second contact;

receiving first instruction information generated by the instruction contact in response to detecting the movement operation of the movement contact on the fixation point and detecting that the movement is completed;

according to the first instruction information and the position of the fixation point, executing interactive operation;

and in response to detecting that the interaction step is completed, acquiring the face information again through the eye movement assembly.

2. The method of claim 1, wherein the gaze point is one of: cursor point, regional special effect, display content highlighting.

3. The method of claim 1, wherein the preset area is any area in the preset touch display assembly, and the touch display assembly is one of the following: a positive touch screen, a negative touch screen and a touch pad.

4. The method of claim 1, wherein the determining gaze point location information from the face information comprises:

and inputting the face information into a pre-trained fixation point positioning model to obtain fixation point position information.

5. The method of claim 1, wherein the interacting step further comprises:

responsive to detecting a first user operation for a preset area, and the first user operation characterizes movement of the gaze point, confirming whether the movement is complete;

generating post-movement gaze point position information according to the movement position of the gaze point in response to movement completion;

acquiring second face information through the eye movement assembly, and determining the second face information and the post-movement gaze point position information as gaze point movement information;

and adding the gaze point movement information to a preset movement information sequence, wherein the preset movement information sequence is initially empty.

6. The method of claim 4, wherein the method further comprises:

in response to the current time being a preset correction time, or in response to the preset movement information sequence meeting a first preset condition, performing the following correction steps:

correcting the fixation point positioning model according to the preset movement information sequence to generate a corrected fixation point positioning model, wherein the first preset condition is that the number of preset movement information in the preset movement information sequence is greater than or equal to the number of preset information;

and acquiring third face information through the eye movement assembly, and inputting the third face information into the corrected gaze point positioning model to obtain gaze point position information.

7. The method of claim 1, wherein the interacting step further comprises:

responding to the first user operation to characterize single-finger operation, and detecting movement operation of a single-finger contact point to the fixation point, and determining whether movement is completed;

receiving second instruction information generated by the single-finger contact point within a preset time period in response to the completion of the movement, and executing interactive operation according to the second instruction information and the position of the fixation point;

And responding to the fact that the second instruction information generated by the single-finger contact is not received within the preset time, and performing clicking interaction operation on the position of the gaze point.

8. A touch screen-based visual-touch combined interaction device, comprising:

a first acquisition unit configured to acquire face information through the eye movement assembly;

a determining unit configured to determine gaze point position information from the face information;

a generating unit configured to generate a gaze point and display the gaze point on a touch display assembly according to the gaze point position information;

an execution unit configured to execute, in response to detection of a first user operation for a preset area, the following interaction steps: determining a first contact and a second contact in response to the first user operation characterizing a two-finger operation; determining a moving contact and an instruction contact according to the first contact and the second contact; receiving first instruction information generated by the instruction contact in response to detecting the movement operation of the movement contact on the fixation point and detecting that the movement is completed; according to the first instruction information and the position of the fixation point, executing interactive operation;

And a second acquisition unit configured to acquire face information again through the eye movement assembly in response to detection of completion of the execution of the interaction step.

9. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon;

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1 to 7.

10. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1 to 7.