CN115019382A - Area determination method, apparatus, device, storage medium and program product - Google Patents

Abstract

The application relates to a region determination method, a region determination device, a computer device, a storage medium and a computer program product, which can be used in the field of financial technology or other related fields, and comprise the following steps: acquiring eye movement data; inputting the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtaining a plurality of estimation results of each region aiming at the eye movement data estimation models through the eye movement data estimation models; taking the estimation result corresponding to the eye movement data estimation model corresponding to each region as a target estimation result; screening out candidate watching areas from all the areas based on the target estimation result; constructing a gaze region identification framework based on a plurality of estimation results corresponding to the candidate gaze region; and determining a target gazing area from the candidate gazing area based on the gazing area identification framework. By adopting the method, the target watching area can be accurately determined.

Description

Area determination method, apparatus, device, storage medium and program product

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a method, apparatus, computer equipment, storage medium and computer program product for determining an area.

Background technique

With the development of artificial intelligence technology, the automatic confirmation technology of the interface area of the intelligent terminal has appeared. This technology confirms the interface area that the user pays attention to by collecting the user's voice and extracting keywords based on the voice.

In the above technical solution, if the keywords extracted based on the user's voice are related to multiple areas of the above-mentioned intelligent terminal interface, the intelligent terminal will provide multiple areas for the user to choose, or if the user cannot pronounce due to special reasons or the pronunciation is inaccurate When there is a user, the intelligent terminal cannot accurately determine the area that the user pays attention to on the interface.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide an area determination method, apparatus, computer equipment, computer-readable storage medium and computer program product that can accurately determine the user's attention area in response to the above technical problems.

In a first aspect, the present application provides a region determination method. The method includes:

obtaining eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes a plurality of regions;

The eye movement data is input into a plurality of pre-built eye movement data estimation models, and through the plurality of eye movement data estimation models, a plurality of eye movement data estimation models for each region is obtained. Estimation results; the multiple eye movement data estimation models are in one-to-one correspondence with the multiple regions, and the multiple estimation results are used to represent the probability that the respective regions are the gaze regions of the human eye;

Taking the estimated result corresponding to the eye movement data estimation model corresponding to each area as the target estimated result; based on the target estimated result, screen out candidate gaze areas from the various areas;

Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area identification framework is constructed; based on the gaze area identification framework, a target gaze area is determined from the candidate gaze areas.

In one embodiment, the selection of candidate gaze regions from the respective regions based on the target estimation result includes: obtaining a final estimation result corresponding to the current region based on the target estimation result ; The final estimated result is the first preset value or the second preset value; If the final estimated result of the current area is the first preset value, the current area is determined as the candidate gaze area .

In one embodiment, the constructing a gaze area identification framework based on the multiple estimation results corresponding to the candidate gaze areas includes: using a calibration method constructed based on a non-parametric method, The estimated results are calibrated to obtain multiple calibration results corresponding to the respective candidate fixation regions; the multiple calibration results corresponding to the respective candidate fixation regions are fused to obtain fusion results corresponding to the respective candidate fixation regions; The fusion results corresponding to each candidate fixation region are described, and a fixation region recognition framework is constructed.

In one embodiment, the determining a target gaze area from the candidate gaze areas based on the gaze area identification framework includes: obtaining, based on the gaze area identification framework, an identification corresponding to each candidate gaze area Result; if the value of the recognition result corresponding to the current candidate fixation area is the maximum value among all the values of the recognition results, and the value of the recognition result corresponding to the current candidate fixation area is greater than a preset threshold, then determine the current candidate fixation area gaze area for the target.

In one embodiment, the acquiring eye movement data includes: acquiring an image of a human eye, acquiring the coordinates of the pupil center on the human eye image, and acquiring a corneal reflection spot on the human eye image center coordinates; based on the center coordinates of the pupil and the center coordinates of the corneal reflection spot, the line of sight direction of the human eye is obtained; based on the line of sight direction, the gaze of each gaze point of the human eye on the interface is obtained point coordinates; obtain the dwell time of each fixation point, and the distance between each fixation point and the next fixation point corresponding to each fixation point; take the fixation point coordinates, the stay time and the distance as the eye movement data.

In one embodiment, the acquiring the coordinates of the pupil center on the human eye image includes: acquiring a pupil area in the human eye image; acquiring a boundary of a preset number of boundary points on the boundary of the pupil area Point coordinates; based on the boundary point coordinates, the pupil center coordinates are obtained.

In one embodiment, acquiring the center coordinates of the corneal reflection spot on the human eye image includes: filtering the human eye image to obtain multiple corneal reflection spot images; acquiring the multiple corneal reflection spot images The center coordinates of the corresponding multiple corneal reflection light spot images; the center coordinates of the multiple corneal reflection light spot images are fused to obtain the center coordinates of the corneal reflection light spot.

In one embodiment, after the target gaze area is determined from the candidate gaze areas, the method further includes: in response to a trigger request for the target gaze area, switching the interface to be corresponding to the target gaze area target interface.

In one embodiment, the above method further includes: setting the display brightness of the target gaze area on the interface to be greater than the display brightness of the non-target gaze area; the non-target gaze areas are the multiple areas in the area other than the target gaze area.

In a second aspect, the present application also provides an apparatus for determining an area. The device includes:

an eye movement data acquisition module, used for acquiring eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes a plurality of regions;

The estimation result acquisition module is used for inputting the eye movement data into a plurality of pre-built eye movement data estimation models, and through the plurality of eye movement data estimation models, each area is obtained for the plurality of eye movement data estimation models. Multiple prediction results of the movement data prediction model; the multiple eye movement data prediction models are in one-to-one correspondence with the multiple regions, and the multiple prediction results are used to characterize the respective regions as the people the probability of the gaze area of the eye;

The candidate gaze area judgment module is used to use the estimated result corresponding to the eye movement data prediction model corresponding to each area as the target estimated result; based on the target estimated result, select the candidate gaze from the various areas area;

A target gaze area determination module, configured to construct a gaze area identification framework based on multiple estimation results corresponding to the candidate gaze areas; and determine a target gaze area from the candidate gaze areas based on the gaze area identification framework.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

obtaining eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes a plurality of regions;

The eye movement data is input into a plurality of pre-built eye movement data estimation models, and through the plurality of eye movement data estimation models, a plurality of eye movement data estimation models for each region is obtained. Estimation results; the multiple eye movement data estimation models are in one-to-one correspondence with the multiple regions, and the multiple estimation results are used to represent the probability that the respective regions are the gaze regions of the human eye;

Taking the estimated result corresponding to the eye movement data estimation model corresponding to each area as the target estimated result; based on the target estimated result, screen out candidate gaze areas from the various areas;

Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area identification framework is constructed; based on the gaze area identification framework, a target gaze area is determined from the candidate gaze areas.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by the processor, the following steps are implemented:

obtaining eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes a plurality of regions;

The eye movement data is input into a plurality of pre-built eye movement data estimation models, and through the plurality of eye movement data estimation models, a plurality of eye movement data estimation models for each region is obtained. Estimation results; the multiple eye movement data estimation models are in one-to-one correspondence with the multiple regions, and the multiple estimation results are used to represent the probability that the respective regions are the gaze regions of the human eye;

Taking the estimated result corresponding to the eye movement data estimation model corresponding to each area as the target estimated result; based on the target estimated result, screen out candidate gaze areas from the various areas;

Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area identification framework is constructed; based on the gaze area identification framework, a target gaze area is determined from the candidate gaze areas.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

obtaining eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes a plurality of regions;

The eye movement data is input into a plurality of pre-built eye movement data estimation models, and through the plurality of eye movement data estimation models, a plurality of eye movement data estimation models for each region is obtained. Estimation results; the multiple eye movement data estimation models are in one-to-one correspondence with the multiple regions, and the multiple estimation results are used to represent the probability that the respective regions are the gaze regions of the human eye;

Taking the estimated result corresponding to the eye movement data estimation model corresponding to each area as the target estimated result; based on the target estimated result, screen out candidate gaze areas from the various areas;

Based on a plurality of estimation results corresponding to the candidate gaze areas, a gaze area identification framework is constructed; based on the gaze area identification framework, a target gaze area is determined from the candidate gaze areas.

The above-mentioned area determination method, device, computer equipment, storage medium and computer program product are obtained by obtaining eye movement data; the eye movement data is used to represent the change of the gaze point of the human eye on the gaze interface, and the interface includes multiple areas; The eye movement data is input into multiple pre-built eye movement data prediction models, and through multiple eye movement data prediction models, multiple prediction results for each region for multiple eye movement data prediction models are obtained; The prediction model of movement data corresponds to multiple regions one-to-one, and multiple prediction results are used to represent the probability that each region is the gaze region of the human eye; the prediction result corresponding to the prediction model of eye movement data corresponding to each region is used as the target prediction. Based on the target estimation results, the candidate gaze areas are selected from each area; based on multiple prediction results corresponding to the candidate gaze areas, the gaze area recognition framework is constructed; target gaze area. The present application extracts eye movement data, then inputs the eye movement data into a pre-built eye movement data prediction model, selects candidate fixation regions, and finally uses the fixation region recognition framework to accurately determine the target fixation region.

Description of drawings

1 is a schematic flowchart of a method for determining a region in one embodiment;

2 is a schematic flowchart of screening candidate gaze regions in one embodiment;

3 is a schematic flowchart of constructing a gaze area recognition framework in one embodiment;

4 is a schematic flowchart of determining a target gaze area in one embodiment;

Fig. 5 is the user operation flow chart of the intelligent terminal in one embodiment;

Fig. 6-1 is a system interface rendering diagram of an intelligent terminal in one embodiment;

Figure 6-2 is a system interface rendering diagram of an intelligent terminal in another embodiment;

7 is a structural block diagram of an apparatus for determining an area in an embodiment;

FIG. 8 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

It should be noted that the term "first\second" involved in the embodiments of the present invention is only to distinguish similar objects, and does not represent a specific ordering of objects. It is understandable that "first\second" may be used when permitted The specific order or sequence may be interchanged below. It should be understood that the "first\second" distinctions may be interchanged under appropriate circumstances to enable the embodiments of the invention described herein to be practiced in sequences other than those illustrated or described herein.

In one embodiment, as shown in FIG. 1 , a method for determining an area is provided. In this embodiment, the method is applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a terminal. A system including a terminal and a server, and is realized through the interaction of the terminal and the server. In this embodiment, the method includes the following steps:

Step S101 , obtaining eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes multiple regions.

Among them, the eye movement data is the data generated by the change of the human eye sight. For example, the eye movement data can be the specific direction of the human eye sight, the time that the human eye sight stays in a certain direction, and the change of the human eye sight from one direction to another direction. angle, etc. The interface is a pre-set plane on which the human eye gazes, and the interface includes a number of pre-set areas. As for the gaze point, it is the intersection of the human eye's line of sight and the above-mentioned plane, that is, the point where the human eye gazes on the interface. .

Specifically, a human eye image is captured by an infrared camera device to obtain a human eye image, and then the human eye image is binarized to obtain a result of the binarized image processing, and the result is filtered by a Gaussian function, thereby The noise of the human eye image is removed, and the human eye gaze direction recognition model is pre-built with the noise-removed image input value to obtain the gaze direction information of the human eye, and based on the gaze direction information, the eye movement data is obtained.

Step S102, input the eye movement data into a plurality of pre-built eye movement data estimation models, and obtain a plurality of estimation results for each region for the plurality of eye movement data estimation models through the plurality of eye movement data estimation models ; Multiple eye movement data prediction models are in one-to-one correspondence with multiple regions, and multiple prediction results are used to characterize the probability that each region is the gaze region of the human eye.

The multiple pre-built eye movement data prediction models are multiple pre-built two-class models, the two-class models are in one-to-one correspondence with the areas in the above interface, the two-class model is constructed based on a gradient boosting iterative decision tree, the The two-class model is used to identify whether the corresponding area is a gaze area of the human eye, and the estimated result is the output value of the two-class model, and the output value is used to represent the probability that each of the above-mentioned areas is the gaze area of the human eye.

Specifically, by inputting the eye movement data into the two-class model corresponding to a certain area, the estimated result of each of the above-mentioned areas can be obtained, and the estimated result is the output value of the two-class model, wherein the estimated result corresponding to the above-mentioned one area The error is the smallest, and the error of the estimation results corresponding to the remaining areas is relatively large. After that, repeat this operation to input the eye movement data into the two-class model corresponding to the remaining areas, and obtain multiple prediction models for multiple eye movement data in each area. Estimated results. For example, the above interface has four areas A, B, C, and D, then there are four two-category models of A, B, C, and D. Now input the eye movement data into model A to obtain the estimated result A and the estimated result. B. Estimated result C, and estimated result D, where the error of the estimated result A corresponding to the area A is small, while the error of the estimated result B, the estimated result C, and the estimated result D is relatively large. Enter model B, model C, and model D, and finally get a total of 16 estimated results, each area corresponds to 4 estimated results corresponding to 4 models.

In step S103, the prediction result corresponding to the eye movement data prediction model corresponding to each area is used as the target prediction result; based on the target prediction result, candidate fixation areas are selected from each area.

The target prediction result is the prediction result with the smallest error corresponding to each area, that is, the prediction result corresponding to the prediction model corresponding to each area. For example, the target prediction result of the above-mentioned area A is the prediction result A , and the candidate gaze area is the area in which the probability of human eye gaze exceeds 50% among the above-mentioned areas.

Specifically, from the plurality of estimation results corresponding to the current area, the estimation result corresponding to the recognition model corresponding to the current area is selected as the target estimation result, and if the target estimation result is greater than the preset threshold, it is determined that the current area is Candidate gaze areas.

Step S104 , constructing a gaze area identification framework based on the multiple estimation results corresponding to the candidate gaze areas; and determining a target gaze area from the candidate gaze areas based on the gaze area identification framework.

Among them, the gaze area identification framework is an identification framework constructed based on D-S evidence theory reasoning, which is used to determine the target gaze area, and the target gaze area is the most likely area of the human eye in the above interface.

Specifically, multiple prediction results corresponding to the current candidate gaze area are fused to obtain the fusion result of the current candidate gaze area, and similarly, the fusion results of the remaining areas are obtained, and a gaze area identification framework is constructed based on the fusion results, and then the gaze area The recognition framework outputs the corresponding number of the target gaze area.

In the above area determination method, the eye movement data is obtained; the eye movement data is used to represent the change of the gaze point of the human eye on the gaze interface, and the interface includes multiple areas; the eye movement data is input into a plurality of pre-built eyes. Movement data prediction model, through multiple eye movement data prediction models, multiple prediction results for each area for multiple eye movement data prediction models are obtained; multiple eye movement data prediction models are associated with multiple areas one by one Correspondingly, multiple prediction results are used to represent the probability that each area is the gaze area of the human eye; the prediction result corresponding to the eye movement data prediction model corresponding to each area is used as the target prediction result; The candidate gaze area is screened out from the area; the gaze area identification framework is constructed based on multiple prediction results corresponding to the candidate gaze area; the target gaze area is determined from the candidate gaze area based on the gaze area identification framework. The present application extracts eye movement data, then inputs the eye movement data into a pre-built eye movement data prediction model, selects candidate fixation regions, and finally uses the fixation region recognition framework to accurately determine the target fixation region.

In one embodiment, as shown in FIG. 2 , based on the target estimation result, a candidate gaze area is selected from each area, including the following steps:

Step S201, based on the target estimation result, obtain a final estimation result corresponding to the current area; the final estimation result is a first preset value or a second preset value.

Wherein, the final estimated result is two preset results, and which preset result is closer to the target estimated result, the preset result is the final estimated result, and the first preset value or the second preset value is the above-mentioned Two preset results, for example, the first preset value is 1, and the second preset value is 0.

Specifically, when the target estimation result is close to 1, the final estimation result corresponding to the target estimation result is 1; when the target estimation result is close to 0, the final estimation result corresponding to the target estimation result is 0;

Step S202, if the final estimation result of the current area is the first preset value, determine the current area as a candidate gaze area.

Specifically, if the final estimation result of the current area is 1, the current area is determined as a candidate gaze area.

In this embodiment, by converting the target estimation result into the final estimation result, based on the final estimation result, the candidate gaze regions can be accurately screened.

In one embodiment, as shown in Figure 3, based on multiple estimation results corresponding to the candidate gaze area, constructing a gaze area identification framework, comprising the following steps:

Step S301 , using a calibration method constructed based on a non-parametric method, calibrate multiple prediction results corresponding to each candidate fixation area, and obtain multiple calibration results corresponding to each candidate fixation area.

Among them, the calibration method constructed based on the non-parametric method is a calibration result of the output value of a two-class model, which can be the Histogram Binning non-parametric method, and the calibration result is the estimated result after calibration and optimization through this scheme, and the calibration result is the candidate fixation. The region is the probability value of the target gaze region.

Specifically, calibration and optimization are performed on multiple prediction results corresponding to each candidate fixation region by the non-parametric method Histogram Binning, and multiple calibration results corresponding to each candidate fixation region are obtained.

Step S302 , fuse multiple calibration results corresponding to each candidate fixation region to obtain a fusion result corresponding to each candidate fixation region.

The fusion result is an orthogonal sum of multiple calibration results corresponding to each candidate fixation region based on the inference of D-S evidence theory.

Specifically, the fusion rules of D-S evidence theory reasoning are as follows:

m_j(A_i)为候选注视区域A对应的多个校准结果，m(A)为候选注视区域A的融合结果，通过该融合规则将各个候选注视区域对应的多个校准结果进行融合，得到各个候选注视区域对应的融合结果。例如，假设上述区域A为候选注视区域，将区域A对应的4个校准结果A、B、C、D进行融合，得到区域A对应的融合结果，也即区域A为目标注视区域的最终概率。in,

m _j (A _i ) is the multiple calibration results corresponding to the candidate gaze area A, m(A) is the fusion result of the candidate gaze area A, and the multiple calibration results corresponding to each candidate gaze area are fused through the fusion rule to obtain The fusion results corresponding to each candidate gaze area. For example, assuming that the above-mentioned area A is a candidate gaze area, the four calibration results A, B, C, and D corresponding to area A are fused to obtain the fusion result corresponding to area A, that is, the final probability that area A is the target gaze area.

Step S303 , based on the fusion results corresponding to each candidate gaze area, construct a gaze area identification framework.

Specifically, a set of fusion results corresponding to each candidate gaze area is formed, and the set is the above gaze area identification framework. That is to say, the fusion result corresponding to each candidate gaze area is the value corresponding to each event in the above gaze area identification framework. For example, the candidate gaze area A is an event in the target gaze area, and the candidate gaze area A corresponds to The fusion result A of is the value corresponding to the event.

In this embodiment, by fusing multiple calibration results corresponding to each candidate gaze area, a gaze area identification framework can be accurately constructed.

In one embodiment, as shown in FIG. 4 , the following steps are included: determining a target gaze area from candidate gaze areas based on a gaze area identification framework.

Step S401 , based on the gaze area identification framework, obtain identification results corresponding to each candidate gaze area.

Wherein, the recognition result is the probability that the candidate gaze area is the target gaze area.

Specifically, based on the above-mentioned gaze area identification framework, a dark set of the above-mentioned gaze area identification framework, that is, a set of all subsets of the above-mentioned gaze area identification frame, is obtained. For example, if the existing candidate fixation regions A, B, and C have corresponding fusion results of a, b, and c, respectively, the fixation region recognition framework is the set {a, b, c}, and the dark set of the recognition framework is {a, b, c, {a, b}, {a, c}, {c, d}, {a, b, c}}, then the recognition result corresponding to the candidate fixation area is, in the corresponding dark set, the candidate fixation The sum of the probabilities of all related events in the area, for example, event A is the candidate gaze area A is the target gaze area, Bel(A) represents the sum of the probabilities of all related events in event A, which is the total trust degree of event A, that is, the candidate Look at the recognition result corresponding to area A.

Step S402, if the value of the recognition result corresponding to the current candidate fixation area is the maximum value among all the values of the recognition results, and the value of the recognition result corresponding to the current candidate fixation area is greater than a preset threshold, then determine that the current candidate fixation area is the target fixation area.

Specifically, the maximum value in the above identification results is selected. If the maximum value is greater than a preset threshold (for example, it may be 0.5), the candidate gaze area corresponding to the maximum value is the target gaze area. If the maximum value is not greater than the preset threshold Threshold, the recognition result is that there is no target gaze area.

In this embodiment, the target gaze area can be accurately identified through the gaze area identification framework.

In one embodiment, obtaining eye movement data includes the following steps:

A human eye image captured with human eyes is acquired, and the pupil center coordinates on the human eye image are acquired, and the center coordinates of the corneal reflection spot on the human eye image are acquired.

Among them, the human eye image is the preprocessed image of the human eye, and the pupil center coordinates are the pixel coordinates of the human eye pupil center in the human eye image. As for the center coordinates of the corneal reflection spot, it is the corneal reflection spot center in the human eye image The pixel coordinates of , where the center of the corneal reflection spot is the reflection spot of the light source on the human cornea. Generally, there are multiple reflection spots corresponding to multiple light sources, and the shapes of these spots are generally different.

Specifically, the human eye is photographed by an infrared camera device to obtain a human eye image, and then the human eye image is binarized, and then the Gaussian function is used to remove the noise of the human eye image, and the human eye image is extracted by the human eye feature extraction. The features are extracted to obtain the center coordinates of the pupil and the center coordinates of the corneal reflection spot.

Based on the pupil center coordinates and the corneal reflection spot center coordinates, the line of sight direction of the human eye is obtained.

The line of sight direction of the human eye is the gaze direction of the human eye.

Specifically, based on the pupil center coordinates and the corneal reflection spot center coordinates, the initial direction vector of the line of sight of the human eye is obtained, and the initial direction vector is obtained in the pixel coordinate system of the human eye image. Through the corresponding coordinate conversion function, the The initial direction vector is converted into the direction vector in the world coordinate system, and based on the direction vector, the line of sight direction of the human eye is obtained.

Based on the gaze direction, the gaze point coordinates of each gaze point of the human eye on the interface are obtained.

Wherein, the gaze point is the point at which the human eye gazes on the above-mentioned interface, and the coordinates of the gaze point are the coordinates of the gaze point on the interface.

Specifically, the coordinates of the intersection point of the line where the line of sight is located and the above-mentioned interface are taken as the coordinates of the gaze point on the interface.

The dwell time of each fixation point and the distance between each fixation point and the next fixation point corresponding to each fixation point are obtained; the fixation point coordinates, dwell time and distance are used as eye movement data.

Among them, the dwell time is the time that the human eye stays at a certain fixation point, and the distance is the distance between the two fixation points when the human eye's line of sight jumps from a certain fixation point to the next fixation point.

Specifically, the eye movement data can be represented in various ways. Here, the coordinates of the gaze point, the dwell time and the distance are selected as the eye movement data.

In this embodiment, the gaze direction of the human eye is obtained through the center coordinates of the pupil and the center coordinates of the corneal reflection spot, and then the eye movement data can be accurately obtained through the gaze direction.

In one embodiment, acquiring the coordinates of the pupil center on the image of the human eye includes the following steps:

Obtain the pupil area in the image of the human eye.

Among them, the pupil area is the area where the pupil is located in the human eye image, which is an elliptical boundary.

Specifically, the pupil region is extracted through a human eye image feature extraction tool.

Obtain the boundary point coordinates of a preset number of boundary points on the boundary of the pupil area.

The preset number of boundary points are pre-set 6 points on the boundary of the pupil ellipse area, and the coordinates of the boundary points are the pixel coordinates of the boundary points on the human eye image.

Specifically, the pixel coordinates of the above-mentioned boundary points are extracted through a human eye image feature extraction tool.

Based on the boundary point coordinates, the pupil center coordinates are obtained.

Specifically, the following is the boundary curve fitting formula of the pupil ellipse region:

Ax ² +Bxy+Cy ² +Dx+Ey+F=0;

By substituting the coordinates of the above six points into the above formula, the values of A, B, C, D, and E can be solved, thereby solving the above formula, and then through the ellipse center coordinate formula, the pupil center coordinate can be obtained.

In this embodiment, the coordinates of the pupil center can be accurately obtained by using the boundary curve fitting formula of the pupil ellipse region.

In one embodiment, acquiring the center coordinates of the corneal reflection spot on the human eye image includes the following steps: filtering the human eye image to obtain multiple corneal reflection spot images; acquiring multiple corneal reflection spot images corresponding to the multiple corneal reflection spot images The center coordinates of the reflected light spot image; the center coordinates of the multiple corneal reflected light spot images are fused to obtain the center coordinates of the corneal reflected light spot.

Specifically, the human eye image is filtered through the human eye image feature extraction tool, leaving only multiple corneal reflection spot images, and then the center coordinates of multiple corneal reflection spot images are obtained through the human eye image feature extraction tool, and the average value of these coordinates is obtained. , to obtain the coordinates of the center of the corneal reflection spot.

In this embodiment, by acquiring the center coordinates of a plurality of corneal reflection light spot images, the center coordinates of the corneal reflection light spot can be accurately obtained.

In one embodiment, after the target gaze area is determined from the candidate gaze areas, the following step is further included: in response to a trigger request for the target gaze area, switching the interface to a target interface corresponding to the target gaze area.

The trigger request is a request to enter the target interface, and the target interface is the interface corresponding to the target gaze area. Specifically, the user clicks on the target gaze area in the interface to switch the interface to the target interface corresponding to the target gaze area.

In this embodiment, through the trigger request for the target gaze area, it is possible to accurately enter the target interface corresponding to the target gaze area.

In one embodiment, the above method further includes the following steps: setting the display brightness of the target gaze area on the interface to be greater than the display brightness of the non-target gaze area; the non-target gaze area is a plurality of areas other than the target gaze area Area.

Wherein, the display brightness is the brightness of each area on the above-mentioned interface, and the non-target gaze area is an area other than the target gaze area among the multiple areas. Specifically, the target gaze area is highlighted, so that the display brightness of the target gaze area on the interface is greater than the display brightness of the non-target gaze area.

In this embodiment, by highlighting the target gaze area, the user can accurately find the click target gaze area.

In one embodiment, a method for determining the area of an interface business module of a bank intelligent terminal system is also provided. By collecting the user's eye movement data during business operations, corresponding preprocessing and feature extraction are performed, and then based on a binary classification algorithm The progressive gradient regression numerical model performs cognitive calculations such as interactive intention reasoning, and at the same time, the classification results are optimally selected, and displayed as visual instructions on the interactive interface to achieve more accurate prompts for customer business processing. The above method includes the following steps:

1. Eye movement data collection, the eye movement data is obtained by the method based on the pupil center-cornea reflection point (PCCR), and the camera device with infrared light is built in the self-service machine to shoot the human eye image, and obtain the human eye image. The image of the human eye is then binarized to obtain the result of the binarized image processing, and the result is filtered by a Gaussian function to achieve the purpose of removing the noise of the human eye image. At this point, the eye image is obtained, and the next step is to extract the features of the human eye image to obtain the center of the pupil of the human eye and the center of the infrared light reflection spot (also known as the Purchin spot), and calculate the PCCR vector formed by the two, and perform the target position. Calibration, establishing the mapping relationship between the human eye and the interface, and finally calculating by fitting.

1.1. Pupil center extraction, the extraction of the pupil geometry in this paper is based on the pupil area and the pupil geometry. Coarse positioning is performed on the pupil geometry first, and then the pupil geometry after the rough positioning is processed for precise positioning.

The pupil is not a regular circle, but an ellipse, so an ellipse is used for fitting, and the fitting curve is as follows:

Ax ² +Bxy+Cy ² +Dx+Ey+F=0;

Take any six points in the pupil, substitute them into the above formula, calculate the coefficients of the ellipse A, B, C, D, E, and F by the least square method, fit the exit pupil curve, and then obtain the center of the ellipse with the following formula :

1.2. Purkin spot center extraction, extracting the coordinates of the center point of the Purkin spot is a key part. The line of sight direction is determined by the position of the pupil center in the human eye image and the coordinate position of the center point of the Purkin spot, including two main stages, namely the pupil The image preprocessing and the coordinates of the center point of the Purchen patch are used to obtain the Purchen patch geometry, and then the center of the geometry is obtained. The geometric figure center data is obtained by the following formula operations, where x _f and y _f are the center coordinates:

1.3. PCCR vector fitting, PCCR vector is obtained from the pupil center data and Purchin's spot center data obtained by the above calculation, and then the PCCR vector fitting data is obtained according to the image calibration point of the virtual scene. In the field of eye tracking research, the calibration point coordinates are usually used to fit the PCCR vector, and the calibration is used to obtain the correspondence between the system fixation point coordinates and the actual fixation point coordinates.

2. Eye movement feature extraction, select three eye movement features of fixation point coordinates, fixation point duration and saccade to analyze eye movement data.

3. The calculation of the classification result of the eye movement data two-classification model, in the present invention, the gradient boosting iterative decision tree is used to classify the eye movement gaze pattern of the user when operating the self-service machine, and the classification algorithm model is established as follows:

3.1. Initialize the learner:

3.2. Establish M classification and regression trees:

Calculate the corresponding response value of the mth tree:

For the leaf node area, bring the formula to calculate the best fit value:

Based on the best fit value, the strong learner can be updated:

The expression to get the final strong learner:

Its classification model can be expressed as:

Input the above eye movement feature into the above two-class model, the corresponding expected output is 1 or 2, when the expected output is 1, it means that the user of the self-service machine module does not need to select the result, and when the expected output is 2, it means that the user of the self-service machine module The result that needs to be selected.

4. Multi-module optimal calculation, obtain the above-mentioned intelligent terminal operation user-required module operation set, assign BPA to the classifier through the identification framework θ of D-S evidence theory reasoning, and obtain the final judgment result.

Input the above-mentioned eye movement features into the above-mentioned two-class model, and obtain the decision results Z _p of the above-mentioned two-class model as Z _p1 and Z _p2 , and use Z _p as the input of the decision-making layer. For any test sample Z _p , estimate its class probability :

Among them, f is the output function of the classifier, and the parameters λ and B can be obtained by calculating the log-likelihood function that minimizes the training sample and the output function f of the above two-class model, and solves the optimization problem of the following two equations to obtain the category Probability p _i :

Among them, t represents the t-th target category, and s represents the s-th above-mentioned two-category model;

According to the error upper bound theorem, the BPA is allocated to the classifier through the identification framework θ of D-S evidence theory inference, and the final decision result is obtained:

m _s (Θ)=P _ε ;

m _s (C _t )=P _t ^s (1-P _ε );

5. Confirm the corresponding area in the visual interactive interface, apply the eye movement data to the banking self-service machine system, design the system function interface for the user's self-service operation, and carry out the module functions that the user may need through the eye movement data of the user's operation of the self-service machine. hint. As shown in Figure 6-1, the system interface displays multiple business function modules. By acquiring the user's eye movement data, and calculating the time that the user does not operate the interface in real time, if the user does not operate the self-service machine for more than 5 seconds, the five The user's eye movement data collected within seconds is classified and judged (as shown in Figure 5), and the module required by the user is selected to be highlighted and displayed (as shown in Figure 6-2).

It should be understood that, although the steps in the flowcharts involved in the above embodiments are sequentially displayed according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but may be performed at different times The execution order of these steps or phases is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or phases in the other steps.

Based on the same inventive concept, an embodiment of the present application further provides an area determination device for implementing the above-mentioned area determination method. The implementation solution for solving the problem provided by the device is similar to the implementation solution described in the above method, so the specific limitations in one or more area determination device embodiments provided below can refer to the above limitations on the area determination method, It is not repeated here.

In one embodiment, as shown in FIG. 7 , a region determination device is provided, comprising: an eye movement data acquisition module 701, an estimated result acquisition module 702, a candidate gaze region determination module 703 and a target gaze region determination module 704, in:

The eye movement data acquisition module 701 is used for acquiring eye movement data; the eye movement data is used to represent the change of the fixation point of the human eye on the fixation interface, and the interface includes a plurality of regions;

The estimation result acquisition module 702 is used for inputting the eye movement data into a plurality of pre-built eye movement data estimation models, and through the plurality of eye movement data estimation models, each region is obtained for a plurality of eye movement data estimation models multiple prediction results; multiple eye movement data prediction models correspond to multiple regions one-to-one, and multiple prediction results are used to characterize the probability that each region is the gaze region of the human eye;

The candidate gaze area judgment module 703 is used to use the estimated result corresponding to the eye movement data prediction model corresponding to each area as the target estimated result; based on the target estimated result, select the candidate gaze area from each area;

The target gaze area determination module 704 is configured to construct a gaze area identification framework based on multiple prediction results corresponding to the candidate gaze areas; and based on the gaze area identification framework, determine the target gaze area from the candidate gaze areas.

In one embodiment, the candidate gaze region determination module 703 is further configured to obtain the final estimation result corresponding to the current region based on the target estimation result; the final estimation result is the first preset value or the second preset value; If the final estimation result of the current area is the first preset value, the current area is determined as a candidate gaze area.

In one embodiment, the target gaze area determination module 704 is further configured to calibrate the multiple prediction results corresponding to each candidate gaze area through a calibration method constructed based on a non-parametric method to obtain multiple A calibration result is obtained; multiple calibration results corresponding to each candidate fixation area are fused to obtain a fusion result corresponding to each candidate fixation area; based on the fusion results corresponding to each candidate fixation area, a fixation area recognition framework is constructed.

In one embodiment, the target gaze area determination module 704 is further configured to obtain the recognition result corresponding to each candidate gaze area based on the gaze area identification framework; if the value of the recognition result corresponding to the current candidate gaze area is the value of all the recognition results and the value of the recognition result corresponding to the current candidate gaze area is greater than the preset threshold, then it is determined that the current candidate gaze area is the target gaze area.

In one embodiment, the eye movement data acquisition module 701 is further configured to acquire an image of a human eye, and acquire the center coordinates of the pupil on the image of the human eye, and acquire the center coordinates of the corneal reflection spot on the image of the human eye; Based on the pupil center coordinates and the center coordinates of the corneal reflection spot, the line of sight direction of the human eye is obtained; based on the line of sight direction, the fixation point coordinates of each fixation point of the human eye on the interface are obtained; the dwell time of each fixation point and the fixation point of each fixation point are obtained The distance of the next fixation point corresponding to each fixation point; the fixation point coordinates, dwell time and distance are used as eye movement data.

In one embodiment, the eye movement data acquisition module 701 is further configured to acquire the pupil area in the human eye image; acquire the boundary point coordinates of a preset number of boundary points on the boundary of the pupil area; based on the boundary point coordinates, obtain the pupil Center coordinates.

In one embodiment, the eye movement data acquisition module 701 is further configured to filter the image of the human eye to obtain multiple corneal reflection spot images; obtain the center coordinates of the multiple corneal reflection spot images corresponding to the multiple corneal reflection spot images ; Integrate the center coordinates of multiple corneal reflection spot images to obtain the center coordinates of the corneal reflection spot.

In one embodiment, the target gaze area determination module 704 is further configured to switch the interface to the target interface corresponding to the target gaze area in response to a trigger request for the target gaze area.

In one embodiment, the target gaze area determination module 704 is further configured to set the display brightness of the target gaze area on the interface to be greater than the display brightness of the non-target gaze area; Area outside the gaze area.

Each module in the above-mentioned area determination apparatus may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 8 . The computer equipment includes a processor, memory, a communication interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer equipment is used for wired or wireless communication with an external terminal, and the wireless communication can be realized by WIFI, mobile cellular network, NFC (Near Field Communication) or other technologies. The computer program, when executed by a processor, implements an area determination method. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the foregoing method embodiments.

In one embodiment, a computer program product is provided, including a computer program, which implements the steps in each of the foregoing method embodiments when the computer program is executed by a processor.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all Information and data authorized by the user or fully authorized by the parties.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to a memory, a database or other media used in the various embodiments provided in this application may include at least one of a non-volatile memory and a volatile memory. Non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random Memory) Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The database involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the present application should be determined by the appended claims.

Claims (13)

1. A method for determining a region, the method comprising:

acquiring eye movement data; the eye movement data is used for representing the change of a gazing point of human eyes on a gazing interface, and the interface comprises a plurality of areas;

inputting the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtaining a plurality of estimation results of each region aiming at the eye movement data estimation models through the eye movement data estimation models; the eye movement data estimation models correspond to the regions one by one, and the estimation results are used for representing the probability that each region is the watching region of the human eyes;

taking the estimation result corresponding to the eye movement data estimation model corresponding to each region as a target estimation result; screening out candidate gazing areas from the areas based on the target estimation result;

constructing a gaze region identification framework based on a plurality of estimation results corresponding to the candidate gaze region; and determining a target gazing area from the candidate gazing area based on the gazing area identification framework.

2. The method of claim 1, wherein the screening candidate regions of gaze from the respective regions based on the target predictor comprises:

obtaining a final estimation result corresponding to the current area based on the target estimation result; the final estimated result is a first preset value or a second preset value;

and if the final estimation result of the current area is the first preset value, judging the current area as the candidate gazing area.

3. The method of claim 2, wherein constructing a gaze region identification framework based on a plurality of estimates corresponding to the candidate gaze regions comprises:

calibrating a plurality of estimated results corresponding to each candidate gazing area through a calibration method constructed based on a nonparametric method to obtain a plurality of calibration results corresponding to each candidate gazing area;

fusing a plurality of calibration results corresponding to each candidate watching area to obtain a fusion result corresponding to each candidate watching area;

and constructing the gaze region identification framework based on the fusion result corresponding to each candidate gaze region.

4. The method of claim 3, wherein said determining a target gaze region from said candidate gaze region based on said gaze region identification framework comprises:

obtaining an identification result corresponding to each candidate watching area based on the watching area identification frame;

and if the value of the identification result corresponding to the current candidate watching area is the maximum value of all the values of the identification result, and the value of the identification result corresponding to the current candidate watching area is greater than a preset threshold value, judging that the current candidate watching area is the target watching area.

5. The method of claim 1, wherein the acquiring eye movement data comprises:

acquiring a human eye image shot with the human eyes, acquiring pupil center coordinates on the human eye image, and acquiring cornea reflection light spot center coordinates on the human eye image;

obtaining the sight line direction of the human eyes based on the pupil center coordinates and the cornea reflection light spot center coordinates;

based on the sight line direction, obtaining fixation point coordinates of each fixation point of the human eyes on the interface;

obtaining the staying time of each fixation point and the distance between each fixation point and the next fixation point corresponding to each fixation point; and taking the fixation point coordinate, the stay time and the distance as the eye movement data.

6. The method of claim 5, wherein the obtaining pupil center coordinates on the image of the human eye comprises:

acquiring a pupil area in the human eye image;

acquiring boundary point coordinates of a preset number of boundary points on the boundary of the pupil area;

and obtaining the pupil center coordinate based on the boundary point coordinate.

7. The method of claim 6, wherein the obtaining of the corneal reflection spot center coordinates on the image of the human eye comprises:

filtering the human eye image to obtain a plurality of cornea reflection light spot images;

obtaining central coordinates of a plurality of cornea reflection light spot images corresponding to the plurality of cornea reflection light spot images;

and fusing the central coordinates of the plurality of cornea reflection light spot images to obtain the central coordinates of the cornea reflection light spots.

8. The method of claim 1, wherein after determining the target gaze region from the candidate gaze region, further comprising:

and responding to a trigger request aiming at the target watching area, and switching the interface to a target interface corresponding to the target watching area.

9. The method of claim 8, further comprising:

setting the display brightness of the target gazing area on the interface to be greater than the display brightness of a non-target gazing area; the non-target gazing area is an area other than the target gazing area among the plurality of areas.

10. An area determination apparatus, characterized in that the apparatus comprises:

the eye movement data acquisition module is used for acquiring eye movement data; the eye movement data is used for representing the change of a gazing point of human eyes on a gazing interface, and the interface comprises a plurality of areas;

the estimation result acquisition module is used for inputting the eye movement data into a plurality of pre-constructed eye movement data estimation models, and obtaining a plurality of estimation results of each area aiming at the eye movement data estimation models through the eye movement data estimation models; the eye movement data estimation models correspond to the regions one by one, and the estimation results are used for representing the probability that each region is the watching region of the human eyes;

the candidate staring area judging module is used for taking the estimation result corresponding to the eye movement data estimation model corresponding to each area as a target estimation result; screening out candidate gazing areas from the areas based on the target estimation result;

the target gazing area judging module is used for constructing a gazing area identification frame based on a plurality of estimated results corresponding to the candidate gazing areas; and determining a target gazing area from the candidate gazing area based on the gazing area identification framework.

11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method of any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 9 when executed by a processor.

Images