WO2023231663A1

WO2023231663A1 - Eye type detection method and apparatus, computer device, storage medium and computer program product

Info

Publication number: WO2023231663A1
Application number: PCT/CN2023/091183
Authority: WO
Inventors: 马文辉; 吴阳平; 许亮
Original assignee: 上海商汤智能科技有限公司
Priority date: 2022-05-31
Filing date: 2023-04-27
Publication date: 2023-12-07
Also published as: CN114903424A

Abstract

Disclosed are an eye type detection method and an apparatus, a computer device, a storage medium and a computer program product. The method comprises: in a case of determining that a target object faces an image acquisition device, acquiring a front face image of the target object acquired by the image acquisition device; determining a pupil position of the left eye and a pupil position of the right eye of the target object in the front face image, respectively; determining, according to the pupil position of the left eye, a gaze angle of the left eye, and determining, according to the pupil position of the right eye, a gaze angle of the right eye of the target object; determining an angle difference between the gaze angle of the left eye and the gaze angle of the right eye; in a case that the angle difference exceeds a first predetermined angle range, determining a larger one of absolute values of the gaze angles of the left eye and the right eye as a strabismus eye in the front face image; and according to the strabismus eye in the front face image, calibrating an eye strabismus type of the target object. According to the embodiment of the present disclosure, the eye strabismus type of the target object can be conveniently and accurately detected in an image processing mode.

Description

An eye type detection method, device, computer equipment, storage medium and computer program product

Cross-references to related applications

This disclosed embodiment is based on a Chinese patent application with application number 202210615649.9, application date is May 31, 2022, and the application name is "Eye type detection method and device, computer equipment, storage medium", and requires that the Chinese patent application Priority, the entire content of this Chinese patent application is hereby incorporated by reference into this disclosure.

Technical field

The present disclosure relates to but is not limited to the field of artificial intelligence, and in particular, to an eye type detection method, device, computer equipment, storage medium and computer program product.

Background technique

Gaze usually represents the content of human attention or the intention of interaction. Currently, artificial intelligence technology can support line-of-sight-based interaction scenarios, such as in smart home scenes, smart mobile electronic devices, and smart driving scenarios, it supports line-of-sight detection of areas or objects that people are interested in, or line-of-sight control for air-to-air control.

For example, in a driving scene, the Driver Monitor System (DMS) will determine parameters such as eye gaze direction or gaze area based on the detection of the driver's eyes to determine whether the driver is distracted.

However, for people with strabismus, because their eyesight directions are inconsistent, current technology is difficult to determine the user's true gaze. Therefore, eye type calibration for people with strabismus is of great significance.

Contents of the invention

The present disclosure provides an eye type detection method, device, computer equipment, storage medium and computer program product.

In a first aspect, an embodiment of the present disclosure provides an eye type detection method, which method includes:

When it is determined that the target object is facing the image acquisition device, obtain the front face image of the target object collected by the image acquisition device;

Determine respectively the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image;

Determine the gaze angle of the left eye based on the pupil position of the left eye, and determine the gaze angle of the right eye of the target object based on the pupil position of the right eye;

determining an angular difference between the gaze angle of the left eye and the gaze angle of the right eye;

When the angle difference exceeds the first predetermined angle range, determine the one with the larger absolute value of the gaze angle of the left eye and the right eye as the strabismus eye in the front face image;

The eye strabismus type of the target object is calibrated according to the strabismus eye in the front face image.

In a second aspect, an embodiment of the present disclosure provides an eye type detection device, which includes:

The first acquisition module is configured to acquire the front face image of the target object collected by the image acquisition device when it is determined that the target object is facing the image acquisition device;

A first determination module configured to respectively determine the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image;

a second determination module configured to determine the gaze angle of the left eye based on the pupil position of the left eye, and determine the gaze angle of the right eye of the target object based on the pupil position of the right eye;

a third determination module configured to determine the angle difference between the gaze angle of the left eye and the gaze angle of the right eye;

The fourth determination module is configured to determine the one with a larger absolute value of the gaze angles of the left eye and the right eye as the front face when the angle difference exceeds the first predetermined angle range. squinting eyes in the image;

A calibration module configured to calibrate the eye strabismus type of the target object according to the strabismus eye in the front face image.

In a third aspect, an embodiment of the present disclosure provides a computer device, including: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the above eye type detection method.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above eye type detection method is implemented.

In a fifth aspect, embodiments of the present disclosure provide a computer program product, the computer program product comprising a computer program, when the computer program is run on a computer device, causing the computer device to perform the eye type as described above Steps in the detection method.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

In an embodiment of the present disclosure, the angle difference between the gaze angle of the left eye and the gaze angle of the right eye is determined based on the pupil position of the target object's left eye and the pupil position of the right eye in the front face image. When the angle difference exceeds the third In the case of a predetermined angle range, the one with the larger absolute value of the gaze angle of the left eye and the right eye is determined as the strabismus eye in the front face image, and then the target object's eye strabismus type is calibrated. Compared with using cover inspection method, strabismus angle measurement method, or using a detection instrument for strabismus to detect strabismus. The embodiment of the present disclosure can conveniently and accurately detect the strabismus type of the target object's eyes through image processing.

It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and do not limit the present disclosure.

Description of the drawings

Figure 1 is a flowchart 1 of the implementation of an eye type detection method provided by an embodiment of the present disclosure;

Figure 2 is an example diagram of the gaze angle of the left eye and the gaze angle of the right eye in a camera coordinate system provided by an embodiment of the present disclosure;

Figure 3 is a flowchart 2 of the implementation of an eye type detection method provided by an embodiment of the present disclosure;

Figure 4 is a flowchart 3 of the implementation of an eye type detection method provided by an embodiment of the present disclosure;

Figure 5 is a schematic structural diagram of an eye type detection device provided by an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a hardware entity of a computer device provided by an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. The following description refers to the attached In the drawings, the same numbers in the different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with aspects of the disclosure as detailed in the appended claims.

Embodiments of the present disclosure provide an eye type detection method, the execution subject of which may be an eye type detection device. For example, the eye type detection method may be executed by a terminal device or a server or other electronic device, where the terminal device may be a user device (User). Equipment, UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. In some embodiments, the eye type detection method may be implemented by a processor calling computer readable instructions stored in a memory.

In embodiments of the present disclosure, the eye type detection device may include an image acquisition device, thereby obtaining an image using the image acquisition device. In addition, the eye type detection device can also receive images sent from other image acquisition devices.

Figure 1 is an implementation flowchart 1 of an eye type detection method provided by an embodiment of the present disclosure. As can be seen from Figure 1, the method includes the following steps S11 to S16:

Step S11: When it is determined that the target object is facing the image acquisition device, obtain the front face image of the target object collected by the image acquisition device;

Step S12: Determine the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image respectively;

Step S13: Determine the gaze angle of the left eye according to the pupil position of the left eye, and determine the gaze angle of the right eye of the target object according to the pupil position of the right eye;

Step S14: Determine the angle difference between the gaze angle of the left eye and the gaze angle of the right eye;

Step S15: When the angle difference exceeds the first predetermined angle range, determine the one with the larger absolute value of the gaze angle of the left eye and the right eye as the strabismus in the front face image. Eye;

Step S16: Calibrate the eye strabismus type of the target object based on the strabismus eye in the front face image.

In the embodiment of the present disclosure, step S11 determines that the target object is facing the image acquisition device (such as a camera). It may be to output a prompt message requesting the camera to face the target object and then determine that the target object is facing the image acquisition device; or to determine the target in a specific scene. The object is facing the image acquisition device, such as the image acquisition device installed in the cabin facing the driver's seat. In the scene of the vehicle driving in a straight line at high speed, it is determined that the driver is facing the image acquisition device; or during the online consultation process, it is determined that the doctor is facing the computer. Camera. Therefore, when it is determined that the target object is facing the image acquisition device, a front face image of the target object facing the image acquisition device collected by the image acquisition device is obtained.

The front face image obtained by the present disclosure of the target object may be a front face image filtered out based on the facial features in the extracted image among the images collected by the image acquisition device. For example, the face rotation angle can be determined based on the facial features, and the image with the face rotation angle smaller than the threshold can be used as the front face image; it can also be determined based on the facial features, whether the nose is on the central axis of the face outline, and the nose is placed on the central axis of the face. The image on the central axis of the facial contour is used as the front face image.

In this disclosure, after acquiring the front face image of the target object, step S12 is executed to respectively determine the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image. Among them, the pupil position (including the pupil position of the left eye and the pupil position of the right eye) can be the position of the pupil outline in the image coordinate system of the frontal image, or it can be the pupil center The position in the image coordinate system of the front face image; the image coordinate system can be a coordinate system in pixels with the upper left corner of the image as the origin. The abscissa and ordinate of the pixel are the number of columns in its image array respectively. and the row number, the image coordinate system is a two-dimensional rectangular coordinate system.

If the pupil position is the position of the pupil outline in the image coordinate system of the frontal image, in some embodiments, the image can be converted into a grayscale image and binarized on the image, based on the binarized image Determine the pupil outline and obtain the pupil position.

If the pupil position is the position of the pupil center in the image coordinate system of the frontal image, in some embodiments, after the pupil outline is determined by the above method, the centroid method can be used to determine the position of the pupil center in the pupil outline.

In some embodiments, after the pupil contour is obtained through the above method, the average value of the sum of all directions can also be calculated based on the sum of the displacement vectors in all directions and the gradient direction of each pixel within the pupil range, and the pupil range can be The coordinates of the point corresponding to the maximum average of the sums of all pixel points are used as the position of the pupil center.

In this disclosure, after determining the pupil position of the left eye and the pupil position of the right eye, step S13 is performed to determine the gaze angle of the left eye based on the pupil position of the left eye, and determine the gaze angle of the right eye based on the pupil position of the right eye. Among them, the gaze angle represents the angle between the gaze direction of the left eye or the right eye and the normal direction of the imaging surface of the image acquisition device. The gaze angle can be positive or negative. For example, the gaze direction is located in the normal direction of the imaging surface of the image acquisition device. On the left, the gaze angle is a negative value; the gaze direction is on the right side of the normal direction of the imaging surface of the image acquisition device, the gaze angle is a positive value, etc.

The gaze angle of the present disclosure may be an angle in the image coordinate system, the angle in the image acquisition device coordinate system, or the angle in the world coordinate system, and the disclosure does not limit this. Among them, the coordinate system of the image acquisition device is a three-dimensional rectangular coordinate system established with the focus center of the image acquisition device as the origin and the optical axis as the Z-axis.

Taking the pupil position as the position of the pupil outline in the image coordinate system of the front face image as an example, the present disclosure can use the feature point detection method to extract the eye contour features in the front face image. According to the relative position of the pupil outline and the eye outline, Determine the gaze angle of the eye in the image coordinate system. Among them, feature point detection methods include but are not limited to any of the following methods: local-based methods, global-based methods, hybrid-based methods, Active Shape Model (ASM) and Active Appearance Model (Active Appearance Model). , AAM).

Illustratively, the present disclosure determines a first distance between the leftmost end of the pupil outline and the leftmost end of the eye outline and/or a second distance between the rightmost end of the pupil outline and the rightmost end of the eye outline based on the relative position of the pupil outline and the eye outline. Distance; determine the gaze angle of the eye based on the correspondence between the predetermined first distance and/or the second distance and the gaze angle. For example, under normal circumstances, the width of the eyes is about 30 mm, and the horizontal gaze angle of one eye is between [-75, 75] degrees (the gaze angle when looking straight ahead is set to 0° (degrees)), so the first A distance of 15 mm corresponds to a gaze angle of 0 degrees, a first distance of 5 mm corresponds to a gaze angle of -50 degrees (i.e. 50 degrees to the left), and a first distance of 20 mm corresponds to a gaze angle of 50 degrees (i.e. to the left). 50 degrees to the right).

Taking the pupil position as the position of the pupil center in the image coordinate system of the front-face image as an example, the present disclosure can use the aforementioned feature point detection method to extract the eye contour features in the front-face image. According to the position of the pupil center in the eye contour area Relative position, determine the gaze angle of the eye in the image coordinate system corresponding to the relative position.

For example, determine a third distance from the pupil center to the leftmost end of the eye contour and/or a fourth distance from the pupil center to the rightmost end of the eye contour based on the relative position of the pupil center within the eye contour area; based on the predetermined third distance and / or The fourth correspondence relationship between distance and gaze angle determines the gaze angle of the eyes.

After the present disclosure determines the gaze angle of the eye in the image coordinate system according to the pupil position, in some embodiments, the gaze angle of the eye in the coordinate system of the image acquisition device can be determined based on the internal parameters of the image acquisition device. In some embodiments, after determining the gaze angle in the coordinate system of the image acquisition device, the gaze angle of the eye in the world coordinate system may be determined based on external parameters of the image acquisition device.

In some embodiments, when determining the gaze angle of the corresponding eye according to the pupil position, the front face image can also be input into the gaze angle detection model to obtain the gaze angle of the left eye and the gaze angle of the right eye; wherein, the gaze angle detection model It can be trained based on deep learning networks, for example, based on training sample data and label values, training and parameter adjustment of networks such as convolutional neural networks (Convolutional Neural Networks, CNN), deep learning networks (Deep Neural Networks, DNN), etc. Finally, the gaze angle detection model is obtained; where the label value is the predetermined gaze angle. The present disclosure is based on a trained gaze angle detection model. By inputting the front face image of the target object into the model, the gaze angle of the left eye and the gaze angle of the right eye can be obtained.

According to the above embodiments, the gaze angle of the left eye can be determined according to the pupil position of the left eye, and the gaze angle of the right eye can be determined according to the pupil position of the right eye, but the present disclosure is not limited to the above method.

In this disclosure, after determining the gaze angle of the left eye and the gaze angle of the right eye, step S14 is performed to determine the angle difference between the gaze angle of the left eye and the gaze angle of the right eye. The angle difference may be the difference between the gaze angle of the left eye and the gaze angle of the right eye, or may be the result of a weighted calculation of the gaze angle of the left eye and the gaze angle of the right eye.

Because the gaze directions of the eyes of people with strabismus are different, for example, under normal circumstances, the gaze angles of the eyes are parallel, but the gaze angles of the eyes of people with strabismus are quite different. Based on the physiological characteristics of strabismus people, step S15 of the present disclosure determines the one with the larger absolute value of the gaze angle of the left eye and the right eye as the strabismus eye in the front face image when the angle difference exceeds the first predetermined angle range. . Among them, the angle difference is the difference between the gaze angle of the left eye and the gaze angle of the right eye. If the angle difference is a negative value, correspondingly, if the angle difference exceeds the first predetermined angle range, it can be less than the predetermined negative value; if the angle difference is a positive value, , correspondingly, the first predetermined angle range may be greater than the predetermined positive value, and the present disclosure does not limit the specific content of the first predetermined angle range.

After the strabismus eye in the front face image is determined, step S16 is performed to calibrate the strabismus type of the target object's eyes based on the determined strabismus eye, where the eye corner strabismus type includes at least one of the following: non-strabismus in both eyes, strabismus in the left eye, and strabismus in the right eye. . Specifically, the eye strabismus type of the target object can be calibrated directly based on the strabismus in a single front face image. Alternatively, after it is determined that the same eye of the target object is a strabismus in multiple consecutively collected front face images, the strabismus type of the target object's eyes can be calibrated.

Figure 2 is an example diagram of the gaze angle of the left eye and the gaze angle of the right eye in a camera coordinate system provided by an embodiment of the present disclosure. In Figure 2, 21 is a camera, corresponding to the aforementioned image acquisition device, and 22 is the left eye of the target object. The pupil of The gaze angle of the normal line of the camera's imaging surface corresponds to the aforementioned gaze angle of the right eye, and β is greater than α. According to the present disclosure, the angle difference is determined as β-α according to the gaze angle of the left eye and the right eye; when the angle difference exceeds the first predetermined angle range, the right eye corresponding to β is determined The eye is determined to be strabismus, thereby determining that the target object's eye strabismus type is right eye strabismus.

For example, the normal of the camera imaging plane in Figure 2 can be set to 0°, the gaze angle β of the right eye is 30°, the gaze angle α of the left eye is -10°, and the angle threshold θ in the first angle range of the predetermined angle is is 10°, and the angle difference is the difference between the absolute value of the gaze angle of the right eye and the gaze angle of the left eye, that is, 20°, which exceeds the angle threshold θ, so it can be determined that the right eye corresponding to β is a strabismus eye.

It can be understood that in the embodiment of the present disclosure, the angle difference between the gaze angle of the left eye and the gaze angle of the right eye is determined based on the pupil position of the left eye and the pupil position of the right eye in the front face image. In the angle difference When the first predetermined angle range is exceeded, the one with the larger absolute value of the gaze angle of the left eye and the right eye is determined to be a strabismus. Compared with using the cover inspection method, the strabismus angle measurement method, or using the method for strabismus The detection instrument detects strabismus. The embodiment of the present disclosure can conveniently and accurately detect the strabismus type of the target object's eyes through image processing.

In some embodiments, in a medical scenario, it is possible to determine whether a medical seeker has strabismus according to the eye type detection method of the present disclosure; in addition, the number of people with strabismus can also be counted based on the eye type detection results. In a driver's license physical examination scenario, whether the driver has strabismus can be determined based on the eye type detection method of the present disclosure. In a vehicle driving scene, the eye type detection method of the present disclosure can be used to determine whether the driver has strabismus. This can be based on the monitoring of the gaze of the non-strabismus eye (i.e., the normal eye), such as the gaze angle and the gaze angle. The degree of change, gaze duration, etc., determine whether the driver is distracted, thereby improving the accuracy of the driver monitoring system.

In related technologies, for example, in a vehicle driving scenario, the driver's eyes are normal to detect whether the driver is driving safely, such as judging whether the driver is driving safely based on the difference between the gaze angle of the eyes and the deflection angle of the face. Consider the situation where the driver is a person with strabismus. When a person with strabismus is the driver, it is easy to mistakenly detect the result of unsafe driving.

Therefore, it can be understood that in the present disclosure, when the angle difference between the gaze angle of the left eye and the gaze angle of the right eye exceeds the first predetermined angle range, the absolute value of the gaze angle of the left eye and the right eye is larger. One is determined to be a strabismus, and the target object's eye strabismus type is calibrated so that in vehicle driving scenarios, the driver can be detected to drive safely based on non-strabismus eyes, which can improve the accuracy of driver status assessment in a DMS system, for example.

In some embodiments, the frontal face image includes multiple frames of frontal face images in the video stream of the target object collected by the image acquisition device;

Calibrating the eye strabismus type of the target object based on the strabismus eye in the front face image includes:

In the multi-frame front face images, if the number of frames in which it is determined that the left eye is strabismus reaches a predetermined threshold, calibrate the eye type of the target object as left eye strabismus; or

In the multi-frame frontal face images, if the number of frames in which it is determined that the right eye is strabismus reaches a predetermined threshold, the eye type of the target object is calibrated as right eye strabismus.

In the embodiment of the present disclosure, the acquired frontal face image includes multiple frames of frontal face images in the video stream of the target object collected by the image acquisition device, wherein the multiple frames of frontal face images may be consecutive frames of frontal face images, or may be Multi-frame front face images obtained by sampling at certain time intervals in the video stream.

The present disclosure can count multiple frames when calibrating the eye strabismus type of the target object based on the strabismus in the front face image. The number of frames in the front face image in which the left eye is strabismus and the number of frames in the image in which the right eye is strabismus. When the number of frames of the image in which the left eye is determined to be strabismus reaches a predetermined threshold, the eye type of the target object is calibrated to be left eye strabismus; or, when the number of frames in the image in which the right eye is determined to be strabismus reaches the predetermined threshold, The eye type of the calibration target object is right eye strabismus. It can be understood that the number of frames of the image of the left eye with strabismus is positively correlated with the probability that the left eye is strabismus, and the number of frames of the image of the right eye with strabismus is positively correlated with the probability of the right eye with strabismus.

In some embodiments, each front-face image can obtain a corresponding strabismus detection result. The present disclosure detects multiple front-face images, and the multiple detection results obtained may include those whose left eye is strabismus. The results also include the result that the right eye is strabismus. The number of frames of images in which the left eye is a strabismus and the number of frames in which the right eye is a strabismus can be obtained by counting the corresponding multiple detection results. The larger the number of frames, the more likely it is that the eye corresponding to the number of frames is a strabismus. The bigger. Therefore, determining the eye corresponding to the number of frames reaching the predetermined threshold as a strabismus and calibrating the eye type of the target object accordingly can further improve the accuracy of determining the eye type.

In some embodiments, the frontal face image includes N frames of frontal face images in the video stream of the target object collected by the image acquisition device, where N is a predetermined positive integer;

The method also includes:

When the angle difference does not exceed the first predetermined angle range, determine that both eyes in the front face image are non-strabismus;

In the N frames of frontal face images, the first frame number of the image in which both eyes are determined to be non-strabismus, the second frame number of the image in which the left eye is determined to be a strabismus, and the third frame number of the image in which the right eye is determined to be strabismus are determined. Frame numbers are compared;

In the case where the first frame number is greater than the second frame number and the third frame number, calibrating the binocular type of the target object as binocular non-strabismus; or

When the first frame number is less than any one of the second frame number and the third frame number, calibrate the corresponding one of the second frame number and the third frame number. The strabismus eye in the front face image is the strabismus eye of the target object.

In this embodiment of the present disclosure, the frontal face image includes N frames of frontal face images obtained based on the foregoing method in the video stream of the target object collected by the image acquisition device, where N is a predetermined positive integer.

In the present disclosure, when the angle difference does not exceed the first predetermined angle range, that is, the difference in the gaze angles of the two eyes is small and does not conform to the gaze angle characteristics of the strabismus human eye, therefore, it is determined that both eyes in the front face image are non-strabismus.

In the present disclosure, in N frames of frontal face images, the first frame number of the image in which both eyes are non- strabismus is determined. It can be understood that the first frame number is positively related to the probability that both eyes are non-strabismus.

After determining the first frame number, combining the second frame number (the number of frames corresponding to the image of the left eye with strabismus) and the third frame number (corresponding to the number of frames of the image of the right eye with strabismus) obtained based on the aforementioned method, Compare the first frame number, the second frame number and the third frame number. When the first frame number is greater than the second frame number and the third frame number, that is, when the probability of both eyes being non-strabismus is the highest, Calibrate the target object's binocular type to be binocular non-strabismus; or, when the first frame number is less than either of the second frame number and the third frame number, that is, when the probability of strabismus exists is high, calibrate the third The strabismus eye in the front face image corresponding to the larger of the second frame number and the third frame number is the strabismus eye of the target object.

It can be understood that in order to reduce the misdetection of strabismus by normal eyes due to occasional strabismus or low image quality of the front face image (such as overexposure, low resolution), etc., in the present disclosure, in N frames of front face images, The first frame number of the image in which both eyes are determined to be non-squinting eyes, the second frame number of the image in which the left eye is determined to be strabismus, and the third frame number of the image in which the right eye is determined to be strabismus are compared, and the strabismus is calibrated based on the comparison results. , improve the accuracy of calibrating strabismus.

In some embodiments, the N frames of frontal face images are N frames of frontal face images whose timing difference between any two frames in the video stream does not exceed the preset duration, so that eye type detection is performed on the N frames of frontal face images within the preset duration. .

In the embodiment of the present disclosure, if the images of any two frames are images of the first and last frames, then the N frames of frontal images of the present disclosure are N frames of frontal images whose total timing does not exceed the preset duration.

For example, N is 10 and the preset duration is 15 minutes. In the vehicle driving scene, the detection of 10 frames of frontal images with a total duration of no more than 15 minutes is completed, and the first frame number, the second frame number and The third frame number is used to calibrate the eye type of the target object based on the comparison results of the first frame number, the second frame number and the third frame number. Among them, the preset time period can be started from the time when the vehicle starts and the first frame of the front face image is collected, or it can be started from the time under other circumstances (such as the vehicle speed is greater than the predetermined speed threshold) and the first frame of the front face image is collected. This disclosure does not do this. limited.

It can be understood that the larger N is in this disclosure, the higher the accuracy of the eye type calibration result. The setting of the preset duration needs to take into account the time required to complete the acquisition of N frames of frontal face images. On this basis, the shorter the preset duration, the higher the immediacy of the final determination of the eye type.

It should be noted that if N frames of frontal face images are not acquired within a preset time period, the detection of eye types is stopped, thereby reducing the power consumption caused by the eye type detection device being stuck in the detection of frontal face images for a long time.

It can be understood that, first, the present disclosure detects N frames of frontal face images and calibrates the strabismus of the target object. Compared with calibrating strabismus based on only one frame of frontal face images, the present disclosure can improve the accuracy of calibrating strabismus. Secondly, the timing difference between any two frames in the N-frame frontal face image of the present disclosure does not exceed the preset time, which improves the immediacy of eye type calibration while taking into consideration the improvement of eye type accuracy.

In some embodiments, determining the gaze angle of the left eye based on the pupil position of the left eye, and determining the gaze angle of the right eye based on the pupil position of the right eye includes:

According to the internal parameters of the image acquisition device, it is determined that the position of the pupil center of the left eye in the front face image is the first position in the coordinate system of the image acquisition device, and the position of the right pupil center in the front face image is determined. The second position in the coordinate system of the image acquisition device;

The gaze angle of the left eye is determined based on the angle between the connection line between the first position and the coordinate origin of the coordinate system of the image acquisition device and the normal line of the imaging plane of the image acquisition device. ;

The gaze angle of the right eye is determined based on the angle between the connection line between the second position and the coordinate origin of the coordinate system of the image acquisition device and the normal line of the imaging surface of the image acquisition device. .

In embodiments of the present disclosure, the position of the pupil center (including the position of the left pupil center and the position of the right pupil center) may be the position of the center of the outline of the pupil area in the front face image in the image coordinate system.

It should be noted that the image capture device of the present disclosure can be installed directly in front of the head of the target subject to capture the front face image.

According to the internal parameters of the image acquisition device (such as focal length), the present disclosure places the pupil center in the two-dimensional image coordinate system. The coordinates are mapped to the three-dimensional coordinates in the coordinate system of the image acquisition device, so that the position of the center of the left pupil is mapped to the first position in the coordinate system of the image acquisition device, and the position of the center of the right pupil is mapped to the coordinate system of the image acquisition device. second position. According to the connection between the first position and the coordinate origin of the coordinate system of the image acquisition device, the angle between the normal line of the imaging plane of the image acquisition device is determined as the gaze angle of the left eye. According to the second position and the image acquisition device The angle between the line connecting the coordinate origins of the device's coordinate system and the normal line of the imaging surface of the image acquisition device is determined as the gaze angle of the right eye. Then, the eye type is calibrated based on the gaze angle of the left eye and the gaze angle of the right eye.

In some embodiments, since during the process of driving the vehicle, the driving direction or the head direction of the vehicle is likely to be consistent with the direction of the driver's face, the present disclosure may also be based on the first position and the coordinate system of the image acquisition device. The angle between the connection between the coordinate origins and the vehicle driving direction or the head direction is determined as the gaze angle of the left eye; according to the connection between the second position and the coordinate origin of the coordinate system of the image acquisition device, and The angle between the vehicle's driving direction or the vehicle's head direction is determined as the gaze angle of the right eye. Among them, the driving direction or head direction of the vehicle can be determined based on the rotation angle of the steering wheel, or the driving direction of the vehicle can be obtained by docking the vehicle DMS system.

It can be understood that since the image capture device that captures the front face image is installed directly in front of the head of the target subject, the present disclosure determines the gaze angle of the left eye and the gaze angle of the right eye based on the normal of the imaging surface of the image capture device, When the angle difference between the gaze angle of the left eye and the gaze angle of the right eye exceeds the first predetermined angle range, one of the absolute larger gaze angles of the left eye and the right eye is determined to be a strabismus eye to improve the calibration Accuracy of strabismus.

In some embodiments, the image acquisition device is an image acquisition device provided inside a vehicle, and the target object is the driver of the vehicle;

The method also includes:

Obtain the speed of the vehicle and the twist angle of the steering wheel of the vehicle;

When the speed is greater than a predetermined speed threshold and the twist angle of the steering wheel is less than a predetermined angle threshold, it is determined that the target object is facing the image acquisition device;

When it is determined that the target object is facing the image acquisition device, obtaining the front face image of the target object collected by the image acquisition device includes:

When it is determined that the target object is looking directly at the image acquisition device, in response to the driver's squint calibration function being turned on, the front face image of the driver captured by the image acquisition device is acquired.

In the embodiment of the present disclosure, the image capture device can be installed inside the vehicle, such as directly in front of the driver's seat, to facilitate capturing the driver's front face image.

This disclosure can obtain the speed of the vehicle and the torsion angle of the steering wheel of the vehicle. It can obtain the values collected by the speed sensor and the angle sensor installed in the vehicle. It can also obtain the speed of the vehicle and the torsion angle of the steering wheel by docking the vehicle DMS system.

The present disclosure determines that the target object is facing the image acquisition device based on the acquired vehicle speed and the steering wheel's twist angle, when the vehicle's speed is greater than a predetermined speed threshold and the steering wheel's twist angle is less than a predetermined angle threshold. And in response to the driver's squint calibration function being turned on, the facial image collected by the image acquisition device is used as a front face image, so that the driver's squint eyes are calibrated based on the front face image based on the aforementioned method. Among them, the driver's squint calibration function can be turned on by default when the vehicle is started, or it can be turned on after a preset period of time after the vehicle is started, or it can be turned on based on the driver's squint calibration function. The issued operation command turns on the strabismus calibration function, which is not limited in this disclosure.

It can be understood that when the speed of the vehicle is greater than the predetermined speed threshold and the steering wheel twist angle is less than the predetermined angle threshold, it means that the vehicle is traveling straight at high speed. At this time, the probability of the driver looking straight ahead is relatively high. Therefore, in this case, image collection The facial image collected by the device has a high probability of being a frontal image of the target subject looking straight ahead, which can reduce the process of filtering out the frontal face image from the images collected by the image collection device, thereby improving the real-time determination of eye type based on the frontal face image. sex. In addition, since the front-face image is acquired in this scenario, the user cannot perceive it. Compared with outputting prompt information to the user to ask the user to cooperate in collecting the front-face image, the embodiments of the present disclosure can reduce the disturbance to the user and improve the user experience.

In some embodiments, the method further includes:

During the face image registration process, output a first prompt message requesting the target subject to look directly at the image collection device;

Based on the first prompt information, obtain a facial image;

The facial image that satisfies the predetermined image quality condition is used as the front face image of the target object acquired when the target object faces the image acquisition device.

In the embodiment of the present disclosure, during the face image registration process, the first prompt information indicating that the target subject is required to face the image collection device can be output through display screen text output or voice broadcast, etc., prompting the user to look towards the device that collects the face image. The image collection device collects the user's facial image after outputting the first prompt information. Among them, facial recognition technology, such as FaceID technology, can be used to register facial images.

After acquiring the facial image, the present disclosure determines whether the facial image meets the predetermined image quality conditions, uses the facial image that meets the predetermined image quality conditions as the front face image, and then performs the aforementioned eye type calibration based on the acquired front face image; The predetermined image quality conditions at least include that the face image is a frontal image, and may also include that the resolution is greater than a predetermined resolution threshold, the exposure is within a predetermined exposure range, and the signal-to-noise ratio is greater than a predetermined signal-to-noise ratio threshold, etc. The present disclosure calibrates eye types based on images that meet predetermined image quality conditions, which can improve the accuracy of determining eye types.

In addition, based on the features in the collected facial images, the features can be used as the unique identification of the user, so that the calibrated strabismus eye can be associated and stored with the unique identification of the user. The next time the user's eye type is detected, the acquired features in the user's facial image can be directly compared with the unique identifier. In the case where the facial image and the unique identifier belong to the same individual, for example, the feature similarity is greater than a predetermined The similarity threshold can directly determine the eye type detection result associated with the unique identifier as the eye type detection result of the target object in the facial image, without having to perform the aforementioned eye type detection method, thereby improving the immediacy of calibrating the eye type.

Therefore, the present disclosure obtains the facial image during the facial image registration process, and uses the facial image that meets the predetermined image quality conditions as the front face image, which can improve the accuracy and immediacy of the subsequent eye type calibration based on the front face image.

In some embodiments, since the registration of facial images usually obtains the user's front face image, the eye type detection device of the present disclosure can also directly obtain the facial image after successful facial image registration by other devices, and convert the facial image to as a frontal image.

It can be understood that the present disclosure directly uses the facial image after successfully registering the facial image obtained from other devices as the front face image, which can improve the reuse degree of the facial image and eliminates the need to output to the user a request for the target object to look directly at the target object. The first prompt information of the image collection device reduces the disturbance to the user and improves the user experience.

In some embodiments, the method further includes:

When the eye type of the target object is calibrated as left-eye strabismus or right-eye strabismus, at least one of the following interactions is performed based on the gaze information of the target object's non-strabismus eye:

Output information about the interactive object corresponding to the gaze angle of the non-strabismic eye in the space where the target object is located;

In the case where the target object is the driver of the vehicle and the vehicle speed is not 0, the second prompt information is output in response to the gaze angle of the non-strabismic eye exceeding the second predetermined angle range within a predetermined period of time.

In the embodiment of the present disclosure, interactive information can be output based on the gaze information of the non-strabismic eye; wherein the gaze information includes gaze angle and gaze duration.

Because when people with strabismus gaze in a certain direction, the gaze angle of the strabismus eye deviates greatly from the gaze angle of the normal eye, therefore, the interactive object corresponding to the gaze angle of the strabismus eye in the space is not the object that the strabismus eye really wants to gaze at.

In this regard, in some embodiments, the present disclosure shields the strabismus eye and determines the corresponding interactive object in the space based on the gaze angle of the non- strabismus eye, thereby determining the object that the strabismus person really wants to gaze at, and outputs the object's information. Determining the interactive object based on the gaze angle of the non-strabismic eye may include determining the gaze point of the gaze angle in the space where the target object is located based on the gaze angle of the non-strabismic eye, thereby determining the object corresponding to the gaze point as the interactive object; The gaze range can also be determined based on the gaze angle of the non-strabismic eye, for example, the gaze angle range of plus or minus 20 degrees is used as the gaze range, and then the object whose gaze range covers the space where the target object is located is determined as the interactive object.

Taking the vehicle driving scene as an example, if the non-squinting eye's gaze angle interacts with the vehicle dashboard in the cabin, the dashboard information, such as vehicle speed, rotational speed, etc., can be displayed on the display screen of the vehicle center console.

In some embodiments, the present disclosure shields the strabismus eye when the target object is the driver of the vehicle and the vehicle speed is not 0, and when the gaze angle of the non-strabismus eye exceeds the second predetermined angle range within a predetermined period of time, it represents The driver may be distracted (such as not looking in the direction of travel for a long time), and the second prompt message is output. Wherein, if the gaze angle is a positive value, exceeding the second predetermined angle range may be greater than the predetermined positive value; if the gaze angle is a negative value, exceeding the second predetermined angle range may be less than the predetermined negative value.

It can be understood that the present disclosure blocks the strabismic eye after determining the strabismic eye, and outputs interactive information based on the gaze information of the non-strabismic eye, which can reduce the erroneous output of interactive information based on the gaze information of the strabismic eye and improve the accuracy of interactive information output.

Figure 3 is a flowchart 2 of the implementation of an eye type detection method provided by an embodiment of the present disclosure. As can be seen from Figure 3, the method includes the following steps S31 to step S33:

Step S31: Whether the facial image registration is successful; if yes, proceed to step S32; if not, continue to execute step S31.

In some embodiments, if the facial image registration is successful, the facial image that satisfies the predetermined image quality condition will be used as the front face image of the target object acquired when the target object faces the image acquisition device.

Step S32: Whether the difference between the left and right pupil directions is greater than the threshold; if so, execute step S33; if not, the detection process ends and it is determined that there is no strabismus.

In some embodiments, the left pupil direction is the gaze angle of the aforementioned left eye, and the right pupil direction is the gaze angle of the aforementioned right eye. At the gaze angle, the difference between the left and right pupil directions is the aforementioned angle difference, and the threshold is the aforementioned first predetermined angle range. The present disclosure determines whether the difference between the left and right pupil directions is greater than a threshold, that is, the aforementioned determination of the angle difference between the gaze angle of the left eye and the gaze angle of the right eye, and whether the angle difference exceeds the first predetermined angle range.

Step S33: The eye with a greater deviation from the normal direction of the camera is regarded as a strabismus eye.

In some embodiments, the normal line of the camera is the normal line of the imaging surface of the image acquisition device, and the direction deviation from the normal line of the camera is the gaze angle of the left eye and the gaze angle of the right eye. In this disclosure, the eye with a greater deviation from the normal direction of the camera is regarded as a strabismus. That is, when the angle difference exceeds the first predetermined angle range, the one with the larger absolute value of the gaze angle of the left eye and the right eye is determined. It is the squint in the front face image.

In some embodiments, on the one hand, when the face image registration is successful, the present disclosure uses the image after the face image registration is successful as a front face image for eye type detection, which can improve the degree of image reuse. On the other hand, this disclosure determines the difference between the left and right pupil directions based on the acquired front face image. When the difference between the left and right pupil directions is greater than the threshold, the eye with a larger deviation from the normal direction of the camera is determined to be a strabismus. Compared with Instead of detecting strabismus using cover inspection, strabismus angle measurement, or using detection instruments for strabismus, embodiments of the present disclosure can quickly and accurately detect the strabismus type of the target object's eyes through image processing.

Figure 4 is a flowchart 3 of the implementation of an eye type detection method provided by an embodiment of the present disclosure. As can be seen from Figure 4, the method includes the following steps S41 to step S45:

Step S41: Whether the whole vehicle is in a high-speed scene; if so, execute step S42; if not, end the detection process.

In some embodiments, the high-speed scene is a scene in which the vehicle speed is greater than a predetermined speed threshold and the steering wheel twist angle is less than a predetermined angle threshold.

If the whole vehicle is in a high-speed scene, N frames of facial images in the video stream collected by the image acquisition device are used as front-face images.

Step S42: Whether the difference between the left and right pupil directions in this frame is greater than the threshold; if so, execute step S43; if not, execute step S44.

In some embodiments, the present disclosure determines whether the difference in direction between the left and right pupils in the current frame of the face image is greater than a threshold, that is, the aforementioned determination of the angle difference between the gaze angle of the left eye and the gaze angle of the right eye, and whether the angle difference exceeds the third A predetermined angle range.

Step S43: The eye with a larger deviation angle from the driving direction ahead is regarded as a strabismus eye;

In some embodiments, the forward driving direction is the aforementioned vehicle driving direction, and the deviation angle from the forward driving direction is the gaze angle of the left eye and the gaze angle of the right eye based on the vehicle traveling direction.

In the present disclosure, when the difference in direction between the left and right pupils is greater than a threshold, the eye with a larger deviation angle from the driving direction ahead is regarded as a strabismus. That is, when the angle difference exceeds the first predetermined angle range, the left eye and the right eye are regarded as squinting. The one with the larger absolute value of the angle is determined to be the squinting eye in the front face image.

Based on this method, for N frames of frontal face images, the number of image frames in which the left eye is strabismus is determined, corresponding to the aforementioned second frame number; and the number of frames in the image in which the right eye is strabismus is determined, corresponding to the aforementioned third frame number.

Step S44: The driver is considered to be a normal person;

In some embodiments, the present disclosure determines that the driver is driving when the difference between the left and right pupil directions is less than or equal to a threshold. If the person is a normal person and the aforementioned angle difference does not exceed the first predetermined angle range, the target object's binocular type is binocular non-strabismus. Based on this method, for N frames of frontal face images, the number of frames in which both eyes are non-squinting eyes is determined, corresponding to the aforementioned first number of frames.

Steps S42 to S44 are executed in a loop N times, that is, the detection of N frames of images is completed. After executing steps S42 to S44 N times in a loop, the first frame number, the second frame number, and the third frame number are obtained, and step S45 is executed.

Step S45: Vote on N results and output the result with the highest number of votes.

In some embodiments, N results are voted on to obtain the number of votes, where the number of votes for the left eye with strabismus corresponds to the second frame number, the number of votes for the right eye with strabismus corresponds to the third frame number, and the number of votes for both eyes without strabismus. Corresponding to the first frame number, the result with the highest number of votes is output; the result with the highest number of votes is the detection result of the eye type.

Wherein, in the present disclosure, when the first frame number is greater than the second frame number and the third frame number, the binocular type of the target object is calibrated to binocular non-strabismus; or, when the first frame number is less than the second frame number and the third frame In the case of any one of the numbers, the strabismus eye in the front face image corresponding to the larger of the second frame number and the third frame number is calibrated as the strabismus eye of the target object.

In some embodiments, on the one hand, when the vehicle is in a high-speed scene, the driver is more likely to look straight ahead. Therefore, in this case, the facial image collected by the image acquisition device is more likely to be a frontal face image, so that The process of filtering out front-face images from images collected by an image acquisition device can be reduced, thereby improving the immediacy of determining eye types based on front-face images. In addition, since the front-face image is acquired in this scenario, the user cannot perceive it. Compared with outputting prompt information to the user to ask the user to cooperate with the front-face image collection device to collect the front-face image, the embodiment of the present disclosure can reduce the disturbance to the user. Improve user experience.

On the other hand, the present disclosure determines the first frame number, the second frame number and the third frame number based on N frames of frontal images obtained when the whole vehicle is in a high-speed scene. Based on the first frame number, the second frame number and the third frame number, The comparison result of the third number of frames determines the final eye type. Compared with determining the eye type based only on one frame of the front face image, the embodiment of the present disclosure can improve the accuracy of determining the eye type.

This disclosure is based on the FaceID registration process (i.e., the above-mentioned facial image registration). First, when the driver (i.e., the above-mentioned target object) is facing the camera (i.e., the above-mentioned image acquisition device), the camera collects the driver's front face image; then; , detect the pupil direction angles of the driver's eyes in the front face image (i.e., the gaze angle of the left eye and the right eye), and obtain the difference in gaze directions of the two pupils (i.e., the gaze angle of the left eye and the right eye) (angle difference between the gaze angles); finally, based on the angle difference and the first predetermined angle range, it is determined whether the driver is a strabismus person. In addition, in order to avoid random errors in the FaceID registration process, during the operation of DMS, a period of dynamic strabismus calibration will also be performed when the vehicle is running at high speed.

The present disclosure detects the pupil direction angle of the driver's eyes on the picture that passes the FaceID registration requirement (ie, the above-mentioned front face image), that is, the picture that meets the driver's requirement of facing the camera. If the difference in pupil direction angles of both eyes (i.e., the angle difference between the gaze angle of the left eye and the gaze angle of the right eye) exceeds a certain threshold (i.e., the first predetermined angle range), it is considered that the driver has both pupils. There are non-parallel situations. Based on this scenario, the driver should be looking directly at the camera, so the eye with a larger deviation angle from the normal direction of the camera's imaging plane is output as a strabismus eye. Although the driver's head posture and eye orientation are relatively stable and standardized when FaceID is registered, it cannot be completely Fully control whether the driver actually looks at the camera at this time. In order to further reduce random errors in the FaceID registration process, the DMS system is used to perform dynamic calibration of strabismus in certain scenarios during driving. During the operation of DMS, by setting the dynamic calibration scene of strabismus, a scene is created in which the driver looks towards a certain area with a high probability (that is, the above scene of the vehicle driving straight at high speed), and then in this scene, the driver can The eye with a larger pupil direction deviation area is output as a strabismus.

This disclosure sets the DMS dynamic calibration scene as a high-speed driving scene, that is, when the vehicle is running at a high speed (>80km/h) and the steering wheel angle deflection is small, it is considered that the driver is basically looking ahead (i.e., the above-mentioned front-facing image acquisition device) driving status. Since the dynamic calibration process of DMS is the result of multiple rounds of voting, it has stronger robustness. Therefore, if the DMS dynamic calibration process is completely executed, the results of the DMS dynamic calibration can be used to replace the detection results during the FaceID registration process. Whether it is the FaceID registration process or the DMS dynamic calibration process, there is no need to explicitly inform the driver to cooperate in the detection, and they are all non-sensory strabismus detection processes. In order to make the calibration results more accurate, the number of testing rounds N can be appropriately increased; at the same time, in order to prevent the calibration process from being stuck in the calibration process for too long, a calibration time limit can be added. If N testing is not performed within a certain period of time, the current round of calibration will be abandoned.

Compared with related technologies, the present disclosure has the following advantages: (1) The present disclosure provides a driver-insensitive strabismus detection method; (2) The DMS dynamic calibration process provided by the present disclosure supplements the potential problems caused by the FaceID detection process. random error problem.

Figure 5 is a schematic structural diagram of an eye type detection device provided by an embodiment of the present disclosure. As can be seen from Figure 5, the eye type detection device 500 includes:

The first acquisition module 501 is configured to acquire the front face image of the target object collected by the image acquisition device when it is determined that the target object is facing the image acquisition device;

The first determination module 502 is configured to respectively determine the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image;

The second determination module 503 is configured to determine the gaze angle of the left eye according to the pupil position of the left eye, and determine the gaze angle of the right eye of the target object according to the pupil position of the right eye;

The third determination module 504 is configured to determine the angle difference between the gaze angle of the left eye and the gaze angle of the right eye;

The fourth determination module 505 is configured to determine the larger absolute value of the gaze angle of the left eye and the right eye as the correct angle when the angle difference exceeds the first predetermined angle range. squinting eyes in face images;

The calibration module 506 is configured to calibrate the eye strabismus type of the target object according to the strabismus eye in the front face image.

The calibration module 506 is configured to calibrate the eye type of the target object as left eye strabismus when the number of frames in the multi-frame frontal face image in which it is determined that the left eye is strabismus reaches a predetermined threshold; or when In the multi-frame frontal face images, if the number of frames in which it is determined that the right eye is strabismus reaches a predetermined threshold, the eye type of the target object is calibrated as right eye strabismus.

The eye type detection device 500 also includes:

The fifth determination module 507 is configured to determine that both eyes in the front face image are non-strabismus eyes when the angle difference does not exceed the first predetermined angle range;

Calibration module 506 includes:

The determination sub-module is configured to determine, in the N frames of front face images, the first frame number of the image in which both eyes are non-strabismus, the second frame number of the image in which the left eye is determined to be strabismus, and the determination of the right eye to be strabismus. The third frame number of the eye image is compared;

A calibration submodule configured to calibrate the binocular type of the target object as binocular non-strabismus when the first frame number is greater than the second frame number and the third frame number; or in the first When the number of frames is less than any one of the second number of frames and the third number of frames, calibrate the front face image corresponding to the larger one of the second number of frames and the third number of frames. The squinting eye is the squinting eye of the target subject.

In some embodiments, the second determination module 503 includes:

The first determination sub-module is configured to determine the first position of the pupil center of the left eye in the front face image in the coordinate system of the image acquisition device according to the internal parameters of the image acquisition device, and the The position of the center of the right pupil in the frontal image is at the second position in the coordinate system of the image acquisition device;

The second determination sub-module is configured to determine the angle between the connection line between the first position and the coordinate origin of the coordinate system of the image acquisition device and the normal line of the imaging surface of the image acquisition device, Determine the gaze angle of the left eye;

The third determination sub-module is configured to determine the angle between the connection line between the second position and the coordinate origin of the coordinate system of the image acquisition device and the normal line of the imaging surface of the image acquisition device, Determine the gaze angle of the right eye.

The eye type detection device 500 also includes:

The second acquisition module 508 is configured to acquire the speed of the vehicle and the twist angle of the steering wheel of the vehicle;

The sixth determination module 509 is configured to determine that the target object is facing the image acquisition device when the speed is greater than a predetermined speed threshold and the twist angle of the steering wheel is less than a predetermined angle threshold;

The first acquisition module 501 is configured to acquire the driver's front face image collected by the image acquisition device in response to the driver's squint calibration function being turned on when it is determined that the target object is looking directly at the image acquisition device.

In some embodiments, the eye type detection device 500 further includes:

The output module 510 is configured to, during the facial image registration process, output the first prompt information requesting the target subject to face the image collection device;

The third acquisition module 511 is configured to acquire a facial image based on the first prompt information;

The seventh determination module 512 is configured to use the facial image that meets the predetermined image quality condition as a target pair. A frontal face image of the target object acquired by the image acquisition device.

In some embodiments, the eye type detection device 500 further includes:

The interaction module 513 is configured to perform at least one of the following interactions based on the gaze information of the target object's non-strabismus eyes when the eye type of the target object is calibrated as left-eye strabismus or right-eye strabismus: output the non-strabismus eye. Information about the interactive object corresponding to the gaze angle of the strabismus in the space where the target object is located; when the target object is the driver of the vehicle and the vehicle speed is not 0, in response to the non-strabismus eye within a predetermined period of time When the gaze angle exceeds the second predetermined angle range, the second prompt information is output.

Figure 6 is a schematic diagram of the hardware entity of a computer device provided by an embodiment of the present disclosure. As shown in Figure 6, the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802 and a memory 803, where: the processor 801 usually Controls the overall operation of computer device 800. The communication interface 802 can enable the computer device to communicate with other terminals or servers through a network.

The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the computer device 800 (for example, image data, audio data, voice communication data and Video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be carried out between the processor 801, the communication interface 802 and the memory 803 through the bus 804. The processor 801 is used to execute some or all steps in the above method.

Embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, some or all of the steps in the above method are implemented.

Embodiments of the present disclosure provide a computer program product, which includes a computer program that, when the computer program is run on a computer device, causes the computer device to execute some or all of the steps in the above method.

It should be pointed out here that the above description of the storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

It will be understood that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that in various embodiments of the present disclosure, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present disclosure. The implementation process constitutes any limitation. The above serial numbers of the embodiments of the present disclosure are only for description and do not represent the advantages and disadvantages of the embodiments.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element.

In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components may be combined, or can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be electrical, mechanical, or other forms. of.

The units described above as separate components may or may not be physically separated; the components shown as units may or may not be physical units; they may be located in one place or distributed to multiple network units; Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure can be all integrated into one processing unit, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed through hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the execution includes: The steps of the above method embodiment; and the aforementioned storage media include: mobile storage devices, read-only memory (Read Only Memory, ROM), magnetic disks or optical disks and other various media that can store program codes.

Alternatively, if the above-mentioned integrated units of the present disclosure are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can be embodied in the form of a software product in essence or that contributes to related technologies. The computer software product is stored in a storage medium and includes a number of instructions to enable a computer. A computer device (which may be a personal computer, a server, a network device, etc.) executes all or part of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: mobile storage devices, ROMs, magnetic disks or optical disks and other media that can store program codes.

The above are only embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, and should are covered by the protection scope of this disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

An eye type detection method, the method includes:

When it is determined that the target object is facing the image acquisition device, obtain the front face image of the target object collected by the image acquisition device;

Determine respectively the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image;

Determine the gaze angle of the left eye based on the pupil position of the left eye, and determine the gaze angle of the right eye of the target object based on the pupil position of the right eye;

determining an angular difference between the gaze angle of the left eye and the gaze angle of the right eye;

When the angle difference exceeds the first predetermined angle range, determine the one with the larger absolute value of the gaze angle of the left eye and the right eye as the strabismus eye in the front face image;

The eye strabismus type of the target object is calibrated according to the strabismus eye in the front face image.
The method according to claim 1, wherein the frontal face image includes multiple frames of frontal face images in the video stream of the target object collected by the image acquisition device;

Calibrating the eye strabismus type of the target object based on the strabismus eye in the front face image includes:

In the multi-frame front face images, if the number of frames in which it is determined that the left eye is strabismus reaches a predetermined threshold, calibrate the eye type of the target object as left eye strabismus; or

In the multi-frame frontal face images, if the number of frames in which it is determined that the right eye is strabismus reaches a predetermined threshold, the eye type of the target object is calibrated as right eye strabismus.
The method according to claim 1, wherein the frontal face image includes N frames of frontal face images in the video stream of the target object collected by the image acquisition device, and N is a predetermined positive integer;

The method also includes:

When the angle difference does not exceed the first predetermined angle range, determine that both eyes in the front face image are non-strabismus;

Calibrating the eye strabismus type of the target object based on the strabismus eye in the front face image includes:

In the N frames of frontal face images, the first frame number of the image in which both eyes are determined to be non-strabismus, the second frame number of the image in which the left eye is determined to be a strabismus, and the third frame number of the image in which the right eye is determined to be strabismus are determined. Frame numbers are compared;

In the case where the first frame number is greater than the second frame number and the third frame number, calibrating the binocular type of the target object as binocular non-strabismus; or

When the first frame number is less than any one of the second frame number and the third frame number, calibrate the corresponding one of the second frame number and the third frame number. The strabismus eye in the front face image is the strabismus eye of the target object.
The method according to claim 3, wherein the N frames of frontal images are N frames of frontal images in which the timing difference between any two frames in the video stream does not exceed a preset time period.
The method according to any one of claims 1 to 4, wherein the pupil position of the left eye Determining the gaze angle of the left eye, and determining the gaze angle of the right eye based on the pupil position of the right eye, includes:

According to the internal parameters of the image acquisition device, it is determined that the position of the pupil center of the left eye in the front face image is the first position in the coordinate system of the image acquisition device, and the position of the right pupil center in the front face image is determined. The second position in the coordinate system of the image acquisition device;

The gaze angle of the left eye is determined based on the angle between the connection line between the first position and the coordinate origin of the coordinate system of the image acquisition device and the normal line of the imaging plane of the image acquisition device. ;

The gaze angle of the right eye is determined based on the angle between the connection line between the second position and the coordinate origin of the coordinate system of the image acquisition device and the normal line of the imaging surface of the image acquisition device. .
The method according to any one of claims 1 to 5, wherein the image acquisition device is an image acquisition device provided inside a vehicle, and the target object is a driver of the vehicle;

The method also includes:

Obtain the speed of the vehicle and the twist angle of the steering wheel of the vehicle;

When the speed is greater than a predetermined speed threshold and the twist angle of the steering wheel is less than a predetermined angle threshold, it is determined that the target object is facing the image acquisition device;

When it is determined that the target object is facing the image acquisition device, obtaining the front face image of the target object collected by the image acquisition device includes:

When it is determined that the target object is looking directly at the image acquisition device, in response to the driver's squint calibration function being turned on, the front face image of the driver captured by the image acquisition device is acquired.
The method according to any one of claims 1 to 5, wherein the method further includes:

During the face image registration process, output a first prompt message requesting the target subject to look directly at the image collection device;

Based on the first prompt information, obtain a facial image;

The facial image that satisfies the predetermined image quality condition is used as the front face image of the target object acquired when the target object faces the image acquisition device.
The method according to any one of claims 1 to 7, wherein the method further includes:

When the eye type of the target object is calibrated as left-eye strabismus or right-eye strabismus, at least one of the following interactions is performed based on the gaze information of the target object's non-strabismus eye:

Output information about the interactive object corresponding to the gaze angle of the non-strabismic eye in the space where the target object is located;

In the case where the target object is the driver of the vehicle and the vehicle speed is not 0, the second prompt information is output in response to the gaze angle of the non-strabismic eye exceeding the second predetermined angle range within a predetermined period of time.
An eye type detection device, the device includes:

The first acquisition module is configured to acquire the front face image of the target object collected by the image acquisition device when it is determined that the target object is facing the image acquisition device;

A first determination module configured to respectively determine the pupil position of the left eye and the pupil position of the right eye of the target object in the front face image;

The second determination module is configured to determine the gaze angle of the left eye according to the pupil position of the left eye, and based on Determine the gaze angle of the target object's right eye based on the pupil position of the right eye;

a third determination module configured to determine the angle difference between the gaze angle of the left eye and the gaze angle of the right eye;

The fourth determination module is configured to determine the one with a larger absolute value of the gaze angles of the left eye and the right eye as the front face when the angle difference exceeds the first predetermined angle range. squinting eyes in the image;

A calibration module configured to calibrate the eye strabismus type of the target object according to the strabismus eye in the front face image.
A computer device, including: a processor; a memory used to store instructions executable by the processor;

Wherein, the processor is configured to perform the method according to any one of claims 1 to 8.
A computer-readable storage medium on which a computer program is stored, which implements the method described in any one of claims 1 to 8 when executed by a processor.
A computer program product, the computer program product comprising a computer program, when the computer program is run on a computer device, causing the computer device to perform an eye type as claimed in any one of claims 1 to 8 Detection method.