JP7227385B2 - Neural network training and eye open/close state detection method, apparatus and equipment - Google Patents

Description

"Cross References to Related Applications"
The present disclosure is based on a Chinese patent application titled No. 201910153463.4 "Neural network training and eye open/closed state detection method, device and apparatus" filed with the Chinese Patent Office on February 28, 2019. priority is claimed, the entire contents of which are incorporated by reference into this disclosure.

The present disclosure relates to computer visual technology, in particular, neural network training method, neural network training device, eye open/close state detection method, eye open/close state detection device, intelligent driving control method, intelligent driving control device, electronic equipment, computer reading It relates to possible storage media and computer programs.

The detection of the open/closed state of the eyes is to detect the open/closed state of the eyes. Eye open/close state detection can be used in fields such as fatigue monitoring, biometric recognition, and facial expression recognition. For example, in driving support technology, it is possible to monitor fatigue driving by detecting whether the driver's eyes are open or closed, and based on the results of the detection, it is possible to determine whether the driver is in a state of fatigue driving. There is a need to. Accurately detecting the open/closed state of the eyes and avoiding misjudgment as much as possible is advantageous for improving the safety of vehicle travel.

Embodiments of the present disclosure provide technical solutions for neural network training, eye open/close state detection and intelligent driving control.

In one aspect of an embodiment of the present disclosure, for each of a plurality of eye images in an image set corresponding to each of at least two eye open/closed detection training tasks, via a trained eye open/closed detection neural network: performing eye open/closed state detection processing and outputting an eye open/closed state detection result; determining a respective loss corresponding to each of at least two eye open/close detection training tasks, and adjusting network parameters of the neural network based on the loss corresponding to each of the at least two eye open/close detection training tasks. and wherein eye images included in different image sets are at least partially different.

In another aspect of the embodiment of the present disclosure, an image to be processed is acquired, an eye open/closed state detection process is performed on the image to be processed via a neural network, and a detection result of the eye open/closed state is output. and, wherein the neural network is obtained by training according to the neural network training method described in the above embodiment.

According to another aspect of the embodiment of the present disclosure, an image to be processed collected by an imaging device mounted on a vehicle is acquired, and eye open/closed state detection processing is performed on the image to be processed via a neural network. and outputting the detection result of the eye open/closed state, and determining the fatigue state of the subject based on at least the detection result of the eye open/closed state of the same subject in a plurality of time-series images to be processed. and generating and outputting a command according to the subject's fatigue state, wherein the neural network is trained by the neural network training method according to the above embodiment. Intelligent driving Provide a control method.

In another aspect of an embodiment of the present disclosure, an eye open/closed state detection process is performed on each of a plurality of eye images in an image set corresponding to each of the at least two eye open/closed detection training tasks to detect an eye open/closed state. Based on the eye open/closed detection neural network of the training target used to output the detection result of the eye image, the eye open/closed labeling information of the eye image, and the eye open/closed state detection result output from the neural network, used to respectively determine a loss corresponding to each of at least two eye open/close detection training tasks, and adjust network parameters of the neural network based on the losses corresponding to each of said at least two eye open/close detection training tasks. and an adjustment module, wherein eye images included in different image sets are at least partially different.

According to another aspect of the embodiment of the present disclosure, an acquisition module used to acquire an image to be processed performs an eye open/closed state detection process on the image to be processed, and outputs a detection result of the eye open/closed state. and a neural network used in particular, wherein the neural network is trained by the neural network training apparatus described in the above embodiments.

In another aspect of the embodiment of the present disclosure, an acquisition module used to acquire an image to be processed collected by an imaging device mounted on a vehicle; and the fatigue state of the subject based on the detection results of the eye open/closed state of the same subject in at least a plurality of time-series images to be processed. and a command module used to generate and output a command according to the fatigue state of the subject, wherein the neural network is described in the above embodiments. provides an intelligent driving controller that has been trained with the neural network training device of .

In another aspect of an embodiment of the disclosure, a memory for storing a computer program, executing the computer program stored in the memory, and when the computer program is executed, any method of the disclosure and a processor that implements an embodiment of .

In another aspect of the embodiments of the disclosure, there is provided a computer readable storage medium storing a computer program that, when executed by a processor, implements an embodiment of any of the methods of the disclosure.

In another aspect of the disclosed embodiments, there is provided a computer program product comprising computer instructions that, when executed in a processor of a device, implements an embodiment of any of the disclosed methods.

In the course of implementing embodiments of the present disclosure, the inventors found that in a conventional neural network that trains a single task, for a neural network trained on the image set of the task, the scene corresponding to the task is Although it has a relatively good eye open/close detection accuracy rate, it is difficult to ensure the eye open/close detection accuracy in other scenes that do not correspond to the task. Simply taking multiple images collected in different scenes as one image set for neural network training, without distinguishing whether the images in the image set are from different scenes or correspond to different training tasks, this one image The distribution of the image subsets (batches) input each time from the set to neural network training is uncontrollable, there may be many images of one scene, but few or even less of others, resulting in different iterations. The distributions of the trained image subsets are also not exactly the same. That is, the distribution of the image subsets in each iteration of the neural network is too random, loss calculation is not performed for different training tasks, and it is impossible to control the ability learning of the neural network considering each different training task in the training process. Therefore, the trained neural network cannot ensure the accuracy of eye open/close detection in different scenes corresponding to different tasks.

According to the present disclosure, a neural network training method and apparatus, an eye open/closed state detection method and apparatus, an intelligent driving control method and apparatus, an electronic device, a computer-readable storage medium, and a computer program enable a plurality of different eye open/close detection tasks to be performed. Determining corresponding image sets respectively, determining multiple eye images in one training session of the neural network from the multiple image sets, and opening and closing the eyes for each training task in the training based on the eye images from the multiple image sets. The losses of the neural network for each detection result are determined, and the network parameters of the neural network are adjusted based on each loss. In this way, the subset of eye images provided to the neural network for each iteration of training of the neural network contains an eye image corresponding to each training task, and the loss is computed for each training task, so that the neural In the training process of the network, ability learning regarding the detection of eye opening/closing ability is possible for each training task, and ability learning can be performed in consideration of different training tasks. This allows the trained neural network to simultaneously increase the accuracy of eye open/close detection in each scene in multiple scenes corresponding to multiple training tasks, and to detect the eyes in different scenes based on the neural network. It is advantageous for promoting the improvement of universality and generalization of the invention of accurately detecting opening and closing, and better meeting the actual application needs for multiple scenes.

The technical solution of the present disclosure will be described in more detail below with reference to the drawings and embodiments.

The drawings herein are used as part of the specification to illustrate embodiments of the disclosure and, together with the description, to interpret the principles of the disclosure.

The following detailed description makes the present disclosure clearer with reference to the drawings.

1 shows a flow chart of one embodiment of a neural network training method of the present disclosure; 1 shows a flowchart of an embodiment of an eye open/closed state detection method of the present disclosure; 1 shows a flowchart of an embodiment of an eye open/closed state detection method of the present disclosure; 1 shows a flowchart of one embodiment of an intelligent driving control method of the present disclosure; 1 shows a schematic diagram of a configuration of an embodiment of a neural network training device of the present disclosure; FIG. 1 shows a configuration schematic diagram of an embodiment of an eye open/closed state detection apparatus of the present disclosure; FIG. 1 shows a configuration schematic diagram of an embodiment of an intelligent driving control device of the present disclosure; FIG. 1 shows a block diagram of an exemplary device according to embodiments of the present disclosure; FIG.

Various exemplary embodiments of the present disclosure are described in detail below with reference to the drawings. Unless otherwise stated, the relative arrangements, formulas and numerical values of the means and steps described in these examples are not intended to limit the scope of the present disclosure.

Also, for convenience of explanation, it should be understood that the dimensions of the parts shown in the drawings are not drawn according to actual proportional relationships.

The description of at least one exemplary embodiment below is merely illustrative in nature and is not in any way limiting on the disclosure and its application or use.

Techniques, methods and equipment known to those of ordinary skill in the relevant fields may not be discussed in detail, but where appropriate, such techniques, methods and equipment should be considered part of this specification. is.

Similar symbols and letters indicate similar elements. As such, it should be noted that once an element is defined in one drawing, it need not be further discussed in subsequent drawings.

The disclosed embodiments may be employed in electronic devices such as terminals, computer systems and servers, and are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of known terminals, computer systems, environments and/or configurations suitable for use with electronic equipment such as terminals, computer systems and servers include personal computer systems, server computer systems, thin clients, thick clients, handhelds or laptops. Top devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, large computer systems and distributed cloud computing technology environments including any of the above systems, etc. , but not limited to.

Electronic devices such as terminals, computer systems and servers may be described in the general context of computer system-executable instructions (such as program modules) being executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in storage media in local or remote computing systems including storage devices.

Illustrative Examples FIG. 1 shows a flowchart of one embodiment of the neural network training method of the present disclosure. As shown in FIG. 1, the method according to this embodiment includes steps: S100 and S110. Each step in FIG. 1 will be described in detail below.

S100, performing an eye open/closed state detection process for each of the plurality of eye images in the image set corresponding to each of the at least two eye open/closed detection training tasks via the eye open/closed detection neural network to be trained; , outputs the detection result of the eye open/closed state.

In one selectable example, the eye open/closed detection neural network to be trained of the present disclosure performs eye open/closed state detection on the processed image after being trained, and outputs the eye open/closed state detection result of the processed image. can be used to For example, for one processed image, the neural network outputs two probability values, one of which indicates the probability that the subject's eyes are open in the processed image, and this probability The larger the value, the closer to the open-eye state. The other probability value indicates the probability that the subject's eyes are closed in the image to be processed. The sum of the two probability values may be one.

In one selectable example, the neural network of the present disclosure may be a convolutional neural network. The neural networks of the present disclosure may include convolution layers, Relu (Rectified Linear Unit) layers (also called activation layers), pooling layers, fully connected layers and layers for classification (e.g., binary classification), etc. Good, but not limited to: The more layers included in this neural network, the deeper the network. This disclosure does not limit the specific configuration of the neural network.

In one selectable example, there are at least two eye open/closed detection training tasks involved in the process of training a neural network in this disclosure, and each eye open/closed detection training task enables the neural network to detect an eye open/closed state. should belong to the overall training task for The training goals corresponding to different eye open/close detection training tasks are not exactly the same. That is, the present disclosure can divide the overall training task of the neural network into multiple training tasks, one training task corresponding to one training target, and different training targets corresponding to different training tasks.

In one selectable example, the at least two eye open/closed detection training tasks of the present disclosure are an eye open/closed detection task with wearable eyes, an eye open/closed detection task with no eye wearables, Eye open/close detection task in the environment, Eye open/close detection task in an outdoor environment, Eye open/close detection task when the wearable object is attached to the eye and a spot is on the wearable object, Eye wearable object is attached to the eye and the wearable object is spotted at least two of the eye open/closed detection tasks in the absence of . The wearable item may be eyeglasses or a transparent plastic sheet. The spot may be a spot formed on the wearable by reflection of the wearable. Eyeglasses of the present disclosure generally refer to eyeglasses that allow the wearer's eyes to be seen through the lenses.

If desired, the eye open/closed detection task when the wearer is wearing eyeglasses may be the eye open/closed detection task when the eyeglasses are worn. The task of detecting open/closed eyes while wearing glasses can realize at least one of detection of open/closed eyes while wearing glasses indoors and detection of open/closed eyes while wearing glasses outdoors.

If desired, the eye open/closed detection task when no wearable object is worn on the eye may be an eye open/closed detection task without eyeglasses. The eye open/close detection task without wearing glasses can realize at least one of indoor open/close eye detection without wearing glasses and open/close eye detection outdoors without wearing glasses.

Optionally, the task of detecting open/closed eyes in an indoor environment includes detection of open/closed eyes without wearing glasses in the room, detection of open/closed eyes in the room with glasses and reflected by the glasses, and detection of open/closed eyes in the room with glasses, In addition, at least one of eye open/close detection without reflection from the spectacles can be realized.

If desired, the eye open/close detection task in an outdoor environment includes eye open/close detection when the user is not wearing eyeglasses outdoors, eye open/close detection when the user is wearing eyeglasses outdoors and reflects the eyeglasses, and eye open/close detection when the user is wearing eyeglasses outdoors. In addition, at least one of eye open/close detection without reflection from the spectacles can be realized.

If desired, the eye open/closed detection task when the wearable is worn on the eye and the wearable has a spot may be the eye open/closed detection task in which the eyeglasses are worn and the eyeglasses are reflected. The eye open/closed detection task of the person wearing glasses and reflected by the glasses includes the task of detecting the opened/closed eyes of the person wearing glasses indoors and the eyes of the person wearing glasses indoors and the open/closed eyes of the person wearing glasses outdoors. At least one of detection can be implemented.

If desired, the eye open/closed detection task when the wearable is worn on the eye and the wearable has no spots may be the eye open/closed detection task in which the eyeglasses are worn and the eyeglasses are not reflective. The eye open/closed detection task for the person wearing the glasses and the eyeglasses not reflecting is the eye open/closed detection task for the person wearing the glasses indoors and the eyeglasses not reflecting, and the eye open/close detection task for the person wearing the glasses indoors and the eyeglasses not reflecting. At least one of non-reflective eye open/close detection can be implemented.

As can be seen from the above content, there is a common part between the different eye open/closed detection training tasks of the present disclosure. Eye open/close detection task, eye open/close detection task when the wearable object is attached to the eye and there is a spot, and eye open/close detection task when the wearable object is attached to the eye and there is no spot on the eye open/closed detection task. There may be Here, the fact that there are common parts among the six eye open/closed detection training tasks listed above will not be explained one by one. Also, the present disclosure does not limit the number of such eye open/close detection training tasks, and the number of eye open/close detection training tasks can be determined according to actual needs. This disclosure does not limit the form in which any eye open/closed detection training task is embodied.

Optionally, as shown in FIG. 2, the at least two eye open/close detection training tasks of the present disclosure may include the following three eye open/close detection training tasks.

Eye open/close detection training task a, eye open/close detection training task in an indoor environment.

Eye open/closed detection training task b, eye open/closed detection task in an outdoor environment.

Eye open/closed detection training task c, eye open/closed detection task when the wearable is worn on the eye and the wearable has a spot.

There is no common portion between eye open/close detection training task a and eye open/close detection training task b, there may be a common portion between training task a and training task c, and training task b and training task c. There may be a common part between

In one selectable example, each of the at least two eye open/close detection training tasks of the present disclosure has a corresponding set of images, such as eye open/close detection training task a, eye open/close detection training task b, and eye open/close detection training task of FIG. c has a corresponding image set. Typically, multiple eye images are included for each image set. The eye images contained in different image sets are at least partially different. That is, for one image set, at least some eye images in this image set are absent from other image sets. If desired, the eye images contained in different image sets may have common parts.

Optionally, the image sets corresponding to each of the six eye open/closed detection training tasks listed above are respectively the eye image set with the eye wearer attached, the eye image set without the eye wearer, and the eye image set in the room environment. A set of acquired eye images, a set of eye images acquired in an outdoor environment, a set of eye images with wearables on the eyes and spots on the wearables, and a set of eye images with wearables on the eyes and no spots on the wearables. may be

If desired, all eye images in the set of eye images with wearables on the eyes may be eye images with eyeglasses, for example, the eye image set may be eyeglasses collected in an indoor environment. It may also include images of eyes wearing glasses and images of eyes wearing glasses collected in an outdoor environment.

If desired, all images in the set of eye images where the eye wear is not worn may be eye images without eyeglasses, e.g. and eye images collected in an outdoor environment without glasses.

If desired, the set of eye images collected in an indoor environment may include eye images without eyeglasses collected in an indoor environment and eye images with eyeglasses collected in an indoor environment.

If desired, the set of eye images collected in the outdoor environment may include eye images without eyeglasses collected in the outdoor environment and eye images with eyeglasses collected in the outdoor environment.

If desired, all eye images in the set of eye images with eye wear and spots on eye wear may be eye images with eyeglasses and spots on eyeglasses. For example, the eye image set may include eye images with spectacles and spots on spectacles collected in an indoor environment and eye images with spectacles and spots on spectacles collected in an outdoor environment. good.

If desired, all eye images in the set of eye images with eyewear on and no spots on the eyewear may be eye images with eyeglasses on and no spots on the eyeglasses. For example, the eye image set may include eye images with glasses and no spots on glasses collected in an indoor environment and eye images with glasses and no spots on glasses collected in an outdoor environment. good.

In one alternative example, the image set included in this disclosure is determined by an eye open/close detection training task included in this disclosure. For example, if the present disclosure includes at least two of the six eye open/closed detection training tasks described above, the present disclosure will include eye image sets corresponding to each of the at least two eye open/closed detection training tasks.

In one alternative example, the eye images used in the neural network training process of the present disclosure may be referred to as eye image samples, and typically the image content of the eye image samples includes eyes. The eye image samples of this disclosure are typically eye image samples based on one eye. That is, the image content of the eye image sample does not include both eyes, it includes one eye. If desired, the eye image samples may be eye image samples based on one eye. For example, it may be an eye image sample based on the left eye. Of course, this disclosure does not exclude cases where the eye image samples are both eye-based eye image samples or either eye-based eye image samples.

In one selectable example, the eye images of the present disclosure may generally be eye image blocks cropped from an image containing the eye taken by an imager. For example, the process of forming an eye image in the present disclosure includes performing eye detection on an image captured by an imaging device, determining eye portions in the image, and cropping the detected eye portions from the image, Optionally, the present disclosure performs operations such as zooming and/or image content mapping on the cropped image blocks (e.g., right-eye image blocks are transformed into left-eye image blocks through image content mapping) and eye opening and closing. It may include forming an eye image for training a detection neural network. Of course, the eye image in the present disclosure does not exclude the possibility that the eye image is a complete image including the eye captured by the imaging device. Also, the eye images in this disclosure may be the eye images in the corresponding training sample set.

In one optional example, the eye images for training the eye open/closed detection neural network in this disclosure typically have labeling information, and the labeling information can represent the eye open/closed state in the eye image. That is, the labeling information can indicate whether the eyes in the eye image are open or closed. In one possible example, an eye image labeling information of 1 indicates that the eye in this eye image is in an open-eye state, and an eye image labeling information of 0 indicates that Indicates that the eyes are closed.

In one selectable example, the present disclosure typically obtains each corresponding number of eye images from sets of eye images corresponding to each of the different training tasks. For example, in FIG. 2, a corresponding number of eye images are acquired from the image set corresponding to the eye open/closed detection training task a and provided to the training target eye open/closed detection neural network, and corresponding to the eye open/closed detection training task b. Acquiring a corresponding number of eye images from the image set and providing them to the neural network for eye open/close detection of the training target, acquiring a corresponding number of eye images from the image set corresponding to the eye open/close detection training task c, and providing them to the training target It provides a neural network for eye open/close detection.

In one selectable example, the present disclosure respectively acquires a corresponding number of eye images from the eye image sets corresponding to each of the different training tasks according to a preset proportion of the number of images for the different training tasks. can be done. The process of acquiring eye images also typically takes into account a preset number of batches. For example, if the ratio of the number of images preset for the eye open/close detection training task a, the eye open/close detection training task b, and the eye open/close detection training task c is 1:1:1, then the preset batch processing is performed. If the number is 600, the present disclosure selects 200 eye images from the eye image set corresponding to eye open/close detection training task a, 200 eye images from the eye image set corresponding to eye open/close detection training task b, eye open/closed. 200 eye images can be obtained from the eye image set corresponding to the detection training task c.

Optionally, if the number of eye images in the eye image set corresponding to an eye open/closed detection training task does not reach the corresponding number (e.g., does not reach 200), another A corresponding number of eye images can be obtained from the eye image set corresponding to the eye open/close detection training task. For example, there are only 100 eye images in the eye image set corresponding to the eye open/close detection training task c, and there are only 100 eye images in the eye image set corresponding to each of the eye open/close detection training task a and the eye open/close detection training task b. When the number of both exceeds 250, 250 eye images from the eye image set corresponding to eye open/close detection training task a, 250 eye images from the eye image set corresponding to eye open/close detection training task b, and eye open/close detection. 100 eye images are obtained from the eye image set corresponding to training task c, and a total of 600 eye images can be obtained. This allows for greater flexibility in acquiring eye images.

It should be noted that the present disclosure can adopt the method of randomly setting the number to obtain a corresponding number of eye images from each eye image set corresponding to each different training task. The present disclosure does not limit the specific implementation method for obtaining each corresponding number of eye images from each corresponding eye image set for each different training task. In addition, in the process of acquiring eye images from the eye image set, acquisition of eye images whose labeling information is in an unknown open/closed state should be avoided, thereby improving the detection accuracy of the eye open/close detection neural network. It is advantageous to

In one optional example, the present disclosure provides an order of acquired eye images to a training subject's eyes open/closed detection neural network to perform an eye open/closed state detection process for each input eye image. Each can be performed by a neural network for detection. As a result, the eye open/close detection neural network to be trained sequentially outputs the detection result of the eye open/closed state of each eye image. For example, a single eye image input to the training target eye open/close detection neural network is sequentially processed by the convolution layer, the fully connected layer, and the classification layer. Two probability values are output by the detection neural network, the range of the two probability values is between 0 and 1, and the sum of the two probability values is one. One of the probability values corresponds to the open-eye state, and the closer the magnitude of this probability value to 1, the closer the eye in the eye image is to the open-eye state. Another probability value among them corresponds to the closed-eye state, and the closer the magnitude of this probability value to 1, the closer the eye in the eye image is to the closed-eye state.

S110, based on the eye open/closed labeling information of the eye image and the eye open/closed state detection result output from the neural network, respectively determine a loss corresponding to each of the at least two eye open/closed detection training tasks; Adjust the network parameters of the neural network based on the losses corresponding to each of the eye open/close detection training tasks.

In one optional example, the present disclosure determines a loss corresponding to each eye open/close detection training task, determines a combined loss based on the loss corresponding to each of all training tasks, and utilizes this combined loss. should be used to adjust the network parameters of the neural network. Network parameters in this disclosure may include, but are not limited to, convolution kernel parameters and/or matrix weights. This disclosure does not limit the specific content included in network parameters.

In one optional example, for any eye open/closed detection training task, the present disclosure provides eye open/close state detection output from a neural network for each of a plurality of eye images in an image set corresponding to the training task. Based on the included angle between the maximum probability value of the results and the boundary plane corresponding to the labeling information of the corresponding eye image in the image set, the loss corresponding to the training task can be determined. Optionally, the present disclosure utilizes an A-softmax (Angular Normalized Exponential) loss function based on the eye open/closed labeling information of the eye image and the eye open/close state detection results output from the neural network. , determine the losses corresponding to each of the different eye open/closed detection training tasks, respectively, determine the overall loss (e.g., the sum of each loss) based on the losses corresponding to each of the different eye open/closed detection training tasks, and perform stochastic gradient descent method can be adopted to tune the network parameters of the neural network. For example, the present disclosure uses the A-softmax loss function to calculate the respective losses corresponding to each eye open/close detection training task, and calculates the backproperty based on the sum of the losses corresponding to all eye open/close detection training tasks. Gation processing can be performed to update the network parameters of the trained eye open/close detection neural network by the method of lossy gradient descent.

As can be seen from the above content, the present disclosure, in the process of training a neural network, all eye images provided to the neural network for each iteration training can form a subset of one eye image. This subset of eye images includes eye images corresponding to each training task. Since the present disclosure calculates the loss for each training task, the neural network is capable of learning the ability to detect eye opening and closing ability for each training task in the process of training, and performs ability learning considering different training tasks. can do. This allows the trained neural network to simultaneously increase the accuracy of eye open/close detection in each scene in multiple scenes corresponding to multiple training tasks, and to detect the eyes in different scenes based on the neural network. It is advantageous for promoting the improvement of universality and generalization of the invention of accurately detecting opening and closing, and better meeting the actual application needs for multiple scenes.

式（１） The A-softmax loss function in the present disclosure can be expressed by Equation (1) below.

formula (1)

は、ｉ番目の目画像について、ニューラルネットワークから出力された目開閉状態の検出結果のうちの最大確率値と、ラベリング値に対応する境界面との間の夾角を表す。

は、ｍと上記夾角との積を表す。 In equation (1) above, _Lang represents the loss corresponding to one training task, N represents the number of eye images in the training task, ||*|| represents the modulus of *, and x _i represents the i-th eye image corresponding to the training task, _yi represents the labeling value of the i-th eye image corresponding to the training task, m is a constant, and the minimum value of m is usually is greater than or equal to a predetermined value, for example, greater than or equal to 2+√3;

represents the included angle between the maximum probability value among the eye open/close state detection results output from the neural network and the boundary surface corresponding to the labeling value for the i-th eye image.

represents the product of m and the included angle.

In one alternative example, the training process is terminated when the training of the eye open/close detection neural network to be trained reaches a predetermined iteration condition. The predetermined iteration condition in the present disclosure is that the difference between the detection result of the eye open/closed state output by the training target eye open/closed detection neural network for the eye image and the labeling information of the eye image satisfies the request for the predetermined difference. may include filling. If the difference satisfies the predetermined difference requirement, the neural network has been successfully trained this time. In addition, the predetermined iteration condition in the present disclosure may include training the eye open/closed detection neural network to be trained and the number of eye images to be used reaches a predetermined number of requests. If the number of eye images used reaches the predetermined number requirement, but the difference does not meet the predetermined difference requirement, then the neural network has not been successfully trained this time. A successfully trained neural network can be used in the eye open/closed state detection process.

The present disclosure forms a combined loss based on the losses of different training tasks, uses the combined loss to adjust the network parameters of a neural network for detecting eye open/close, and the neural network detects eye open/close for each training task in the training process. Ability learning is possible with respect to ability detection, and ability learning can be performed considering different training tasks. This allows the trained neural network to simultaneously increase the accuracy of eye open/close detection in each scene in multiple scenes corresponding to multiple training tasks, and to detect the eyes in different scenes based on the neural network. It is advantageous for promoting the improvement of universality and generalization of the invention of accurately detecting opening and closing, and better meeting the actual application needs for multiple scenes.

FIG. 3 shows a flow chart of one embodiment of the eye open/closed state detection method of the present disclosure.

As shown in FIG. 3, the method of this embodiment includes steps: S300 and S310. Each step in FIG. 3 will be described in detail below.

S300, an image to be processed is obtained.

In one selectable example, the processed images of the present disclosure may be images such as still images or photographs, or video frames of dynamic video, e.g., captured by an imager set on a moving object. In another example, it may be a video frame of a video captured by a capture device set in a fixed position. The moving object may be a vehicle, robot, or robotic arm. The fixed position may be a desk or a wall. This disclosure does not limit the embodied forms of moving objects and fixed locations.

In one optional example, the present disclosure can detect eye location regions in the processed image after acquiring the processed image. For example, a bounding box for the eyes of the processed image can be determined, such as by face detection or face keypoint detection methods. The present disclosure then crops the image of the eye region from the processed image based on the eye bounding box, and the cropped eye image block is provided to the neural network. Of course, the clipped eye image blocks can be provided to the neural network after undergoing certain preprocessing. For example, zooming is performed on the cropped eye image block, and the size of the zoomed eye image block satisfies the size requirements of the input image to the neural network. In another example, after clipping the eye image blocks for both eyes of the subject, the mapping process is performed on the eye image blocks on a given side to form two eye image blocks on the same side of the subject. If desired, zooming can also be performed on two same-side eye image blocks. This disclosure does not limit a specific implementation method for cropping an eye image block from a processed image, nor does it limit a specific implementation method for performing pre-processing on the cropped eye image block.

In step S310, an eye open/closed state detection process is performed on the image to be processed via a neural network, and the detection result of the eye open/closed state is output. The neural network in the present disclosure has been successfully trained using embodiments of the neural network training method in the present disclosure.

In one selectable example, the eye open/closed state detection result output from the neural network in the present disclosure for the input eye image block is at least one probability value, such as a probability value indicating that the eyes are in the open state. and a probability value indicating that the eyes are closed. The two probability values both range from 0 to 1, and the sum of the two probability values for the same eye image block is one. The closer to 1 the probability value indicating that the eye is in the open state, the closer the eye in the eye image block is to the open state. The closer the magnitude of the probability value indicating that the eye is in the closed state is to 1, the closer the eye in the eye image block is to the closed eye state.

In an alternative example, the present disclosure can further determine against time-series eye open/close state detection results output from the neural network. This determines the subject's eye behavior, such as fast blinking, or opening one eye and closing the other, or squinting, in a plurality of time-series processed images. can do.

In a selectable example, the present disclosure is based on the time-series detection result of the eye open/close state output from the neural network and the state of other organs of the subject's face. facial expressions such as smiling, laughing or crying or sad.

In an alternative example, the present disclosure can further determine against time-series eye open/close state detection results output from the neural network. This makes it possible to determine the subject's state of fatigue, such as mild fatigue, dozing off, or deep sleep, in a plurality of time-series processed images.

In an alternative example, the present disclosure can further determine against time-series eye open/close state detection results output from the neural network. Since this allows determination of the eye movement of the subject in the plurality of time-series processed images, the present disclosure at least presents the subject in the plurality of time-series processed images based on the eye movement. dialog control information can be determined.

In one alternative example, eye movements, facial expressions, fatigue status, and interaction control information determined by this disclosure can be used for a variety of purposes. For example, predetermined eye movements and/or facial expressions of a subject may be used to trigger predetermined special effects during live/broadcast, or to achieve corresponding human-computer interactions, etc. It is advantageous to diversify the implementation method. For another example, in intelligent driving technology, real-time detection of the driver's fatigue state is advantageous in preventing the phenomenon of fatigue driving. The present disclosure does not limit the specific application of the eye open/closed state detection result output from the neural network.

FIG. 4 shows a flowchart of one embodiment of the intelligent driving control method of the present disclosure. The intelligent driving control method of the present disclosure can be applied in the autonomous driving environment and can also be applied in the cruise driving environment. This disclosure does not limit the application environment of the intelligent driving control method.

As shown in FIG. 4, the method of this embodiment includes steps: S400, S410, S420 and S430. Each step in FIG. 4 will be described in detail below.

S400, obtaining an image to be processed, which is captured by an imaging device mounted on a vehicle. For the specific implementation method of this step, please refer to the description of S300 in FIG. 3 in the above method embodiment, and the details thereof are omitted here.

In step S410, an eye open/closed state detection process is performed on the image to be processed via a neural network, and the detection result of the eye open/closed state is output. The neural network of this example was successfully trained using the above embodiment of the neural network training method. For the specific implementation method of this step, please refer to the description of S310 in FIG. 3 in the embodiment of the above method, and the details thereof are omitted here.

S420, the subject's fatigue state is determined at least based on the detection result of the same subject's eye open/closed state in a plurality of time-series processed images.

In one selectable example, the subject of this disclosure is typically a vehicle driver. The present disclosure belongs to the same subject, and based on the detection results of multiple eye open/closed states in time series, the number of times this subject (for example, a driver) blinks per unit time, the closed eye time per time, or An indicator parameter, such as the eye-opening time per session, can be determined so that the corresponding indicator parameter can be further determined using the predetermined indicator request to determine whether the subject (e.g., the driver) is in a state of fatigue. can decide whether Fatigue conditions in the present disclosure may include various different degrees of fatigue, such as, for example, mild fatigue, moderate fatigue, or severe fatigue. This disclosure does not limit specific implementations for determining a subject's fatigue status.

S430, generate and output a command according to the subject's fatigue state.

In one selectable example, the present disclosure provides commands generated according to the subject's fatigue state, such as a command to switch to an intelligent driving state, a voice warning command for fatigue driving, a vibration wake-up command, and a dangerous driving information notification command. At least one of such may be included. This disclosure does not limit the embodied form of the directive.

The neural network trained by the neural network training method of the present disclosure is advantageous in improving the accuracy of the detection result of the eye open/closed state of the neural network. Therefore, judging the state of fatigue using the detection results of the eye open/closed state output from this neural network is useful for improving the accuracy of detecting the state of fatigue. This is advantageous for avoiding fatigue driving and further for driving safety.

FIG. 5 shows a structural schematic diagram of an embodiment of the neural network training device of the present disclosure. The neural network training apparatus shown in FIG. 5 includes a training target eye open/close detection neural network 500 and an adjustment module 510 . Optionally, the device may further include an input module 520. FIG.

The eye open/closed detection neural network 500 to be trained performs an open/closed state detection process on each of a plurality of eye images in an image set corresponding to each of at least two eye open/closed detection training tasks to detect the open/closed state of the eye. Used to output results. The eye images contained in different image sets are at least partially different.

In one selectable example, the training target eye open/closed detection neural network 500 of the present disclosure is trained, then performs eye open/close state detection on the processed image, and outputs the eye open/closed state detection result of the processed image to Can be used for output. For example, for one processed image, the neural network 500 outputs two probability values, one of which indicates the probability that the subject's eyes are open in the processed image, and this probability value The larger is, the closer the eye is to the open state. The other probability value indicates the probability that the subject's eyes are closed in the image to be processed. The sum of the two probability values may be one.

In one alternative example, neural network 500 in this disclosure may be a convolutional neural network. Neural network 500 in the present disclosure may include, but is not limited to, convolutional layers, Relu layers (also called activation layers), pooling layers, fully connected layers, and layers for classification (eg, binary classification). The more layers included in this neural network 500, the deeper the network. This disclosure does not limit the specific configuration of neural network 500 .

In one selectable example, there are at least two eye open/closed detection training tasks involved in the process of training the neural network 500 in this disclosure, and each eye open/closed detection training task both enables the neural network to detect an eye open/closed state. should belong to the overall training task for The training targets corresponding to different eye open/close detection training tasks are not exactly the same. That is, the present disclosure can divide the overall training task of neural network 500 into multiple training tasks, one training task corresponding to one training target, and different training targets corresponding to different training tasks.

In one selectable example, the at least two eye open/closed detection training tasks of the present disclosure are an eye open/closed detection task with wearable eyes, an eye open/closed detection task with no eye wearables, Eye open/close detection task in the environment, Eye open/close detection task in an outdoor environment, Eye open/close detection task when the wearable object is attached to the eye and a spot is on the wearable object, Eye wearable object is attached to the eye and the wearable object is spotted at least two of the eye open/closed detection tasks in the absence of . The wearable item may be eyeglasses or a transparent plastic sheet. The spot may be a spot formed on the wearable by reflection of the wearable. For the details of the above listed tasks, please refer to the description of the above method embodiments, and the details thereof are omitted here.

In one alternative example, each of the at least two eye open/closed detection training tasks of the present disclosure has a corresponding image set, typically including multiple eye images per image set. The eye images contained in different image sets are at least partially different. That is, for one image set, at least some eye images in this image set are absent from other image sets. If desired, the eye images contained in different image sets may have common parts.

Optionally, the image sets corresponding to each of the six eye open/closed detection training tasks listed above are respectively the eye image set with the eye wearer attached, the eye image set without the eye wearer, and the eye image set in the room environment. A set of acquired eye images, a set of eye images acquired in an outdoor environment, a set of eye images with wearables on the eyes and spots on the wearables, and a set of eye images with wearables on the eyes and no spots on the wearables. may be For the details of the above-listed image sets, please refer to the description of the above method embodiments, and the details thereof are omitted here.

In one alternative example, the image set included in this disclosure is determined by an eye open/close detection training task included in this disclosure. For example, if the present disclosure includes at least two of the six eye open/closed detection training tasks described above, the present disclosure will include eye image sets corresponding to each of the at least two eye open/closed detection training tasks.

In one selectable example, the eye images in this disclosure may be eye image blocks that are cropped from those that typically include eye images captured by an imager. For the process of forming an eye image in the present disclosure, please refer to the description of the above method embodiments, and the details thereof are omitted here.

In one alternative example, the eye images for training the eye open/closed detection neural network 500 of the present disclosure typically have labeling information, and the labeling information can represent the eye open/closed state in the eye images. . If desired, it is also possible to indicate that the eyes in the labeling information eye image in the present disclosure are in an unknown open/closed state. However, eye images for training the eye open/close detection neural network 500 in the present disclosure usually do not include eye images whose labeling information is in an unknown open/closed state. This is advantageous for avoiding the influence of eye opening and closing, and is advantageous for improving the detection accuracy of the eye open/close detection neural network 500 .

The input module 520 is used to obtain a corresponding number of eye images from different image sets and provide them to the training target eye open/close detection neural network 500 . For example, for different eye open/close detection training tasks, the input module 520 acquires a corresponding number of eye images from different image sets according to the proportion of the number of images preset for the different eye open/close detection training tasks, and trains the corresponding number of eye images. It is used to provide a neural network 500 for eye open/closed detection of a subject. The input module 520 also typically considers a preset number of batches in the process of acquiring eye images. For example, if the ratio of the number of images preset for the eye open/close detection training task a, the eye open/close detection training task b, and the eye open/close detection training task c is 1:1:1, then the preset batch processing is performed. If the number is 600, the input module 520 outputs 200 eye images from the eye image set corresponding to eye open/close detection training task a, 200 eye images from the eye image set corresponding to eye open/close detection training task b, eye 200 eye images can be obtained from the eye image set corresponding to the open/close detection training task c.

Optionally, if the number of eye images in the set of eye images corresponding to an eye open/closed detection training task does not reach the corresponding number (e.g., does not reach 200), the input module 520 is configured to reach the batch number. A corresponding number of eye images can be obtained from eye image sets corresponding to other eye open/close detection training tasks. For example, there are only 100 eye images in the eye image set corresponding to the eye open/close detection training task c, and there are only 100 eye images in the eye image set corresponding to each of the eye open/close detection training task a and the eye open/close detection training task b. If both numbers exceed 250, the input module 520 selects 250 eye images from the eye image set corresponding to eye open/close detection training task a, and 250 eye images from the eye image set corresponding to eye open/close detection training task b. , 100 eye images can be obtained from the eye image set corresponding to the eye open/closed detection training task c. This causes the input module 520 to acquire a total of 600 eye images.

In addition, a method of randomly setting the number of input modules 520 can be adopted to obtain a corresponding number of eye images from each eye image set corresponding to each different training task. The present disclosure does not limit the specific implementation method for the input module 520 to respectively acquire a corresponding number of eye images from each corresponding set of eye images for different training tasks. In addition, in the process of acquiring eye images from the eye image set, the input module 520 should avoid acquiring eye images whose labeling information is in an unknown open/closed state, so that the neural network for detecting eye open/closed detection will not be able to detect such images. This is advantageous for improving accuracy.

In one selectable example, the input module 520 provides an order of the acquired eye images to the trained eye open/closed detection neural network 500 to perform eye open/closed state detection processing on each of the input eye images. This is performed by the eye open/closed detection neural network 500, whereby the training target eye open/closed detection neural network 500 sequentially outputs the detection result of the eye open/closed state of each eye image. For example, a single eye image input to the training target eye open/close detection neural network 500 is sequentially subjected to convolution layer processing, fully connected layer processing, and classification layer processing. Two probability values are output by the open/close detection neural network 500, the range of the two probability values is 0 to 1, and the sum of the two probability values is one. One of the probability values corresponds to the open-eye state, and the closer the magnitude of this probability value to 1, the closer the eye in the eye image is to the open-eye state. Another probability value among them corresponds to the closed-eye state, and the closer the magnitude of this probability value to 1, the closer the eye in the eye image is to the closed-eye state.

The adjustment module 510 determines a loss corresponding to each of the at least two eye open/closed detection training tasks based on the eye open/closed labeling information of the eye image and the eye open/closed state detection result output from the neural network 500, and at least It is used to tune the network parameters of neural network 500 based on the losses corresponding to each of the two eye open/closed detection training tasks.

In one alternative example, adjustment module 510 should determine a corresponding loss for each eye open/close detection training task, and determine a total loss based on the respective losses for all training tasks. Adjustment module 510 uses this total loss to adjust the network parameters of the neural network. Network parameters in this disclosure may include, but are not limited to, convolution kernel parameters and/or matrix weights. This disclosure does not limit the specific content included in network parameters.

In one optional example, for any eye open/closed detection training task, the adjustment module 510 adjusts the eye open/close state output from the neural network for each of the plurality of eye images in the image set corresponding to the training task. A loss corresponding to the training task can be determined based on the included angle between the maximum probability value among the detection results and the boundary surface corresponding to the labeling information of the corresponding eye image in the image set.

Optionally, the adjustment module 510 utilizes an A-softmax (Angular Normalized Exponential) loss function based on the eye open/closed labeling information of the eye image and the eye open/close state detection results output from the neural network. to determine a loss corresponding to each of the different eye open/closed detection training tasks, and a total loss (eg, the sum of each loss) based on the losses corresponding to each of the different eye open/closed detection training tasks. The tuning module 510 can then employ stochastic gradient descent to tune the network parameters of the neural network. For example, the adjustment module 510 may use the A-softmax loss function to calculate the respective losses corresponding to each eye open/close detection training task, and back up based on the sum of the losses corresponding to all eye open/close detection training tasks. Propagation processing may be performed to update the network parameters of the trained eye open/close detection neural network 500 by the method of lossy gradient descent.

In one alternative example, when the training of the trained eye open/close detection neural network 500 reaches a predetermined iteration condition, the adjustment module 510 can control the current training process to end. The predetermined iteration condition in the present disclosure is that the difference between the detection result of the eye open/closed state output by the training target eye open/closed detection neural network 500 for the eye image and the labeling information of the eye image is a predetermined difference. may include filling. If the difference satisfies the predetermined difference requirement, the neural network 500 has been successfully trained this time.

Optionally, the predetermined iteration conditions used by the adjustment module 510 may include training a subject neural network for eye open/close detection, and that the number of eye images used has reached a predetermined number requirement. good. If the number of eye images used reaches the predetermined number requirement, but the difference does not meet the predetermined difference requirement, then the neural network 500 was not successfully trained this time. A successfully trained neural network 500 can be used in the eye open/closed state detection process.

FIG. 6 shows a configuration schematic diagram of an embodiment of an eye open/closed state detection apparatus of the present disclosure. As shown in FIG. 6, the apparatus of this embodiment includes acquisition module 600 and neural network 610 . Optionally, the eye open/closed state detection device may further include a determination module 620 .

Acquisition module 600 is used to acquire the processed image.

In one selectable example, the processed image acquired by the acquisition module 600 may be an image such as a still image or a photograph, or a video frame of a dynamic video, such as a shot set on a moving object. It may be a video frame of a video captured by the device, or in another example, a video frame of a video captured by a capture device set in a fixed position. The moving object may be a vehicle, robot, or robotic arm. The fixed position may be a desk or a wall.

In one selectable example, the acquisition module 600 can detect eye location regions in the processed image after acquiring the processed image. For example, the acquisition module 600 can determine the eye bounding boxes of the processed image, such as by face detection or face keypoint detection methods. The acquisition module 600 then crops the image of the eye region from the processed image based on the eye bounding box, and the cropped eye image block is provided to the neural network 610 . Of course, the acquisition module 600 can perform certain pre-processing on the cropped eye image block before providing it to the neural network 610 . For example, the acquisition module 600 performs a zoom process on the cropped eye image block, and the size of the zoomed eye image block satisfies the size requirements of the input image to the neural network. In another example, after the eye image blocks for both eyes of the subject are cropped, the eye image blocks on a predetermined side of them are mapped by the acquisition module 600 to obtain two eye image blocks on the same side of the subject. form. If desired, the acquisition module 600 can also perform zoom processing on the two same-side eye image blocks. The present disclosure does not limit the specific implementation method for the acquisition module 600 to cut out the eye image block from the processed image, but the specific implementation for the acquisition module 600 to pre-process the cut eye image block. The method is also not limited.

The neural network 610 is used to perform eye open/closed state detection processing on the image to be processed and output the eye open/closed state detection result.

In one selectable example, the eye open/closed state detection result output from the neural network 610 of the present disclosure for the input eye image block has at least one probability value, e.g. value and a probability value indicating that the eye is in an eye-closed state. The two probability values both range from 0 to 1, and the sum of the two probability values for the same eye image block is one. The closer to 1 the probability value indicating that the eye is in the open state, the closer the eye in the eye image block is to the open state. The closer the magnitude of the probability value indicating that the eye is in the closed state is to 1, the closer the eye in the eye image block is to the closed eye state.

The determination module 620 determines at least the subject's eye movement and/or expression and/or fatigue state and/or dialogue control based on the detection result of the same subject's eye open/closed state in a plurality of time-series processed images. Used to determine information.

In one selectable example, the subject's eye action is, for example, a quick blink action, or an action of opening one eye and closing another eye, or a squinting action. The facial expression of the subject is, for example, smiling, laughing, crying or sad. The subject's state of fatigue is, for example, mild fatigue or doze or deep sleep. The interaction control information represented by the target person is, for example, confirmation or refusal.

FIG. 7 shows a configuration schematic diagram of an embodiment of the intelligent driving control device of the present disclosure. The apparatus shown in FIG. 7 mainly includes an acquisition module 600 , a neural network 610 , a fatigue state determination module 700 and a command module 710 .

Acquisition module 600 is used to acquire processed images acquired by an imager mounted on a vehicle.

The neural network 610 is used to perform eye open/closed state detection processing on the image to be processed and output the eye open/closed state detection result.

For the operations specifically performed by the acquisition module 600 and the neural network 610, please refer to the description of the embodiments of the above apparatus, and the details thereof are omitted here.

The fatigue state determination module 700 is used to determine the fatigue state of the subject at least based on the detection results of the eye open/close state of the same subject in a plurality of time-series processed images.

In one selectable example, the subject of this disclosure is typically a driver. The fatigue state determination module 700 belongs to the same subject and determines the number of times the subject (for example, a driver) blinks per unit time, based on the results of monitoring a plurality of eye open/closed states in time series. Indicator parameters such as eye closure time or eye open time per session can be determined. Accordingly, the fatigue state determination module 700 uses the predetermined index request to further determine the corresponding index parameters. The fatigue state determination module 700 can determine whether the subject (eg, the driver) is in a state of fatigue. Fatigue conditions in the present disclosure may include various different degrees of fatigue, such as, for example, mild fatigue, moderate fatigue, or severe fatigue. This disclosure does not limit the specific implementation method for the fatigue state determination module 700 to determine the fatigue state of the subject.

The command module 710 is used to generate and output commands according to the fatigue state of the subject.

In one selectable example, the command generated by the command module 710 according to the subject's fatigue state includes a command to switch to an intelligent driving state, a voice warning command for fatigue driving, a vibration wake-up command, and a notification of dangerous driving information. It may include at least one of instructions and the like. This disclosure does not limit the embodied form of the directive.

The neural network 610 trained by the neural network training method of the present disclosure is advantageous in improving the accuracy of the detection result of the eye open/close state of the neural network. Therefore, the fatigue state determination module 700 determines the fatigue state using the detection result of the eye open/closed state output from the neural network 610, which helps improve the accuracy of the fatigue state detection. Accordingly, the command module 710 generates a command according to the detected fatigue state, which is advantageous for avoiding fatigue driving and further driving safety.

Exemplary Equipment FIG. 8 shows a block diagram of an exemplary equipment of an embodiment of the present disclosure. This device 800 may be a control system/electronic system installed in an automobile, a mobile terminal (e.g., smart phone, etc.), a personal computer (PC, e.g., desktop computer or notebook computer, etc.), a tablet computer, a server, etc. . In FIG. 8, device 800 includes one or more processors, communications, etc., wherein said one or more processors are one or more central processing units (CPUs) 801 and/or one or more may be the acceleration unit 813 of Acceleration unit 813 may be a graphics processor (GPU) or the like. The processor performs various appropriate operations and processes based on executable instructions stored in read only memory (ROM) 802 or loaded from storage 808 into random access memory (RAM) 803. can. The communication unit 812 may include, but is not limited to, a network card, and the network card may include, but is not limited to, an IB (InfiniBand) network card. The processor communicates with read-only memory 802 and/or random-access memory 803 to execute executable instructions, is connected to communication portion 812 via bus 804, and communicates with other target devices via communication portion 812. complete the corresponding steps of this disclosure by doing.

For the operations performed by each of the above instructions, please refer to the related descriptions of the above method embodiments, and the details thereof are omitted here. The RAM 803 can also store various programs and data necessary for the operation of the device. CPU 801 , ROM 802 and RAM 803 are interconnected via bus 804 .

If RAM 803 is present, ROM 802 is an optional module. RAM 803 stores or writes executable instructions to ROM 802 during operation, which cause central processing unit 801 to perform the steps included in the methods described above. Input/output (I/O) interface 805 is also connected to bus 804 . The communication unit 812 may be arranged integrally, or may have multiple sub-modules (eg, multiple IB network cards), each configured to be connected to a bus.

The following means are connected to the I/O interface 805: an input section 806 including a keyboard and mouse, etc., an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LDC) and speakers, etc., a memory including a hard disk. section 808, and communication section 809 including network interface cards such as LAN cards, modems, and the like. A communication unit 809 performs communication processing via a network such as the Internet. Drivers 810 are also connected to I/O interfaces 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory can be attached to the drive 810 as needed, and a computer program read therefrom can easily be attached to the storage unit 808 as needed. become.

Note that the architecture shown in FIG. 8 is only an alternative implementation. In specific practice, the number and type of means in FIG. 8 above can be selected, deleted, added or replaced according to actual needs. Regarding the arrangement of members with different functions, a method of realizing such as distributed installation or integrated installation may be adopted. For example, the acceleration unit 813 and the CPU 801 can be distributed. In another example, acceleration unit 813 can be integrated into CPU 801 . The communication part can be distributed and can be integrated into the CPU 801 or the acceleration unit 813 . All these alternative embodiments fall within the protection scope of the present disclosure.

In particular, according to embodiments of the present disclosure, the processes described below with reference to flowcharts may be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program embodied on a machine-readable medium, the computer program including program code for performing the steps shown in the flowcharts, the program code described in the present disclosure. It may contain instructions corresponding to performing the steps of such methods.

In such embodiments, the computer program may be downloaded and installed from a network via communications portion 809 and/or installed from removable media 811 . When this computer program is executed by central processing unit (CPU) 801, the instructions described in this disclosure for performing the corresponding steps above are executed.

In one or more optional embodiments, the embodiments of the present disclosure further, when executed, cause a computer to perform a neural network training method or an eye open/closed state detection method or an intelligent method according to any of the above embodiments. A computer program product is provided for storing computer readable instructions for performing an operational control method.

This computer program product may be embodied in hardware, software, or a combination thereof. In one option, the computer program product is embodied in a computer storage medium. In another alternative, the computer program product is embodied as a software product, such as a Software Development Kit (SDK).

In one or more optional embodiments, the examples of the present disclosure further provide another eye open/closed state detection method, intelligent driving control method and neural network training method and corresponding devices and electronics, computer storage. Provide a medium, a computer program and a computer program product, the method of which is a method for training a neural network or a method for detecting an eye open/closed state or an intelligent driving control in any of the above possible practicable embodiments by a first device Sending to the second device a neural network training command or an eye open/close state detection command or an intelligent driving control command for causing the second device to perform the method; receiving a neural network training result or an eye open/close state detection result or an intelligent driving control result.

In some embodiments, this neural network training command or eye open/closed state detection command or intelligent driving control command may specifically be a calling command, and the first device trains the neural network to call the command. The second device can be caused to perform the operation or eye open/closed state detection operation or intelligent driving control operation, and correspondingly, in response to the received call command, the second device performs the above neural network training method or eyes. The steps and/or flows in either embodiment of the open/closed state detection method or the intelligent driving control method can be performed.

It should be understood that terms such as "first" and "second" in embodiments of the present disclosure are for distinction only and should not be construed as limiting examples of the present disclosure. It should also be understood that, in this disclosure, "plurality" can refer to two or more, and "at least one" can refer to one, two, or more. Further, it should also be understood that any member, data or structure referred to in this disclosure is generally to be understood as one or more unless there is a clear limitation or the context suggests otherwise. Also, the description of various embodiments in this disclosure emphasizes the differences between each embodiment, and the same or similar references may be made to each other, and for the sake of brevity, they are repeated one by one. It should also be understood that no

The methods and apparatus, electronics, and computer-readable storage media of the disclosure may be implemented in many ways. For example, the methods and apparatus, electronic devices and computer readable storage media of the present disclosure can be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps used in the method is for illustration only, and unless otherwise stated, the steps of the method of the present disclosure are not limited to the above specifically listed order. Further, in some embodiments the present disclosure may be implemented as a program recorded on a recording medium. These programs contain machine-readable instructions for implementing the methods of the present disclosure. Accordingly, the present disclosure also covers recording media for storing programs for performing the methods of the present disclosure.

The description of this disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or to limit the disclosure to each form presented. Various modifications and alterations will be apparent to those skilled in the art. The selection and description of the embodiments better explain the principles and practical applications of the disclosure, and enable those skilled in the art to understand the embodiments of the disclosure and design each embodiment with various modifications to suit a particular application. This is to ensure that

Claims (12)

An eye open/closed state detection process is performed on each of a plurality of eye images in an image set corresponding to each of at least two eye open/closed detection training tasks via an eye open/close detection neural network to be trained. outputting the detection result of the open/closed state;
determining a loss corresponding to each of the at least two eye open/closed detection training tasks based on eye open/closed labeling information of the eye image and eye open/closed state detection results output from the neural network; adjusting network parameters of the neural network based on losses corresponding to each of two eye open/close detection training tasks;
including
the eye images in the different image sets are at least partially different,
The at least two eye open/closed detection training tasks include an eye open/closed detection task when the wearable object is worn on the eye, an eye open/closed detection task when the wearable object is not worn on the eye, an eye open/closed detection task in an indoor environment, Eye open/closed detection task in an outdoor environment, eye open/closed detection task when the wearable object is attached to the eye and there is a spot, eye open/closed detection task when the wearable object is attached to the eye and there is no spot on the eyewear and corresponding to each of the at least two eye open/closed detection training tasks are an eye image set with the wearer attached to the eye and an eye image set without the wearer attached to the eye. , an eye image set collected in an indoor environment, an eye image set collected in an outdoor environment, an eye image set with a wearable on the eye and a spot on the wearable, and a set of eye wear with the wearable on the eye and a spot on the wearable. including at least two of the non-eye image sets;
adjusting network parameters of the neural network based on losses corresponding to each of the at least two eye open/close detection training tasks;
determining a combined loss of the at least two eye open/closed detection training tasks based on losses corresponding to each of the at least two eye open/closed detection training tasks; and determining network parameters of the neural network based on the combined loss. A method of training a neural network , comprising: adjusting performing an eye open/closed state detection process on each of a plurality of eye images in an image set corresponding to each of at least two eye open/closed detection training tasks via the eye open/closed detection neural network to be trained; Outputting the detection result of the eye open/closed state is
respectively obtaining a corresponding number of eye images from different image sets for different eye open/close detection training tasks according to the proportion of the number of images preset for the different eye open/close detection training tasks;
An eye open/closed state detection process is performed on each of the corresponding number of eye images via the eye open/closed detection neural network to be trained, and the eye open/closed state detection result corresponding to each eye image is obtained. to output;
2. The method of claim 1 , comprising: Determining a loss corresponding to each of the at least two eye open/closed detection training tasks based on eye open/closed labeling information of the eye image and eye open/closed state detection results output from the neural network, respectively;
For any eye open/closed detection training task, the maximum probability value among the eye open/closed state detection results output from the neural network for each of a plurality of eye images in the image set corresponding to the training task; 3. The method of claim 1 or 2 , comprising determining a loss corresponding to the training task based on an included angle between a boundary surface corresponding to the labeling information of the corresponding eye image in the image set. the method of. obtaining a processed image;
performing an eye open/closed state detection process on the image to be processed via a neural network and outputting a detection result of the eye open/closed state;
including
A method for detecting an open/closed state of eyes, wherein the neural network is trained by the method according to any one of claims 1 to 3 . Determining eye movement and/or facial expression and/or fatigue state and/or dialogue control information of the subject based on at least detection results of eye open/closed states of the same subject in a plurality of time-series processed images. 5. The method of claim 4 , further comprising: Acquiring an image to be processed collected by an imaging device mounted on a vehicle;
performing an eye open/closed state detection process on the image to be processed via a neural network and outputting a detection result of the eye open/closed state;
Determining the state of fatigue of the subject at least based on detection results of the eye open/closed state of the same subject in a plurality of time-series images to be processed;
generating and outputting a command according to the subject's fatigue state;
including
An intelligent driving control method, wherein the neural network is trained by the method according to any one of claims 1 to 3 . Training used for performing eye open/closed state detection processing on each of a plurality of eye images in an image set corresponding to each of at least two eye open/closed detection training tasks and outputting eye open/closed state detection results. a neural network for detecting eye opening and closing of a subject;
determining a loss corresponding to each of the at least two eye open/closed detection training tasks based on eye open/closed labeling information of the eye image and eye open/closed state detection results output from the neural network; an adjustment module used to adjust network parameters of the neural network based on losses corresponding to each of two eye open/close detection training tasks;
including
the eye images in the different image sets are at least partially different,
The at least two eye open/closed detection training tasks include an eye open/closed detection task when the wearable object is worn on the eye, an eye open/closed detection task when the wearable object is not worn on the eye, an eye open/closed detection task in an indoor environment, Eye open/closed detection task in an outdoor environment, eye open/closed detection task when the wearable object is attached to the eye and there is a spot, eye open/closed detection task when the wearable object is attached to the eye and there is no spot on the eyewear and corresponding to each of the at least two eye open/closed detection training tasks are an eye image set with the wearer attached to the eye and an eye image set without the wearer attached to the eye. , an eye image set collected in an indoor environment, an eye image set collected in an outdoor environment, an eye image set with a wearable on the eye and a spot on the wearable, and a set of eye wear with the wearable on the eye and a spot on the wearable. including at least two of the non-eye image sets;
adjusting network parameters of the neural network based on losses corresponding to each of the at least two eye open/close detection training tasks;
determining a combined loss of the at least two eye open/closed detection training tasks based on losses corresponding to each of the at least two eye open/closed detection training tasks; and determining network parameters of the neural network based on the combined loss. and adjusting the neural network training device. an acquisition module used to acquire a processed image;
a neural network used to perform eye open/closed state detection processing on the image to be processed and to output a detection result of the eye open/closed state;
including
8. An eye open/closed state detection apparatus, wherein the neural network is trained by the apparatus according to claim 7 . an acquisition module used to acquire an image to be processed acquired by an imaging device mounted on a vehicle;
a neural network used to perform eye open/closed state detection processing on the image to be processed and to output a detection result of the eye open/closed state;
a fatigue state determination module used to determine the fatigue state of the subject based on detection results of the eye open/closed state of the same subject in at least a plurality of time-series images to be processed;
a command module used to generate and output a command according to the fatigue state of the subject;
including
8. An intelligent driving control system, wherein said neural network is trained by the system of claim 7 . a memory for storing a computer program;
a processor executing a computer program stored in said memory and, when said computer program is executed, realizing the method according to any one of the preceding claims 1 to 6 ;
electronic equipment including; A computer readable storage medium storing a computer program which, when executed by a processor, implements the method of any one of claims 1 to 6 . A computer program product comprising computer instructions which, when executed in a processor of an apparatus, implements the method of any one of the preceding claims 1-6 .

Images