CN106295474A - The fatigue detection method of deck officer, system and server - Google Patents

Abstract

The invention provides a fatigue detection method, system and server for ship drivers. Wherein, the fatigue detection method includes: receiving a video stream collected by a wearable device; converting a plurality of video frames in the video stream into a plurality of image information; acquiring the human eye area in the plurality of image information; and analyzing the plurality of image information Fatigue analysis is carried out in the human eye area to determine whether the ship driver is in a state of fatigue, and the analysis results are sent to the wearable device. In the fatigue detection method of the embodiment of the present invention, the wearable device acquires the video image of the ship driver, which can avoid the influence of various objective factors and ensure the quality of the collected video image, thereby improving the reliability of server fatigue detection. Moreover, when the ship driver is in a fatigue state, the wearable device reminds him, thereby greatly improving the safety of the ship driver when driving the ship, avoiding accidents, and ensuring the safety of the ship driver's life and property.

Description

Fatigue detection method, system and server for ship driver

technical field

The invention relates to the technical field of shipping, in particular to a fatigue detection method, system and server for ship drivers.

Background technique

In recent years, with the rapid development of my country's shipping industry, its comprehensive strength has been significantly improved overall, and conditions for safe and scientific development have been prepared. However, there are still many problems in the process of rapid development of the shipping industry, and safety problems are more prominent. When major safety hazards cannot be effectively controlled, safety accidents occur from time to time, and the life and property safety of ship drivers are greatly affected. threaten.

Compared with the rapid development of fatigue recognition technology for vehicle drivers, the fatigue recognition technology for ship drivers is still in its infancy. At present, the fatigue recognition technology for vehicle drivers mainly includes, for example, the driver warning system launched by Volvo Cars to assist drivers in improving driving safety, and to give warnings in time before the driver enters a sleep state; by Carnegie Mellon University The developed PERCLOS system can judge the driver's fatigue status by analyzing the position and opening of the driver's eyes; The driver's fatigue status can be monitored in real time; the European Union's AWAKE system can monitor the driver's eye condition, line of sight direction, steering wheel grip and other information in real time through comprehensive monitoring of driver behavior and by using various sensors such as image and pressure.

However, compared with vehicle driver fatigue recognition technology, the development of ship driver fatigue recognition technology is mainly affected by the following aspects:

(1) The area of the cockpit of the ship is relatively large, and the ship driver usually takes behaviors such as looking sideways to observe the water surface environment during the driving process. Therefore, the range of activities of the ship driver is relatively large, and the existing vehicle driver fatigue recognition technology is difficult to comprehensively and accurately collect the status information of the ship driver.

(2) The ship driver has the characteristics of simple and single operation during the driving process, and the slow speed of the ship makes the ship driver more fault-tolerant during the driving process, which leads to the standardized operation of the ship driver during the driving process. Awareness is weak.

(3) The driving environment of the ship is mainly affected by the natural environment and the ship environment. Because the water environment is affected by many factors such as heavy fog and water surface fluctuations, the water surface environment is usually more complex and changeable than the road environment. Furthermore, the noise and vibration of the equipment in the ship are relatively complex, and the influence of many environmental factors increases the labor level and psychological pressure of the ship driver, which can easily cause the fatigue of the ship driver.

Therefore, compared with the fatigue detection technology for vehicle drivers, the fatigue detection technology for ship drivers is relatively complicated, and there are many factors to be considered. However, the speed of the ship is relatively slow compared to the speed of the vehicle, and it has a strong fault tolerance for ship fatigue driving, so the real-time requirements for ship fatigue detection are not high.

Contents of the invention

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

For this reason, the first purpose of the present invention is to propose a fatigue detection method for ship pilots. The fatigue detection method acquires video images of ship pilots through wearable devices, which can avoid the influence of various objective factors and ensure the accuracy of collection. The quality of video images, thereby improving the reliability of server fatigue detection. Moreover, when the ship driver is in a fatigue state, the wearable device reminds him, thereby greatly improving the safety of the ship driver when driving the ship, avoiding accidents, and ensuring the safety of the ship driver's life and property.

The second object of the present invention is to propose a fatigue detection system for ship drivers.

A third object of the present invention is to propose a server.

In order to achieve the above purpose, the embodiment of the first aspect of the present invention proposes a fatigue detection method for ship drivers, including the following steps: receiving a video stream collected by a wearable device; converting a plurality of video frames in the video stream into A plurality of image information; obtaining the human eye area in the plurality of image information; and performing fatigue analysis on the human eye area in the plurality of image information to determine whether the ship driver is in a fatigue state, and sending the analysis result to The wearable device.

The fatigue detection method of the ship driver in the embodiment of the present invention can avoid the influence of various objective factors, including the lighting environment, the water surface fluctuation environment, the operating environment and the water surface in front of the ship driver by acquiring the video image of the ship driver through the wearable device. Field of view, etc., based on the front-end acquisition system of wearable devices, can collect clear video images, and can also guarantee the quality of the collected video images under harsh conditions such as shaking inside the ship and insufficient light, thereby improving the reliability of server fatigue detection . Moreover, the machine vision technology is integrated into the fatigue detection method, and the video image is sent to the server through the wearable device, and the video image is processed, human eye positioning and fatigue detection are performed through the server, and when the ship driver is in a fatigue state, the wearable The equipment reminds it to warn the ship driver, thus greatly improving the safety of the ship driver when driving the ship, avoiding accidents, and ensuring the safety of the ship driver's life and property.

In order to achieve the above purpose, the embodiment of the second aspect of the present invention proposes a fatigue detection system for ship drivers, including a server and a wearable device, wherein the wearable device is used to collect video streams and store the video streams Send to the server, and receive the analysis result sent by the server; the server is used to receive the video stream collected by the wearable device, and convert multiple video frames in the video stream into multiple image information , and acquire the human eye area in the plurality of image information, and perform fatigue analysis on the human eye area in the plurality of image information to determine whether the ship driver is in a fatigue state, and send the analysis result to the available wearable device.

The fatigue detection system of the ship driver in the embodiment of the present invention can avoid the influence of various objective factors by acquiring the video image of the ship driver through the wearable device, including the lighting environment, the water surface fluctuation environment, the operating environment and the water surface in front of the ship driver Field of view, etc., based on the front-end acquisition system of wearable devices, can collect clear video images, and can also guarantee the quality of the collected video images under harsh conditions such as shaking inside the ship and insufficient light, thereby improving the reliability of server fatigue detection . Moreover, the machine vision technology is integrated into the fatigue detection method, and the video image is sent to the server through the wearable device, and the video image is processed, human eye positioning and fatigue detection are performed through the server, and when the ship driver is in a fatigue state, the wearable The equipment reminds it to warn the ship driver, thus greatly improving the safety of the ship driver when driving the ship, avoiding accidents, and ensuring the safety of the ship driver's life and property.

To achieve the above purpose, the embodiment of the first aspect of the present invention proposes a server, including: a receiving module, used to receive a video stream collected by a wearable device; a conversion module, used to convert multiple video frames in the video stream converted into a plurality of image information; the acquisition module is used to obtain the human eye area in the plurality of image information; and the analysis module is used to perform fatigue analysis on the human eye area in the plurality of image information to judge ship driving Whether the employee is in a state of fatigue, and the analysis result is sent to the wearable device.

The server of the embodiment of the present invention processes video images, locates human eyes and detects fatigue, and reminds the ship driver through a wearable device when he is in a fatigued state, so as to warn the ship driver, thereby greatly improving the safety of the ship driver. The safety when driving the ship avoids the occurrence of accidents and ensures the safety of the life and property of the ship driver.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is the flowchart of the fatigue detection method of the ship driver of an embodiment of the present invention;

Fig. 2 is the flowchart of the fatigue detection method of the ship driver of a specific embodiment of the present invention;

Fig. 3 is the schematic diagram of Haar characteristic in the embodiment of the present invention;

Fig. 4 is the schematic diagram of integral diagram in the embodiment of the present invention;

Fig. 5 is a structural schematic diagram of a ship driver's fatigue detection system according to an embodiment of the present invention; and

Fig. 6 is a schematic structural diagram of a server according to an embodiment of the present invention.

detailed description

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

Fig. 1 is a flow chart of a fatigue detection method for a ship driver according to an embodiment of the present invention, and Fig. 2 is a flow chart of a fatigue detection method for a ship driver according to a specific embodiment of the present invention.

As shown in Figure 1 and Figure 2, the fatigue detection methods for ship drivers include:

S101. Receive a video stream collected by a wearable device.

In one embodiment of the present invention, the wearable device may be glasses. Specifically, the wearable device-based ship driver fatigue detection method of the present invention is developed based on the Raspberry Pi B+ hardware device platform using machine vision, and the front-end device for collecting video streams can be in the form of glasses. Set with RPiCamera infrared camera. The use of glasses can effectively avoid the influence of factors such as the range of activities of the crew, driving habits and the environment of the ship. That is to say, the human eye image of the ship driver can be directly obtained through wearable glasses, which can not only avoid the influence of various objective factors, but also improve the quality of the collected human eye image, and provide guidance for subsequent human eye positioning and fatigue detection. Provide high-quality image information and reduce image noise. In addition, the use of an infrared camera can meet the needs of collecting clear human eye images under the condition of insufficient light at night.

Furthermore, after the wearable device collects the video image, it can first preprocess the video image, for example, compress the video image, or set the frame rate of the video image, etc., so that it can meet the requirements of the video image Under the premise of meeting the quality requirements, the video image transmission rate is improved. Then, the wearable device transmits the video image to the back-end image processing server in the form of a video stream, wherein the server and the wearable device can communicate through a wireless network, and the communication methods may include but are not limited to Wifi, infrared, bluetooth, One of the 3G networks. After the server receives the video stream collected by the wearable device, the video stream is backed up on the server.

S102. Convert multiple video frames in the video stream into multiple image information.

Specifically, the server acquires multiple video frames from the received video stream, and converts the multiple video frames into image information according to a preset threshold in the server. For example, a wearable device collects video images continuously for 10 minutes. Since the blinking time of normal people is about 0.2-0.4 seconds, and the blinking speed is generally slow in the fatigue state, it is a process of gradually closing the eyes, from opening to closing. Generally, it takes at least 1 second or so. Therefore, the server can set the video frame rate to 10 (that is, FPS=10) to meet the real-time capture of the state of human eyes. Under this condition, 6000 pieces of sample image information can be generated.

It should be understood that the server needs to perform feature learning on the ship driver's eye information in advance, so as to improve the accuracy of human eye detection and fatigue detection.

S103. Acquire human eye regions in multiple pieces of image information.

In an embodiment of the present invention, before the server obtains the human eye area in the multiple image information, preprocessing may be performed on the multiple image information, so that the server can obtain image information of better quality. Wherein, the preprocessing includes one or more of image denoising processing, equalization processing, and contrast processing.

Specifically, the server locates the human eye area in the plurality of image information, that is, locates the position of the human eye, so as to obtain an accurate image partially containing the human eye area from the image information, and remove useless information in the image information.

In one embodiment of the present invention, the server obtains the human eye area in the plurality of image information specifically includes:

S1031. The server performs human eye positioning on multiple image information according to the Haar feature-based Adaboost algorithm, and acquires a first human eye area.

S1032. The server performs binarization processing on a plurality of image information, and performs human eye positioning on the binarized image information according to the Adaboost algorithm based on Haar features, so as to obtain a second human eye area.

S1033. The server judges whether the first human eye area matches the second human eye area, and takes the first human eye area and/or the second human eye area as the human eye area in the plurality of image information when matching.

Specifically, the server uses the Adaboost algorithm based on Haar features to initially locate the human eye according to the image information combined with the learning results of the ship driver's eye features, and obtains the first human eye area. Then, the server uses image processing technology to analyze and process the image information, obtains the binary image of the image information, and uses the Haar-based Adaboost algorithm to locate the human eye again according to the generated binary image, and obtains the second human eye area. Next, the server matches the first human eye area with the second human eye area, and if the image set of the first human eye area includes the image set of the second human eye area, it is judged that the human eye detection is successful; otherwise, the image information is deleted.

Specifically, the server first performs Haar-based human eye feature extraction on the image information. Among them, the characteristics of the human eye in the image information can be expressed as coordinates, distance, color, brightness, shape and other information. The Haar feature belongs to the matrix feature, so it can be abstracted as points and lines. Simple graphics composed of basic collection elements such as faces. Among them, as shown in Figure 3, Haar features can be divided into three categories: edge features, line features and surround features. The basic idea of the Haar feature is to first divide the rectangular frame into blocks, and then combine the gray pixels of the blocks with the edge features to analyze a feature analysis method. In the target image, the rectangular image area at a specific position can be abstracted as Haar features, and the image features of the target area can be quantified by this method. The pixel gray value of the white area in the image is subtracted from the sum of the pixel gray value of the black area, and the obtained value is the feature value of the covered area.

The speed of feature calculation can be improved by using integral graph calculation. Integral graph is a matrix representation method that can describe global information, which is defined as:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, f(x,y) is the integral image of the original image at (x,y), and g(x′,y′) is the original image at (x,y). Therefore, as shown in Figure 4, the integral image at point (x, y) is equal to the sum of all pixel values in the upper left gray area of this point.

Furthermore, the server identifies the position of human eyes in the image information according to the Adaboost algorithm. For the captured image of 24*24 pixels, the number of Haar features is as many as tens of thousands of image matches, and there are only a few available features. In the present invention, the fast human eye detection is realized by using the Adaboost algorithm. The basic idea is to use a large number of training sets to train weak classifiers, and finally form a strong classifier through algorithm superposition.

If the human eye area image has k features, it can be expressed as f _j ( _xi ), where, 1≤j≤k, x _i is represented as the i-th sample image. Then the feature set of each image can be expressed as {f ₁ ( _xi ),f ₂ ( _xi ),f ₃ ( _xi ),…f _j ( _xi ),…f _k ( _xi )}, where , each feature corresponds to a weak classifier.

The server forms a weak classifier h _j (x) into three parts: feature f _j (x), threshold θ _j and symbol p _j , where a feature corresponds to a weak classifier, and the classification threshold is a method to classify all matrices The eigenvalue of , and the classification symbol is a symbol indicating that it has a positive and negative direction. The server represents the weak classifier for the jth feature as:

Among them, h _j (x) is the value of the weak classifier, θ _j is the threshold, p _j is used to control the direction of the inequality sign, and the value is +1 or -1, and f _j (x) is the feature value.

Based on the Adaboost algorithm, the following steps are performed on known n training samples (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), where y _i ={0,1 } corresponds to the true and false of the sample.

(1) Take n training samples, including m human eye samples and l non-human eye samples, expressed as (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), Wherein, y _i =0 and y _i =1 correspond to human eye samples and non-human eye samples respectively.

(2) Initialize the error weight, for samples with y _i =0, For samples with y _i =1,

(3) Initialize t=1, where t≤T, T is the number of training sample classifiers.

(4) Normalize the weights to

(5) Train a weak classifier h(x,f,p,θ) for each feature f, and calculate the weighted (q _i ) error rate of the corresponding weak classifier ε _f =∑|h _j (x _j )-q _i |, and select the classifier h _t with the smallest error ε _f , and update the weight Among them, e _i =0 means it is correctly classified, e _i =1 means it is wrongly classified,

(6) Another t=t+1, repeat step (4) until t>T.

(7) Finally, the strong classifier is obtained as:

S104, performing fatigue analysis on the human eye area in multiple image information to determine whether the ship driver is in a fatigue state, and sending the analysis result to the wearable device.

In an embodiment of the present invention, the server performs fatigue analysis on the human eye area to determine whether the ship driver is in a fatigue state specifically includes: the server calculates the PERCLOS values of the human eye area in multiple image information according to the PERCLOS algorithm, and converts the PERCLOS The value is compared with the threshold value for judging the fatigue degree, and when the PERCLOS value is greater than or equal to the threshold value for judging the fatigue degree, it is judged that the ship driver is in a fatigue state. Wherein, the server calculates the PERCLOS value according to the following formula:

$\mathrm{<span\; class="notranslate">\; PERCLOS</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; N</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; 100</span>}<span\; class="notranslate">\; \%</span><span\; class="notranslate">\; ,</span>$

Among them, N is the total number of human eye area samples in continuous time,

Specifically, since the state of the eyes has a high correlation with the fatigue degree of the ship driver, the PERCLOS algorithm (ie, Percentage of Eyelid Closure Over the Pupil Over time) is a method to detect fatigue by analyzing the opening and closing of the eyes . Among them, the P80 standard has the highest correlation with the degree of fatigue and is recognized as the "golden judgment" standard.

After the server locates the human eye area in the image information, the degree of opening and closing of the human eye in the human eye area is determined by image processing technology. That is to say, after the server calculates P(i), the server can compare P(i) with the threshold T for judging the degree of fatigue, where the threshold T is obtained after a comprehensive evaluation of the ship driving environment based on experiments. An ideal numerical parameter, if P(i)≥T, it is judged that the human eyes are closed, that is, it is judged that the ship driver is in a fatigued state. If P(i)<T, it is judged that the human eyes are in an open state, that is, it is judged that the ship driver is not in a fatigued state. Then, the server sends the analysis result of judging whether the ship driver is in a fatigued state to the wearable device.

In one embodiment of the present invention, after the server sends the analysis result to the wearable device, if the server judges that the ship driver is in a fatigue state, the wearable device will give an alarm prompt. Wherein, the alarm prompt includes one or more of light prompt, voice prompt and vibration prompt.

The fatigue detection method of the ship driver in the embodiment of the present invention can avoid the influence of various objective factors, including the lighting environment, the water surface fluctuation environment, the operating environment and the water surface in front of the ship driver by acquiring the video image of the ship driver through the wearable device. Field of view, etc., based on the front-end acquisition system of wearable devices, can collect clear video images, and can also guarantee the quality of the collected video images under harsh conditions such as shaking inside the ship and insufficient light, thereby improving the reliability of server fatigue detection .

Moreover, the machine vision technology is integrated into the fatigue detection method, and the video image is sent to the server through the wearable device, and the video image is processed, human eye positioning and fatigue detection are performed through the server, and when the ship driver is in a fatigue state, the wearable The equipment reminds it to warn the ship driver, thus greatly improving the safety of the ship driver when driving the ship, avoiding accidents, and ensuring the safety of the ship driver's life and property.

In order to realize the above-mentioned embodiments, the present invention also proposes a ship driver's fatigue detection system.

FIG. 5 is a schematic structural diagram of a ship driver's fatigue detection system according to an embodiment of the present invention. As shown in FIG. 5 , the ship driver's fatigue detection system includes a server 10 and a wearable device 20 .

Specifically, the wearable device 20 is used to collect video streams, send the video streams to the server 10, and receive analysis results sent by the server. Wherein, the wearable device 20 may be glasses. After the wearable device 20 captures the video image, it can first preprocess the video image, for example, compress the video image, or set the frame rate of the video image, etc., so that it can meet the quality requirements of the video image Under the premise, improve the video image transmission rate. Then, the wearable device 20 transmits the video image to the back-end image processing server 10 in the form of a video stream, wherein the server 10 and the wearable device 20 can communicate through a wireless network, and the communication method can include but not limited to Wifi, One of infrared, bluetooth, 3G network.

The server 10 is used to receive the video stream collected by the wearable device 20, and convert multiple video frames in the video stream into multiple image information, and obtain the human eye area in the multiple image information, and analyze the multiple image information Fatigue analysis is performed on the human eye area to determine whether the ship driver is in a fatigue state, and the analysis result is sent to the wearable device 20 . Specifically, after the server 10 receives the video stream collected by the wearable device 20 , it backs up the video stream on the server 10 . Then, the server 10 acquires a plurality of video frames from the received video stream, and converts the plurality of video frames into image information according to a preset threshold in the server 10 . For example, the wearable device 20 collects continuous 10-minute video images. Since the blinking time of normal people is about 0.2-0.4 seconds, and the blinking speed is generally slow in the fatigue state, it is a process of gradually closing the eyes. Closing generally takes at least 1 second or so. Therefore, the server 10 can set the video frame rate to 10 (that is, FPS=10) to satisfy the real-time capture of the state of human eyes. Under this condition, 6000 pieces of sample image information can be generated.

Wherein, the server 10 is further configured to preprocess the plurality of image information before obtaining the human eye area in the plurality of image information, so that the server 10 can obtain image information of better quality. Wherein, the preprocessing includes one or more of image denoising processing, equalization processing, and contrast processing.

Then, the server 10 locates the human eye area in the plurality of image information, that is, locates the human eye position, so as to obtain an accurate image partially containing the human eye area from the image information, and remove useless information in the image information.

In one embodiment of the present invention, the server 10 is specifically configured to perform human eye positioning on multiple image information according to the Adaboost algorithm based on Haar features, obtain the first human eye area, and perform binarization processing on multiple image information , and according to the Adaboost algorithm based on Haar features, the binarized image information is used to locate the human eyes to obtain the second human eye area, and to judge whether the first human eye area and the second human eye area match, and in the matching In this case, the first human eye area or the second human eye area is used as the human eye area in the plurality of image information. Specifically, the server 10 uses the Haar feature-based Adaboost algorithm to initially locate the human eye according to the image information combined with the learning result of the ship driver's eye features, and obtains the first human eye area. Then, the server 10 uses image processing technology to analyze and process the image information, obtains the binary image of the image information, and uses the Adaboost algorithm based on the Haar feature to locate the human eye again according to the generated binary image, and obtains the second Two eye area. Next, the server 10 matches the first human eye area with the second human eye area, and if the image set of the first human eye area includes the image set of the second human eye area, it is judged that the human eye detection is successful, otherwise, the image information is deleted .

Furthermore, the server 10 first performs Haar-based human eye feature extraction on the image information. Among them, the characteristics of the human eye in the image information can be expressed as coordinates, distance, color, brightness, shape and other information. The Haar feature belongs to the matrix feature, so it can be abstracted as points and lines. Simple graphics composed of basic collection elements such as faces. Among them, as shown in Figure 3, Haar features can be divided into three categories: edge features, line features and surround features. The basic idea of the Haar feature is to first divide the rectangular frame into blocks, and then combine the gray pixels of the blocks with the edge features to analyze a feature analysis method. In the target image, the rectangular image area at a specific position can be abstracted as Haar features, and the image features of the target area can be quantified by this method. The pixel gray value of the white area in the image is subtracted from the sum of the pixel gray value of the black area, and the obtained value is the feature value of the covered area.

The server 10 can increase the feature calculation speed by using the integral graph for calculation. Integral graph is a matrix representation method that can describe global information, which is defined as:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, f(x,y) is the integral image of the original image at (x,y), and g(x′,y′) is the original image at (x,y). Therefore, as shown in Figure 4, the integral image at point (x, y) is equal to the sum of all pixel values in the upper left gray area of this point.

Furthermore, the server 10 identifies the position of the human eyes in the image information according to the Adaboost algorithm. For the captured image of 24*24 pixels, the number of Haar features is as many as tens of thousands of image matches, and there are only a few available features. In the present invention, the fast human eye detection is realized by using the Adaboost algorithm. The basic idea is to use a large number of training sets to train weak classifiers, and finally form a strong classifier through algorithm superposition.

If the human eye area image has k features, it can be expressed as f _j ( _xi ), where, 1≤j≤k, x _i is represented as the i-th sample image. Then the feature set of each image can be expressed as {f ₁ ( _xi ),f ₂ ( _xi ),f ₃ ( _xi ),…f _j ( _xi ),…f _k ( _xi )}, where , each feature corresponds to a weak classifier.

The server 10 makes a weak classifier h _j (x) consist of three parts: feature f _j (x), threshold θ _j and symbol p _j , where one feature corresponds to a weak classifier, and the classification threshold is a set of all matrices The eigenvalue of the classification, and the classification symbol is a symbol indicating that it has a positive and negative direction. The server 10 represents the weak classifier of the jth feature as:

Among them, h _j (x) is the value of the weak classifier, θ _j is the threshold, p _j is used to control the direction of the inequality sign, and the value is +1 or -1, and f _j (x) is the feature value.

Based on the Adaboost algorithm, the following steps are performed on known n training samples (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), where y _i ={0,1 } corresponds to the true and false of the sample.

(1) Take n training samples, including m human eye samples and l non-human eye samples, expressed as (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), Wherein, y _i =0 and y _i =1 correspond to human eye samples and non-human eye samples respectively.

(2) Initialize the error weight, for samples with y _i =0, For samples with y _i =1,

(3) Initialize t=1, where t≤T, T is the number of training sample classifiers.

(4) Normalize the weights to

(5) Train a weak classifier h(x,f,p,θ) for each feature f, and calculate the weighted (q _i ) error rate of the corresponding weak classifier ε _f =∑|h _j (x _j )-q _i |, and select the classifier h _t with the smallest error ε _f , and update the weight Among them, e _i =0 means it is correctly classified, e _i =1 means it is wrongly classified,

(6) Another t=t+1, repeat step (4) until t>T.

(7) Finally, the strong classifier is obtained as:

In one embodiment of the present invention, the server 10 is specifically configured to calculate the PERCLOS value of the human eye area in multiple image information according to the PERCLOS algorithm, and compare the PERCLOS value with the threshold value for judging the degree of fatigue, and when the PERCLOS value is greater than or equal to When the threshold value of the fatigue degree is determined, it is judged that the ship pilot is in a fatigue state. Wherein, the server 10 calculates the PERCLOS value according to the following formula: Among them, N is the total number of human eye area samples in continuous time, Since the state of the eyes has a high correlation with the degree of fatigue of the ship's pilot, the PERCLOS algorithm is a method to detect fatigue by analyzing the opening and closing of the eyes. Among them, the P80 standard has the highest correlation with the degree of fatigue and is recognized as the "golden judgment" standard.

After the server 10 locates the human eye area in the image information, the degree of opening and closing of the human eye in the human eye area is determined by image processing technology. That is to say, after the server 10 calculates P(i), the server 10 can compare P(i) with the threshold T for judging the degree of fatigue, wherein the threshold T is obtained after comprehensively evaluating the driving environment of the ship according to experiments. If P(i)≥T, it is judged that the human eyes are closed, that is, it is judged that the ship driver is in a fatigued state. If P(i)<T, it is judged that the human eyes are in an open state, that is, it is judged that the ship driver is not in a fatigued state. Then, the server 10 sends the analysis result of judging whether the ship driver is in a fatigue state to the wearable device 20 .

In one embodiment of the present invention, the wearable device 20 is also used to give an alarm prompt when the server 10 judges that the ship driver is in a fatigue state. Wherein, the alarm prompt includes one or more of light prompt, voice prompt and vibration prompt.

The fatigue detection system of the ship driver in the embodiment of the present invention can avoid the influence of various objective factors by acquiring the video image of the ship driver through the wearable device, including the lighting environment, the water surface fluctuation environment, the operating environment and the water surface in front of the ship driver Field of view, etc., based on the front-end acquisition system of wearable devices, can collect clear video images, and can also guarantee the quality of the collected video images under harsh conditions such as shaking inside the ship and insufficient light, thereby improving the reliability of server fatigue detection .

Moreover, the machine vision technology is integrated into the fatigue detection method, and the video image is sent to the server through the wearable device, and the video image is processed, human eye positioning and fatigue detection are performed through the server, and when the ship driver is in a fatigue state, the wearable The equipment reminds it to warn the ship driver, thus greatly improving the safety of the ship driver when driving the ship, avoiding accidents, and ensuring the safety of the ship driver's life and property.

In order to realize the foregoing embodiments, the present invention further proposes a server.

FIG. 6 is a schematic structural diagram of a server according to an embodiment of the present invention. As shown in FIG. 6, the server includes a receiving module 110, a conversion module 120, an acquisition module 130, an analysis module 140, and a preprocessing module 150, wherein the acquisition module 130 includes a first An acquisition unit 131 , a second acquisition unit 132 and a judgment unit 133 , and the analysis module 140 includes a calculation unit 141 , a comparison unit 142 and a judgment unit 143 .

Specifically, the receiving module 110 is configured to receive video streams collected by wearable devices.

The converting module 120 is used for converting multiple video frames in the video stream into multiple image information. Specifically, the converting module 120 acquires multiple video frames from the video stream received by the receiving module 110, and converts the multiple video frames into image information according to a preset threshold. For example, a wearable device collects video images continuously for 10 minutes. Since the blinking time of normal people is about 0.2-0.4 seconds, and the blinking speed is generally slow in the fatigue state, it is a process of gradually closing the eyes, from opening to closing. Generally, it takes at least 1 second or so. Therefore, the conversion module 120 can set the video frame rate to 10 (that is, FPS=10) to satisfy the real-time capture of the state of human eyes. Under this condition, 6000 pieces of sample image information can be generated.

The obtaining module 130 is used for obtaining human eye regions in multiple image information.

In one embodiment of the present invention, the server further includes a preprocessing module 150, and the preprocessing module 150 is used to preprocess a plurality of image information, wherein the preprocessing includes image denoising processing, equalization processing, and contrast processing. One or more.

In an embodiment of the present invention, the obtaining module 130 includes a first obtaining unit 131 , a second obtaining unit 132 and a judging unit 133 . Wherein, the first acquisition unit 131 is configured to perform human eye positioning on a plurality of image information according to the Adaboost algorithm based on Haar features, and acquire the first human eye area. The second acquisition unit 132 is used to perform binarization processing on a plurality of image information, and perform human eye positioning on the binarized image information according to the Adaboost algorithm based on Haar features, so as to obtain the second human eye area. The judging unit 133 is configured to judge whether the first human eye area and the second human eye area match, and take the first human eye area and/or the second human eye area as the human eye area in the plurality of image information when matching. Specifically, the first acquiring unit 131 uses the Haar feature-based Adaboost algorithm to initially locate the human eye according to the image information combined with the learning result of the ship driver's eye features, and obtains the first human eye area. Then, the second acquisition unit 132 uses image processing technology to analyze and process the image information, obtains the binary image of the image information, and uses the Adaboost algorithm based on the Haar feature to locate the human eye again according to the generated binary image , to get the second eye area. Next, the judging unit 133 matches the first human eye area with the second human eye area, and if the image set of the first human eye area includes the image set of the second human eye area, it is judged that the human eye detection is successful, otherwise the image is deleted information.

Specifically, the first acquisition unit 131 and the second acquisition unit 132 first perform Haar-based human eye feature extraction on the image information. Among them, the characteristics of the human eye in the image information can be expressed as coordinates, distance, color, brightness, shape and other information. The Haar feature belongs to the matrix feature, so it can be abstracted as points and lines. Simple graphics composed of basic collection elements such as faces. Among them, as shown in Figure 3, Haar features can be divided into three categories: edge features, line features and surround features. The basic idea of the Haar feature is to first divide the rectangular frame into blocks, and then combine the gray pixels of the blocks with the edge features to analyze a feature analysis method. In the target image, the rectangular image area at a specific position can be abstracted as Haar features, and the image features of the target area can be quantified by this method. The pixel gray value of the white area in the image is subtracted from the sum of the pixel gray value of the black area, and the obtained value is the feature value of the covered area.

The speed of feature calculation can be improved by using integral graph calculation. Integral graph is a matrix representation method that can describe global information, which is defined as:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

Among them, f(x,y) is the integral image of the original image at (x,y), and g(x′,y′) is the original image at (x,y). Therefore, as shown in Figure 4, the integral image at point (x, y) is equal to the sum of all pixel values in the upper left gray area of this point.

Further, the first acquiring unit 131 and the second acquiring unit 132 identify the positions of human eyes in the image information according to the Adaboost algorithm. For the captured image of 24*24 pixels, the number of Haar features is as many as tens of thousands of image matches, and there are only a few available features. In the present invention, the fast human eye detection is realized by using the Adaboost algorithm. The basic idea is to use a large number of training sets to train weak classifiers, and finally form a strong classifier through algorithm superposition.

If the human eye area image has k features, it can be expressed as f _j ( _xi ), where, 1≤j≤k, x _i is represented as the i-th sample image. Then the feature set of each image can be expressed as {f ₁ ( _xi ),f ₂ ( _xi ),f ₃ ( _xi ),…f _j ( _xi ),…f _k ( _xi )}, where , each feature corresponds to a weak classifier.

The first acquisition unit 131 and the second acquisition unit 132 make a weak classifier h _j (x) consist of three parts: feature f _j (x), threshold θ _j and symbol p _j , wherein one feature corresponds to a weak classifier , the classification threshold is an eigenvalue that classifies all matrices, and the classification sign is a sign indicating that it has a positive and negative direction. The server represents the weak classifier for the jth feature as:

Among them, h _j (x) is the value of the weak classifier, θ _j is the threshold, p _j is used to control the direction of the inequality sign, and the value is +1 or -1, and f _j (x) is the feature value.

Based on the Adaboost algorithm, the following steps are performed on known n training samples (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), where y _i ={0,1 } corresponds to the true and false of the sample.

(1) Take n training samples, including m human eye samples and l non-human eye samples, expressed as (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _n ,y _n ), Wherein, y _i =0 and y _i =1 correspond to human eye samples and non-human eye samples respectively.

(2) Initialize the error weight, for samples with y _i =0, For samples with y _i =1,

(3) Initialize t=1, where t≤T, T is the number of training sample classifiers.

(4) Normalize the weights to

(5) Train a weak classifier h(x,f,p,θ) for each feature f, and calculate the weighted (q _i ) error rate of the corresponding weak classifier ε _f =∑|h _j (x _j )-q _i |, and select the classifier h _t with the smallest error ε _f , and update the weight Among them, e _i =0 means it is correctly classified, e _i =1 means it is wrongly classified,

(6) Another t=t+1, repeat step (4) until t>T.

(7) Finally, the strong classifier is obtained as:

The analysis module 140 is used to perform fatigue analysis on the human eye area in multiple image information to determine whether the ship driver is in a fatigue state, and send the analysis result to the wearable device.

In one embodiment of the present invention, the analyzing module 140 includes a calculating unit 141 , a comparing unit 142 and a judging unit 143 . Wherein, the calculating unit 141 is used for calculating the PERCLOS value of the human eye area in the plurality of image information according to the PERCLOS algorithm. Wherein, the calculation unit 141 calculates the PERCLOS value according to the following formula: Among them, N is the total number of human eye area samples in continuous time, The comparing unit 142 is used to compare the PERCLOS value with the threshold value for judging the degree of fatigue. The judging unit 143 is used for judging that the ship driver is in a fatigue state when the PERCLOS value is greater than or equal to the threshold for judging the fatigue degree.

The server of the embodiment of the present invention processes video images, locates human eyes and detects fatigue, and reminds the ship driver through a wearable device when he is in a fatigued state, so as to warn the ship driver, thereby greatly improving the safety of the ship driver. The safety when driving the ship avoids the occurrence of accidents and ensures the safety of the life and property of the ship driver.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the above described embodiments, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

In the present invention, unless otherwise clearly specified and limited, the terms "installation", "connection", "connection" and other terms should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or a Integral; it may be mechanically connected or electrically connected; it may be directly connected or indirectly connected through an intermediary, and it may be the internal communication of two elements or the interaction relationship between two elements, unless otherwise clearly defined. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (21)

1. A fatigue detection method for a ship driver, comprising the following steps: Receive video streams collected by wearable devices; converting a plurality of video frames in the video stream into a plurality of image information; Acquiring human eye regions in the plurality of image information; and Fatigue analysis is performed on the human eye area in the plurality of image information to determine whether the ship driver is in a fatigue state, and the analysis result is sent to the wearable device. 2. the fatigue detection method of ship driver as claimed in claim 1, is characterized in that, obtaining the human eye area in described multiple image information specifically comprises: According to the Adaboost algorithm based on Haar feature, human eye positioning is performed on the plurality of image information, and the first human eye area is obtained; Carrying out binarization processing on the plurality of image information, and performing human eye positioning on the image information after binarization processing according to the Adaboost algorithm based on Haar feature, to obtain the second human eye area; judging whether the first human eye area and the second human eye area match, and when matching, use the first human eye area and/or the second human eye area as one of the plurality of image information human eye area. 3. the fatigue detection method of ship driver as claimed in claim 1, is characterized in that, carrying out fatigue analysis to described human eye area to judge whether ship driver is in fatigue state specifically comprises: Calculate the PERCLOS value of the human eye area in the plurality of image information according to the PERCLOS algorithm, and compare the PERCLOS value with the threshold value for judging the degree of fatigue, and when the PERCLOS value is greater than or equal to the threshold value for judging the degree of fatigue It is judged that the ship driver is in a state of fatigue. 4. the fatigue detection method of ship driver as claimed in claim 3, is characterized in that, calculates described PERCLOS value according to following formula: Among them, N is the total number of human eye area samples in continuous time, 5. the ship driver's fatigue detection method as claimed in claim 1, is characterized in that, after analysis result is sent to described wearable device, also comprises: When it is judged that the ship driver is in a state of fatigue, the wearable device will give an alarm prompt. 6 . The fatigue detection method for ship pilots according to claim 5 , wherein the alarm prompts include one or more of light prompts, voice prompts and vibration prompts. 7 . 7. the fatigue detection method of ship driver as claimed in claim 1, is characterized in that, before obtaining the human eye area in described multiple image information, also comprises: Perform preprocessing on the plurality of image information, wherein the preprocessing includes one or more of image denoising processing, equalization processing, and contrast processing. 8. The method for detecting fatigue of a ship driver according to any one of claims 1-7, wherein the wearable device is glasses. 9. A fatigue detection system for ship drivers, comprising: a server and a wearable device, wherein, The wearable device is used to collect video streams, send the video streams to the server, and receive analysis results sent by the server; and The server is configured to receive the video stream collected by the wearable device, convert a plurality of video frames in the video stream into a plurality of image information, and obtain human eye regions in the plurality of image information, and Fatigue analysis is performed on the human eye area in the plurality of image information to determine whether the ship driver is in a fatigue state, and the analysis result is sent to the wearable device. 10. the ship driver's fatigue detection system as claimed in claim 9, is characterized in that, described server is specifically used for: According to the Adaboost algorithm based on the Haar feature, the human eyes are positioned on the multiple image information, and the first human eye area is obtained, and the multiple image information is binarized, and according to the Adaboost algorithm based on the Haar feature. The binarized image information is subjected to human eye positioning to obtain a second human eye area, and it is judged whether the first human eye area and the second human eye area match, and when matching, the first human eye area is The human eye area or the second human eye area is used as the human eye area in the plurality of image information. 11. the ship driver's fatigue detection system as claimed in claim 9, is characterized in that, described server is specifically used for: Calculate the PERCLOS value of the human eye area in the plurality of image information according to the PERCLOS algorithm, and compare the PERCLOS value with the threshold value for judging the degree of fatigue, and when the PERCLOS value is greater than or equal to the threshold value for judging the degree of fatigue It is judged that the ship driver is in a state of fatigue. 12. the ship driver's fatigue detection system as claimed in claim 11, is characterized in that, server calculates described PERCLOS value according to following formula: Among them, N is the total number of human eye area samples in continuous time, 13. The ship driver's fatigue detection system as claimed in claim 9, wherein the wearable device is also used for: When the server judges that the ship driver is in a fatigue state, an alarm prompt is given. 14. The ship driver's fatigue detection system according to claim 13, wherein the alarm prompt includes one or more of light prompt, voice prompt and vibration prompt. 15. the ship driver's fatigue detection system as claimed in claim 1, is characterized in that, described server is also used for: Perform preprocessing on the plurality of image information, wherein the preprocessing includes one or more of image denoising processing, equalization processing, and contrast processing. 16. The ship driver's fatigue detection system according to any one of claims 9-15, wherein the wearable device is glasses. 17. A server, characterized in that, comprising: The receiving module is used to receive the video stream collected by the wearable device; A conversion module, configured to convert a plurality of video frames in the video stream into a plurality of image information; an acquisition module, configured to acquire human eye regions in the plurality of image information; and The analysis module is used to perform fatigue analysis on the human eye area in the plurality of image information to determine whether the ship driver is in a fatigue state, and send the analysis result to the wearable device. 18. The server according to claim 17, wherein the obtaining module comprises: The first acquisition unit is used to perform human eye positioning on the plurality of image information according to the Adaboost algorithm based on Haar features, and acquire the first human eye region; The second acquisition unit is used to perform binarization processing on the plurality of image information, and perform human eye positioning on the binarized image information according to the Adaboost algorithm based on Haar features, so as to obtain the second human eye area; A judging unit, configured to judge whether the first human eye area and the second human eye area match, and use the first human eye area and/or the second human eye area as the multiple The human eye area in the image information. 19. The server according to claim 17, wherein the analysis module comprises: A calculation unit, configured to calculate the PERCLOS value of the human eye area in the plurality of image information according to the PERCLOS algorithm; A comparison unit for comparing the PERCLOS value with a threshold value for judging the degree of fatigue; and A judging unit, configured to judge that the ship driver is in a fatigue state when the PERCLOS value is greater than or equal to the fatigue degree judgment threshold. 20. The server according to claim 19, wherein the calculation unit calculates the PERCLOS value according to the following formula: Among them, N is the total number of human eye area samples in continuous time, 21. The server according to claim 17, further comprising: A preprocessing module, configured to preprocess the plurality of image information, wherein the preprocessing includes one or more of image denoising processing, equalization processing, and contrast processing.