CN117351470A - Operator fatigue detection method based on space-time characteristics - Google Patents

Operator fatigue detection method based on space-time characteristics Download PDF

Info

Publication number
CN117351470A
CN117351470A CN202311638769.1A CN202311638769A CN117351470A CN 117351470 A CN117351470 A CN 117351470A CN 202311638769 A CN202311638769 A CN 202311638769A CN 117351470 A CN117351470 A CN 117351470A
Authority
CN
China
Prior art keywords
isolated
units
value
feature
gray
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311638769.1A
Other languages
Chinese (zh)
Other versions
CN117351470B (en
Inventor
王晗宇
陈登凯
周垚
孙意为
黄悦欣
肖江浩
乔一丹
王憬鸾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwestern Polytechnical University
Original Assignee
Northwestern Polytechnical University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwestern Polytechnical University filed Critical Northwestern Polytechnical University
Priority to CN202311638769.1A priority Critical patent/CN117351470B/en
Publication of CN117351470A publication Critical patent/CN117351470A/en
Application granted granted Critical
Publication of CN117351470B publication Critical patent/CN117351470B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Human Computer Interaction (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to the technical field of image feature extraction, in particular to a method for detecting fatigue of operators based on space-time features. The method comprises the following steps: dividing a facial gray level image of a tower crane operator into cell units, and determining a feature vector of each cell unit; further determining the similarity of adjacent cell units, and dividing the facial gray scale image into a similar area and isolated units according to the similarity; determining distribution discreteness and feature expressivity of the isolated units; determining the merging probability of the isolated units according to the distribution discreteness, the characteristic expressive property and the gray value distribution of the pixel points in the isolated units; further determining a target area; and carrying out feature analysis on the target area, determining a feature descriptor, and detecting the fatigue degree of tower crane operators in the operation process according to the feature descriptor to obtain a detection result. The invention can effectively screen out noise influence, and can obtain more accurate and objective detection results when fatigue detection is carried out.

Description

Operator fatigue detection method based on space-time characteristics
Technical Field
The invention relates to the technical field of image feature extraction, in particular to a method for detecting fatigue of operators based on space-time features.
Background
The tower crane operator fatigue detection method based on the space-time characteristics is used for extracting relevant space-time characteristics in the images and judging the fatigue state of operators by analyzing the space-time characteristics.
In the related art, facial images of tower crane operators are acquired, and then feature descriptors are acquired by using an HOG feature extraction algorithm, so that fatigue states of the operators are analyzed based on the feature descriptors, but because tower crane operations are usually at high altitudes, the transmission of images is usually wireless, the image transmission process is affected by signal attenuation so as to contain a large amount of noise, and the accuracy and reliability of the acquisition of the feature descriptors and the accuracy of the analysis of the fatigue states of the operators are affected by the noise.
Disclosure of Invention
In order to solve the technical problem that the existence of noise influences the accuracy and reliability of feature descriptor acquisition and further influences the accuracy of analysis of the fatigue state of an operator during feature extraction and analysis, the invention provides an operator fatigue detection method based on space-time features, which adopts the following specific technical scheme:
the invention provides an operator fatigue detection method based on space-time characteristics, which comprises the following steps:
acquiring face gray images of at least two frames of tower crane operators during operation; dividing the facial gray image into at least two cell units with the same size, and determining the feature vector of each cell unit according to the gradient features of all pixel points of each cell unit;
determining the similarity of two cell units according to the feature vectors of the two cell units which are closest to each other, traversing all the face gray images, and dividing the face gray images into similar areas and isolated units according to the similarity;
determining the distribution discreteness of each isolated unit according to the distribution of the gradient directions of all pixel points in each isolated unit and the information entropy; determining the characteristic expressive property of each isolated unit according to the difference of distribution discreteness and the difference of gray values between each isolated unit and other isolated units in a preset neighborhood range; determining the merging probability of the isolated units according to the distribution discreteness, the characteristic expressive property and the gray value distribution of the pixel points in the isolated units; screening the isolated units according to the merging probability, merging the isolated units into similar areas which are closest to each other, and taking all the similar areas after merging as target areas;
and carrying out feature analysis on the target area, determining a feature descriptor, and detecting the fatigue degree of the tower crane operator in the operation process by combining the feature descriptors of at least two frame surface gray level images to obtain a detection result.
Further, the determining the distribution discreteness of the isolated units according to the distribution of the gradient directions of all the pixel points in each isolated unit and the information entropy comprises the following steps:
constructing gradient direction histograms of all pixel points in the isolated units based on a preset included angle range, and calculating normalized values of variances of gradient direction distribution of all pixel points in each isolated unit according to the gradient direction histograms to obtain first discrete influence coefficients of the isolated units;
taking the extremely poor normalized values of the gradient directions of all pixel points in the isolated units within different preset included angle ranges as second discrete influence coefficients of the isolated units;
calculating normalized values of information entropy of the quantity of gradient directions of all pixel points in the isolated unit in different preset included angle ranges to obtain a third discrete influence coefficient of the isolated unit;
determining the distribution discreteness of the isolated units according to the first discrete influence coefficient, the second discrete influence coefficient and the third discrete influence coefficient, wherein the first discrete influence coefficient and the distribution discreteness are in positive correlation, the second discrete influence coefficient and the third discrete influence coefficient are in inverse correlation with the distribution discreteness, and the value of the distribution discreteness is a normalized numerical value.
Further, the determining the feature representation of each isolated cell according to the difference of the distribution discreteness and the difference of the gray value between the isolated cell and other isolated cells in the preset neighborhood range includes:
calculating the average value of the absolute value of the difference value of the distribution discreteness between any isolated unit and other isolated units in a preset neighborhood range to obtain a neighborhood discrete coefficient;
calculating the gray value average value of the pixel points of the isolated units to obtain a central gray value average value, and calculating the gray value average value of all the pixel points in other cell units in a preset neighborhood range as a neighborhood gray value average value; taking a normalized value of the difference absolute value of the center gray average value and the neighborhood gray average value as a neighborhood gray coefficient;
and taking an inverse proportion normalized value of the product of the neighborhood gray scale coefficient and the neighborhood discrete coefficient as the characteristic expressive property of the isolated unit.
Further, the determining the merging probability of the isolated units according to the distribution discreteness, the feature expressiveness and the gray value distribution of the pixel points in the isolated units includes:
gaussian filtering is carried out on pixel points in the isolated units based on a preset filtering scale to obtain a filtered gray value of each pixel point after Gaussian filtering, and a normalized value of a mean value of an absolute value of a difference value between an original gray value and the filtered gray value of each pixel point is calculated to obtain a first weight coefficient; calculating the difference value between the positive integer 1 and the first weight coefficient to obtain a second weight coefficient;
and taking the first weight coefficient as the characteristic expressive weight value, taking the second weight coefficient as the distribution discreteness weight value, and carrying out normalization processing after weighted addition to obtain the merging probability of the isolated units.
Further, the screening and merging the isolated units according to the merging probability into the similar areas closest to each other, and taking all the similar areas after merging as target areas, including:
and taking the isolated units with the merging probability larger than a preset probability threshold as units to be merged, merging the units to be merged into any similar area which is closest to the units to be merged, and taking all the similar areas after merging as target areas.
Further, the performing feature analysis on the target area to determine a feature descriptor includes:
and carrying out feature analysis on the target area based on an HOG feature extraction algorithm to obtain a feature descriptor.
Further, the detecting the fatigue degree of the tower crane operator in the operation process by combining the feature descriptors of the grayscale images of at least two frames to obtain a detection result includes:
and inputting the feature descriptors into a pre-trained long-period and short-period memory network model, analyzing the feature descriptors through the long-period and short-period memory network model, and outputting a detection result.
Further, the determining the feature vector of each cell unit according to the gradient features of all pixel points of each cell unit includes:
and calculating the feature vector of each cell unit according to the gradient amplitude and the gradient direction of all pixel points of each cell unit by using an HOG operator.
Further, the determining the similarity of the two cell units according to the feature vectors of the two cell units closest to each other includes:
and calculating cosine similarity values of feature vectors of two nearest cell units based on a cosine similarity calculation formula, and taking the cosine similarity values as the similarity of the two cell units.
Further, the dividing the face gray-scale image into a similar region and an isolated unit according to the similarity includes:
and combining the cell units with the similarity larger than a preset similarity threshold value to obtain a similar region, and taking the cell units which do not belong to any similar region as isolated units.
The invention has the following beneficial effects:
the invention aims to eliminate the noise of the face image acquired by the tower crane worker during working, so as to facilitate more accurate and objective fatigue detection of the tower crane worker. According to the invention, the face gray level image is divided into the cell units, then, the similarity of the feature vector of each cell unit is analyzed, so that the isolated units are determined, the cell units corresponding to noise and the cell units corresponding to face textures are contained in the isolated units, so that the distribution discreteness and the feature expressivity of each isolated unit are specifically analyzed, the pixel point distribution discreteness degree of the isolated units can be effectively determined through the analysis of the distribution discreteness, the consistency of the isolated units and other surrounding cell units can be effectively determined through the analysis of the feature expressivity, the merging probability of the isolated units is determined through the combination of the distribution discreteness and the feature expressivity, the isolated units are further screened and merged according to the merging probability, and a target area is obtained, so that the area greatly influenced by the noise can be effectively screened out, the accuracy and the reliability of the subsequent feature descriptor acquisition are effectively improved, the feature descriptor with great influence on the noise is avoided, and then, the fatigue degree of a tower crane operator in the operation process is detected according to the feature descriptor, and a detection result is obtained. In summary, the invention can effectively screen out the influence of noise on the feature descriptors, thereby obtaining more accurate and effective feature descriptors and further obtaining more accurate and objective detection results when fatigue detection is carried out.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions and advantages of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are only some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a method for detecting fatigue of an operator based on space-time characteristics according to an embodiment of the present invention.
Detailed Description
In order to further describe the technical means and effects adopted by the present invention to achieve the preset purpose, the following detailed description refers to the specific implementation, structure, characteristics and effects of a method for detecting fatigue of operators based on space-time characteristics according to the present invention, with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" means that the embodiments are not necessarily the same. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The invention provides a specific scheme of an operator fatigue detection method based on space-time characteristics, which is specifically described below with reference to the accompanying drawings.
Referring to fig. 1, a flowchart of a method for detecting fatigue of an operator based on space-time features according to an embodiment of the present invention is shown, where the method includes:
s101: acquiring face gray images of at least two frames of tower crane operators during operation; dividing the face gray level image into at least two cell units with the same size, and determining the feature vector of each cell unit according to the gradient features of all pixel points of each cell unit.
The specific use scene of the embodiment of the invention is that when a tower crane driver works, fatigue detection is carried out on the tower crane driver by collecting the facial information of the tower crane driver, and it can be understood that the working state of the tower crane driver in the working process needs to be detected due to the danger of the tower crane operation, and the specific detection content is fatigue detection.
In the embodiment of the invention, an image acquisition device, such as a camera, a video camera and other equipment, can be used for acquiring the original data of at least two frames of tower crane operators in real time when the tower crane operators operate, and then, the original data is subjected to image preprocessing to obtain the facial gray-scale image only comprising faces. It can be understood that, since the preprocessing is usually performed in the central processing unit, the collected original image needs to be transmitted to the central processing unit, in this process, due to the particularity of the tower crane, that is, the tower crane is usually in an overhead operation and is usually in a remote area to be developed, therefore, the signal stability during transmission is greatly affected, and when the signal is unstable, a larger noise is generated during transmission, so that the link of the subsequent feature extraction is affected.
The preprocessing of the original image may specifically be, for example: firstly, carrying out histogram equalization processing on an original image, amplifying local detail characteristics, then carrying out semantic segmentation on each frame of the original image in a semantic segmentation mode to obtain a face area and a background area, deleting the background area to obtain an image of the face area, and then carrying out image graying processing on the image of the face area to obtain a face gray image. Of course, the implementation process of obtaining the face gray image by image preprocessing in the embodiment of the present invention is a technology well known in the art, and is not further limited and described in detail herein.
In the embodiment of the present invention, after the face gray image is obtained, in order to facilitate specific analysis of the face gray image, the face gray image may be divided into at least two cell units with the same size, where the cell units may be, for example, specifically 8×8 in size, and of course, in other embodiments of the present invention, the cell units may be specifically set according to the actual detection requirement, for example, multiple sizes such as 11×11, 5×5, etc., which are not limited thereto.
The gradient features can be specifically, for example, gradient amplitude and gradient direction features, and in order to facilitate analysis of fatigue detection features of the face gray image, the embodiment of the invention introduces gradient features to realize feature description of the face gray image.
Further, in some embodiments of the present invention, determining the feature vector of each cell unit based on the gradient features of all pixels of each cell unit includes: and calculating the feature vector of each cell unit according to the gradient amplitude and the gradient direction of all pixel points of each cell unit by using an HOG operator.
The HOG operator is an operator corresponding to a direction gradient histogram (Histogram of Oriented Gradients, HOG) algorithm, and in this embodiment of the present invention, the HOG operator may be used to perform a direction gradient histogram analysis on the gradient magnitude and the gradient direction of all pixel points of each cell unit, so as to obtain a feature vector of each cell unit, where the direction gradient histogram algorithm is an algorithm well known in the art, and the analysis and calculation of the feature vector is a calculation process well known in the art, which is not described herein.
In the embodiment of the invention, after the feature vector of each cell unit is determined, the feature vector can be specifically analyzed by combining with the HOG algorithm, and it can be understood that the image interference noise such as spiced salt noise can be generated in the image acquisition and transmission process, and the image interference noise can influence the subsequent analysis result, so that the embodiment of the invention removes the image noise interference in the subsequent process.
S102: and determining the similarity of two cell units according to the feature vectors of the two cell units which are closest to each other, traversing all the face gray images, and dividing the face gray images into similar areas and isolated units according to the similarity.
In the embodiment of the present invention, the similarity analysis is performed on two cell units that are closest to each other, and it can be understood that, in the embodiment of the present invention, the cell units that are closest to each other may be specifically, for example, two adjacent cell units, that is, cell units within the eight neighborhood range of the cell units, and in the embodiment of the present invention, the more similar the features between two adjacent cell units, the higher the corresponding similarity, and based on this, the similarity is calculated.
Further, in some embodiments of the invention, determining the similarity of two cell units from the eigenvectors of the two cell units closest to each other comprises: and calculating cosine similarity values of feature vectors of two nearest cell units based on a cosine similarity calculation formula, and taking the cosine similarity values as the similarity of the two cell units.
The calculation formula of the similarity of the cell units may specifically be, for example:
the method comprises the steps of carrying out a first treatment on the surface of the In (1) the->And->Feature vectors representing two adjacent cell units, respectively, < >>Representing the similarity of the corresponding two cell units, < >>Representing cosine functions, in the present embodiment of the invention, the eigenvectors +.>And->The closer the corresponding cosine value is, the closer 1 is.
Further, in some embodiments of the present invention, dividing the face gray image into similar areas and isolated units according to the similarity includes: and combining the cell units with the similarity larger than a preset similarity threshold value to obtain a similar region, and taking the cell units which do not belong to any similar region as isolated units.
The preset similarity threshold is a similarity threshold, and because the similarity is cosine similarity, the value range of the similarity is between [0,1], so that the corresponding preset similarity threshold can be set to be 0.9, and in other embodiments of the invention, the preset similarity threshold can be adjusted according to the actual detection requirement, and the method is not limited.
It may be understood that in the embodiment of the present invention, the cell units with similarity greater than the preset similarity threshold are combined to obtain the similar area, that is, when the similarity between two adjacent cell units is detected to be greater than the preset similarity threshold, the two cell units are combined and are grown outwards until the whole face gray image is traversed, and the area including at least two cell units is taken as the similar area, and in the whole face gray image, there still remain some cell units not belonging to any similar area, and the some cell units may be those cell units affected by salt and pepper noise greatly or be texture features of the face of the relevant tower crane staff.
S103: determining the distribution discreteness of the isolated units according to the distribution of the gradient directions of all pixel points in each isolated unit and the information entropy; determining the characteristic expressive performance of each isolated unit according to the difference of distribution discreteness and the difference of gray values between each isolated unit and other isolated units in a preset neighborhood range; determining the merging probability of the isolated units according to the distribution discreteness, the characteristic expressive property and the gray value distribution of the pixel points in the isolated units; and screening the isolated units according to the merging probability, merging the isolated units into the similar areas which are closest to each other, and taking all the similar areas after merging as target areas.
The invention specifically analyzes the characteristics of the isolated units obtained in S102, thereby deleting the isolated units related to noise. The specific analysis process can comprise distribution of gradient direction histograms and calculation of information entropy.
Further, in some embodiments of the present invention, determining the distribution discreteness of the isolated units according to the distribution of the gradient directions and the information entropy of all the pixel points in each isolated unit includes: constructing gradient direction histograms of all pixel points in the isolated units based on a preset included angle range, and calculating normalized values of variances of gradient direction distribution of all pixel points in each isolated unit according to the gradient direction histograms to obtain first discrete influence coefficients of the isolated units; taking the extremely poor normalized values of the gradient directions of all pixel points in the isolated units within different preset included angle ranges as second discrete influence coefficients of the isolated units; calculating normalized values of information entropy of the quantity of gradient directions of all pixel points in the isolated unit in different preset included angle ranges, and obtaining a third discrete influence coefficient of the isolated unit; determining the distribution discreteness of the isolated units according to the first discrete influence coefficient, the second discrete influence coefficient and the third discrete influence coefficient, wherein the first discrete influence coefficient and the distribution discreteness are in positive correlation, the second discrete influence coefficient and the third discrete influence coefficient are in inverse correlation with the distribution discreteness, and the value of the distribution discreteness is a normalized numerical value.
In one embodiment of the present invention, the normalization process may specifically be, for example, maximum and minimum normalization processes, and the normalization in the subsequent steps may be performed by using the maximum and minimum normalization processes, and in other embodiments of the present invention, other normalization methods may be selected according to a specific range of values, which will not be described herein.
The positive correlation relationship indicates that the dependent variable increases along with the increase of the independent variable, the dependent variable decreases along with the decrease of the independent variable, and the specific relationship can be multiplication relationship, addition relationship, idempotent of an exponential function and is determined by practical application; the negative correlation indicates that the dependent variable decreases with increasing independent variable, and the dependent variable increases with decreasing independent variable, which may be a subtraction relationship, a division relationship, or the like, and is determined by the actual application.
The gradient direction histogram of all the pixel points in the isolated unit is constructed based on the preset included angle range, and the embodiment of the invention can set the preset included angle range to be a range corresponding to a 20-degree angle, that is, the gradient directions of all the pixel points are divided into 9 preset included angle ranges in the total range of 0-180 degrees, and then the gradient direction histogram is specifically analyzed.
Thus, the calculation formula of the distribution discreteness in the embodiment of the present invention may specifically be, for example:
；
wherein,representing the distribution discreteness of isolated units +.>Represents the total number of pixel points contained in each isolated unit, +.>Represents the side length of each isolated cell, +.>Represents the total number of the preset included angle ranges, +.>Index indicating the range of preset angles, +.>Indicate->The number of pixel points included in the range of the preset included angle is +.>Representing the maximum value of the number of pixel points contained in all preset included angles, +.>Minimum value representing the number of pixels included in all preset angles, +.>Representing normalization processing->The super parameter is represented by a security value set to prevent the denominator from being 0, which may be specifically 0.01.
In an embodiment of the present invention,the first discrete influence coefficient representing an isolated cell, the variance, i.e. characterizes the corresponding degree of dispersion, so that the larger the variance, the larger the first discrete influence coefficient value, i.e. the larger the distribution dispersion of the corresponding isolated cell. />The larger the range of the second discrete influence coefficient representing an isolated bin, the more concentrated the histogram gradient number value in a certain angular range, the lower the dispersion thereof, i.e. the larger the range, the larger the second discrete influence coefficient number value, the smaller the corresponding distribution dispersion, ideally the range is 0, the highest the dispersion is because the histogram distribution is more uniform. />Representing information entropy->Indicate->The probability of the number of gradients in the range of the preset included angle is 1 in ideal case, thenThe entropy of (2) is the smallest and the corresponding distribution discreteness is the largest.
After determining the distribution discreteness, a specific analysis is performed on the feature expressiveness of the isolated units.
Further, in some embodiments of the present invention, determining the feature representation of an isolated cell based on the difference in distribution dispersion and the difference in gray value between each isolated cell and other isolated cells within a predetermined neighborhood comprises: calculating the average value of the absolute value of the difference value of the distribution discreteness between any isolated unit and other isolated units in a preset neighborhood range to obtain a neighborhood discrete coefficient; calculating the gray value average value of the pixel points of the isolated units to obtain a central gray value average value, and calculating the gray value average value of all the pixel points in other cell units in a preset neighborhood range as a neighborhood gray value average value; taking a normalized value of the difference absolute value of the center gray average value and the neighborhood gray average value as a neighborhood gray coefficient; and taking an inverse proportion normalized value of the product of the neighborhood gray scale coefficient and the neighborhood discrete coefficient as the characteristic expressive property of the isolated unit.
The feature representation characterizes the consistency of features represented by the isolated area and other surrounding areas, so that the embodiment of the invention calculates the average value of the absolute value of the difference value of the distribution discreteness between any isolated unit and other isolated units in a preset neighborhood range to obtain a neighborhood discrete coefficient.
The preset neighborhood range may be, for example, an eight-neighborhood range specifically, or may be set according to practical situations, which is not limited in particular, and in the embodiment of the present invention, the preset neighborhood range is specifically exemplified as an eight-neighborhood.
The mean value of the absolute value of the difference between the distribution discreteness of the isolated unit of the central position and the distribution discreteness of other isolated units in the eight neighborhood range of the isolated unit is calculated, and it is understood that the eight neighborhood range of the isolated unit may not have other isolated units, at this time, the distribution discreteness of the isolated unit is taken as a neighborhood discrete coefficient, and the larger the value is, the less similar the corresponding isolated unit is to other surrounding isolated units, the lower the feature consistency is, that is, the smaller the corresponding feature expressiveness is.
The method comprises the steps of calculating the average value of gray values of all cell units in a preset neighborhood range, and taking a normalized value of the difference absolute value of the center gray average value and the neighborhood gray average value as a neighborhood gray coefficient, wherein the larger the neighborhood gray coefficient is, the more dissimilar the corresponding isolated unit is with other surrounding isolated units, the lower the feature consistency is, namely the smaller the corresponding feature expressive is.
Therefore, the invention takes the inverse proportion normalized value of the product of the neighborhood gray coefficient and the neighborhood discrete coefficient as the characteristic expressivity of the isolated units, and the larger the characteristic expressivity is, the corresponding isolated units are characterized as still belonging to other similar areas which can be supplemented to the surrounding or can form similar areas with the surrounding other isolated units. Thus, the merging probability of each isolated cell is calculated.
Further, in some embodiments of the present invention, determining the merging probability of an isolated bin based on the distribution discreteness, the feature expressiveness, and the gray value distribution of pixel points within the isolated bin includes: gaussian filtering is carried out on pixel points in the isolated units based on a preset filtering scale to obtain a filtered gray value of each pixel point after Gaussian filtering, and a normalized value of a mean value of an absolute value of a difference value between an original gray value and the filtered gray value of each pixel point is calculated to obtain a first weight coefficient; calculating the difference value between the positive integer 1 and the first weight coefficient to obtain a second weight coefficient; and taking the first weight coefficient as a characteristic expressive weight value, taking the second weight coefficient as a distribution discreteness weight value, and carrying out normalization processing after weighted addition to obtain the merging probability of the isolated units.
The preset filtering scale is specifically, for example, 0.2, that is, the gaussian filter with the scale of 0.2 is used to perform gaussian filtering on the pixel points in the isolated unit, the filtered gray value is used as a filtered gray value, for the same pixel point, the absolute value of the difference between the original gray value and the filtered gray value is calculated, the average value of the absolute values of the difference of the gray values corresponding to all the pixel points in the isolated unit is obtained, and the average value is normalized and listed to obtain the first weight coefficient.
It can be understood that the larger the first weight coefficient is, the stronger the corresponding gaussian filtering effect is represented, and because the transmission of image acquisition is usually realized by using a wireless transmission mode when the tower crane works, the corresponding noise is usually represented by salt and pepper noise and white noise, the larger the difference of the gaussian filtering on the gray level change of the pixel point is, the stronger the suppression effect of the gaussian filtering is represented, and based on the difference, the stronger the representation of the corresponding salt and pepper noise and white noise is represented, and the larger the corresponding first weight coefficient is.
The calculation formula of the combining probability in the embodiment of the present invention may specifically be, for example:
the method comprises the steps of carrying out a first treatment on the surface of the In (1) the->Representing the merging probability of isolated units, +.>Representing the first weight coefficient,/->Representing the distribution discreteness of isolated units +.>The characteristic expressivity of the isolated cell is represented,the normalization process is represented.
In an embodiment of the present invention,representing the second weight coefficient, since the larger the feature expressive value is, the higher the correspondence between the corresponding isolated cell and the surrounding is, that is, the more likely to be merged, the larger the merging probability is, and the larger the distribution discreteness is, the more uniform the distribution of the pixel points in the isolated cell is, and thus the greater the merging probability with other surrounding cell cells is, the embodiments of the present invention perform the mergingAnd (5) carrying out specific weighted analysis to obtain the merging probability.
Further, in some embodiments of the present invention, the filtering and merging the isolated units into the similar regions closest to each other according to the merging probability, and taking all the similar regions after merging as the target regions includes: and taking the isolated units with the merging probability larger than the preset probability threshold as units to be merged, merging the units to be merged into any similar region which is closest to the units to be merged, and taking all the similar regions after merging as target regions.
The preset probability threshold is a threshold value of merging probability, and the preset probability threshold in the embodiment of the invention may specifically be, for example, 0.85, or may be adjusted according to actual detection requirements, when the merging probability of an isolated unit is greater than 0.85, a texture expression unit with a normal large probability still exists in the corresponding isolated unit is indicated, and is taken as a unit to be merged, the unit to be merged is merged into any similar region which is closest to the unit to be merged, and all the similar regions after merging are taken as target regions.
It can be understood that in the embodiment of the present invention, when the merging probability of the isolated units is less than or equal to the preset probability threshold, the corresponding isolated units may be further characterized as isolated units with larger influence of noise, so that the corresponding isolated units are deleted, and no subsequent feature analysis is performed on the isolated units, so that feature analysis on cell units with more noise is prevented, and the final feature analysis effect is affected.
S104: and carrying out feature analysis on the target area, determining feature descriptors, and detecting the fatigue degree of tower crane operators in the operation process by combining the feature descriptors of at least two frame surface gray level images to obtain a detection result.
Further, in some embodiments of the present invention, performing feature analysis on the target area to determine a feature descriptor includes: and carrying out feature analysis on the target area based on the HOG feature extraction algorithm to obtain feature descriptors.
The HOG feature extraction algorithm is a feature extraction algorithm commonly used by those skilled in the art, and will not be further described.
In the embodiment of the invention, the HOG feature extraction algorithm can be used for extracting the features of the target area so as to obtain the feature descriptors, or a plurality of other arbitrary possible feature extraction algorithms can be used for realizing the analysis of the feature descriptors so as to facilitate the follow-up fatigue detection according to the feature descriptors.
Further, in some embodiments of the present invention, the detecting the fatigue degree of the tower crane operator in the operation process by combining the feature descriptors of the grayscale images of at least two frames of the grayscale images to obtain a detection result includes: the feature descriptors are input into a pre-trained long-period and short-period memory network model, and are analyzed through the long-period and short-period memory network model, and detection results are output.
In the embodiment of the invention, the fatigue degree can be detected by using a pre-trained long-short-period memory network model, namely the detection result can be specifically, for example, the fatigue degree, the expression form can be a numerical value or a severity grade, and the like, and the method is not limited.
In the embodiment of the invention, various labeled fatigue detection images can be used as a training set, and the long-term memory network model is pre-trained to obtain a pre-trained long-term memory network model, wherein the model training process is a model training mode commonly used in the field, and further description and limitation are omitted.
The embodiment of the invention inputs the feature descriptors into the pre-trained long-term and short-term memory network model for analysis, and outputs the detection result, wherein the process is a common neural network processing process, and the method is not limited to the process.
The invention aims to eliminate the noise of the face image acquired by the tower crane worker during working, so as to facilitate more accurate and objective fatigue detection of the tower crane worker. According to the invention, the face gray level image is divided into the cell units, then, the similarity of the feature vector of each cell unit is analyzed, so that the isolated units are determined, the cell units corresponding to noise and the cell units corresponding to face textures are contained in the isolated units, so that the distribution discreteness and the feature expressivity of each isolated unit are specifically analyzed, the pixel point distribution discreteness degree of the isolated units can be effectively determined through the analysis of the distribution discreteness, the consistency of the isolated units and other surrounding cell units can be effectively determined through the analysis of the feature expressivity, the merging probability of the isolated units is determined through the combination of the distribution discreteness and the feature expressivity, the isolated units are further screened and merged according to the merging probability, and a target area is obtained, so that the area greatly influenced by the noise can be effectively screened out, the accuracy and the reliability of the subsequent feature descriptor acquisition are effectively improved, the feature descriptor with great influence on the noise is avoided, and then, the fatigue degree of a tower crane operator in the operation process is detected according to the feature descriptor, and a detection result is obtained. In summary, the invention can effectively screen out the influence of noise on the feature descriptors, thereby obtaining more accurate and effective feature descriptors and further obtaining more accurate and objective detection results when fatigue detection is carried out.
It should be noted that: the sequence of the embodiments of the present invention is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments.

Claims (10)

1. A method for detecting fatigue of an operator based on spatiotemporal features, the method comprising:
acquiring face gray images of at least two frames of tower crane operators during operation; dividing the facial gray image into at least two cell units with the same size, and determining the feature vector of each cell unit according to the gradient features of all pixel points of each cell unit;
determining the similarity of two cell units according to the feature vectors of the two cell units which are closest to each other, traversing all the face gray images, and dividing the face gray images into similar areas and isolated units according to the similarity;
determining the distribution discreteness of each isolated unit according to the distribution of the gradient directions of all pixel points in each isolated unit and the information entropy; determining the characteristic expressive property of each isolated unit according to the difference of distribution discreteness and the difference of gray values between each isolated unit and other isolated units in a preset neighborhood range; determining the merging probability of the isolated units according to the distribution discreteness, the characteristic expressive property and the gray value distribution of the pixel points in the isolated units; screening the isolated units according to the merging probability, merging the isolated units into similar areas which are closest to each other, and taking all the similar areas after merging as target areas;
and carrying out feature analysis on the target area, determining a feature descriptor, and detecting the fatigue degree of the tower crane operator in the operation process by combining the feature descriptors of at least two frame surface gray level images to obtain a detection result.
2. The method for detecting fatigue of operators based on space-time features as claimed in claim 1, wherein said determining the dispersion of the distribution of each isolated unit based on the distribution of gradient directions of all pixels in the isolated unit and the entropy of information comprises:
constructing gradient direction histograms of all pixel points in the isolated units based on a preset included angle range, and calculating normalized values of variances of gradient direction distribution of all pixel points in each isolated unit according to the gradient direction histograms to obtain first discrete influence coefficients of the isolated units;
taking the extremely poor normalized values of the gradient directions of all pixel points in the isolated units within different preset included angle ranges as second discrete influence coefficients of the isolated units;
calculating normalized values of information entropy of the quantity of gradient directions of all pixel points in the isolated unit in different preset included angle ranges to obtain a third discrete influence coefficient of the isolated unit;
determining the distribution discreteness of the isolated units according to the first discrete influence coefficient, the second discrete influence coefficient and the third discrete influence coefficient, wherein the first discrete influence coefficient and the distribution discreteness are in positive correlation, the second discrete influence coefficient and the third discrete influence coefficient are in inverse correlation with the distribution discreteness, and the value of the distribution discreteness is a normalized numerical value.
3. The method for detecting fatigue of operators based on space-time features as claimed in claim 1, wherein said determining the feature representation of each isolated cell based on the difference in distribution dispersion and the difference in gray value between the isolated cell and other isolated cells within a predetermined neighborhood comprises:
calculating the average value of the absolute value of the difference value of the distribution discreteness between any isolated unit and other isolated units in a preset neighborhood range to obtain a neighborhood discrete coefficient;
calculating the gray value average value of the pixel points of the isolated units to obtain a central gray value average value, and calculating the gray value average value of all the pixel points in other cell units in a preset neighborhood range as a neighborhood gray value average value; taking a normalized value of the difference absolute value of the center gray average value and the neighborhood gray average value as a neighborhood gray coefficient;
and taking an inverse proportion normalized value of the product of the neighborhood gray scale coefficient and the neighborhood discrete coefficient as the characteristic expressive property of the isolated unit.
4. The method for detecting fatigue of operators based on space-time features as claimed in claim 1, wherein said determining a merging probability of said isolated units based on said distribution discreteness, said feature expressiveness, and a gray value distribution of pixels in said isolated units comprises:
gaussian filtering is carried out on pixel points in the isolated units based on a preset filtering scale to obtain a filtered gray value of each pixel point after Gaussian filtering, and a normalized value of a mean value of an absolute value of a difference value between an original gray value and the filtered gray value of each pixel point is calculated to obtain a first weight coefficient; calculating the difference value between the positive integer 1 and the first weight coefficient to obtain a second weight coefficient;
and taking the first weight coefficient as the characteristic expressive weight value, taking the second weight coefficient as the distribution discreteness weight value, and carrying out normalization processing after weighted addition to obtain the merging probability of the isolated units.
5. The method for detecting fatigue of operators based on space-time features as claimed in claim 1, wherein said screening and merging the isolated units into the nearest similar region according to the merging probability, and taking all the merged similar regions as target regions, comprises:
and taking the isolated units with the merging probability larger than a preset probability threshold as units to be merged, merging the units to be merged into any similar area which is closest to the units to be merged, and taking all the similar areas after merging as target areas.
6. The method for detecting fatigue of an operator based on space-time features according to claim 1, wherein the performing feature analysis on the target area to determine a feature descriptor includes:
and carrying out feature analysis on the target area based on an HOG feature extraction algorithm to obtain a feature descriptor.
7. The method for detecting fatigue of an operator based on space-time features according to claim 1, wherein the step of detecting the fatigue degree of the tower crane operator in the operation process by combining the feature descriptors of at least two frame part gray level images to obtain a detection result comprises the steps of:
and inputting the feature descriptors into a pre-trained long-period and short-period memory network model, analyzing the feature descriptors through the long-period and short-period memory network model, and outputting a detection result.
8. The method for detecting fatigue of operators based on space-time features as claimed in claim 1, wherein said determining the feature vector of each cell unit based on the gradient features of all pixels of each cell unit comprises:
and calculating the feature vector of each cell unit according to the gradient amplitude and the gradient direction of all pixel points of each cell unit by using an HOG operator.
9. The method for detecting fatigue of operators based on space-time features as claimed in claim 1, wherein said determining similarity of two cell units based on feature vectors of two cell units closest to each other comprises:
and calculating cosine similarity values of feature vectors of two nearest cell units based on a cosine similarity calculation formula, and taking the cosine similarity values as the similarity of the two cell units.
10. The method for detecting fatigue of operators based on spatiotemporal features of claim 1, wherein said dividing said face gray scale image into similar areas and isolated units based on said similarity comprises:
and combining the cell units with the similarity larger than a preset similarity threshold value to obtain a similar region, and taking the cell units which do not belong to any similar region as isolated units.
CN202311638769.1A 2023-12-04 2023-12-04 Operator fatigue detection method based on space-time characteristics Active CN117351470B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311638769.1A CN117351470B (en) 2023-12-04 2023-12-04 Operator fatigue detection method based on space-time characteristics

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311638769.1A CN117351470B (en) 2023-12-04 2023-12-04 Operator fatigue detection method based on space-time characteristics

Publications (2)

Publication Number Publication Date
CN117351470A true CN117351470A (en) 2024-01-05
CN117351470B CN117351470B (en) 2024-03-19

Family

ID=89366958

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311638769.1A Active CN117351470B (en) 2023-12-04 2023-12-04 Operator fatigue detection method based on space-time characteristics

Country Status (1)

Country Link
CN (1) CN117351470B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008033626A (en) * 2006-07-28 2008-02-14 Kddi Corp Face detector
CN101320004A (en) * 2008-07-03 2008-12-10 西北工业大学 Bamboo strip defect on-line detection method based on machine vision
CN109308445A (en) * 2018-07-25 2019-02-05 南京莱斯电子设备有限公司 A kind of fixation post personnel fatigue detection method based on information fusion
CN111583178A (en) * 2020-04-02 2020-08-25 电子科技大学 Method for segmenting adhesive algae cells in algae cell microscopic image

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008033626A (en) * 2006-07-28 2008-02-14 Kddi Corp Face detector
CN101320004A (en) * 2008-07-03 2008-12-10 西北工业大学 Bamboo strip defect on-line detection method based on machine vision
CN109308445A (en) * 2018-07-25 2019-02-05 南京莱斯电子设备有限公司 A kind of fixation post personnel fatigue detection method based on information fusion
CN111583178A (en) * 2020-04-02 2020-08-25 电子科技大学 Method for segmenting adhesive algae cells in algae cell microscopic image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
XIAOPENG LIU: "Lamb Wave-based Monitoring of Fatigue Crack Propagation using Principal Component Regression", 《IEEE》, no. 08 *
王侃伟, 曹双胜, 方宗德: "区域模糊增强算法在列车轮对检测中的应用", 计算机应用, no. 1 *

Also Published As

Publication number Publication date
CN117351470B (en) 2024-03-19

Similar Documents

Publication Publication Date Title
CN115861135B (en) Image enhancement and recognition method applied to panoramic detection of box body
CN108363970B (en) Method and system for identifying fish species
CN112686331B (en) Forged image recognition model training method and forged image recognition method
CN110059700B (en) Image moire recognition method and device, computer equipment and storage medium
CN107481374B (en) Intelligent terminal fingerprint unblock door opener
CN110717922A (en) Image definition evaluation method and device
Krishnan et al. A survey on different edge detection techniques for image segmentation
Cui et al. An image edge detection method based on haar wavelet transform
CN113822279B (en) Infrared target detection method, device, equipment and medium based on multi-feature fusion
CN109993744B (en) Infrared target detection method under offshore backlight environment
Chitaliya et al. An efficient method for face feature extraction and recognition based on contourlet transform and principal component analysis using neural network
CN107977608B (en) Method for extracting road area of highway video image
CN117351470B (en) Operator fatigue detection method based on space-time characteristics
Abdelkawy et al. Wavelet-based image target detection methods
Soumya et al. Self-organized night video enhancement for surveillance systems
CN108537105B (en) Dangerous behavior identification method in home environment
CN116228712A (en) Multi-scale slope disaster monitoring method, system and device
CN113850252A (en) Method and device for detecting vibration damper bolt and electronic equipment
He et al. A Pedestrian Detection Method Using SVM and CNN Multistage Classification.
Marrón Texture filters and fractal dimension on image segmentation
CN114694090A (en) Campus abnormal behavior detection method based on improved PBAS algorithm and YOLOv5
CN113705672A (en) Threshold value selection method, system and device for image target detection and storage medium
CN113822361A (en) SAR image similarity measurement method and system based on Hamming distance
Wu et al. Recognizing Moving Objects Based on Gaussian-Hermite Moments and ART Neural Networks.
CN118038280B (en) Building construction progress monitoring and early warning method based on aerial image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant