CN114219687A - Intelligent identification method for potential construction safety hazards by fusing human-computer vision - Google Patents
Intelligent identification method for potential construction safety hazards by fusing human-computer vision Download PDFInfo
- Publication number
- CN114219687A CN114219687A CN202111290335.8A CN202111290335A CN114219687A CN 114219687 A CN114219687 A CN 114219687A CN 202111290335 A CN202111290335 A CN 202111290335A CN 114219687 A CN114219687 A CN 114219687A
- Authority
- CN
- China
- Prior art keywords
- hidden danger
- image
- target
- construction
- potential safety
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
- G06Q50/265—Personal security, identity or safety
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
Abstract
An intelligent identification method for potential safety hazards in construction integrating human-computer vision comprises the steps of firstly, tracking an eye jump process by using an eye tracker to obtain a target saliency map based on human eye experience; then identifying a hidden danger image training database to obtain a target saliency map based on primary visual features; training the parameters of the convolutional neural network of the hidden danger part; and finally, establishing a hidden danger knowledge semantic discrimination model, and calculating a similarity matrix of hidden danger information input and a hidden danger knowledge base to realize automatic discrimination of hidden dangers. According to the intelligent identification method for the construction potential safety hazard fusing human-computer vision, the traditional manual inspection construction potential safety hazard is converted into automatic detection of the construction potential safety hazard by a machine, and the labor cost is saved. The mode that detects the potential safety hazard with the expert is applied to the machine for the machine possesses the ability that is similar to the expert and detects the potential safety hazard and goes to potential safety hazard automated inspection, and hidden danger discernment degree is high.
Description
Technical Field
The invention relates to the technical field of construction safety, in particular to an intelligent identification method for potential construction safety hazards by fusing human-computer vision.
Background
The hidden construction safety hazards are mainly caused by unsafe states of people, objects and management, and have the characteristics of concealment, danger, contingency, causality and the like. The construction potential safety hazard is difficult to discover, shows gradually along with the going on of construction, arouses under certain condition, evolves into the incident gradually. In the construction, the hidden trouble has the characteristic of contingency, and the contingency can be developed into the inevitable event without paying attention to the hidden trouble. For potential safety hazards, early discovery, early treatment and early prevention should be achieved.
At present, the potential safety hazard of construction needs to be identified, detected and rectified on site by technicians, depends on the experience and professional level of the technicians, and is difficult to realize real-time, comprehensive and efficient. In the common potential safety hazard and prevention and treatment measures in building construction, the safety precaution measures are as follows: (1) the safety management consciousness is improved (2), a sound and safe production responsibility system is established (3), safety education work is increased (4), acceptance is carried out according to standards and specifications (5), construction equipment is used according to specifications (6), and safety supervision strength is increased. Articles, namely research on methods for investigating hidden dangers in building construction, common hidden dangers and precautionary measures in high-rise building construction, and discussion on measures for coping with the common hidden dangers in building construction, introduce the precautionary measures for the hidden dangers from the management perspective. In the existing patent document CN111145046A, a digital management method and system for the construction process of water conservancy and hydropower engineering are introduced, which realize the monitoring of the site and improve the operation level and engineering quality of the construction process through a digital management platform. The above method has the disadvantages that: the hidden danger cannot be automatically identified, manual identification is still needed, the accuracy of the hidden danger identification depends on the experience of people, and the dependence on the people is too high.
The automatic identification of the potential safety hazards during construction improves the safety management efficiency on one hand, and supervises the operators on the other hand, so that the safety violation behaviors of the construction site are reduced, and the method has the characteristics of high potential safety hazard identification degree, all weather and low cost, and has great significance for reducing safety accidents.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides an intelligent construction potential safety hazard identification method fusing human-machine vision.
The technical scheme adopted by the invention is as follows:
an intelligent identification method for potential safety hazards in construction integrating human-computer vision comprises the steps of firstly, tracking an eye jump process by using an eye tracker to obtain a target saliency map based on human eye experience; then identifying a hidden danger image training database to obtain a target saliency map based on primary visual features; training the parameters of the convolutional neural network of the hidden danger part; and finally, establishing a hidden danger knowledge semantic discrimination model to realize automatic discrimination of hidden dangers.
The intelligent identification method of the construction potential safety hazard integrating human-computer vision comprises the following steps:
the method comprises the following steps: firstly, according to an eye tracker experiment, obtaining eye movement characteristic parameters of a tested population;
secondly, performing coordinate conversion on the gazing clustering central point, solving an optimal path of hidden danger detection, performing image splicing on a hidden danger scene on the optimal path of hidden danger detection to obtain a large-visual-field image, further accurately positioning the hidden danger position in the large-visual-field image according to the group gazing point gathering characteristics, and acquiring a target saliency map S based on primary visual characteristics1；
Thirdly, analyzing the eye movement data, distinguishing a fixation salient region from a non-fixation salient region, inputting a training potential safety hazard image, and acquiring a target salient image S based on the human eye fixation experience2；
Fourthly, further identifying an image target on the basis of accurately positioning the hidden danger exposed part, and training the convolutional neural network parameters of the hidden danger part by combining an established hidden danger image database;
step five, finally establishing a hidden danger knowledge semantic discrimination model, wherein the process is as follows: and calculating a similarity matrix Sim of the hidden danger information input and the hidden danger knowledge base, combining to form hidden danger judgment information, and realizing automatic hidden danger judgment.
The intelligent identification method of the construction potential safety hazard integrating human-computer vision comprises the following steps:
step 1: a large number of hidden danger pictures and videos acquired on site are used as pre-experiment materials, a group visual cognition experiment sample library is established, and a training database is provided for a construction potential safety hazard machine visual identification model.
Step 2: and (3) recruiting construction safety managers with rich experience as tested groups, displaying the sample images one by one in front of the tested groups, and dictating hidden danger positions and characteristics of the tested groups after each sample image is stimulated for a certain time. And recording the hidden danger identification accuracy of each tested group, and selecting the tested group with high hidden danger identification accuracy and good identification stability. The test population screened by the preliminary experiment is used as a test population of the official experiment.
And step 3: according to the construction process, inviting the experimental group to be tested to wear eye tracker equipment to enter an experimental scene, and searching potential safety hazards for all experimental groups to be tested according to respective experiences. And acquiring eye movement characteristic parameters such as a first visual angle video, a fixation point coordinate, fixation point duration and the like when the hidden danger of the tested group is identified through an eye movement instrument.
And 4, step 4: and converting the fixation point coordinates acquired by the eye tracker into target point coordinates in a world coordinate system, performing coordinate conversion on the fixation clustering center point, using the fixation clustering center point as an automatic scanning path key positioning point of a camera monitoring construction site, and solving an optimal path for hidden danger detection by using the shortest time sum of scanning all construction sites as a target function. And carrying out image splicing on a plurality of clear local hidden danger scenes of the hidden danger detection path.
And 5: further accurately positioning the position of the hidden danger in the large-view image according to the gathering characteristics of the group fixation points, and extracting the red, the green and the blue of the input image according to the training database of the hidden danger imageAnalyzing the scene image brightness I, color RGB, direction LB and other basic characteristics of different scales l by using the color channel, the pixel point coordinates of the image and other parameters, obtaining the significant images I (c theta s), RGB (c theta s) and LB (c theta s) of the brightness, color and direction characteristics, normalizing the three significant images, and combining the characteristic weight wI、wRGB、wLBEstablishing a target saliency map S based on primary visual features1：
S1＝wI·I(cΘs)+wRGB·RGB(cΘs)+wLB·LB(cΘs) (4)
Step 6: respectively establishing a fixation time weighted Gaussian map GS by taking the time and the sequence of each fixation point r and the Euclidean distance d (r, v) from the clustering central point v as weightstGaze sequence weighted graph GSrAnd gazing-center-distance weighted graph GSd. Inputting the GS image into a linear support vector machine for training, distinguishing a fixation salient region from a non-fixation salient region, constructing an attention model based on a human eye fixation experience target salient image, solving the optimal w and b parameters of a linear hyperplane model, and extracting a target salient image S based on human eye fixation experience2：
And 7: and (4) integrating the two types of target saliency map construction methods to establish a visual bionic perception model of the construction potential safety hazard. And (4) according to the hidden danger image training database, optimizing the weight parameters of the model, and training a machine vision algorithm to learn the experience of identifying hidden dangers by human eyes.
And 8: classifying a large number of experimental materials such as hidden danger images, videos and eye movement data collected on site according to hidden danger classification to form a convolutional neural network framework, collecting image characteristic information deeply, finding out a candidate region possibly containing a target, acquiring image characteristics contained in the candidate region, inputting the image characteristics into a full connection layer, accessing a softmax classification function to realize classification and identification of the target image, and training convolutional neural network parameters of a hidden danger part by combining an established hidden danger image database.
And step 9: adopting a bounding box algorithm to establish a two-dimensional plane enveloping space of a construction target, converting the spatial position and the relation data thereof into semantic concept expression, establishing a hidden danger knowledge semantic discrimination model, wherein the establishing process comprises the following steps: comparing the hidden danger standard knowledge base, extracting the basic semantic concept V of the hidden danger information input and the hidden danger standard knowledge base1、V2Calculating the semantic element V1、V2And the similarity S between the hidden danger semantic similarity concepts outputs a maximum matching semantic concept set max { Sim } through a hidden danger semantic similarity algorithm, and the hidden danger judgment information is formed through combination, so that the automatic hidden danger judgment is realized.
The invention discloses an intelligent identification method for potential construction safety hazards by fusing human-computer vision, which has the following technical effects:
1) the invention converts the traditional manual inspection construction potential safety hazard into automatic detection construction potential safety hazard by a machine, thereby saving the labor cost.
2) The invention applies the mode of detecting the potential safety hazard by an expert to the machine, so that the machine has the capability similar to the potential safety hazard detection by the expert to automatically detect the potential safety hazard, and the potential safety hazard recognition degree is high.
Drawings
Fig. 1 is a diagram of a hidden danger vision bionic perception model.
Fig. 2 is a flow chart of hidden danger image target identification.
FIG. 3 is a flow chart of semantic discrimination of hidden troubles in construction.
Detailed Description
The intelligent identification method of the construction potential safety hazard integrating human-computer vision comprises the following steps:
1. a large number of hidden danger pictures and videos acquired on site are used as pre-experiment materials, a group visual cognition experiment sample library is established, and a training database is provided for a construction potential safety hazard machine visual identification model.
2. And (3) recruiting construction safety managers with rich experience as tested groups, displaying the sample images one by one in front of the tested groups, and dictating hidden danger positions and characteristics of the tested groups after each sample image is stimulated for a certain time. And recording the hidden danger identification accuracy of each tested group, and selecting the tested group with high hidden danger identification accuracy and good identification stability. The test population screened by the preliminary experiment is used as the test population for the official experiment.
3. According to the construction process, inviting expert groups to wear eye tracker equipment to enter an experiment scene, and all experiments are tested to search routes according to respective experiences to patrol in a construction field. And acquiring eye movement characteristic parameters such as a first visual angle video, a fixation point coordinate, fixation point duration and the like when the hidden danger is identified through an eye movement instrument.
4. According to the principle of eye imaging and data acquisition of an eye tracker, the principle uses a pupil-cornea reflection technology, and the basic principle of the technology is that (1) infrared rays are used for irradiating eyes; (2) collecting infrared rays reflected from the cornea and retina by using a camera; (3) due to the physiological structure and physical properties of the eyeball, on the premise that the relative position of the light source and the head is not changed, the light spot formed by corneal reflection cannot move, and (4) the direction of the light reflected on the retina marks the direction of the pupil (the light of the light source enters from the pupil, and the light reflected by the retina exits from the pupil); and finally (5) calculating the eye movement direction according to the angle between the cornea and the pupil reflected light. Combining the eye movement horizontal visual angle alpha and the vertical visual angle beta of the tested group to the target point, and calculating the scaling ratio px of the visual angles of the x, y and z coordinate axes according to the measurement distance d between the space point of the hidden danger part and the tested pointα(d)、pyβ(d)、pzα(d) In that respect Fixation point coordinates (v) based on eye tracker coordinate systemx,vy,vz) According to the scaling, the space coordinate (vz) of the fixation point under the same coordinate origin is calculatedx,vzy,vzz) And the spatial information mapping of the fixation point of the eye tracker is realized.
vzx＝pxα×vx,vzy＝pyβ×vy,vzz＝pzα×vz (1)；
Wherein: pxα、pyβ、pzαRespectively expressed by an eye trackerThe origin of coordinates, the ratio of the coordinates of the fixation point in the actual spatial position to the coordinates in the x, y, z direction in the eye tracker. vzx、vzy、vzzRespectively representing the actual space position coordinates of the fixation point by taking the eye tracker as the origin of coordinates. Considering the difference between the coordinate system of the eye tracker and the world space coordinate system, a space coordinate system conversion equation is established by taking the space coordinate of the fixation point as the basis and the coordinates (X, Y, Z) of the fixation point object based on the origin of the camera as the reference, a coordinate system conversion matrix M is obtained, and the fixation point coordinate collected by the eye tracker is converted into the target point coordinate under the world coordinate system.
[X,Y,Z]＝M[vzx,vzy,vzz] (2)；
And (4) carrying out coordinate transformation on a watching cluster central point, wherein the watching cluster central point refers to a central point of a watching point gathered in a certain range, and the watching cluster central point is used as a key positioning point of an automatic scanning path of a camera monitoring construction site so as to scan the shortest time sum of all the construction sites as a target function and solve the optimal path of hidden danger detection. And carrying out image splicing on a plurality of clear local hidden danger scenes of the hidden danger detection path.
The specific process of solving the optimal path for detecting the hidden danger is as follows:
calculating the retention time TS of each scene according to the accumulated time sum sigma ms of injection points in each watching clustering regionKThe shortest total time min sigma TS required for scanning all construction areasKConsidering the priority order ζ of the detection sites as an objective functionKAnd establishing a construction potential safety hazard detection traversal model based on the problem of the traveling salesman, and solving an optimal traversal path for detecting the optimal potential safety hazard by using a particle swarm algorithm. According to the hidden danger detection path, a large number of hidden danger local images are collected by combining key watching areas on the path, and an image data basis is provided for hidden danger identification.
The problem of the traveling salesman who is detected by the construction potential safety hazard: the method takes the best hidden danger detection effect (time for detecting all construction areas and recognition rate) as an objective function, and comprehensively considers the constraint conditions such as the priority of detection parts, the retention time of key watching areas, the number of collected images (the number of personnel and mechanical equipment, the operation type and the occurrence frequency and the severity of hidden dangers) and the like.
The image splicing process for the hidden danger scene of the hidden danger detection path is as follows:
firstly, setting shooting overlapping rates of two adjacent scene images on a hidden danger detection path, and ensuring that the shot adjacent images have overlapping areas.
Secondly, carrying out multi-scale space division on the hidden trouble image, extracting a region with obvious pixel change amplitude between the same scale plane and different scales, and acquiring a stable characteristic point set P (P) under the scale change1,p2,...,pn}；
Then, calculating the gradient direction theta (x, y) and the gradient amplitude m (x, y) of the pixel (x, y) of each feature point, and describing the feature points according to the feature vectors p (theta (x, y), m (x, y)) of the feature points to form a multi-dimensional feature description subset;
and finally, traversing and calculating the Euclidean distance l between the feature points in the adjacent images according to the feature vectors of the feature points, and taking the minimum distance as a matched feature point:
based on an image feature point matching strategy algorithm, splicing adjacent images with overlapping regions into a large-view hidden danger image, and completing image acquisition work of a construction scene on a path most possibly having construction potential safety hazards to cover the construction potential safety hazards.
5. Acquiring a primary visual feature target saliency map S1The specific process is as follows:
and (5.1) further accurately positioning the hidden danger positions in the large-view images according to the gathering characteristics of the group fixation points. Extracting red r of input image according to hidden danger image training databaselG green glB, blue blThree color channels and parameters such as pixel point coordinates (x, y) of the image, extracting basic characteristics such as scene image brightness I, color RGB, direction LB and the like of different scales l, and training a target significant attention model based on primary visual characteristics:
in formula (3): i islRepresenting the scene luminance in the l scale, rlRed channel, g, representing the input image at the l-scalelGreen channel representing the input image at the l-scale, blRepresenting the blue channel, RGB, of an input image at the l-scalelFeatures representing three colors on the l scale, fRGBl(rl,gl,bl) Representing the color characteristic function of three colors, LB (x, y) representing the direction characteristic of the pixel point coordinate (x, y), fLB(x, y) represents a function of the directional characteristic of the pixel point coordinates (x, y).
(5.2) obtaining saliency maps I (c Θ s), RGB (c Θ s) and LB (c Θ s) of brightness, color and direction features aiming at scale factors of high resolution c and low resolution s in an image scale space by calculating gradient difference Θ between central pixels and peripheral pixels of an image area, normalizing the three saliency maps, and combining feature weights wI、wRGB、wLBEstablishing a target saliency map S based on primary visual features1：
S1＝wI·I(cΘs)+wRGB·RGB(cΘs)+wLB·LB(cΘs) (4)
In formula (4): s1Representing a target saliency map, w, based on primary visual featuresI、wRGB、wLBFeature weights of saliency maps representing features of luminance, color, and orientation, I (c Θ s), RGB (c Θ s), and LB (c Θ s) represent saliency maps representing features of luminance, color, and orientation.
6. Obtaining a target saliency map S based on human eye gaze experience2The specific process is as follows:
(6.1) respectively establishing a fixation time weighted Gaussian map GS by taking the time and the sequence of each fixation point r and the Euclidean distance d (r, v) from the clustering central point v as weightstGaze sequence weighted graph GSrAnd gazing-center-distance weighted graph GSd. Weighting the three graphs based on the weight alpha, beta and gamma to form a multi-graphTarget detection gazing gaussian GS:
GS＝αGSt+βGSr+γGSd (5)
and (6.2) carrying out secondary classification on the multi-target detection gazing Gaussian map GS by adopting a linear support vector machine, wherein the secondary classification is to find a hyperplane and distinguish a gazing salient region from a non-gazing salient region. Inputting the GS image into a linear support vector machine for training, finding out a hyperplane, and distinguishing a fixation significant point from a non-fixation significant point, wherein a solving model of the linear hyperplane is as follows:
in the formula (6), w is a weight, b is a deviation vector, and y is a sample true category label.
By inputting training potential safety hazard images, solving optimal parameters w and b, constructing an attention model based on the eye-gazing experience target saliency map, and extracting a target saliency map S based on the eye-gazing experience2。
7. A target significant attention model based on primary visual features and human eye gazing experience is synthesized, a construction safety hidden danger visual bionic perception model is constructed, and the process is shown in figure 1. And (3) training the hidden danger identification experience of a machine vision learner, and accurately positioning the exposed part of the hidden danger. The process of constructing the visual bionic perception model of the construction potential safety hazard comprises the following steps:
firstly, inputting images in a hidden danger image training database, then obtaining a primary visual characteristic target saliency map, and forming a construction safety hidden danger visual bionic perception model by combining the primary visual characteristic target saliency map and the experience target saliency map watched by human eyes.
8. On the basis of accurately positioning the hidden danger exposed part, further identifying an image target, and specifically carrying out the following process:
and (8.1) classifying a large number of experimental materials such as hidden danger pictures, videos and eye movement data collected on site according to hidden danger classification, and providing basic data for training parameters of the convolutional neural network.
(8.2) inputting the original image of the hidden trouble part into a convolutional neural network VGG16, alternately and repeatedly processing the convolutional layer Conv and the Pooling layer Pooling through a VGG16 network, and then extracting an image feature vector through a full connecting layer and a softmax classification layer to obtain a convolutional feature map of the image of the hidden trouble part. The process of generating the convolution signature is as follows:
red, green and blue are three primary colors, which can basically synthesize colors that human eyes can distinguish, convolution operation is carried out on an input original image of a hidden danger part, three characteristic filters of red, green and blue are utilized to obtain three-channel output, the strength of a numerical value at a certain position in a channel is a reaction to the strength of the current characteristic, namely a convolution characteristic diagram,
and (8.3) aiming at the extracted convolution feature map, determining the number of contours in the candidate frame and the number of edges overlapped with the edges of the candidate frame by adopting an EdgeBoxes method and utilizing the information such as the texture, the edge color and the like of the feature map, finding out a candidate region possibly containing a target, and generating a regression boundary of the candidate region. And recombining the candidate regions and the convolution characteristic graphs corresponding to the candidate regions in a pooling layer, accessing a full connection layer and a softmax classification layer, and detecting the target class and the region boundary. And training parameters of the convolutional neural network by combining the established hidden danger image database. The hidden danger area image target identification flow is shown in fig. 2.
9. The hidden danger semantic relation reasoning process comprises the following steps:
(9.1) taking irregular target shape characteristics of constructors, machinery, equipment and the like into consideration, establishing a two-dimensional plane envelope space of a construction target by adopting a bounding box algorithm, and respectively extracting a maximum value coordinate point (x, y) of the envelope spacemaxCoordinate with the minimum value (x, y)min。
(9.2) detecting the image coordinate data of the target through the trained convolutional neural network parameters, and calculating an image scaling conversion matrix a1Symmetric iso-transform matrix a2Angle transformation matrix a3Constructing a transformation matrix of coordinates (x, y) and actual coordinates (x ', y') on the image target graph
Realizing the position mapping transformation of the image target and the actual space:
(9.3) establishing a hidden danger knowledge semantic discrimination model, wherein the process is as follows:
converting the spatial position and the relation data thereof into semantic concept expression, comparing the hidden danger standard knowledge base, extracting the hidden danger information input and the basic semantic concept V of the hidden danger standard knowledge base1、V2V. location1、V2Structural position VL in semantic network1、VL2Measure both network node distance Dist (VL)1,VL2) And the semantic network maximum level maxVL. Synthesizing the parameters to calculate semantic element V1、V2Similarity between S:
traversing m semantic concepts VM (virtual machine) of hidden danger text input by using a semantic element similarity algorithm11,V12,···,V1mAnd n semantic concepts VN ═ V in hidden danger specification knowledge base21,V22,···,V2nCalculating the similarity between the hidden danger information input and a similarity matrix Sim of a knowledge base:
in equation (9), VM is m semantic concepts of hidden text input, VM ═ V11,V12,···,V1mVN is n semantic concepts VN ═ V in the hidden danger standard knowledge base21,V22,···,V2nIn Sim matrix
The similarity between any hidden danger text and a hidden danger standard knowledge base.
And outputting a maximum matching degree semantic concept set max { Sim } through a hidden danger semantic similarity algorithm, combining to form hidden danger judgment information, and realizing automatic hidden danger judgment. As shown in fig. 3.
The hidden danger is automatically distinguished, namely, a machine vision is utilized to identify a construction site, the obtained information is the distinguishing information which needs to judge whether the hidden danger exists, similarity calculation is carried out on the distinguishing information and a hidden danger standard knowledge base, if the obtained information is matched with the hidden danger information in the knowledge base, the hidden danger is identified in the construction site, early warning and rectification are needed, then a knowledge case is formed and integrated into the hidden danger standard knowledge base, and if the obtained information is not matched with the hidden danger information in the knowledge base, the obtained information is not the hidden danger. The automatic identification of the hidden danger can automatically monitor and identify the construction potential safety hazard, has the advantages of high identification degree, all weather and low cost, and has great significance for reducing safety accidents.
Claims (9)
1. The intelligent identification method of the construction potential safety hazard by fusing human-computer vision is characterized by comprising the following steps of: firstly, tracking an eye jump process by using an eye tracker to obtain a target saliency map based on human eye experience; then identifying a hidden danger image training database to obtain a target saliency map based on primary visual features; training the parameters of the convolutional neural network of the hidden danger part; and finally, establishing a hidden danger knowledge semantic discrimination model to realize automatic discrimination of hidden dangers.
2. The intelligent identification method of the construction potential safety hazard integrating human-computer vision is characterized by comprising the following steps of:
the method comprises the following steps: firstly, according to an eye movement instrument, obtaining an eye movement characteristic parameter;
secondly, performing coordinate conversion on the gazing clustering central point, solving an optimal path of hidden danger detection, performing image splicing on a hidden danger scene on the optimal path of hidden danger detection to obtain a large-visual-field image, further accurately positioning the hidden danger position in the large-visual-field image according to the group gazing point gathering characteristics, and acquiring a target saliency map S based on primary visual characteristics1；
Thirdly, analyzing the eye movement data, distinguishing the watching salient region from the non-watching salient region, and inputting a training potential safety hazard graphImage, obtain the target saliency map S based on the human eye' S gaze experience2；
Fourthly, further identifying an image target on the basis of accurately positioning the hidden danger exposed part, and training the convolutional neural network parameters of the hidden danger part by combining an established hidden danger image database;
and fifthly, establishing a hidden danger knowledge semantic discrimination model, calculating a similarity matrix of hidden danger information input and a hidden danger knowledge base, combining to form hidden danger discrimination information, integrating the whole process and realizing automatic discrimination of hidden dangers.
3. The intelligent identification method for the potential safety hazard in construction integrating human-computer vision as claimed in claim 2, characterized in that: in the first step, eye movement characteristic parameters such as a first visual angle video, a fixation point coordinate, fixation point duration and the like when the hidden danger of the tested group is identified are collected through an eye movement instrument.
4. The intelligent identification method for the potential safety hazard in construction integrating human-computer vision as claimed in claim 2, characterized in that: in the second step, according to the principle of eye imaging and data acquisition of an eye tracker, combining the horizontal visual angle alpha and the vertical visual angle beta of the tested group to the target point, and according to the measured distance d between the space point of the hidden danger part and the tested group, calculating the scaling ratio px of the visual angles of the x, y and z coordinate axesα(d)、pyβ(d)、pzα(d) The gaze point coordinates (v) based on the eye tracker coordinate systemx,vy,vz) According to the scaling, the space coordinate (vz) of the fixation point under the same coordinate origin is calculatedx,vzy,vzz) Realizing the spatial information mapping of the fixation point of the eye tracker;
vzx＝pxα×vx,vzy＝pyβ×vy,vzz＝pzα×vz (1)；
considering the difference between the coordinate system of the eye tracker and the world space coordinate system, establishing a space coordinate system conversion equation by taking the fixation point space coordinate as the basis and the fixation point physical coordinate (X, Y, Z) based on the origin of the camera as the reference, solving a coordinate system conversion matrix M, and converting the fixation point coordinate collected by the eye tracker into the target point coordinate under the world coordinate system;
[X,Y,Z]＝M[vzx,vzy,vzz] (2)
performing coordinate transformation on the gazing clustering center point, taking the gazing clustering center point as an automatic scanning path key positioning point of a camera monitoring construction site, and solving an optimal path for hidden danger detection by taking the shortest time sum of scanning all the construction sites as a target function; and carrying out image splicing on a plurality of clear local hidden danger scenes of the hidden danger detection path.
5. The intelligent identification method for the potential safety hazard in construction integrating human-computer vision as claimed in claim 2, characterized in that:
in the second step, a target saliency map S based on the primary visual features is obtained1The method comprises the following steps:
step 2.1: extracting parameters such as red, green and blue color channels of an input image and pixel point coordinates of the image according to a hidden danger image training database, analyzing basic characteristics such as scene image brightness I, color RGB, direction LB and the like of different scales l, and training a target significant attention model based on primary visual characteristics:
step 2.2: calculating gradient difference theta of central pixels and peripheral pixels of an image area according to scale factors of high resolution c and low resolution s in an image scale space, obtaining saliency maps I (c theta s), RGB (c theta s) and LB (c theta s) of brightness, color and direction features, normalizing the three saliency maps, and combining feature weights wI、wRGB、wLBEstablishing a target significant attention model S based on the primary visual characteristics1：
S1＝wI·I(cΘs)+wRGB·RGB(cΘs)+wLB·LB(cΘs) (4)。
6. The intelligent identification method for the potential safety hazard in construction integrating human-computer vision as claimed in claim 2, characterized in that:
in the third step, a target saliency map S based on human eye gazing experience is obtained2The method comprises the following steps:
step 3.1: respectively establishing a fixation time weighted Gaussian map GS by taking the time and the sequence of each fixation point r and the Euclidean distance d (r, v) from the clustering central point v as weightstGaze sequence weighted graph GSrAnd gazing-center-distance weighted graph GSd(ii) a And forming a multi-target detection fixation Gaussian map GS based on the weighted alpha, beta and gamma maps:
GS＝αGSt+βGSr+γGSd (5)；
step 3.2: inputting the GS image into a linear support vector machine for training, constructing an attention model based on the eye-gazing experience target saliency, solving the optimal w and b parameters of a linear hyperplane model, carrying out secondary classification on the multi-target detection gazing Gaussian image GS, and extracting the target saliency S based on the eye-gazing experience2Finding out hyperplane distinguishing staring salient points and non-staring salient points, and distinguishing staring salient areas and non-staring salient areas;
s.t y(wTx+b)＞1 (6)。
7. the intelligent identification method for the potential safety hazard in construction integrating human-computer vision as claimed in claim 2, characterized in that: in the fourth step, the image target is further identified, and the method specifically comprises the following steps:
step 4.1: classifying a large number of experimental materials such as hidden danger pictures, videos and eye movement data collected on site according to hidden danger classification, and providing basic data for training convolutional neural network parameters of hidden danger parts;
step 4.2: inputting the original image of the hidden trouble part into a convolutional neural network VGG16, alternately and repeatedly processing a convolutional layer Conv and a Pooling layer Pooling through a VGG16 network, then extracting an image feature vector through a full connection layer and a softmax classification layer, and obtaining a convolutional feature map of the hidden trouble part image;
step 4.3: aiming at the extracted convolution feature map, determining the number of contours in the candidate frame and the number of edges overlapped with the edges of the candidate frame by adopting an EdgeBoxes method and utilizing the texture, edge color and other information of the feature map, finding out a candidate region possibly containing a target, and generating a regression boundary of the candidate region; recombining the candidate regions and the convolution characteristic graphs corresponding to the candidate regions in a pooling layer, accessing a full connection layer and a softmax classification layer, and detecting a target class and a region boundary; and training the parameters of the convolutional neural network of the hidden danger part by combining the established hidden danger image database.
8. The intelligent identification method for the potential safety hazard in construction integrating human-computer vision as claimed in claim 2, characterized in that:
the fifth step comprises the following steps:
step 5.1: considering irregular target shape characteristics of constructors, machinery, equipment and the like, establishing a two-dimensional plane envelope space of a construction target by adopting a bounding box algorithm, and respectively extracting a maximum value coordinate point (x, y) of the envelope spacemaxCoordinate with the minimum value (x, y)min；
Step 5.2: detecting the image coordinate data of the target through the convolution neural network parameters of the hidden trouble positions, and calculating an image scaling conversion matrix a1Symmetric iso-transform matrix a2Angle transformation matrix a3Constructing a transformation matrix of coordinates (x, y) and actual coordinates (x ', y') on the image target graph
Realizing the position mapping transformation of the image target and the actual space:
step 5.3: establishing a hidden danger knowledge semantic discrimination model to realize automatic discrimination of hidden dangers, wherein the process is as follows:
converting the spatial position and the relation data thereof into semantic concept expression, comparing the hidden danger standard knowledge base, extracting the hidden danger information input and the basic semantic concept V of the hidden danger standard knowledge base1、V2V. location1、V2Structural position VL in semantic network1、VL2Measure both network node distance Dist (VL)1,VL2) And a semantic network maximum level maxVL; synthesizing the parameters to calculate semantic element V1、V2Similarity between S:
traversing m semantic concepts VM (virtual machine) of hidden danger text input by using a semantic element similarity algorithm11,V12,…,V1mAnd n semantic concepts VN ═ V in hidden danger specification knowledge base21,V22,…,V2nCalculating the similarity between the hidden danger information input and a similarity matrix Sim of a knowledge base:
and outputting a maximum matching degree semantic concept set max { Sim } through a hidden danger semantic similarity algorithm, combining to form hidden danger judgment information, integrating the whole process, and realizing automatic hidden danger judgment.
9. The intelligent identification method of the construction potential safety hazard integrating human-computer vision is characterized by comprising the following steps of:
step 1: taking the hidden danger pictures and videos collected on site as pre-experiment materials, and establishing a group visual cognition experiment sample library as a training database;
step 2: displaying the sample images one by one in front of the tested group, stimulating each sample image for a certain time, and dictating hidden danger parts and characteristics of the tested group; recording the hidden danger identification accuracy of each tested population, and selecting the tested population with high hidden danger identification accuracy and good identification stability as the tested population for experiments;
and step 3: acquiring a first visual angle video, a fixation point coordinate and a fixation point duration eye movement characteristic parameter when a tested group identifies hidden danger through an eye movement instrument;
and 4, step 4: transforming the fixation point coordinates acquired by the eye tracker into target point coordinates in a world coordinate system, performing coordinate transformation on the fixation clustering center point, using the fixation clustering center point as an automatic scanning path key positioning point of a camera monitoring construction site, and solving an optimal path for hidden danger detection by scanning the shortest time sum of all construction sites as a target function; carrying out image splicing on a plurality of clear local hidden danger scenes of the hidden danger detection path;
and 5: according to the gathering characteristics of group fixation points, the position of hidden danger in a large-view image is further accurately positioned, parameters such as red, green and blue color channels of an input image and pixel point coordinates of the image are extracted according to a hidden danger image training database, basic characteristics such as scene image brightness I, color RGB and direction LB of different scales l are analyzed, a significant graph I (c theta s), RGB (c theta s) and LB (c theta s) of brightness, color and direction characteristics are obtained, the three significant graphs are normalized, and the characteristic weight w is combinedI、wRGB、wLBEstablishing a target saliency map S based on primary visual features1：
S1＝wI·I(cΘs)+wRGB·RGB(cΘs)+wLB·LB(cΘs) (4)
Step 6: respectively establishing a fixation time weighted Gaussian map GS by taking the time and the sequence of each fixation point r and the Euclidean distance d (r, v) from the clustering central point v as weightstGaze sequence weighted graph GSrAnd gazing-center-distance weighted graph GSd(ii) a Inputting the GS image into a linear support vector machine for training, distinguishing a fixation salient region from a non-fixation salient region, constructing an attention model based on a human eye fixation experience target salient image, solving the optimal w and b parameters of a linear hyperplane model, and extracting the human eye fixation experience based attention modelTarget saliency map S2：
s.t y(wTx+b)＞1 (6)
And 7: integrating two types of target saliency map construction methods, and establishing a construction potential safety hazard visual bionic perception model; optimizing model weight parameters according to a hidden danger image training database, and training a machine vision algorithm to learn hidden danger identification experience of human eyes;
and 8: classifying potential hazard pictures, videos and eye movement data experiment materials acquired on site according to potential hazard classification to form a convolutional neural network framework, deeply acquiring image characteristic information, finding out a candidate region possibly containing a target, acquiring image characteristics contained in the candidate region, inputting the image characteristics into a full connection layer, accessing a softmax classification function to realize classification and identification of the target image, and training convolutional neural network parameters of a potential hazard part by combining an established potential hazard image database;
and step 9: adopting a bounding box algorithm to establish a two-dimensional plane enveloping space of a construction target, converting the spatial position and the relation data thereof into semantic concept expression, establishing a hidden danger knowledge semantic discrimination model, wherein the establishing process comprises the following steps: comparing the hidden danger standard knowledge base, extracting the basic semantic concept V of the hidden danger information input and the hidden danger standard knowledge base1、V2Calculating the semantic element V1、V2The similarity S between the hidden danger semantic similarity concepts is obtained by a hidden danger semantic similarity algorithm, a maximum matching semantic concept set max { Sim } is output, hidden danger judgment information is formed by combination, and automatic hidden danger judgment is achieved;
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111290335.8A CN114219687A (en) | 2021-11-02 | 2021-11-02 | Intelligent identification method for potential construction safety hazards by fusing human-computer vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111290335.8A CN114219687A (en) | 2021-11-02 | 2021-11-02 | Intelligent identification method for potential construction safety hazards by fusing human-computer vision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114219687A true CN114219687A (en) | 2022-03-22 |
Family
ID=80696456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111290335.8A Pending CN114219687A (en) | 2021-11-02 | 2021-11-02 | Intelligent identification method for potential construction safety hazards by fusing human-computer vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114219687A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115908954A (en) * | 2023-03-01 | 2023-04-04 | 四川省公路规划勘察设计研究院有限公司 | Geological disaster hidden danger identification system and method based on artificial intelligence and electronic equipment |
| CN116310760A (en) * | 2023-04-06 | 2023-06-23 | 河南禹宏实业有限公司 | Intelligent water conservancy monitoring system based on machine vision |
| CN116630752A (en) * | 2023-07-25 | 2023-08-22 | 广东南方电信规划咨询设计院有限公司 | Construction site target object identification method and device based on AI algorithm |
-
2021
- 2021-11-02 CN CN202111290335.8A patent/CN114219687A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115908954A (en) * | 2023-03-01 | 2023-04-04 | 四川省公路规划勘察设计研究院有限公司 | Geological disaster hidden danger identification system and method based on artificial intelligence and electronic equipment |
| CN115908954B (en) * | 2023-03-01 | 2023-07-28 | 四川省公路规划勘察设计研究院有限公司 | Geological disaster hidden danger identification system and method based on artificial intelligence and electronic equipment |
| CN116310760A (en) * | 2023-04-06 | 2023-06-23 | 河南禹宏实业有限公司 | Intelligent water conservancy monitoring system based on machine vision |
| CN116310760B (en) * | 2023-04-06 | 2023-09-22 | 河南禹宏实业有限公司 | Intelligent water conservancy monitoring system based on machine vision |
| CN116630752A (en) * | 2023-07-25 | 2023-08-22 | 广东南方电信规划咨询设计院有限公司 | Construction site target object identification method and device based on AI algorithm |
| CN116630752B (en) * | 2023-07-25 | 2023-11-17 | 广东南方电信规划咨询设计院有限公司 | Construction site target object identification method and device based on AI algorithm |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114219687A (en) | Intelligent identification method for potential construction safety hazards by fusing human-computer vision | |
| US20230383232A1 (en) | Method and system for automated microbial colony counting from streaked sample on plated media | |
| CN112288706B (en) | Automatic chromosome karyotype analysis and abnormality detection method | |
| CN103886308B (en) | A kind of pedestrian detection method of use converging channels feature and soft cascade grader | |
| US8577151B2 (en) | Method, apparatus, and program for detecting object | |
| CN109635875A (en) | A kind of end-to-end network interface detection method based on deep learning | |
| CN106980852B (en) | Based on Corner Detection and the medicine identifying system matched and its recognition methods | |
| CN105844621A (en) | Method for detecting quality of printed matter | |
| CN110837784A (en) | Examination room peeping cheating detection system based on human head characteristics | |
| CN106845502A (en) | It is a kind of to visualize guidance method for the Wearable servicing unit of overhaul of the equipments and overhaul of the equipments | |
| CN109255298A (en) | Safety cap detection method and system in a kind of dynamic background | |
| CN107273891A (en) | A kind of target category detection method based on click supervised training | |
| CN109558825A (en) | A kind of pupil center's localization method based on digital video image processing | |
| CN113435282B (en) | Unmanned aerial vehicle image ear recognition method based on deep learning | |
| CN115272652A (en) | Dense object image detection method based on multiple regression and adaptive focus loss | |
| CN113033315A (en) | Rare earth mining high-resolution image identification and positioning method | |
| CN110032932A (en) | A kind of human posture recognition method based on video processing and decision tree given threshold | |
| CN114170686A (en) | Elbow bending behavior detection method based on human body key points | |
| CN104063689B (en) | Face image identification method based on binocular stereoscopic vision | |
| CN114049589A (en) | Transformer substation monitoring system based on artificial intelligence | |
| CN110660187B (en) | Forest fire alarm monitoring system based on edge calculation | |
| KR100543706B1 (en) | Vision-based humanbeing detection method and apparatus | |
| CN114330477B (en) | Power equipment defect detection system and method based on mixed reality equipment | |
| CN115995097A (en) | Deep learning-based safety helmet wearing standard judging method | |
| CN109886325A (en) | A kind of stencil-chosen and acceleration matching process of non linear color space classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |