CN116665305A - Method and system for detecting worker behaviors based on computer vision and knowledge graph - Google Patents
Method and system for detecting worker behaviors based on computer vision and knowledge graph Download PDFInfo
- Publication number
- CN116665305A CN116665305A CN202310708247.8A CN202310708247A CN116665305A CN 116665305 A CN116665305 A CN 116665305A CN 202310708247 A CN202310708247 A CN 202310708247A CN 116665305 A CN116665305 A CN 116665305A
- Authority
- CN
- China
- Prior art keywords
- behavior
- construction site
- worker
- unsafe
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000006399 behavior Effects 0.000 title claims abstract description 161
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000010276 construction Methods 0.000 claims abstract description 102
- 238000001514 detection method Methods 0.000 claims abstract description 38
- 230000033001 locomotion Effects 0.000 claims abstract description 30
- 230000011218 segmentation Effects 0.000 claims abstract description 24
- 238000007418 data mining Methods 0.000 claims abstract description 8
- 230000003542 behavioural effect Effects 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 13
- 230000000694 effects Effects 0.000 claims description 10
- 230000003993 interaction Effects 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 4
- 238000000605 extraction Methods 0.000 claims description 3
- 239000003999 initiator Substances 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- CDBYLPFSWZWCQE-UHFFFAOYSA-L Sodium Carbonate Chemical compound [Na+].[Na+].[O-]C([O-])=O CDBYLPFSWZWCQE-UHFFFAOYSA-L 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000011449 brick Substances 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 231100000279 safety data Toxicity 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 239000002023 wood Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
- G06N5/025—Extracting rules from data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
- G06Q50/265—Personal security, identity or safety
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Tourism & Hospitality (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Computer Security & Cryptography (AREA)
- Social Psychology (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Development Economics (AREA)
- Educational Administration (AREA)
- Psychiatry (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a worker behavior detection method based on computer vision and a knowledge graph, which comprises the following steps: acquiring a construction site image, and identifying a construction site target entity according to a target detection model; panoramic segmentation is carried out on the construction site image by adopting a panoramic segmentation model, so as to obtain scene information of workers; tracking a motion trail of a worker in a construction site image by adopting a trail tracking recognition model, and acquiring motion state information of the worker; performing secondary data mining according to information obtained from the construction site image, and obtaining image characteristic information corresponding to the ontology model, thereby creating a graphic data instance; according to the graphic data instance, the unsafe behavior type, the unsafe behavior result and the intervention or management measures of the unsafe behavior are judged through the condition matching with the rules in the rule base. Compared with the prior art, the method has the advantages of realizing automatic judgment of unsafe behaviors of workers, being easy to expand, being capable of flexibly adapting to the requirements of different construction scenes, and the like.
Description
Technical Field
The invention relates to the technical field of detection of worker behaviors in construction sites, in particular to a method and a system for detecting worker behaviors based on computer vision and a knowledge graph.
Background
In recent years, construction safety problems have been paid attention to, and according to accident cause statistics, 80-90% of construction safety accidents are caused by unsafe behaviors of people, so that unsafe behaviors of a construction site are considered to be main causes of accidents. Behavior safety management is an effective method for normalizing worker behaviors and avoiding accidents.
Along with the continuous development of technology, construction safety intelligent supervision is a trend of future development. The computer vision, the deep learning and other technologies are gradually applied to unsafe behavior recognition of a construction site, and the unsafe behavior recognition method has excellent performance in the unsafe behavior recognition of workers in the scenes of intrusion into construction dangerous areas, wearing of worker safety equipment, high-place operation of the workers and the like. Compared with manual supervision, automatic supervision by means of computer vision technology has significant advantages: high efficiency, accuracy and labor saving, and can not interfere the first-line construction operation of the construction site.
However, there are some common limitations to current computer vision based construction site worker unsafe behavior identification techniques, including: the application scene range is limited, and only single unsafe behavior can be judged; the expansibility is poor, and the recognition requirement of new unsafe behavior caused by the change specified by the specification cannot be flexibly met; semantic relationships between targets cannot be further extracted, and requirements of knowledge-intensive safety supervision are difficult to meet.
Disclosure of Invention
The invention aims to overcome the defects of limited applicable scenes, poor expansibility and incapability of further extracting semantic relations among targets in the prior art, and provides a worker behavior detection method and system based on computer vision and a knowledge graph.
The aim of the invention can be achieved by the following technical scheme:
a worker behavior detection method based on computer vision and knowledge graph comprises the following steps:
acquiring a construction site image, and identifying a construction site target entity according to a pre-established and trained target detection model; performing panoramic segmentation on the construction site image by adopting a pre-established and trained panoramic segmentation model to acquire scene information of workers; tracking a motion trail of a worker in the construction site image by adopting a pre-established and trained trail tracking recognition model to acquire motion state information of the worker;
performing secondary data mining according to the information obtained from the construction site image, and obtaining image characteristic information corresponding to a pre-constructed body model, thereby creating a graphic data instance;
according to the graphic data instance, judging the type of unsafe behavior, the result of unsafe behavior and giving intervention or management measures of unsafe behavior by performing condition matching with rules in a rule base established in advance.
Further, the semantic concepts in the ontology model include:
behavior bodies, major initiators of unsafe behavior, including workers;
an object, an entity other than a behavior host existing in the construction site image;
activity, changes caused by a behavioural subject;
location, location and interface to which unsafe behavior relates;
time, time involved in unsafe behavior;
an attribute, a specific description of a property of an entity;
behavioral consequences, consequences resulting from the subject's behavior;
feedback, intervention or management measures acting on the behavioural subject based on the behavioural consequences.
Further, the judging process of the unsafe behavior type comprises the following steps: at a specific time and place, a behavior subject with specific attributes engages in specific activities and specific interactions and associations with objects with specific attributes occur, leading to a predisposed behavior outcome and giving corresponding feedback.
Further, the rule includes a result item and a condition item, both of which are in the form of triples;
the process of performing condition matching according to the rule so as to judge the unsafe behavior type comprises the following steps: and matching the information in the graphic data instance according to the condition items in the rule, so as to obtain a corresponding result item, and outputting the type of unsafe behavior.
Further, the identified construction site target entity comprises a behavior subject and an object, and interaction and association between the subject and the object are judged according to the overlapping ratio of the identified subject and the object;
according to the scene information of the identified worker, the scene information is used as attribute information of the corresponding position of the behavior main body;
according to the result of tracking the movement track of the worker, acquiring attribute information of a movement state corresponding to the behavior main body; the location and time are pre-input information.
Further, the acquiring process of the located position includes: and taking a point in the detection frame of the detected target entity as a key point, and acquiring the category of the image pixel point corresponding to the key point by using the panoramic segmentation model so as to judge the position.
Further, the process of acquiring the motion state includes: and calculating the average pixel speed of the target entity according to the coordinate change information of the detection frame of the target entity in different image frames, so that the average pixel speed is mapped into the actual speed, and the corresponding motion state is obtained.
Further, the rules in the rule base include:
an internal logic rule for judging the unsafe behavior type;
an unsafe behavior result judging rule for judging the result of unsafe behavior;
the unsafe behavior feedback judgment rule is used for feeding back intervention or management measures of unsafe behavior.
Further, the unsafe behavior result judgment rule and the unsafe behavior feedback judgment rule are obtained by coding and converting based on corresponding clauses in the construction safety standard.
The invention also provides a worker behavior detection system based on computer vision and a knowledge graph, which comprises:
the camera is arranged on a construction site and used for acquiring an image of the construction site;
the image feature extraction module is used for identifying a construction site target entity on the construction site image by adopting a pre-established and trained target detection model; performing panoramic segmentation on the construction site image by adopting a pre-established and trained panoramic segmentation model to acquire scene information of workers; tracking a motion trail of a worker in the construction site image by adopting a pre-established and trained trail tracking recognition model to acquire motion state information of the worker;
the ontology model matching module is used for carrying out secondary data mining according to the information obtained from the construction site image, obtaining image characteristic information corresponding to a pre-constructed ontology model and creating a graphic data instance;
and the knowledge reasoning module is used for judging the type of unsafe behavior, the result of unsafe behavior and giving intervention or management measures of unsafe behavior by carrying out condition matching with rules in a rule base established in advance according to the graphic data instance.
Compared with the prior art, the invention provides a construction site worker unsafe behavior detection system based on a computer vision technology and a knowledge graph. Firstly, classifying unsafe behavior entities and definitely defining relations among unsafe behavior entities, and constructing an unsafe behavior ontology model of a construction site worker on the basis of the classification; secondly, extracting characteristic information of different dimensions of unsafe behaviors of workers in a construction site from the image by using a computer vision algorithm; and finally, constructing a construction site worker unsafe behavior graphic database by taking the body model as a framework and taking the image characteristic information as filling, and developing a construction site worker unsafe behavior rule reasoning module matched with the knowledge graph, wherein the construction site worker unsafe behavior rule reasoning module and the knowledge graph jointly form a construction site worker unsafe behavior knowledge graph. By utilizing the knowledge graph, unsafe behaviors of workers in an image shot by the construction site monitoring equipment can be automatically detected, and the knowledge graph has the following advantages:
(1) According to the invention, a knowledge graph of unsafe behaviors of a construction site worker is constructed by combining a computer vision technology and an ontology, and a matched knowledge reasoning module is developed, so that a detection system of unsafe behaviors of the construction site worker based on the knowledge graph is established. The system introduces diversified computer visual information for judging unsafe behavior, is easy to expand, has the capability of processing complex and changeable unsafe behavior rules, and effectively solves the problem of automatic recognition of unsafe behaviors of workers on construction sites with complex rules.
(2) The construction site worker unsafe behavior knowledge graph constructed by the invention provides a way for integrating and structuring safety information and knowledge related to unsafe behaviors in stored images, and the requirements of backtracking, inquiring, analyzing and mining the data and the like in construction safety management are met by automatically accumulating the safety data. Secondly, the automatic reasoning function of the knowledge graph in the invention realizes the automatic identification of unsafe behaviors of workers on the construction site from the semantic level, can flexibly adapt to the requirements of different construction scenes through the definition of rules, and gives corresponding management suggestions (such as intervention measure recommendation), thereby reducing accident hidden trouble on the construction site, preventing the occurrence of accidents and providing technical support for the optimization and the improvement of the efficiency of the safety supervision method on the construction site.
Drawings
Fig. 1 is a schematic flow chart of a worker behavior detection method based on computer vision and knowledge graph provided in an embodiment of the invention;
FIG. 2 is a schematic diagram of an architecture of a construction site worker unsafe behavior ontology model according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a specific construction flow of an inferable knowledge graph according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an example of a computer vision algorithm detection result of an image sample and creation of a graphic database instance thereof according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a rule base according to an embodiment of the present invention;
fig. 6 is an integrated schematic diagram of outputting the reasoning result to the graphic database according to the embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
Example 1
As shown in fig. 1, the present embodiment provides a worker behavior detection method based on computer vision and knowledge graph, including the steps of:
s1: acquiring a construction site image, and identifying a construction site target entity according to a pre-established and trained target detection model; panoramic segmentation is carried out on the construction site image by adopting a pre-established and trained panoramic segmentation model, so as to obtain scene information of workers; tracking a motion trail of a worker in a construction site image by adopting a pre-established and trained trail tracking recognition model, and acquiring motion state information of the worker;
s2: performing secondary data mining according to information obtained from a construction site image, and obtaining image characteristic information corresponding to a pre-constructed ontology model, thereby creating a graphic data instance;
s3: according to the graphic data instance, the unsafe behavior type, the unsafe behavior result and the intervention or management measures of the unsafe behavior are judged by carrying out condition matching with rules in a rule base which is established in advance.
The method is equivalent to the steps of:
(1) Based on relevant literature, specifications, expert experience and the like, specific connotations of unsafe behaviors are defined, classification of unsafe behavior entities and relations among the entities are defined, and accordingly an ontology model of unsafe behaviors of workers on a construction site is constructed.
(2) Taking a picture shot by a monocular camera as an object, and extracting image characteristic information of unsafe behaviors of workers from multiple dimensions by utilizing various computer vision technologies including a target recognition algorithm, a panoramic segmentation algorithm, a track tracking algorithm and the like. On one hand, training a target detection model to realize the identification of various entities related to unsafe behaviors of workers on a construction site; on the other hand, the acquisition of semantic information such as attribute information, image background information and the like of the entities is realized by utilizing the existing computer vision algorithm, including a panorama segmentation algorithm under a Detectron2 framework and a deep SORT multi-target track tracking algorithm.
(3) Taking a construction site worker unsafe behavior ontology model as a framework, taking image characteristic information of the construction site worker unsafe behavior obtained based on a computer vision technology as filling, and constructing a structured graphic database in a graphic database system Neo4j to serve as a basic knowledge graph.
(4) The development knowledge reasoning module is used for automatically reasoning the unsafe behaviors of workers in the shot images of the construction site monitoring equipment based on the unsafe behavior knowledge of the workers in the construction site stored in the graphic database in the Neo4j system, so that the unsafe behavior type is detected, and behavior management knowledge (i.e. intervention and management measures) corresponding to the unsafe behavior type is obtained by reasoning.
Specifically, semantic concepts in the ontology model include:
behavior bodies, major initiators of unsafe behavior, including workers;
an object, an entity other than a behavior host existing in the construction site image;
activity, changes caused by a behavioural subject;
location, location and interface to which unsafe behavior relates;
time, time involved in unsafe behavior;
an attribute, a specific description of a property of an entity;
behavioral consequences, consequences resulting from the subject's behavior;
feedback, intervention or management measures acting on the behavioural subject based on the behavioural consequences.
The judging process of the unsafe behavior type comprises the following steps: at a specific time and place, a behavior subject with specific attributes engages in specific activities and specific interactions and associations with objects with specific attributes occur, leading to a predisposed behavior outcome and giving corresponding feedback.
The rule comprises a result item and a condition item, wherein the result item and the condition item are in the form of triples;
the process of performing condition matching according to the rule so as to judge the unsafe behavior type comprises the following steps: and matching the information in the graphic data instance according to the condition items in the rule, so as to obtain a corresponding result item, and outputting the type of unsafe behavior.
The identified construction site target entity comprises a behavior subject and an object, and interaction and association between the subject and the object are judged according to the overlapping ratio of the identified subject and the object;
according to the scene information of the identified worker, the scene information is used as attribute information of the corresponding position of the behavior main body;
according to the result of tracking the movement track of the worker, acquiring attribute information of a movement state corresponding to the behavior main body; the location and time are pre-entered information.
The acquisition process of the located position comprises the following steps: and taking a point in the detection frame of the detected target entity as a key point, and acquiring the category of the image pixel point corresponding to the key point by using the panoramic segmentation model so as to judge the position.
The acquisition process of the motion state comprises the following steps: and calculating the average pixel speed of the target entity according to the coordinate change information of the detection frame of the target entity in different image frames, so that the average pixel speed is mapped into the actual speed, and the corresponding motion state is obtained.
The embodiment also provides a worker behavior detection system based on computer vision and a knowledge graph, which comprises:
the camera is arranged on a construction site and used for acquiring an image of the construction site;
the image feature extraction module is used for identifying a construction site target entity on the construction site image by adopting a pre-established and trained target detection model; panoramic segmentation is carried out on the construction site image by adopting a pre-established and trained panoramic segmentation model, so as to obtain scene information of workers; tracking a motion trail of a worker in a construction site image by adopting a pre-established and trained trail tracking recognition model, and acquiring motion state information of the worker;
the ontology model matching module is used for carrying out secondary data mining according to information obtained from a construction site image, obtaining image characteristic information corresponding to a pre-constructed ontology model and creating a graphic data instance;
the knowledge reasoning module is used for judging the unsafe behavior type, the unsafe behavior result and the unsafe behavior intervention or management measures through the condition matching with the rules in the rule base established in advance according to the graphic data instance.
The details and advantages of the system of the present invention can be found in the above method embodiments, and are not described herein.
The implementation of the above method is described below in a specific example.
The worker behavior detection method based on computer vision and knowledge graph of the embodiment comprises the following steps:
(1) The modeling of an ontology may be broken down into two parts, entity definition and relationship definition between entities. First, in combination with related documents and specifications, unsafe behavior of a construction site worker is specifically defined as: at a particular time and place, a behavioral subject (i.e., worker) with a particular attribute engages in a particular activity (i.e., construction production job) and a particular interaction and association with an object (i.e., machine, material, environment, etc.) with a particular attribute occurs, with (possibly) a predisposing outcome (i.e., unsafe behavior), corresponding feedback (i.e., intervention and management actions) being given by the manager.
(2) From the definition of the unsafe behavior of the workers in the construction site, 8 keywords are captured as semantic concepts in the ontology model, and the semantic concepts are respectively as follows: behavioral subjects, objects, activities, places, time, attributes, behavioral consequences, and feedback. They were taken as the primary classification of the entities in the ontology model, and their connotation and secondary classification below are given in table 1.
TABLE 1 definition of semantic concepts (i.e., entities) for worker unsafe behavior at a job site and secondary classifications thereof
(3) According to the characteristics of computer vision technology, the relationship between the entities is described by adopting a graphic database language, so that the visual requirement is better met. Fig. 2 shows the architecture of a job site worker unsafe behavior ontology model, which mainly contains 4 pieces of information: pre-input information, image subject information, entity attribute information, and security knowledge.
(4) In a job site worker unsafe behavior ontology model, pre-input information provides construction scene supplemental information in the image that is difficult to directly obtain, e.g., time and location define the spatiotemporal properties of the construction scene in which the behavior subject (i.e., worker) is located. The action subject (i.e., worker) and the object (i.e., machine, material, environment, etc.) together constitute an image subject information section, whose spatial positions are directly related. Meanwhile, the behavior subject (i.e., worker) and the object (i.e., machine, material, environment, etc.) are each accompanied by corresponding attribute information for recording their behavior characteristics. The above entities and relationships between entities together constitute an activity (i.e., a construction production job). Finally, security knowledge is mined and deduced from the activity (i.e., construction production job), and feedback is acted on security management for the behavioural subject (i.e., worker).
(5) After the ontology model is built, in order to realize accurate unsafe behavior recognition of construction site workers, various computer vision technologies are needed to be utilized, including a YOLO V5s target detection algorithm, a panorama segmentation algorithm under a Detectron2 framework and a deep sort multi-target track tracking algorithm, target recognition, panorama segmentation and track tracking tasks are respectively completed to recognize target entities, and multidimensional image characteristic information of unsafe behaviors of the construction site workers is acquired to serve as attribute information of the target entities. The specific method comprises the following steps:
(6) First, performing construction site entity target identification. By means of a YOLO V5s target detection algorithm, an SODA open source data set is utilized, 100 rounds are trained to obtain detection weights, and recognition of 14 kinds of construction site target entities such as personnel, safety helmets, safety vests, scaffolds, fences, plates, bricks, wood, tower crane hooks, cutters, electric boxes, trolleys, hoppers, slogans and the like is achieved.
(7) And secondly, performing panoramic segmentation on the construction site image. And acquiring context information of scenes and the like where workers are located in the images by adopting a panoramic segmentation algorithm under a Detectron2 framework so as to construct the relation between a foreground target and a background and realize acquisition of main body attributes such as 'location'.
(8) And thirdly, tracking and identifying the track of the unsafe behavior main body (namely, the worker). And a Deep Simple Online and Realtime Tracking (deep SORT) multi-target tracking algorithm is adopted to acquire the motion trail of the 'worker' type target, so as to acquire dynamic attributes such as 'motion state', 'displacement'.
(9) And fourthly, performing secondary data mining on the discrete image characteristic information, and refining relation information between information corresponding to the ontology model, namely, the general spatial position relation and the subordinate relation of the attribute. The specific method comprises the following steps:
(10) On the other hand, according to the calculation result of the overlapping ratio (Intersection over Union, ioU) of the detection frames of the two targets, as shown in the formula (1), the spatial position relationship of the two targets is determined. Spatial positional relationships are classified into 3 categories: contain (witin), overlap, far away (away).
(11) On the other hand, a subordinate relationship of an attribute refers to a relationship in which an entity has a specific attribute, or a relationship in which an attribute belongs to a specific property entity. The following will take two typical properties of workers as examples, and their definition and acquisition modes will be described respectively.
(12) Firstly, regarding the static attribute of the 'position' of the worker, setting the midpoint of the bottom edge of the personnel detection frame as a key point, acquiring the category of the pixel point of the image corresponding to the key point by using a panoramic segmentation algorithm, and judging the position of the worker according to the pixel category.
(13) Secondly, regarding the dynamic attribute of the "motion state" of the worker, with 15 frames as a unit, it is assumed that coordinates of the 1 st frame and the 15 th frame in the point of the worker detection frame are (x 1 ,y 1 ) And (x) 15 ,y 15 ) By using the deep SORT algorithm, the coordinate information of different frames of the same object can be matched according to the id of the object detection frame, so as to obtain (x) 1 ,y 1 ) And (x) 15 ,y 15 ) The average pixel speed of the worker within these 15 frames can be calculated as in equation (2). Then, the pixel speed is mapped to the real speed with reference to the general height of the person of 1.6m, as shown in expression (3).
In equation (3), H is the human detection frame pixel height. In consideration of possible errors in the speed measurement, a threshold is set to simplify the speed attribute of the worker into a motion state attribute, and the mapping relation of the speed attribute is shown in a formula (4).
(14) In order to realize automatic reasoning of unsafe behaviors of construction site workers in the knowledge graph, a graph database of unsafe behavior knowledge and a rule reasoning module are respectively constructed, a data transmission path between the graph database and the rule reasoning module is established, and fig. 3 is a specific construction flow of the reasonable knowledge graph.
(15) And selecting the Neo4j graphic database system as a carrier of the knowledge graph, carrying out data modeling, and converting the obtained construction site entity target identification result, entity attribute and entity relation information into an instance in a database. Firstly, uniformly outputting image characteristic information of unsafe behaviors of construction site workers, which is obtained through a computer vision technology, into a structured text with suffix name of 'json', and endowing each subject or object with a non-repeated id; then, a Python program is written to realize automatic reading and input of the designated json file information, and the Neo4j system processes the information in a structured manner according to the input content to complete the creation of the graphic data instance.
(16) In the creation of graphic data instances, to facilitate representing pre-input information in the same Image, an "Image ID" node is introduced as an initial node for a single graphic data instance, which connects the behavior subject, object, and pre-input information. Meanwhile, each subject or object correspondingly establishes an 'Attribute: < entity name >' node, and Attribute information of each subject or object is attributed to the 'Attribute: < entity name >' node corresponding to the subject or object, so that subsequent security knowledge reasoning and query are facilitated. Fig. 4 illustrates an example of computer vision algorithm detection results for one image sample and its graphical database instance creation.
(17) The rule reasoning module contains 3 parts: rule base, inference mechanism, and reading and outputting of triples.
(18) Rules in the rule base are stored in text form in a specified text file. Each rule contains a result term and a condition term, with a derivable relationship between the two. The basic format of each item is the same and is a triplet shaped as a relation (X, R, Y), wherein X and Y respectively represent two objects in sequence, and R represents the relation between the two objects. Items are positively connected by "&" symbol. For example, a simple equivalent substitution rule is expressed as follows:
when the condition item of the rule is satisfied, the rule is triggered to perform knowledge reasoning, and the result obtained by the reasoning is stored in the database as a new fact in the form of the triples.
(19) From the reasoning requirement, the rule reasoning in the invention comprises two parts: the type of unsafe behavior and its possible consequences, as well as intervention or management measures recommended for unsafe behavior. Fig. 5 shows a specific composition of a rule base, containing 3 types of rules in total: the first type of rules are internal logic, and the rules are always established and play a role in simplifying the reasoning process; the second type and the third type of rules respectively correspond to the reasoning contents of the two items, and the two types of rules are obtained by coding and converting corresponding terms in the construction safety specification.
(20) The reasoning mechanism in the rule reasoning module can be briefly explained as: according to rules in the rule base, matching objects meeting the conditions in the fact database, reasoning to generate new triplet relations, and storing the new triplet relations in the complete database (namely, fact + reasoning results). And finally, outputting the results to a graphic database so as to complement the knowledge framework in the unsafe behavior patterns of constructors.
(21) By inference, each principal of action (i.e., worker) will be given the result of an independent judgment of action. The inference results are integrated into the "Result: < entity name >" nodes connected to the corresponding behavior body (i.e., worker) after being output into the graphic database, as shown in fig. 6.
The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention by one of ordinary skill in the art without undue burden. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.
Claims (10)
1. The worker behavior detection method based on computer vision and knowledge graph is characterized by comprising the following steps:
acquiring a construction site image, and identifying a construction site target entity according to a pre-established and trained target detection model; performing panoramic segmentation on the construction site image by adopting a pre-established and trained panoramic segmentation model to acquire scene information of workers; tracking a motion trail of a worker in the construction site image by adopting a pre-established and trained trail tracking recognition model to acquire motion state information of the worker;
performing secondary data mining according to the information obtained from the construction site image, and obtaining image characteristic information corresponding to a pre-constructed body model, thereby creating a graphic data instance;
according to the graphic data instance, judging the type of unsafe behavior, the result of unsafe behavior and giving intervention or management measures of unsafe behavior by performing condition matching with rules in a rule base established in advance.
2. The method for detecting the behavior of a worker based on computer vision and knowledge graph according to claim 1, wherein the semantic concepts in the ontology model include:
behavior bodies, major initiators of unsafe behavior, including workers;
an object, an entity other than a behavior host existing in the construction site image;
activity, changes caused by a behavioural subject;
location, location and interface to which unsafe behavior relates;
time, time involved in unsafe behavior;
an attribute, a specific description of a property of an entity;
behavioral consequences, consequences resulting from the subject's behavior;
feedback, intervention or management measures acting on the behavioural subject based on the behavioural consequences.
3. The method for detecting the behaviors of workers based on computer vision and knowledge graph according to claim 2, wherein the judging process of the unsafe behavior type comprises the following steps: at a specific time and place, a behavior subject with specific attributes engages in specific activities and specific interactions and associations with objects with specific attributes occur, leading to a predisposed behavior outcome and giving corresponding feedback.
4. A worker behavior detection method based on computer vision and knowledge graph according to claim 3, wherein the rule includes a result item and a condition item, both of which are in the form of triples;
the process of performing condition matching according to the rule so as to judge the unsafe behavior type comprises the following steps: and matching the information in the graphic data instance according to the condition items in the rule, so as to obtain a corresponding result item, and outputting the type of unsafe behavior.
5. The method for detecting the behaviors of workers based on computer vision and knowledge graph according to claim 2, wherein the identified target entities of the construction site comprise a behavior subject and an object, and interaction and association between the subject and the object are judged according to the overlapping ratio of the identified subject and object;
according to the scene information of the identified worker, the scene information is used as attribute information of the corresponding position of the behavior main body;
according to the result of tracking the movement track of the worker, acquiring attribute information of a movement state corresponding to the behavior main body; the location and time are pre-input information.
6. The method for detecting the behaviors of workers based on computer vision and knowledge graph according to claim 5, wherein the process of acquiring the positions comprises the following steps: and taking a point in the detection frame of the detected target entity as a key point, and acquiring the category of the image pixel point corresponding to the key point by using the panoramic segmentation model so as to judge the position.
7. The method for detecting the behavior of a worker based on computer vision and knowledge graph according to claim 5, wherein the process of acquiring the movement state comprises: and calculating the average pixel speed of the target entity according to the coordinate change information of the detection frame of the target entity in different image frames, so that the average pixel speed is mapped into the actual speed, and the corresponding motion state is obtained.
8. The method for detecting the behaviors of workers based on computer vision and knowledge graph according to claim 1, wherein the rules in the rule base comprise:
an internal logic rule for judging the unsafe behavior type;
an unsafe behavior result judging rule for judging the result of unsafe behavior;
the unsafe behavior feedback judgment rule is used for feeding back intervention or management measures of unsafe behavior.
9. The method for detecting the behaviors of workers based on computer vision and knowledge patterns according to claim 8, wherein the unsafe behavior result judging rules and unsafe behavior feedback judging rules are obtained by coding and converting based on corresponding terms in construction safety standards.
10. A computer vision and knowledge graph based worker behavior detection system, comprising:
the camera is arranged on a construction site and used for acquiring an image of the construction site;
the image feature extraction module is used for identifying a construction site target entity on the construction site image by adopting a pre-established and trained target detection model; performing panoramic segmentation on the construction site image by adopting a pre-established and trained panoramic segmentation model to acquire scene information of workers; tracking a motion trail of a worker in the construction site image by adopting a pre-established and trained trail tracking recognition model to acquire motion state information of the worker;
the ontology model matching module is used for carrying out secondary data mining according to the information obtained from the construction site image, obtaining image characteristic information corresponding to a pre-constructed ontology model and creating a graphic data instance;
and the knowledge reasoning module is used for judging the type of unsafe behavior, the result of unsafe behavior and giving intervention or management measures of unsafe behavior by carrying out condition matching with rules in a rule base established in advance according to the graphic data instance.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310708247.8A CN116665305A (en) | 2023-06-15 | 2023-06-15 | Method and system for detecting worker behaviors based on computer vision and knowledge graph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310708247.8A CN116665305A (en) | 2023-06-15 | 2023-06-15 | Method and system for detecting worker behaviors based on computer vision and knowledge graph |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116665305A true CN116665305A (en) | 2023-08-29 |
Family
ID=87716961
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310708247.8A Pending CN116665305A (en) | 2023-06-15 | 2023-06-15 | Method and system for detecting worker behaviors based on computer vision and knowledge graph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116665305A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117273129A (en) * | 2023-10-11 | 2023-12-22 | 上海歆广数据科技有限公司 | Behavior pattern creation and generation method and system |
-
2023
- 2023-06-15 CN CN202310708247.8A patent/CN116665305A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117273129A (en) * | 2023-10-11 | 2023-12-22 | 上海歆广数据科技有限公司 | Behavior pattern creation and generation method and system |
| CN117273129B (en) * | 2023-10-11 | 2024-04-05 | 上海峻思寰宇数据科技有限公司 | Behavior pattern creation and generation method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | A critical review of vision-based occupational health and safety monitoring of construction site workers | |
| Wu et al. | Combining computer vision with semantic reasoning for on-site safety management in construction | |
| Zhong et al. | Mapping computer vision research in construction: Developments, knowledge gaps and implications for research | |
| Liu et al. | Deep learning-based data analytics for safety in construction | |
| Nath et al. | Deep convolutional networks for construction object detection under different visual conditions | |
| CN117115147B (en) | Textile detection method and system based on machine vision | |
| CN110222672A (en) | The safety cap of construction site wears detection method, device, equipment and storage medium | |
| Zhang | Safety management of civil engineering construction based on artificial intelligence and machine vision technology | |
| CN113642474A (en) | Hazardous area personnel monitoring method based on YOLOV5 | |
| CN117035419B (en) | Intelligent management system and method for enterprise project implementation | |
| CN116665305A (en) | Method and system for detecting worker behaviors based on computer vision and knowledge graph | |
| WO2023104557A1 (en) | Machine-learning for safety rule violation determination | |
| CN114997279A (en) | Construction worker dangerous area intrusion detection method based on improved Yolov5 model | |
| CN116883880B (en) | Crane identification method and device based on AR technology and electronic equipment | |
| CN114037873A (en) | Transformer substation secondary circuit protection pressing plate detection system and method based on artificial intelligence | |
| CN111832475B (en) | Face false detection screening method based on semantic features | |
| Kim et al. | Image generation of hazardous situations in construction sites using text-to-image generative model for training deep neural networks | |
| CN113191320A (en) | Food safety index monitoring method based on deep learning | |
| Peng et al. | [Retracted] Helmet Wearing Recognition of Construction Workers Using Convolutional Neural Network | |
| CN116957310A (en) | Intelligent manufacturing data management system and method based on cloud computing technology | |
| CN116503809A (en) | Post-processing method for intelligent factory behavior wearing false alarm filtering | |
| CN116798117A (en) | Video understanding-based method for identifying abnormal actions under mine | |
| Kim et al. | On-site safety management using image processing and fuzzy inference | |
| Yang et al. | A real-time framework for dangerous behavior detection based on deep learning | |
| Praganingrum et al. | Image Processing Applications in Construction Projects: Challenges and Opportunities |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |