CN115171216A - Method for detecting and identifying collaboration behavior and related system - Google Patents
Method for detecting and identifying collaboration behavior and related system Download PDFInfo
- Publication number
- CN115171216A CN115171216A CN202210858482.9A CN202210858482A CN115171216A CN 115171216 A CN115171216 A CN 115171216A CN 202210858482 A CN202210858482 A CN 202210858482A CN 115171216 A CN115171216 A CN 115171216A
- Authority
- CN
- China
- Prior art keywords
- behavior
- human
- cooperative
- interaction
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Human Computer Interaction (AREA)
- Biophysics (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Psychiatry (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Social Psychology (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method for detecting and identifying cooperative behavior and a related system, wherein the method comprises the following steps: selecting a video data source which needs to be subjected to cooperative behavior detection; obtaining detection frame information of objects and people by adopting a target detection algorithm; obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm; detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula; detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment; and calculating the information entropy of the cooperative behavior according to the cooperative confidence of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors. The method can effectively detect the human-human interaction behavior and the human-equipment interaction collaboration behavior in the collaboration scene, and can push the collaboration behavior recognition analysis result to the collaboration members in real time, so that the office efficiency of the collaborative working environment is finally improved.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a method for detecting and identifying cooperative behaviors and a related system.
Background
Human Action Recognition (HAR), i.e. understanding and recognizing human behavior, is crucial in many practical applications. It can be used for visual monitoring systems to identify dangerous human activities; the system can also be used for an autonomous navigation system, understanding human behaviors and working cooperatively with human beings; HAR is also important in video search, human-computer interaction, and the like. In general, data patterns used for human behavior recognition are various and classified into two major categories, namely, visual patterns and sensor patterns according to their modalities. The data patterns encode different information sources to realize behavior recognition, and have respective advantages and applicable application scenes.
Although human behavior recognition design has been successful in some common frameworks, the following problems still exist in research and application in specific scenes:
for complex scenarios such as collaboration scenarios where specific behaviors exist: human behavior recognition in collaborative scenarios remains challenging. For example, in a meeting scene, not only human-human interaction but also human-device interaction is involved, and detection of the two types of interaction behaviors is still a quite challenging task; the research on the human behavior recognition technology of overlapping detection areas of a plurality of targets in a specific scene is not deep enough. The application of the existing human behavior recognition model is disabled, and the unreasonable reconstruction of the human posture is caused.
Most current technology-implemented group behavior identification has not been extensively studied and implemented. It is difficult for the user to comprehensively acquire and analyze such information of the complex scene, at what time and what things the collaborative scene is doing, in a short time. However, the human body posture in the collaboration scene is analyzed, and the human body posture analysis method has great significance for understanding the collaboration behavior, knowing the personnel activity in the collaboration scene and finally effectively improving the collaboration efficiency.
Therefore, how to accurately identify human behaviors in a complex collaboration scene becomes an urgent problem to be solved by practitioners of the same industry.
Disclosure of Invention
It is an object of the present invention to provide a method and related system for collaborative behavior detection and identification that at least partially addresses the above technical problems.
In order to realize the purpose, the invention adopts the technical scheme that:
in a first aspect, the present invention provides a method for detecting and identifying collaboration behavior, including the following steps: selecting a video data source which needs to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behavior of people and equipment and the interaction behavior of people and people;
obtaining detection frame information of objects and people by adopting a target detection algorithm;
obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm;
detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula;
detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment;
and calculating the information entropy of the cooperative behavior according to the cooperative confidence of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors.
Further, selecting a video data source needing to perform cooperative behavior detection, comprising:
acquiring video data in a collaborative scene in a preset mode;
preprocessing the video data to obtain key frame data;
judging whether the key frame data meet preset requirements or not;
and outputting format data meeting the preset requirement when the preset requirement is met.
Further, selecting a video data source needing to perform cooperative behavior detection, further comprising:
when the preset requirement is not met, sending an error report; and records the error-processed data in a database.
Further, the method for obtaining the detection frame information of the object and the person by adopting the target detection algorithm comprises the following steps:
and detecting the format data meeting the preset requirement by adopting a YOLOV5 or SSD target detection algorithm, and outputting video key frame detection box information containing detection personnel or equipment.
Further, obtaining corresponding bone key point data by adopting a multi-person posture estimation algorithm, wherein the method comprises the following steps:
14 key point coordinate data of the human skeleton are extracted through an OpenPose algorithm.
Further, detecting the cooperative behavior of human and human interaction according to the human skeleton key point data and a predefined human and human interaction behavior formula, wherein the detecting comprises the following steps of:
determining human body key point positions corresponding to predefined human and human interaction behaviors; human-to-human interaction behavior includes: delivery, receipt, join, and leave;
forming a human body part displacement characteristic vector by the corresponding human body key point part coordinate data and the displacement at two different moments, and estimating the motion state of the human body part;
on the basis of determining the motion state of the human body part, judging whether the change of the vector included angle of the human body part is within a preset range; judging whether the orientation of the head and the motion direction of the human body part are kept within a second preset range or not;
and when the change of the vector included angle of the human body part is within a preset range and the head orientation and the motion direction of the human body part are kept within a second preset range, determining that the cooperative behavior of human-human interaction occurs.
Further, the cooperative behavior detection of the interaction between the person and the device through a predefined calculation formula of the cooperative behavior of the person and the device comprises the following steps:
determining human body key point positions corresponding to predefined human and equipment interaction behaviors; human and device interaction behaviors include: operating a keyboard, an operating screen and an operating mouse;
judging whether the corresponding coordinate data of the key point part of the human body falls in a keyboard, a screen and a mouse detection frame corresponding to the identified equipment or not;
when falling within the detection box, it is determined that a collaborative behavior of human and device interaction has occurred.
Further, according to the collaboration confidence of the collaboration behavior of the user, calculating the information entropy of the collaboration behavior to obtain feedback results of two types of collaboration behavior recognition, including:
counting the frequency of the motion of the human body part, the frequency of the change of the included angle of the vector, the judgment frequency of the head orientation in the same direction as the motion direction of the human body part and the frequency of the times of the sitting, standing and arm stretching behaviors which occur simultaneously in two successively preset time periods, and calculating the probability of the judgment frequency as the cooperative confidence coefficient of the human-human interaction behaviors;
taking the area overlapping rate of a human body key point part boundary rectangular frame corresponding to human and equipment interaction behaviors and a keyboard, screen or mouse detection frame corresponding to equipment as the cooperative confidence coefficient of human and equipment exchange;
counting the categories of the cooperative behaviors, the corresponding confidence degrees and the cooperative liveness, and generating the information entropy of the cooperative behaviors;
and obtaining feedback results of the two types of cooperative behavior recognition according to the information entropy of the cooperative behavior.
In a second aspect, the present invention further provides a system for detecting and identifying collaboration actions, including:
the selection acquisition module is used for selecting a video data source which needs to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behavior of people and equipment and the interaction behavior of people and people;
the cooperative behavior detection module is used for obtaining detection frame information of objects and people by adopting a target detection algorithm; obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm; detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula; detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment;
the data storage module is used for storing the video frame information, the cooperative behavior identification result, the corresponding cooperative confidence coefficient and the cooperative behavior information entropy;
and the identification feedback module is used for calculating the information entropy of the cooperative behavior according to the cooperative confidence coefficient of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors.
Compared with the prior art, the invention has the following beneficial effects:
the method for detecting and identifying the cooperation behavior provided by the embodiment of the invention comprises the following steps: selecting a video data source which needs to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behaviors of people and equipment and the interaction behaviors of people and people; obtaining detection frame information of objects and people by adopting a target detection algorithm; obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm; detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula; detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment; and calculating the information entropy of the cooperative behavior according to the cooperative confidence of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors. The method can effectively detect the human-human interaction behavior and the human-equipment interaction collaboration behavior in the collaboration scene, and can push the collaboration behavior recognition analysis result to the collaboration members in real time, so that the office efficiency of the collaborative working environment is finally improved.
Drawings
FIG. 1 is a flow chart of a method for detecting and identifying collaboration-oriented behavior according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of video source selection and data preprocessing according to an embodiment of the present invention.
Fig. 3 is a schematic flow chart of target detection and human body posture estimation according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of key points of human bones according to an embodiment of the present invention.
Fig. 5 is a schematic diagram of a flow of computing and identifying human and interactive behaviors and human and device interactive behaviors according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of a computing unit of human transfer behavior according to an embodiment of the present invention.
Fig. 7 is a schematic diagram of a computing unit for a person to operate a keyboard of a notebook computer according to an embodiment of the present invention.
Fig. 8 is a schematic flow chart illustrating analysis of a recognition result of a cooperative behavior and system feedback according to an embodiment of the present invention.
Fig. 9 is a communication diagram of a system for detecting and identifying collaboration activities according to an embodiment of the present invention.
Detailed Description
In order to make the technical means, the creation characteristics, the achievement purposes and the effects of the invention easy to understand, the invention is further described with the specific embodiments.
In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front", "rear", "both ends", "one end", "the other end", and the like indicate orientations or positional relationships based on orientations or positional relationships shown in the drawings, and are only for convenience of description and simplification of description, but do not indicate or imply that the device or element referred to must have a specific orientation, be configured in a specific orientation, and operate, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present invention, it should be noted that, unless explicitly stated or limited otherwise, the terms "mounted," "disposed," "connected," and the like are to be construed broadly, such as "connected," which may be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Example 1:
the invention provides a method for detecting and identifying a cooperative behavior, which comprises the following steps:
step 1: selecting a video data source needing to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behavior of people and equipment and the interaction behavior of people and people;
and 2, step: obtaining detection frame information of objects and people by adopting a target detection algorithm;
and step 3: obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm;
and 4, step 4: detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula;
and 5: detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment;
step 6: and calculating the information entropy of the cooperative behavior according to the cooperative confidence of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors.
In the step 1, the collaboration behavior is specifically subdivided into: cooperative behavior of human-to-human interaction, and cooperative behavior of human-to-device interaction. In a collaboration scenario, a collaboration action refers to a process of performing an action or a sequence of actions of something to achieve a common goal in a human-to-human interaction or a human-to-device interaction manner. For the purpose of clearly illustrating the technical solution of the present invention, 7 kinds of collaboration actions are taken as examples to describe in detail. As shown in table 1 below, defined for key collaboration behavior categories in a collaboration scenario.
TABLE 1 Key collaboration behavior categories in collaboration scenarios
And 1-5, preprocessing video data by selecting different video sources to serve as input data of a target detection algorithm (such as a Yolov5 target detection algorithm, SSD and the like), and outputting video key frame information containing detection personnel or equipment. And then extracting key point information of human skeleton points through a human posture estimation algorithm (such as an OpenPose algorithm). The target detection algorithm and the human body posture estimation algorithm may also adopt other existing related methods, which are not limited in the embodiments of the present disclosure, and detailed descriptions of the specific detection process are not repeated.
And then, judging the cooperative behavior of each type, and giving out cooperative behavior judgment through a computing unit. For example, as shown in table 1 above, where the cooperative behavior of human-to-human interaction analyzes 4 cooperative behaviors; the cooperative behavior of human and equipment interaction analyzes 3 cooperative behavior categories, and 7 cooperative behaviors are calculated in total.
In step 6, the obtained categories of the cooperative behaviors of the human-human interaction and the human-device interaction and the corresponding cooperative confidence degrees and the information entropies of the cooperative behaviors are analyzed and used as important quantitative indexes of the cooperative participation liveness in the cooperative scene, and finally the analysis information is transmitted to the terminal user in real time, so that the cooperative efficiency of cooperative personnel in the cooperative scene is improved.
The principle of the method is shown in fig. 1, and the collaboration behavior in the collaboration scene needs to be analyzed, and specifically, the collaboration behavior can be divided into human-device interaction behavior and human-human interaction behavior. After the video data source is selected, firstly, a target detection algorithm is adopted to obtain the detection frame information of objects and people; then, obtaining skeleton key point data by adopting a multi-person posture estimation algorithm, and detecting the cooperative behavior of human-person interaction according to the preprocessed human skeleton key points and a defined human-person interaction behavior formula; the method is applied to the cooperative behavior detection of the interaction of the human and the equipment through a defined calculation formula of the cooperative behavior of the human and the equipment. And then, according to the collaboration confidence of the collaboration behavior of the user, calculating the information entropy of the collaboration behavior, and providing a feedback result of the recognition of the two types of collaboration behaviors, so that a more visual understanding is provided for the user, and the efficiency of collaborative work is improved.
The above steps are described in detail below:
in the step 1, the specific process is as shown in fig. 2:
step S201, a video in a collaboration scene can be obtained from a monocular camera, and the camera is used for shooting a collaboration behavior video of collaborators in the collaboration scene; the camera of the personal notebook can be selected for shooting, and the video can be uploaded in a local video file uploading mode.
In step S202, the key frame data after video preprocessing is output. Here, video pre-processing generally refers to: the video is cut into individual image frames, and the specific cutting can be processed by one frame per second or one frame per second.
Step S203 is to check whether the video data format meets the specification requirement of the system. The specification requirement of the system mainly refers to that the size of the image frame which is cut into the image frame is required to meet the requirement of the selected target detection module and the selected human body posture estimation module on the frame size.
Step S204, which refers to sending an error report if the data format does not meet the system requirements.
Step S205 is to record the error-processed data in the database for later inspection.
In step S206, data conforming to the system format is output.
As shown in fig. 3, in steps 2 and 3, the main task is to obtain the detection frame information of the target and the person and the coordinate information of the human skeleton point, and prepare necessary data for the calculation of the cooperative behavior of the next sub-process. The method specifically comprises the following steps:
step S301, completing target detection on video frame data;
step S302, detecting whether the current video frame contains a person, if the person exists, executing step S303, otherwise, executing step S308.
Step S303, carrying out human body posture estimation on the person of the current frame;
step S304, obtaining information such as a human skeleton point coordinate point of the current frame;
step S305, continuously detecting whether the current frame includes an object, if the current frame includes an object, the process branches to step S306, otherwise, the process branches to step S307.
Step S306, predicting to obtain detection frame information bbox of the object or the equipment.
In step S307, the bounding box information of the object or person and the coordinate information of the human skeleton point are output.
Step S308, whether the next frame data still exists is detected, if so, the process continues to step S301, otherwise, the process is finished.
Wherein, the above skeleton key points are as shown in fig. 4, and are key point information of 14 different parts of human body obtained by human body posture estimation calculation method, and each key point corresponds to a two-dimensional coordinate (x) i ，y i ) And a confidence conf i Where i represents the number of the key points of the human body, and the value is from 0 to 13.
In particular, nose (x) 0 ,y 0 ),conf 0 (ii) a Neck part (x) 1 ,y 1 ),conf 1 (ii) a Right shoulder (x) 2 ,y 2 ),conf 2 (ii) a Right elbow (x) 3 ,y 3 ),conf 3 (ii) a Right wrist (x) 4 ,y 4 ),conf 4 (ii) a Left shoulder (x) 5 ,y 5 ),conf 5 (ii) a Left elbow (x) 6 ,y 6 ),conf 6 (ii) a Left wrist (x) 7 ,y 7 ),conf 7 (ii) a Right crotch (x) 8 ,y 8 ),conf 8 (ii) a Right knee (x) 9 ,y 9 ),conf 9 (ii) a Right ankle (x) 10 ,y 10 ),conf 10 (ii) a Left crotch (x) 11 ,y 11 ),conf 11 (ii) a Left knee (x) 12 ,y 12 ),conf 12 (ii) a Left ankle (x) 13 ,y 13 ),conf 13 。
The above step 4 and step 5, as shown in fig. 5, are schematic flow charts of calculation and identification of human and device and human interaction behaviors of the present invention, and include:
step S501, inputting data information obtained by the sub-processes of target detection and human body posture estimation.
Step S502 is used to verify whether the data contains the detection box information of the panelist. If the detection frame and the skeleton point information of the team members are contained, turning to step S503; otherwise, the process ends.
Step S503, the information such as the coordinates of the skeleton points of the human body is preprocessed, and the data preparation of the human-human interactive identification is completed.
And step S504, finishing the calculation of each calculation unit of the human and human interaction behavior.
Step S505 is used to verify whether the input data contains device information. If the device further includes the detection frame information of the device, step S506 is executed; otherwise, the process is finished.
And S506, separating the information of the equipment detection box in the input data to finish the data preparation of the interactive identification of the people and the equipment.
And step S507, finishing the calculation of each calculation unit of the human-human interaction behavior.
Step S508, calculating the frequency of each atomic behavior recognition.
And step S509, outputting the identification results of various cooperation behaviors, and ending the execution of the process.
However, the above-mentioned S504 relates to the analysis of 4 computing units in total, and is a computing unit of a transfer behavior, a computing unit of a receiving behavior, a computing unit of an joining behavior, and a computing unit of a leaving behavior, respectively. Since the steps of these 4 types of computing units are similar, one of the computing units will be described in detail below:
fig. 6 is a schematic diagram of a computing unit for human transfer behavior according to the present invention. When transmitting things to other people, the atomic actions such as arm stretching and head forward tilting are generally accompanied, and the specific calculation is divided into the following 3 steps by taking the cooperative behavior and coordinate data referred to in the above table 1 and fig. 4 as an example:
(1) Specifically, the displacement of the elbow and wrist key points at two different moments forms an arm displacement feature vector { D elbow ，D wrist Estimating the motion state of the human body arm:
wherein Threshold is a Threshold. According to a priori empirical knowledge, the value is set to 0.1 in the invention.
(2) On the basis of arm movement, judging whether the arm is lifted or not, and then judgingThe angle between the vector of the arm is changed,
and
whether the included angle of the head part is enlarged or not and whether the head part orientation and the arm extending direction are kept within a certain range or not.
1) Judging whether the arm is lifted
2) Taking the left shoulder, the left elbow and the left wrist as an example, then judging the angle change of the arm vector,
and
whether the included angle is firstly increased and then decreased:
satisfies alpha 1 (t+t τ )>α 1 (t)andα 1 (t+2t τ )<α 1 (t+t τ ) (6-3)
α 1 (t) representing an arm vector included angle at the moment t; t is t τ Represents a fixed time value; alpha (alpha) ("alpha") 1 (t+t τ ) Is shown at tOn the basis of carving, the time passes t τ The included angle of the arm vector after the time; alpha (alpha) ("alpha") 1 (t+2t τ ) Indicating that 2t has passed based on time t τ The included angle of the arm vector after the time;
3) While the head orientation is maintained within a certain range with respect to the arm extension direction.
The judgment conditions corresponding to the above two steps are satisfied, and the cooperative behavior of the person and the person interaction, which is "transferred" (to other person) can be judged. Defining a time t 1 ，t 2 ＝t 1 +t δ ，t 3 ＝t 1 +2t δ (ii) a Here the statistics are in [ t ] 1 ，t 2 ]，[t 2 ，t 3 ]Within a time period, the arm lift-off frequency fre occurs simultaneously 1 Frequency fre of change of angle of arm vector 2 And the same decision frequency fre as the head orientation and the arm extension direction 3 And frequency of order fre of other atomic behaviors n For example: sitting, standing, arm stretching; calculating its probability as the cooperative confidence CC of the transitive behavior j1 (t)：
Wherein, the subscript j represents the category of the human-to-human and interactive behaviors, and j takes on the values of 1,2,3,4 according to the table 1; lower partThe reference numeral 1 denotes the number of cooperative actions of "delivery", cc j1 (t) represents the collaboration confidence as a function of time t.
Namely: and 4, further calculating the cooperation confidence of the transmission behaviors in the person-person interaction cooperation behaviors through the detected person-person interaction cooperation behaviors.
Since other collaborative behavior calculation units for human and human interaction are similar to those described above, they are only taken as an example and will not be further described.
The step S507 refers to the analysis of 3 computing units, which are respectively a computing unit for operating a keyboard (notebook computer), a computing unit for operating a screen (mobile phone or tablet), and a computing unit for operating a mouse. Since the steps of the 3 types of computing units are similar, the invention will be explained in detail for one of the computing units:
fig. 7 is a schematic diagram of a computing unit of a human-operated (notebook) keyboard. The key points involved in the calculation are the right elbow, the right wrist, the left elbow, and the left wrist. Generally, the left wrist or the right wrist is located in the frame of the notebook, and the notebook is considered to be operated. The method is specifically divided into the following 4 calculation steps:
(1) The two-dimensional coordinates of the right elbow, the right wrist, the left elbow and the left wrist obtained by the human body posture estimation, such as the OpenPose algorithm, are (x) 3 ,y 3 )、(x 4 ,y 4 )、(x 6 ,y 6 ) And (x) 7 ,y 7 ) (ii) a Calculating the coordinates (x) of the upper left corner of the rectangular bounding box where the key points of the elbow and the wrist are located a1 ，y a1 ) And lower right corner coordinates (x) a2 ，y a2 )：
(2) (b) by an object detection algorithm, such as the YOLOV5 algorithm x ,b y ,b w ,b h ) And rectangular frame information of the notebook computer: (b) x ,b y ,b w ,b h ) I.e. the predicted bounding box bbox. Wherein b is w ,b h Then the predicted values (width and height of the border), b, relative to the whole picture x ,b y Represents two-dimensional coordinates which respectively correspond to the coordinates (x) at the upper left corner of the boundary rectangular frame where the notebook keyboard is positioned in turn b1 ，y b1 ) And the coordinates of the lower right corner (x) b2 ，y b2 )。
(3) Calculating the area overlapping rate of the boundary rectangular frame where the elbow and the wrist are located and the rectangular frame of the notebook computer:
(4) And calculating the obtained IOU i Collaboration confidence cc defined as "keyboard for operation (notebook computer) i1 (t):
The subscript i represents the category of the interaction between the human and the equipment, and the value of i is 1,2 and 3 according to the table 1; the subscript numeral 1 represents the number of the cooperative action of "operating (notebook computer) keyboard", and the variable t represents the degree of confidence of cooperation cc i1 (t) is a function of time t; interaction i Representing intersection, union i A union is represented.
Namely: and step 5, further calculating the cooperation confidence of the operation (notebook computer) keyboard behavior in the cooperation behavior of the interaction between the person and the equipment through the detected cooperation behavior of the interaction between the person and the equipment.
In step 6, the information entropy cie based on the collaboration behavior is defined below, as shown in formula (8-1), a collaboration activity degree measurement index cie oriented to the collaboration behavior detection is obtained through calculation, the cie index changing along with different time and different collaboration behavior category statistical conditions are fed back to the terminal user in the collaboration scene, and the collaboration office efficiency in the collaboration scene is finally improved. The specific statistical analysis is as follows:
and counting 7 types of collaboration behavior categories and corresponding confidence degrees and collaboration activity degrees. Defining the information entropy of the interaction between people based on the cooperative behaviors based on the information entropy of the information theory:
cie obtained from equation (8-1) j (t) refers to the entropy of information that contains the collaborative behavior of all detectable human and human interactions of a user over time t. Wherein q refers to the number of users comprising human-to-human interaction behavior, j is the unique class number of human-to-human interaction behavior, c n Is a weight parameter of the human and human interaction behavior category and satisfies
cc jn (t) refers to the collaboration confidence of the collaborative behavior of the human and human interaction at the time t, and n refers to the unique sequential number of the collaborative behavior category of the human and human interaction, and the variation range is {1,2,3,4}.
Information entropy defining collaborative behavior-based human and device interactions:
cie obtained from equation (8-2) i (t) refers to the entropy of information that contains the cooperative behavior of all detectable human and device interactions of a user over time t. Where p refers to the number of users containing human-device interaction behavior, i is the unique class number of human-device interaction behavior, c m Is a weight parameter of different classes of human and equipment interaction behaviors and meets
cc im (t) refers to the collaboration confidence of the collaborative behavior of the human and equipment interaction at the time t, and m refers to the unique sequence number of the collaborative behavior category of the human and equipment interaction, and the variation range is {1,2,3}.
Information entropy based on two types of collaborative behaviors is defined:
cie(t)＝α·cie i (t)+β·cie j (t) (8-3)
from equation (8-3), the resulting cie (t) refers to the entropy of information that contains all detectable human and device interactions of the user and the cooperative behavior of human and human interactions over time t. Here, α refers to a weight of a cooperative behavior information entropy of a human and device interactive behavior, β refers to a weight of a cooperative behavior information entropy of a human and human interactive behavior, and α + β =1 is satisfied.
Fig. 8 is a schematic flow chart illustrating analysis of the recognition result of the cooperative behavior and system feedback according to the present invention. In the last sub-process, statistics of the collaboration confidence degree cc of 7 collaboration behaviors of 2 types of different collaboration behaviors and calculation of the information entropy cie of the collaboration behaviors are mainly completed, and finally, a visualization display result is given. Wherein:
step S601, inputting data such as recognition results of various types of collaboration behaviors of the previous subprocess.
And step S602, calculating the information entropy cie of the cooperative behavior according to the formulas (8-1), (8-2) and (8-3).
Step S603, transmitting the recognition result and the statistical analysis data to the database in real time.
Step S604, the user determines whether the database is stored, if yes, the process goes to step S605, otherwise, the process goes to step S606.
Step S605, the database storage of the data analysis result is completed.
Step S606 is a process of giving feedback or response by the system.
And step S607, finishing the visual display of the final result.
The method for detecting and identifying the cooperative behavior provided by the embodiment of the invention can detect 7 cooperative behaviors of people in the cooperative scene and perform feedback evaluation on the activity degree and the like of the people in the cooperative scene aiming at the problem that the research on the cooperative behavior in the current cooperative scene is not deep enough, so that the participators can more intuitively know the behavior of people and equipment in the cooperative scene, which occurs when the personnel and the equipment share the physical space, and the method helps the participators to adjust the cooperative strategy and improve the cooperative mode.
The method can effectively detect the human-human interaction behavior and the human-device interaction behavior in the collaboration scene, identify the type of the collaboration behavior, and push the collaboration behavior identification result to the collaboration members, thereby finally improving the office efficiency of the collaborative work environment.
Example 2:
the invention also provides a system for detecting and identifying the cooperative behavior, which comprises the following components:
the selection acquisition module is used for selecting a video data source which needs to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behaviors of people and equipment and the interaction behaviors of people and people;
the cooperative behavior detection module is used for obtaining detection frame information of objects and people by adopting a target detection algorithm; obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm; detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula; detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment;
the data storage module is used for storing the video frame information, the cooperative behavior identification result, the corresponding cooperative confidence coefficient and the cooperative behavior information entropy;
and the identification feedback module is used for calculating the information entropy of the cooperative behavior according to the cooperative confidence coefficient of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors.
In the selection acquisition module, for example, the video data can be acquired and verified in 3 different video input modes to determine whether the video data meets the system requirements. The method has the main function of outputting video frame data information so as to facilitate the next processing;
and in the cooperative behavior detection module, human-human interaction cooperative behavior detection and human-equipment cooperative behavior detection are mainly realized, and calculation and identification result information is transmitted to a database for storage and is fed back and processed by a response end in real time through socket communication.
In the data storage module, the main task is to store analysis information such as preprocessed video frame information, a cooperative behavior recognition result, a corresponding cooperative confidence coefficient and a cooperative behavior information entropy.
In the identification feedback module, the detection result after the cooperative behavior identification and other analysis information results are mainly pushed to the terminal user in real time, and finally, scientific reference basis is provided for personnel in the cooperative scene, so that the cooperative efficiency in the cooperative scene is improved.
Fig. 9 is a communication diagram of the system for detecting and identifying cooperative behaviors and other subprocesses according to the present invention. After system service is started, firstly, real-time video data are obtained through a camera and video frame preprocessing is completed on the real-time video data, then video frame data are delivered to a database through a socket communication mechanism, calculation and identification of a cooperation behavior are completed, and a detected cooperation behavior identification result is stored in the database and buffered through the socket communication mechanism; and finally, after receiving a confirmation instruction of a terminal user, a plurality of users of the terminal transmit detailed data such as cooperative behavior detection of the database to each terminal for rendering and then viewing by multiple users, and finally, the efficiency of cooperative work is improved. Otherwise, reference may be made to the detailed description of example 1 above.
In this embodiment, a definition of a collaboration behavior in a collaboration scenario is given. Specifically, the method is divided into cooperative behaviors of human and equipment interaction and cooperative behaviors of human and human interaction, and for each type of cooperative behaviors, the cooperative atomic actions of the cooperative behaviors are analyzed, and related human key points are given out and explained. On the basis, a system prototype is built, a preprocessed video stream is input into the system, and relevant information of a target detection frame and human key points is obtained through a popular target detection algorithm and a human posture estimation algorithm model; and then, performing related interpretable analysis on the two representative calculation processes for judging the cooperative behaviors, so that the detection facing the cooperative behaviors is realized, a scientific reference basis is finally provided for personnel in the cooperative scene, and the cooperative efficiency in the cooperative scene is improved.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (9)
1. A method for collaborative behavior detection and recognition, comprising the steps of:
selecting a video data source needing to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behavior of people and equipment and the interaction behavior of people and people;
obtaining detection frame information of objects and people by adopting a target detection algorithm;
obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm;
detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula;
detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment;
and calculating the information entropy of the cooperative behavior according to the cooperative confidence of the cooperative behavior of the user to obtain the feedback result of the identification of the two types of cooperative behaviors.
2. The method for collaborative behavior detection and identification according to claim 1, wherein selecting a video data source for collaborative behavior detection comprises:
acquiring video data in a collaborative scene in a preset mode;
preprocessing the video data to obtain key frame data;
judging whether the key frame data meet preset requirements or not;
and outputting format data meeting the preset requirement when the preset requirement is met.
3. The method for collaborative behavior detection and identification according to claim 2, wherein selecting a video data source for collaborative behavior detection further comprises:
when the preset requirement is not met, sending an error report; and records the error-processed data in a database.
4. The method for cooperative behavior detection and identification as claimed in claim 2, wherein obtaining the detection frame information of the object and the person by using a target detection algorithm comprises:
and detecting the format data meeting the preset requirement by adopting a YOLOV5 or SSD target detection algorithm, and outputting video key frame detection box information containing detection personnel or equipment.
5. The method for collaborative behavior detection and recognition according to claim 1, wherein obtaining corresponding skeletal key point data of a person using a multi-person pose estimation algorithm comprises:
14 key point coordinate data of the human skeleton are extracted through an OpenPose algorithm.
6. The method for collaborative behavior detection and identification according to claim 5, wherein detecting collaborative behavior of human-human interaction according to the human skeletal key point data and a predefined human-human interaction behavior formula comprises:
determining human body key point positions corresponding to predefined human and human interaction behaviors; human-to-human interaction behavior includes: delivery, receipt, join, and leave;
the corresponding coordinate data of the key points of the human body form displacement characteristic vectors of the human body parts by the displacement at two different moments, and the motion state of the human body parts is estimated;
on the basis of determining the motion state of the human body part, judging whether the change of the vector included angle of the human body part is within a preset range; judging whether the head orientation and the human body part movement direction are kept within a second preset range;
and when the change of the vector included angle of the human body part is within a preset range and the head orientation and the motion direction of the human body part are kept within a second preset range, determining that the cooperative behavior of human-human interaction occurs.
7. The method for collaborative behavior detection and recognition according to claim 6, wherein collaborative behavior detection for human and device interaction through a predefined computational formula of human and device collaborative behavior comprises:
determining human body key point positions corresponding to predefined human and equipment interaction behaviors; human and device interaction behaviors include: operating a keyboard, an operating screen and an operating mouse;
judging whether the corresponding coordinate data of the key point part of the human body falls in a keyboard, a screen and a mouse detection frame corresponding to the identified equipment or not;
when the user is in the detection frame, the cooperative behavior of human and equipment interaction is determined to occur.
8. The method for detecting and identifying collaborative behavior according to claim 7, wherein the feedback results of two types of collaborative behavior identification are obtained by calculating the information entropy of the collaborative behavior according to the collaboration confidence of the collaborative behavior of the user, and the method comprises:
counting the frequency of the motion of the human body part, the frequency of the change of the included angle of the vector, the judgment frequency of the head orientation in the same direction as the motion direction of the human body part and the frequency of the times of the sitting, standing and arm stretching behaviors which occur simultaneously in two successively preset time periods, and calculating the probability of the judgment frequency as the cooperative confidence coefficient of the human-human interaction behaviors;
taking the area overlapping rate of a human body key point part boundary rectangular frame corresponding to human and equipment interaction behaviors and a keyboard, screen or mouse detection frame corresponding to equipment as the cooperative confidence coefficient of human and equipment exchange;
counting the categories of the cooperative behaviors, the corresponding confidence degrees and the cooperative liveness, and generating the information entropy of the cooperative behaviors;
and obtaining feedback results of the two types of cooperative behavior recognition according to the information entropy of the cooperative behavior.
9. A system for collaborative behavior detection and recognition, comprising:
the selection acquisition module is used for selecting a video data source which needs to be subjected to cooperative behavior detection; the collaboration action includes: the interaction behavior of people and equipment and the interaction behavior of people and people;
the cooperative behavior detection module is used for obtaining detection frame information of objects and people by adopting a target detection algorithm; obtaining corresponding bone key point data of a person by adopting a multi-person posture estimation algorithm; detecting the cooperative behavior of human-human interaction according to the human skeleton key point data and a predefined human-human interaction behavior formula; detecting the cooperative behavior of interaction between people and equipment through a predefined calculation formula of the cooperative behavior of people and equipment;
the data storage module is used for storing the video frame information, the cooperative behavior identification result, the corresponding cooperative confidence coefficient and the cooperative behavior information entropy;
and the recognition feedback module is used for calculating the information entropy of the cooperative behavior according to the cooperation confidence of the cooperative behavior of the user to obtain the feedback result of the recognition of the two types of cooperative behaviors.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210858482.9A CN115171216A (en) | 2022-07-20 | 2022-07-20 | Method for detecting and identifying collaboration behavior and related system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210858482.9A CN115171216A (en) | 2022-07-20 | 2022-07-20 | Method for detecting and identifying collaboration behavior and related system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115171216A true CN115171216A (en) | 2022-10-11 |
Family
ID=83495155
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210858482.9A Pending CN115171216A (en) | 2022-07-20 | 2022-07-20 | Method for detecting and identifying collaboration behavior and related system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115171216A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115984966A (en) * | 2023-01-03 | 2023-04-18 | 西南交通大学 | Character interaction detection method based on feature refining and multiple views |
-
2022
- 2022-07-20 CN CN202210858482.9A patent/CN115171216A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115984966A (en) * | 2023-01-03 | 2023-04-18 | 西南交通大学 | Character interaction detection method based on feature refining and multiple views |
| CN115984966B (en) * | 2023-01-03 | 2023-10-13 | 西南交通大学 | Character object interaction action detection method based on feature refining and multiple views |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190392587A1 (en) | System for predicting articulated object feature location | |
| US9852327B2 (en) | Head-pose invariant recognition of facial attributes | |
| AU2020270998A1 (en) | System and method to predict, prevent, and mitigate workplace injuries | |
| WO2019090268A1 (en) | Contextual training systems and methods | |
| CN111656372A (en) | Training neural network model | |
| CN108596148B (en) | System and method for analyzing labor state of construction worker based on computer vision | |
| CN109581981B (en) | Data fusion system and working method thereof | |
| JP2021536609A (en) | Gaze point estimation method and system | |
| JP2011186576A (en) | Operation recognition device | |
| Veenendaal et al. | Dynamic probabilistic network based human action recognition | |
| Iosifidis et al. | Neural representation and learning for multi-view human action recognition | |
| CN115171216A (en) | Method for detecting and identifying collaboration behavior and related system | |
| Wang et al. | Vision sensor based action recognition for improving efficiency and quality under the environment of industry 4.0 | |
| CN104182747A (en) | Object detection and tracking method and device based on multiple stereo cameras | |
| Afroze et al. | DeepFocus: A visual focus of attention detection framework using deep learning in multi-object scenarios | |
| Büsch et al. | Towards recognition of human actions in collaborative tasks with robots: extending action recognition with tool recognition methods | |
| WO2021250808A1 (en) | Image processing device, image processing method, and program | |
| JP2021135977A (en) | Apparatus and method for processing information | |
| Liu | Video-based human motion capture and force estimation for comprehensive on-site ergonomic risk assessment | |
| WO2022079794A1 (en) | Image selection device, image selection method, and program | |
| JP2023543627A (en) | Systems and methods for counting, locating and visualizing acne | |
| Milanova et al. | Recognition of emotional states in natural human-computer interaction | |
| CN115273215A (en) | Job recognition system and job recognition method | |
| Tiwari et al. | Deep learning approach for generating 2D pose estimation from video for motion capture animation | |
| US20240177526A1 (en) | Computer-readable recording medium storing information processing program, information processing method, and information processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |