JP6884819B2 - Safety management equipment, safety management methods and safety management programs - Google Patents

Description

The present invention relates to a safety management device, a safety management method and a safety management program.

Workers working at the manufacturing site are next to each other due to occupational accidents, and there is a possibility that they may hurt their lower back due to continuous standing work for a long time or taking an unsafe working posture. Companies not only comply with the Labor Standards Law to protect the safety of workers, but also promote safety and health activities to prevent occupational accidents. The basis of occupational health activities is the reliable implementation of "safety education," "site inspection," and "safety guidance." Based on this idea, on-site leaders play a central role in conducting safety education, detecting problematic situations (working situations with unsafe potential) through on-site inspections, and linking them to improvement activities through on-site guidance.

In such efforts, there is a limit to manual site patrols. Therefore, in order to find a problem scene, the computer system memorizes and reproduces the image of the site captured by the video camera or the like. However, it takes a lot of time to humanly cut out the necessary video from the reproduced video.

In Patent Document 1, the worker wears the camera on the head and the accelerometer on the waist. The camera captures an object (scaffolding, ladder, etc.) that the worker is actually viewing. The accelerometer detects the movement of the worker. The safety management system of Patent Document 1 stores in advance a "basic behavior pattern" which is a combination of an object and a time series of movements. The basic behavior pattern is a sample of safe behavior. The safety management system of Patent Document 1 issues an alarm to a worker when the combination of an object and an action acquired from moment to moment deviates from the basic behavior pattern.

The posture estimation device of Patent Document 2 collates the captured human image with the sample image. When a certain part (upper arm, thigh, etc.) in the captured image is concealed by another part, the captured image cannot be accurately collated with the sample image. Therefore, the posture estimation device of Patent Document 2 weights each part according to the length of the parallel line component of the part of the captured image, that is, distinguishes whether to pay attention to or ignore the hidden part.

Japanese Patent No. 6429424 Japanese Patent No. 5877053

The safety management system of Patent Document 1 causes a worker to wear a camera and an accelerometer. This may impose a burden (labor) on the workers in the field. Further, the camera of the safety management system of Patent Document 1 cannot capture the whole body posture of the worker. Furthermore, Patent Document 1 does not mention providing materials for safety education for workers. The posture estimation device of Patent Document 2 aims to estimate some posture even if a part of the image is concealed, and Patent Document 2 does not mention the safety of the posture.
Therefore, an object of the present invention is to detect an unsafe whole body posture and effectively perform safety education without imposing a burden on the worker.

The safety management device of the present invention converts a posture estimation model that outputs a character string indicating the posture of the whole body image into a whole body image specified by the user when the skeleton information extracted from the whole body image of a human image captured by the camera is input. On the other hand, a learning processing unit that learns using learning data labeled with a character string indicating an unsafe posture specified by the user, and a posture estimation model that learns skeletal information extracted from a whole-body image whose posture is unknown. An estimation processing unit that specifies a whole-body image that is input and the output of the learned posture estimation model is a character string indicating an unsafe posture, and an output processing unit that displays the specified whole-body image and the time when the specified whole-body image was captured. If, Bei example, estimation processing section, based on the background of the whole body image, with human estimates the height from the floor surface located to detect the presence or absence of operation thereof located around the human, output processing The unit calculates the continuous time of the specified whole body image in the same unsafe posture, and when the calculated time exceeds a predetermined threshold, displays the specified whole body image and the time when the specified whole body image was captured. It is characterized in that a predetermined threshold value according to the height from the floor surface and the presence or absence of a moving object is applied to continuous times.
Other means will be described in the form for carrying out the invention.

According to the present invention, it is possible to detect an unsafe whole body posture and effectively perform safety education without imposing a burden on the worker.

It is a figure explaining the structure of the safety management apparatus. It is a figure explaining the operation flow of a safety management system. It is a figure explaining the learning of the posture estimation model and the estimation of a posture. It is a figure explaining the skeleton information. It is a figure explaining the learning data. It is a figure explaining the posture estimation result information. It is a figure explaining the time series chart. It is a figure explaining the threshold information. It is a figure explaining progress information. It is a flowchart of a learning process procedure. It is a flowchart of the estimation processing procedure. It is a flowchart of an output processing procedure. This is an example of the result display screen.

Hereinafter, a mode for carrying out the present invention (hereinafter, also referred to as “the present embodiment”) will be described in detail with reference to figures and the like. This embodiment is an example of estimating the posture of a worker at a manufacturing site. However, the present invention can also be applied to an example of estimating a human posture in another field where a human moves the body.

(Safety management device)
FIG. 1 is a diagram illustrating a configuration and the like of the safety management device 1. The safety management device 1 is a general computer, and includes a central control device 11, an input device 12 such as a mouse and a keyboard, an output device 13 such as a display, a main storage device 14, an auxiliary storage device 15, and a communication device 16. These are connected to each other by a bus. The auxiliary storage device 15 stores the whole body image 31, the skeleton information 32, the learning data 33, the posture estimation model 34, the posture estimation result information 35, the time series chart 36, the threshold information 37, and the estimation progress information 38 (all of which will be described in detail later). doing.

The skeleton extraction processing unit 21, the learning processing unit 22, the estimation processing unit 23, and the output processing unit 24 in the main storage device 14 are programs. The central control device 11 reads these programs from the auxiliary storage device 15 and loads them into the main storage device 14, thereby realizing the functions of the respective programs (details will be described later). The auxiliary storage device 15 may have a configuration independent of the safety management device 1.

The camera 4 is installed on the ceiling, wall surface, or the like of the manufacturing site 3. Further, the alarm device 5 is installed in a conspicuous place in the work area. The alarm device 5 may be a portable computer carried by a worker (not shown). A video recorder 7 is installed near the manufacturing site 3. The video recorder 7 stores the whole body image 31. The whole body image 31 is an image of a worker captured by the camera 4. The whole body image 31 may imprint a moving object, a background, etc. around the worker, but must include an image of the whole body of the worker. The number of workers imprinted on the whole body image 31 is basically one. However, the whole body image 31 may capture one or more other workers in the vicinity in addition to the one person. The safety management device 1 acquires a whole body image 31 from the video recorder 7.

The terminal device 6 is installed in the vicinity of the manager (user) of the manufacturing site 3. When the user is away from the safety management device 1, the user operates on the terminal device 6. That is, the terminal device 6 complements or replaces the functions of the input device 12 and the output device 13 of the safety management device 1. The safety management device 1, the video recorder 7, the alarm device 5, and the terminal device 6 are connected to each other via the network 2.

The whole body image 31 may be transmitted to the safety management device 1 at the same time as the camera 4 captures the whole body image 31 without temporarily accumulating the whole body image 31 in the video recorder 7. In this case, the camera 4 may be directly connected to the network 2 without going through the video recorder 7. The safety management device 1, the camera 4, the video recorder 7, and the terminal device 6 constitute a safety management system.

(Operation flow)
FIG. 2 is a diagram illustrating an operation flow of the safety management system. The following is an example in which the user operates the terminal device 6.

First, in the learning stage, in step S51, the camera 4 acquires a whole body image and transmits the whole body image to the video recorder 7. The whole body image here is a sample that should be training data. In step S52, the video recorder 7 stores the received whole body image. In step S53, the terminal device 6 accepts the user to instruct learning.

In step S54, the safety management device 1 executes learning. That is, the safety management device 1 extracts the skeleton information 32 from the whole body image acquired from the video recorder 7 and creates learning data. Further, the safety management device 1 learns the posture estimation model using the created learning data and stores it as the learned posture estimation model 34.

Next, in the estimation stage, in step S55, the camera 4 acquires a whole body image and transmits the whole body image to the video recorder 7. The whole body image here is acquired to detect the posture, and of course the posture is unknown. In step S56, the video recorder 7 stores the received whole body image. In step S57, the terminal device 6 accepts the user to instruct the estimation.

In step S58, the safety management device 1 performs the estimation. That is, the safety management device 1 extracts the skeleton information 32 from the whole body image acquired from the video recorder 7, inputs the skeleton information into the trained posture estimation model 34, and outputs the posture as the output of the posture estimation model 34. get. Further, the safety management device 1 stores the acquired posture as the posture estimation result information 35.

Finally, in the output stage, in step S59, the terminal device 6 accepts the user requesting the display of the result. In step S60, the safety management device 1 creates and stores the time series chart 36 and the like. In step S61, the terminal device 6 displays the time series chart 36 and the like.

(Learning of posture estimation model and estimation of posture)
FIG. 3 is a diagram illustrating learning of the posture estimation model 34 and estimation of the posture. Orientation estimation model 34, the skeleton information 32 as input _{X i,} a mathematical model of the output _{Y i} an estimated label 35. The skeleton information 32 is information indicating the human joint position extracted from the whole body image 31 captured by the camera. The estimation label 35 is a character string indicating how the safety management device 1 estimates the posture of the worker reflected in the whole body image 31, and here, “forward bending”, “sonkyo”, “ "Pointing and calling", "working at height" and "other" indicating that they do not belong to any of them. Then, "forward bending", "sonkyo", "pointing and calling", and "high-altitude work" correspond to the four postures in the whole body image 31 in that order from the top.

When the worker takes a forward bending posture, the waist is strained. When the worker takes a good posture, the knee is strained. These lead to the occurrence of occupational accidents due to deterioration of the health condition of workers or falls. Therefore, hereafter, forward bending and sonkyo may be referred to as "unsafe posture". Pointing and calling and work at height are not directly related to occupational accidents, but are attitudes that should be managed in terms of safety measures.

Various embodiments of the posture estimation model 34 are assumed. The first example is the function "Y _i = F (X _i , a _i )". X _i is an input variable, Y _i is an output variable, and “ _ai ” is a parameter (coefficient, intercept, etc.). The subscript "i" indicates that the input variable and the output variable are multidimensional, and that there are a plurality of parameters.

The second example is a neural network. As is well known, a neural network has an input layer, an intermediate layer and an output layer, and each layer has a plurality of nodes. One node propagates the amount of information in a distributed manner to each node in the next layer. The weight vector indicates how much information is propagated to which node in the next layer. The weight vector is defined for each node in the input layer and the intermediate layer.

The learning data 33 artificially added correct labels (“forward bending”, “sonkyo”, “pointing and calling”, and “working at height”) to the skeleton information 32 extracted from the whole body image 31. Teacher data. The safety management device 1 randomly generates a value of a parameter or a weight vector and substitutes it into a function or a neural network. Then, the safety management device 1 inputs the skeleton information of the learning data 33 into the function or the neural network. The function or neural network then outputs either estimated label.

The output estimated label 35 is often different from the correct label. However, the safety management device 1 happens to come across a case where the accuracy rate is improved when the value of the parameter or the weight vector is changed little by little and the substitution is repeated. The learning of the posture estimation model 34 is to specify the value of the parameter or the weight vector that maximizes the correct answer rate by the safety management device 1 executing such trial and error many times.

(Skeletal information)
FIG. 4 is a diagram for explaining the skeleton information 32. The skeleton information 32 (upper right in FIG. 4) is a tabular representation of the graphic type skeleton information 32 (lower right in FIG. 4) extracted from the whole body image 31. In the tabular skeleton information 32, the feature point is in the feature point column 102, the X coordinate value is in the X coordinate value column 103, and the Y coordinate value column 104 is associated with the identification number stored in the identification number column 101. The Y coordinate value is stored in.

The identification number in the identification number column 101 is a number that uniquely identifies the feature points of the skeleton.
The feature points in the feature point column 102 are the feature points of the skeleton, and in the present embodiment, they correspond to the joint points of the skeleton. Here, for convenience of explanation, only the feature points on the left side are described, but the feature points on the right side may exist.
The X coordinate value in the X coordinate value column 103 is the X coordinate value of the feature point. The X-axis is a horizontal axis whose origin is the upper left corner of the graphic type skeleton information 32 (the right direction is positive).
The Y coordinate value in the Y coordinate value column 104 is the Y coordinate value of the feature point. The Y-axis is a vertical axis whose origin is the upper left corner of the graphic type skeleton information 32 (the downward direction is positive).

(Learning data)
FIG. 5 is a diagram for explaining the learning data 33. In the learning data 33, the correct answer label is stored in the correct answer label column 112 and the skeleton information is stored in the skeleton information column 113 in association with the frame number stored in the frame number column 111.
The frame number in the frame number field 111 is a number that uniquely identifies an individual frame of the whole body image 31. The whole body image 31 is captured as a time-series still image (several tens of frames / second), and the time-series still image is continuously reproduced to become a moving image. Hereinafter, according to the convention in the art, each still image belonging to each frame is also referred to as a "frame".

As described above, the correct answer label in the correct answer label column 112 is a label indicating the posture given by the user to the frame having a specific frame number, and is "forward bending", "sonkyo", and "pointing and calling". And "work at height". The terminal device 6 (or the output device 13 of the safety management device 1, the same applies hereinafter) displays the correct answer label column 71, and then slides and displays each frame of the whole body image 31. Then, the terminal device 6 accepts that the user selects a certain frame and then presses any one of the buttons 71a to 71d in the correct answer label column 71.

For example, suppose that the user selects a frame 31a (frame number is “1”) and then presses the forward bending button 71a. Then, the safety management device 1 stores the frame number “1” in association with the correct label “forward bending”. In the whole body image 31 of FIG. 5, a character string such as "forward bending" is described on each frame exclusively for the sake of easy understanding of the explanation. However, the safety management device 1 assigns a correct label such as "forward bending" to each frame only after the user operates it.
The skeleton information in the skeleton information column 113 stores the X coordinate value and the Y coordinate value of FIG. 4 in a predetermined order.

(Posture estimation result information)
FIG. 6 is a diagram for explaining the posture estimation result information 35. In the posture estimation result information 35, the estimation label is in the estimation label column 122, the skeleton information is in the skeleton information column 123, and the skeleton information is in the work position height column 124 in association with the frame number stored in the frame number column 121. The height of the working position is memorized.

The frame number in the frame number field 121 is the same as the frame number in FIG. However, while the frame number in FIG. 5 specifies a frame as a sample, the frame number in FIG. 6 specifies a frame for which the safety management device 1 should estimate the posture (“estimation target frame” described later).
As described above, the estimation label in the estimation label column 122 is a label indicating the posture given to the frame of the specific frame number by the safety management device 1, and is "forward bending", "sonkyo", and "finger". It is either "labeling", "working at height" or "other".

The skeleton information in the skeleton information column 123 is the same as the skeleton information in FIG.
The height of the working position in the height column 124 of the working position is the height from the floor surface (ground) of a predetermined part of the worker's body. The method for the safety management device 1 to acquire the height of the working position is as follows, for example.

-The safety management device 1 can estimate the distance of how many meters ahead the worker by learning the distance of the skeleton information of a large number of postures every few meters (for example, 2 meters). The safety management device 1 learns every 2 meters, for example. When the estimation result is "probability of 4 meters 50%, probability of 6 meters 50%", the safety management device 1 determines that the estimated distance is 5 meters. In addition to the output estimated distance, the safety management device 1 outputs the height of the working position from the vanishing point and the center point of the camera height and the angle of view by using trigonometry.

-The safety management device 1 converts the height of the camera and the vanishing point and the center point of the angle of view, which are parameters for estimating the height of the working position, even if the height or the angle of view of the camera changes. By doing so, the height of the working position is output without newly performing distance learning.

(Time series chart)
FIG. 7 is a diagram illustrating a time series chart 36 (36a, 36b, 36c). First, pay attention to the time series chart 36a. In the time series chart 36a, the number of detections is stored in the detection number column 132 and the detection status is stored in the detection status column 133 in association with the estimated label stored in the estimation label column 131.

The estimated label in the estimated label column 131 is the same as the estimated label in FIG.
The number of detections in the detection number column 132 is the number of times that the safety management device 1 gives the estimated label to the frame of the whole body image 31 (the number of “reports” described later).
The detection status in the detection status column 133 indicates the time when the frame of the whole body image 31 to which the safety management device 1 has the estimated label is imaged is indicated by “●”. The horizontal axis of the detection information column 133 is time, and “10:00” and the like are the times when the camera 4 captures the frame of the whole body image 31 to which the estimated label is attached. Each frame of the whole body image 31 of FIG. 6 is associated with the time when the image was taken (not shown). The number of ● is equal to the number of detections.

(Detection and reporting of designated posture)
The posture specified by the user as the correct label is called "designated posture". The designated posture includes an unsafe posture. Of the designated postures, the unsafe posture is a posture that the user wants the safety management device 1 to detect by all means. However, the importance of detection depends on the continuous time of the posture, the height of the working position, the presence or absence of moving objects in the vicinity, and the like. For example, if a worker takes a "forward bending" posture for a moment on a floor surface with no moving objects in the surroundings, it is complicated for the safety management device 1 to report it to the user.

Therefore, this embodiment distinguishes between "detection" and "reporting" of the designated posture. The safety management device 1 always detects even one piece corresponding to the designated posture. However, the safety management device 1 reports to the user that the designated posture has occurred, for example, in accordance with Rule 1 and Rule 2 below.
<Rule 1> The safety management device 1 reports to the user that the designated posture has occurred once only when the time when the designated posture is continuously detected reaches a predetermined threshold value (FIG. 8).
<Rule 2> The safety management device 1 continuously detects different threshold values (FIG. 8) for each designated posture (estimated label), height of working position, and presence / absence of surrounding moving objects for the time detected. Apply.

The number of detections and the number of ● in the time series chart 36a of FIG. 7 are the number of “reports”. Although it depends on the frame advance speed, in general, the number of frames for which the safety management device 1 detects the designated posture is much larger than the number of detections and the number of ● in the time series chart 36a. In many cases, the detection is not reflected in the number of detections and the number of ● in the time series chart 36a as a result. The user who sees the time series chart 36a knows, for example, the following.

・ During the period from 10:00 to 11:10 one day, the workers took a forward bending posture 21 times in total. The number of times is somewhat concentrated around 10:20 and 11:10.
・ During this period, the workers took a total of seven postures. The number is somewhat concentrated in the second half of the period.
-During this period, the worker took a pointing and calling posture once and a high-altitude work posture once.

Now, the user inputs an operation to the terminal device 6 stating, "I want to know the number of detections for each 10-minute time width for forward bending and the number of detections for each 20-minute time width for Sonkyo." Suppose you did. Then, the safety management device 1 transitions the time series chart 36a to the time series chart 36b. The differences between the time-series chart 36b and the time-series chart 36a are as follows.

-The safety management device 1 displays eight rectangles 136 indicating a time width of 10 minutes in place of ● in the record 134. Each rectangle 136 displays the number of detections for 10 minutes.
-The safety management device 1 displays four rectangles 137 indicating a time width of 20 minutes in place of ● in the record 135. Each rectangle 137 also displays the number of detections for 20 minutes.

Now, the user wants to tell the terminal device 6 "For forward bending, I want to know the number of detections around 10:20 and 11:10 for each time width of 3 minutes and 20 seconds, and the number of detections in other time zones. Suppose you have entered an operation that says "I want to know all at once." Then, the safety management device 1 transitions the time series chart 36b to the time series chart 36c. The differences between the time-series chart 36c and the time-series chart 36b are as follows.

-The safety management device 1 displays three rectangles 138 showing a time width of 3 minutes and 20 seconds in place of the rectangle 136 near 10:20 on the record 134, and displays the rectangle 136 around 11:10. Instead, three rectangles 138 showing a time width of 3 minutes and 20 seconds are displayed. Each rectangle 138 displays the number of detections for 3 minutes and 20 seconds.
-The safety management device 1 displays a rectangle 139 indicating a time width of 20 minutes and a rectangle 140 indicating a time width of 40 minutes one by one in the record 134 in other time zones. The rectangle 139 displays the number of detections in 20 minutes. The rectangle 140 indicates the number of detections in 40 minutes.

(Threshold information)
FIG. 8 is a diagram for explaining the threshold information 37. In the threshold value information 37, the threshold value is stored in the threshold value column 142 in association with the estimated label stored in the estimation label column 141.
The estimated label in the estimated label column 141 is the same as the estimated label in FIG. However, the estimated label here always includes unsafe postures (forward bending and sonkyo). The estimation label here may include an estimation label of other specific postures.

As described above, the threshold value in the threshold value column 142 is a threshold value (unit: seconds) applied to the continuous time of the posture of the estimated label. The threshold value is set for each of the four types of cases (columns) 142a to 142d and for each estimated label. The case is a combination of the height of the working position and the presence or absence of moving objects around the worker.

For example, the safety management device 1 sets the height of the working position in FIG. 6 to "floor surface" meaning "0 ■" and "high place" meaning other than that, based on the setting by the user. It is divided into. Further, the safety management device 1 displays the whole body image 31 of the whole body image 31 based on the background reflected in the whole body image 31 to indicate that there is a moving object around the worker, and the moving object around the worker. It is divided into two categories, "None", which indicates that there is no such thing.

The four types of cases are combinations of these (2 x 2 = 4). These divisions are merely examples, and the height of the working position can be divided into a larger number. The presence or absence of surrounding moving objects can also be classified into a larger number according to the types of moving objects (machines, trolleys, other workers, etc.) and their numbers. As a result, the number of cases as a combination can be larger.

The higher the worker and the more moving objects are around, the more likely the unsafe attitude will lead to an occupational accident. Therefore, based on the case "floor surface, none", the threshold value becomes smaller when the height of the working position becomes higher, and becomes smaller when the presence / absence of surrounding moving objects becomes "present". The smaller the threshold, the more stringent the criteria for preventing the posture from continuing.

(Progress information)
FIG. 9 is a diagram illustrating progress information 38. In the progress information 38, the status is in the status column 152, the image time is in the image time column 153, the registration time is in the registration time column 154, and the start time is associated with the registration number stored in the registration number column 151. The start time is stored in the column 155, the completion time is stored in the completion time column 156, and the link information is stored in the report page column 157.

The registration number in the registration number column 151 is a number that uniquely identifies the record (row) of the progress information 38.
The status of the status column 152 is either "registered", "running", or "completed". “Registered” indicates that a time-series whole-body image whose posture is unknown is stored (registered) in the video recorder 7 or the auxiliary storage device 15 of the safety management device 1 (step S56 in FIG. 2). “Running” indicates that the process proceeds from the “registered” state, and the safety management device 1 is executing the posture estimation using the trained posture estimation model 34 (step S58 in FIG. 2). .. “Completed” indicates that the safety management device 1 has completed the estimation of the posture by further proceeding from the “running” state.

The image time in the image time column 153 is the year, month, day, hour, minute, and second of the time when the first frame of the continuous image of the registered whole body image 31 is captured and the time when the last frame is captured.
The registration time in the registration time field 154 is the year, month, day, hour, minute, and second of the time when the whole body image 31 is registered.

The start time of the start time column 155 is the year, month, day, hour, minute, and second of the time when the safety management device 1 starts executing the posture estimation using the learned posture estimation model 34.
The completion time in the completion time column 156 is the year, month, day, hour, minute, and second of the time when the safety management device 1 completes the posture estimation.
The link information in the report page column 157 is location information on the network 2 such as the auxiliary storage device 15 in which the result display information 81 (described later as an explanation of FIG. 13) to be visually recognized by the user is stored.

(Processing procedure)
Hereinafter, the processing procedure of the present embodiment will be described. The processing procedure includes a learning processing procedure, an estimation processing procedure, and an output processing procedure. These processing procedures are executed in this order.

(Learning process procedure)
FIG. 10 is a flowchart of the learning process procedure.
In step S201, the learning processing unit 22 of the safety management device 1 acquires the whole body image 31 of the sample. Specifically, first, the learning processing unit 22 acquires the first frame of the sample whole body image 31 from the video recorder 7.
Second, the learning processing unit 22 displays the first frame on the terminal device 6.
Third, the learning processing unit 22 displays the correct answer label button 71 (FIG. 5) on the terminal device 6.

In step S202, the learning processing unit 22 determines whether or not the image is a necessary whole body image. Among the frames of the whole body image 31, there are frames that are judged to be unnecessary for management (frames that are working in a safe posture, etc.). Therefore, when the user determines that the displayed frame is not necessary as learning data, the user presses the "skip" button (not shown). When the learning processing unit 22 accepts the pressing of the skip button (step S202 “No”), it proceeds to step S205, and in other cases (step S202 “Yes”), it proceeds to step S203.

In step S203, the learning processing unit 22 receives the correct answer label. Specifically, first, the learning processing unit 22 accepts the user to select the displayed frame and then press any of the correct label buttons 71a to 71 ■ (FIG. 5). For convenience of explanation, it is assumed that the frame 31a is displayed now, and the user selects the frame 31a and then presses the forward bending button 71a.
Second, the learning processing unit 22 creates a new record of the learning data 33 (FIG. 5), stores the frame number “1” of the frame 31a in the frame number column 111, and “bends forward” in the correct label column 112. "Remember. At this stage, the skeleton information column 113 remains blank.

In step S204, the skeleton extraction processing unit 21 of the safety management device 1 extracts the skeleton information 32. Specifically, first, the skeleton extraction processing unit 21 extracts the graphic type skeleton information 32 (lower right of FIG. 4) from the frame 31a.
Secondly, the skeleton extraction processing unit 21 creates the skeleton information 32 of the tabular type (upper part of FIG. 4) based on the extracted skeleton information 32 of the graphic type. The skeleton extraction processing unit 21 assigns a frame number “1” to the skeleton information 32 of the tabular type created here, and then stores the skeleton information 32 in the auxiliary storage device 15.
Thirdly, the skeleton extraction processing unit 21 stores the skeleton information (X coordinate value and Y coordinate value) in the skeleton information column 113 of the new record created in the “second” of step S203.

In step S205, the learning processing unit 22 of the safety management device 1 determines whether or not there is an unprocessed frame. Specifically, when there is an unprocessed frame (frame), the learning processing unit 22 returns to step S201 and repeats the same processing for the next frame. In other cases (step S205 “No”), the learning processing unit 22 proceeds to step S206.

When the iterative processing of steps S201 to S205 is completed, the learning data 33 as shown in FIG. 5 is completed. The correct answer label column 112 of each record of the learning data 33 stores any one of "forward bending", "sonkyo", "pointing and calling", and "high-altitude work". If 10 records whose correct label is "forward bending" are stored, the skeleton information of each of the 10 records shows a forward bending posture, but the details are different from each other. The same applies to other postures.

In step S206, the learning processing unit 22 learns the posture estimation model 34. Specifically, the learning processing unit 22 uses the learning data 33 (FIG. 5) to optimize the parameters (or weight vectors) of the posture estimation model 34 by the method described above.

In step S207, the learning processing unit 22 stores the learned posture estimation model 34. Specifically, the learning processing unit 22 stores the learned posture estimation model 34 in the auxiliary storage device 15.
After that, the learning process procedure is terminated.

(Estimation processing procedure)
FIG. 11 is a flowchart of the estimation processing procedure. As a premise for starting the estimation processing procedure, it is assumed that the threshold information 37 (FIG. 8) is stored in the auxiliary storage device 15 in the completed state.

In step S301, the estimation processing unit 23 of the safety management device 1 acquires a whole body image 31 whose posture is unknown. Specifically, the estimation processing unit 23 acquires the first frame from the entire time-series image 31 (FIG. 6) registered in the video recorder 7. The estimation processing unit 23 may directly acquire a full-body image from the camera 4 without going through the video recorder 7 (real-time estimation case). In either case, the frames acquired here are referred to as "estimated target frames" for convenience of explanation. The whole image 31 (FIG. 6) actually includes many frames in a posture that does not cause any particular problem in terms of safety, but all the frames including such frames are the estimation target frames.

In step S302, the skeleton extraction processing unit 21 of the safety management device 1 extracts the skeleton information 32. Specifically, first, the skeleton extraction processing unit 21 extracts the graphic type skeleton information 32 from the estimation target frame.
Second, the skeleton extraction processing unit 21 executes the "second" process of step S204.

In step S303, the estimation processing unit 23 of the safety management device 1 inputs the skeleton information 32 into the trained posture estimation model 34. Specifically, the estimation processing unit 23 inputs the skeleton model 32 extracted in the “first” of step S302 into the learned posture estimation model 34 stored in step S207 (FIG. 10).

In step S304, the estimation processing unit 23 acquires the estimation label. Specifically, first, the estimation processing unit 23 acquires the estimation label output by the trained posture estimation model 34. The estimated label obtained here is one of "forward bending", "sonkyo", "pointing and calling", "working at height" and "others".

Secondly, when the estimation label acquired in the "first" of step S304 is "forward bending", the estimation processing unit 23 adds "1" to the "forward bending count value". When the estimation label acquired in the “first” of step S304 is “sonkyo”, the estimation processing unit 23 adds “1” to the “sonkyo count value”. The initial value of the forward bending count value and the initial value of the sonkyo count value are "0".

The estimation processing unit 23 returns the forward bending count value to "0" when the estimated label becomes something other than "forward bending" after the forward bending count value is continuously added. The same applies to the Sonkyo count value. The estimation processing unit 23 stores the forward bending count value or the sonkyo count value in the auxiliary storage device 15 in association with the frame number of the estimation target frame.

In step S305, the estimation processing unit 23 calculates the height of the working position. Specifically, the estimation processing unit 23 calculates the height of the working position by the method described above.

In step S306, the estimation processing unit 23 determines whether or not it is necessary to issue an urgent alarm. Specifically, first, the estimation processing unit 23 checks whether or not all of the following conditions 1 and 2 are satisfied.
<Condition 1> The forward bending count value or the sonkyo count value has reached a predetermined threshold value. The estimation processing unit 23 acquires a threshold value from the threshold value information 37 (FIG. 8) using the estimation label, the height of the working position, and the presence / absence of the moving object reflected in the estimation target frame as search keys.
<Condition 2> Currently, a real-time estimation case is being executed.

Second, the estimation processing unit 23 proceeds to step S307 if all of the conditions 1 and 2 are satisfied (step S306 “Yes”), and proceeds to step S308 otherwise (step S306 “No”). ..

In step S307, the estimation processing unit 23 issues an alarm. Specifically, the estimation processing unit 23 transmits an alarm to the alarm device 5. An example of an alarm here is a voice guide that says, "Be careful not to take too much forward bending posture."

In step S308, the estimation processing unit 23 determines whether or not there is an unprocessed frame. Specifically, when the frame that is not the estimation target frame remains (step S308 “Yes”), the learning processing unit 22 returns to step S301 and repeats the same processing for the next estimation target frame. In other cases (step S308 “No”), the learning processing unit 22 proceeds to step S309.
At the stage when the iterative processing of steps S301 to S308 is completed, the frame number, the estimation label, the skeleton information, and the height of the working position are acquired for each estimation target frame and temporarily stored in the main storage device 14. become.

Further, at this stage, as a result of the "second" in step S304, for example, the following data is stored in the auxiliary storage device 15.
(Frame number, forward bending count value) = (11,0), (12,1), (13,2), (14,0), ..., (20,0), (21,1), (22,2), (23,3), (24,0), ...
This data is data for knowing the number of continuous frames of each unsafe posture, that is, the continuous time, and is also referred to as "continuous information" hereafter. The frame numbers of the continuous information here do not match the frame numbers of the whole body image 31 in FIG.

In step S309, the estimation processing unit 23 creates the posture estimation result information 35 (FIG. 6). Specifically, the estimation processing unit 23 creates the posture estimation result information 35 based on the temporarily stored combination of the frame number, the estimation label, the skeleton information, and the height of the working position. In FIG. 6, in the record in which "Other" is stored in the estimation label column 122, the postures of the estimation target frames are "forward bending", "sonkyo", "pointing and calling", and "high-altitude work". It is a record determined by the estimation processing unit 23 to be neither. The estimation processing unit 23 may delete these records.

In step S310, the estimation processing unit 23 stores the posture estimation result information 35. Specifically, the estimation processing unit 23 stores the posture estimation result information 35 created in step S309 in the auxiliary storage device 15.
After that, the estimation processing procedure ends.

(Transition of progress information record)
The estimation processing unit 23 has completed creating a “registered” record of progress information 38 (FIG. 9) for a continuous whole body image 31 (FIG. 6) before starting the estimation processing procedure. The estimation processing unit 23 updates the "registered" record for the whole body image 31 to "execution" when the estimation processing procedure is started, and when the estimation processing procedure is completed, the estimation processing unit 23 updates the whole body image 31. , Update the "running" record to "completed".

(Output processing procedure)
FIG. 12 is a flowchart of the output processing procedure.
In step S401, the output processing unit 24 of the safety management device 1 creates a time series chart 36. Specifically, the output processing unit 24 creates a time series chart 36a (FIG. 7) using the continuous information and the posture estimation result information 35 (FIG. 6) stored in step S310. At this time, the output processing unit 24 refers to the threshold value information 37 (FIG. 8) to specify a frame of the continuous image to be expressed as one ● in the time series chart 36a. That is, even if the output processing unit 24 has the same "forward bending", there are cases where a frame that is continuous for more than 10 seconds is regarded as one ● (high place, there is), and the output processing unit 24 is continuous for more than 30 seconds. In some cases, one piece is ● (floor surface, none).

For example, it is assumed that the output processing unit 24 has acquired a record having the posture estimation result information 35 (FIG. 6). It is assumed that the estimated label of the record is "forward bending", the height of the working position is "10 m", and the moving object is reflected in the background of the frame corresponding to the frame number. Then, the output processing unit 24 searches the threshold value information 37 (FIG. 8) using "forward bending", "high place", and "presence" as search keys, and acquires the threshold value "10 seconds". Then, when the output processing unit 24 determines that the frames of "forward bending", "high place", and "presence" are continuous for more than 10 seconds as a result of referring to the continuous information, one ● is generated.

In step S402, the output processing unit 24 displays the time series chart and summary information. Specifically, first, the output processing unit 24 displays the result display screen 81 (FIG. 13) on the terminal device 6. The process of step S402 may be started in the wake of an active request (browser viewing request) from the terminal device 6.
Second, the output processing unit 24 displays the time series chart 36a created in step S401 on the result display screen 81.
Third, the output processing unit 24 displays the summary information 82 on the result display screen 81. The summary information 82 is a table summarizing the number of detections of the time series chart 36a.

In step S403, the output processing unit 24 receives the image display request. Specifically, the output processing unit 24 accepts the user to select any “●” on the time series chart 36a.
In step S404, the output processing unit 24 displays a whole body image. Specifically, the output processing unit 24 displays the frame of the whole body image 31 corresponding to the ● selected in step S403 (for example, the first frame among the plurality of frames corresponding to one ●) on the result display screen 81. indicate. FIG. 13 is an example of the result display screen 81 displayed here. Although it is in the middle of the flowchart, the explanation will be temporarily moved to FIG.

FIG. 13 is an example of the result display screen 81. The frame 31 of the full-body image of FIG. 13 reflects two workers. The output processing unit 24 recognizes the posture of the worker on the right side, "Bend forward", as the subject of the report, while the posture of the worker on the left side, "Crowch", is recognized as the subject of the report. We do not recognize it as the subject of the report. This can be seen from the fact that "Safe" is displayed in association with "Crowch" and "Unsafe" is displayed in association with "Bend forward". Based on the worker on the right side, the worker on the left side corresponds to the surrounding moving objects. The floor surface is reflected in the frame 31. Therefore, the threshold value (FIG. 8) when this frame 31 is the report target is "20 seconds" corresponding to "forward bending, floor surface, and presence". The description returns to step S405.

In step S405, the output processing unit 24 displays a moving image. Specifically, first, the output processing unit 24 accepts the user to press the moving image button 83.
Secondly, the output processing unit 24 displays the continuous images of all the frames corresponding to ● selected in step S403 as moving images on the result display screen 81.
In step S406, the output processing unit 24 receives the grouping request. Specifically, the output processing unit 24 accepts the user to input one or more combinations of the estimated label and the time width in the grouping field 84. For convenience of explanation, it is assumed that the user inputs "forward bending, every 10 minutes" and "sonkyo, every 20 minutes".

In step S407, the output processing unit 24 creates a time series chart 36 after grouping. Specifically, the output processing unit 24 creates a time series chart 36b (FIG. 7) in response to the grouping request received in step S406.
In step S408, the output processing unit 24 displays the grouped time series chart 36. Specifically, the output processing unit 24 displays the time series chart 36b created in step S407 on the result display screen 81.
After that, the output processing procedure ends.

(Effect of this embodiment)
The effects of the safety management device of this embodiment are as follows.
(1) The safety management device can display the time when the worker takes an unsafe posture and the whole body image thereof.
(2) The safety management device can display a moving image of an unsafe posture.
(3) The safety management device can display the number of unsafe postures for each time width.
(4) The safety management device can display the unsafe posture when the unstable posture continues in the same posture and the continuous time exceeds the threshold value.
(5) The safety management device can apply a threshold value according to the height from the floor surface and the presence / absence of a moving object to the time when the unstable postures are continuous in the same posture.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the configurations described. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

1 Safety management device 2 Network 3 Manufacturing site 4 Camera 5 Alarm device 6 Terminal device 7 Video recorder 11 Central control device 12 Input device 13 Output device 14 Main memory device 15 Auxiliary storage device 16 Communication device 21 Skeletal extraction processing unit 22 Learning processing unit 23 Estimating processing unit 24 Output processing unit 31 Whole body image 32 Skeletal information 33 Learning data 34 Attitude estimation model 35 Attitude estimation result information 36 Time series chart 37 Threshold information 38 Estimated progress information

Claims (5)

The user specifies a posture estimation model that outputs a character string indicating the posture of the whole body image when the skeleton information extracted from the whole body image of a human image captured by the camera is input in the field, with respect to the whole body image specified by the user. A learning processing unit that learns using learning data labeled with a character string indicating an unsafe posture.
The skeleton information extracted from the whole body image whose posture is unknown is input to the learned posture estimation model, and the whole body image whose output of the learned posture estimation model is a character string indicating the unsafe posture is specified. Estimating processing unit and
An output processing unit that displays the specified whole body image and the time when the specified whole body image was captured.
Equipped with a,
The estimation processing unit
Based on the background of the whole body image, the height from the floor on which the human is located is estimated, and the presence or absence of a moving object located around the human is detected.
The output processing unit
The time during which the specified whole body image continues in the same unsafe posture is calculated, and when the calculated time exceeds a predetermined threshold value, the time when the specified whole body image and the specified whole body image are captured is displayed. ,
Applying the predetermined threshold value according to the height from the floor surface and the presence / absence of the moving object to the continuous time.
Safety management apparatus according to claim. The output processing unit
Displaying a continuous moving image of a whole body image including the specified whole body image,
The safety management device according to claim 1. The output processing unit
Accepting the user to specify the size of the time width,
Displaying the number of times that the whole body image corresponds to the unsafe posture for each time width and for each unsafe posture.
The safety management device according to claim 2. The learning processing department of the safety management device
The user specifies a posture estimation model that outputs a character string indicating the posture of the whole body image when inputting skeletal information extracted from a human whole body image captured by a camera at the site, with respect to the whole body image specified by the user. Learning using training data labeled with a string indicating an unsafe posture,
The estimation processing unit of the safety management device
The skeleton information extracted from the whole body image whose posture is unknown is input to the learned posture estimation model, and the whole body image whose output of the learned posture estimation model is a character string indicating the unsafe posture is specified. ,
The output processing unit of the safety management device is
The time when the specified whole body image and the specified whole body image were captured is displayed .
The estimation processing unit
Based on the background of the whole body image, the height from the floor on which the human is located is estimated, and the presence or absence of a moving object located around the human is detected.
The output processing unit
The time during which the specified whole body image continues in the same unsafe posture is calculated, and when the calculated time exceeds a predetermined threshold value, the time when the specified whole body image and the specified whole body image are captured is displayed. ,
Applying the predetermined threshold value according to the height from the floor surface and the presence / absence of the moving object to the continuous time.
A safety management method for a safety management device characterized by. For the learning processing section of the safety management device
The user specifies a posture estimation model that outputs a character string indicating the posture of the whole body image when the skeleton information extracted from the whole body image of a human image captured by the camera is input in the field, with respect to the whole body image specified by the user. Perform the process of learning using the training data labeled with the character string indicating the unsafe posture.
For the estimation processing unit of the safety management device
The skeleton information extracted from the whole body image whose posture is unknown is input to the learned posture estimation model, and the whole body image whose output of the learned posture estimation model is a character string indicating the unsafe posture is specified. Execute the process,
For the output processing unit of the safety management device
A process of displaying the specified whole body image and the time when the specified whole body image was captured is executed .
For the estimation processing unit
Based on the background of the whole body image, the height from the floor on which the human is located is estimated, and the process of detecting the presence or absence of a moving object located around the human is executed.
For the output processing unit
The time during which the specified whole body image continues in the same unsafe posture is calculated, and when the calculated time exceeds a predetermined threshold value, the time when the specified whole body image and the specified whole body image are captured is displayed. ,
To execute a process of applying the predetermined threshold value according to the height from the floor surface and the presence / absence of the moving object to the continuous time.
A safety management program for operating a safety management device characterized by.

Images