JP6849312B2 - Work motion recognition system - Google Patents

Description

The present invention relates to a technique for recognizing a working motion.

In the field of assembly processing, workers are required to assemble products according to the specified standard operation. Since the standard operation is defined as the operation required to maintain the quality of the product, if the operator performs an operation different from the standard operation (deviation operation), the product targeted for work at that time will be affected. ， There is a high possibility that quality problems will occur. Therefore, a function (deviation motion detection function) that automatically detects deviation motion from human motion information acquired by using various sensors is required. When the deviation motion is detected by the deviation motion detection function, the quality can be guaranteed by, for example, re-inspecting the product that is the work target at that time or discarding the product itself.

As a method for automatically analyzing human movements, the invention described in Patent Document 1 discloses a method using moving images. In the above method, first, feature data is calculated from a moving image, and the moving image is divided by finding a time-series change of the feature data, that is, a change in motion. This is a method of acquiring time-series feature data or symbol strings representing time-series feature data from the divided moving images and analyzing the operation using them. This means that complex movements are divided into simpler movements, and there is an advantage that even complicated movements can be analyzed.

Patent Document 1 has a configuration of "calculating feature data based on statistics of local motion information, dividing moving image data from time-series changes in the data, and calculating time-series feature data for each divided section". Is disclosed.

JP-A-2009-205282

In the invention described in Patent Document 1, the moving image is divided by using the change in the operation, but since the timing of the start and end of the work and the change in the operation are not always the same timing, there is a problem that these timings cannot be obtained. There is.

In order to solve the above problems, the work operation analysis system according to the present invention analyzes the device information, the sensor, the device information generator that generates device information including a plurality of time stamps that specify the time when the sensor issues a report, and the device information. A production device including a transmitter for transmitting to the device, a receiver for receiving device information, an analysis information buffer for temporarily storing analysis information acquired from the imaging device, and a plurality of devices included in the received device information. Wait until the analysis information acquired at each time indicated by the time stamp is accumulated in the analysis information buffer, and analyze the analysis information for the leaves specified using the times indicated by multiple time stamps. It is equipped with an analysis unit and an analysis device.

By acquiring the timings of work start, end, or both from the production equipment, it is possible to perform analysis only during the work period, or at least the time zone including the work.

It is a block diagram of the work motion recognition system in Example 1. FIG. It is a flowchart which shows the operation of the production apparatus in Example 1. FIG. It is a flowchart which shows the operation of the analysis apparatus in Example 1. FIG. It is a figure which showed the component of the device information. This is the structure of the data stored in the trigger generation table. It is a figure which showed the component of the trigger. This is the structure of the data stored in the analysis information buffer. This is the structure of the data stored in the camera ID table. This is the first example of the operation of the analysis information division unit. This is the second example of the operation of the analysis information division unit. This is a third example of the operation of the analysis information division unit. This is the structure of the data stored in the operation model storage unit. This is the structure of the data saved in the log storage section. This is an example of a screen for notifying the analysis result displayed on the display unit.

Hereinafter, examples will be described with reference to the drawings.

FIG. 1 is a system configuration diagram showing an overall image of the work motion recognition system. The work operation system is composed of one or more production devices 100 and analysis devices 102.

The production device 100 includes a device information transmission unit 110, a device information generation unit 111, one or more sensors 112, a control unit 113, and a production unit 114.

The analysis device 102 includes a device information receiving unit 120, a trigger generation unit 121, a trigger generation table 123, a trigger buffer 131, an analysis information division unit 132, an analysis information buffer 133, a camera ID table 134, and the like. It is composed of an operation model selection unit 135, an operation model storage unit 136, an analysis unit 137, a log storage unit 138, and a display unit 139. One or more cameras 140 are connected to the analysis information buffer 133. The personal authentication device 150 and the scheduler 151 are connected to the analysis device 102.

The operation of the production apparatus 100 will be described later with reference to FIG. 2, and the operation of the analysis apparatus 102 will be described with reference to FIG.

FIG. 2 is a flowchart showing the operation of the production apparatus 100. This flowchart will be described with reference to FIG.

S200 is a process in which the device information generation unit 111 acquires the sensor ID from the sensor 112. The sensor ID is information that can identify the sensor 112. By this process, it becomes clear which sensor issued the alarm. The sensor 112 is, for example, a contact sensor, and can acquire the timing when the worker takes out the product from the production device or the timing when the product is put in. The former means the work start timing of the worker, and the latter means the work end timing of the worker.

Further, the sensor 112 may be a start switch of the production unit 114. In that case, the control unit 113 receives the alarm of the sensor 112 as the start switch, and controls the production unit 114 to perform processing related to production, such as cutting, caulking, welding, and conveying. That is, the sensor 112 can acquire the timing at which the worker puts the product into the production device and starts the production device. This means the work end timing of the worker.

In S201, the device information generation unit 111 adds at least a production device ID capable of identifying the production device and a time stamp capable of specifying the time to the sensor ID acquired in S200, and the device information 400 shown in FIG. Is the process of generating. The time stamp represents the timing at which the device information generation unit 111 detects the alarm of the sensor 112. Alternatively, if the sensor 112 includes a time-synchronized clock, the S200 sensor ID and the timing issued by the sensor 112 may be received. In that case, in S201, it is only necessary to add the production device ID.

S202 is a process in which the device information transmission unit 110 transmits the device information 400 to the analysis device 102. FIG. 3 is a flowchart showing the operation of the analysis device 102. This flowchart will be described with reference to FIGS. 5 to 14.

S300 is a process in which the device information receiving unit 120 receives the device information 400 from the device information transmitting unit 110.
As shown in FIG. 5, the data 500 stored in the trigger generation table 123 is a production device ID, a sensor ID, a work ID, and a work start / end flag, which are associated with each other.

S301 is a process in which the trigger generation unit 121 refers to the trigger generation table 123 and reads out the work ID and the work start / end flag corresponding to the sensor ID and the production device ID included in the device information 400. As a result of this processing, the trigger 600 shown in FIG. 6 is generated, which is composed of at least those work IDs, the work start / end flags, and the time stamp included in the device information. Hereinafter, the timing indicated by the time stamp included in the trigger 600 is simply referred to as the timing indicated by the trigger 600.

The work start / end flag indicates whether the timing indicated by the trigger 600 represents the work start or end timing. The former is called the start trigger and the latter is called the end trigger. Since two types of work processes are assumed, one is a work process in which both the start trigger and the end trigger are generated (with a pair trigger), and the other is a work process in which only one of the start and end triggers is generated (without a pair trigger). The flag also contains information about with / without pair triggers.

Depending on the combination of the production device ID and the sensor ID, there may be no information in the trigger generation table 123. In this case, it means that the corresponding work does not exist, and the trigger 600 is not generated.

S310 is a process of temporarily storing the trigger 600 in the trigger buffer 131.

S311 is a process of temporarily storing the analysis information read from the camera 140 in the analysis buffer 133. The analysis information stored here may be a moving image or information representing the posture of a person extracted from the moving image. Further, the posture information of a person may be the position information of parts such as a person's hands, feet, shoulders, and head. In either case, as shown in FIG. 7, the data in the analysis information buffer 133 is stored as time-series information for each camera ID.

In FIG. 7, a simulated diagram when the position information of parts such as human hands, feet, shoulders, and head is accumulated as time-series information is shown as simulated diagram 701 of the analysis information.

S312 is a process in which the analysis information dividing unit 132 reads out the trigger 600 stored in the trigger buffer 131. If the read trigger has a pair trigger, read it after both pair triggers are available. If the read trigger does not have a pair trigger, it is read as it is.

After that, the analysis information dividing unit 132 refers to the camera ID table 134 and acquires the corresponding camera ID from the work ID included in the trigger 600.

As shown in FIG. 8, the data 800 in the camera ID table 134 is data in which the work ID and the camera ID are associated with each other.

S313 is a process in which the analysis information dividing unit 132 divides and reads the analysis information from the analysis information buffer 133 according to the acquired camera ID and the time stamp included in the trigger. Hereinafter, the three patterns of division processing will be described with reference to FIGS. 9 to 11, respectively.

The first pattern is when there is a pair trigger. In this case, the analysis information dividing unit 132 divides and reads the analysis information in the range sandwiched between the timings from the analysis information buffer 133 according to the timings indicated by the start trigger and the end trigger.

FIG. 9 describes the first pattern. That is, in the simulated analysis information 701, the first example 700 of the division target, which is the range sandwiched between the first example 701 of the timing indicated by the start trigger and the first example 702 of the timing indicated by the end trigger. Is shown to be divided.

The second pattern is when there is no pair trigger and the trigger is the start trigger. In this case, the timing indicated by the start trigger and the range between the timing indicated by the start trigger and the timing after a predetermined time are divided and read.

The second pattern will be described with reference to FIG. That is, in the simulated analysis information 701, the range is sandwiched between the second example 1001 of the timing indicated by the start trigger and the timing 1002 after a predetermined time from the second example of the timing indicated by the start trigger. The state of dividing the second example 1002 of the division target is shown. Here, the predetermined range may be a value obtained by adding a margin to the standard working time of the corresponding work process.

The third pattern is when there is no pair trigger and the trigger is the end trigger. In this case, the range between the timing indicated by the end trigger and the timing indicated by the end trigger before the predetermined time is divided and read.

FIG. 11 describes the third pattern. That is, in the simulated analysis information 701, the range is sandwiched between the second example 1101 of the timing indicated by the end trigger and the timing 1102 before a predetermined time from the second example of the timing indicated by the end trigger. The state of dividing the third example 1100 of the division target is shown. Here, the predetermined range may be a value obtained by adding a margin to the standard working time of the corresponding work process.

However, in any of the above three patterns, if the analysis information is not accumulated in time, it waits until the analysis information in the range to be read is accumulated.

S314 is a process in which the operation model selection unit 135 selects an operation model corresponding to the work ID from the operation model storage unit 136 according to the work ID included in the trigger 600. Alternatively, the operation model corresponding to the work ID of the trigger may be selected from the work ID of the trigger and the personal ID that can identify the person performing the corresponding work acquired from the personal authentication device 150 or the scheduler 151. The personal authentication device 150 acquires the personal ID of the person who is performing the work corresponding to the work ID by using the recognition device using biometric information or the authentication device held by the person concerned. The scheduler 151 has information on when each person performs work in which work process, and identifies the personal ID of the person who is performing the work by the work ID and the time stamp included in the trigger 600. be able to.

As shown in FIG. 12, the data 1200 stored in the operation model storage unit 136 is data in which the work ID, the personal ID, and the operation model expressed numerically are associated with each other. When the personal ID is not used, the operation model is saved for each work ID.
The motion model may represent the motion information of a person as the time series information of the motion vector on the image or its representative value, or may be the time series information regarding the posture of the person or its representative value. Furthermore, it may be expressed as a probability distribution related to them. When expressing as a probability distribution, use a method using a Gaussian distribution or a Gaussian mixture distribution, which is a parametric expression method, a method using a frequency distribution, which is a nonparametric expression method, or a method using a Parzen window. can do.

S315 is a process in which the analysis unit 137 performs analysis using the analysis information divided in step S313 and the operation model selected in S314. Specifically, the degree of deviation, which indicates how much the motion indicated by the divided analysis information deviates from the selected motion model, is calculated and output as the analysis result. As a method of calculating the deviance, the dynamic programming method can be used if the selected motion model is represented by time series information, and the Euclidean distance can be used if it is represented by a representative value. Also, when the motion model is expressed as a probability distribution, the Mahalanobis distance can be used if the probability distribution is a Gaussian distribution, and if it is expressed as another probability distribution, divided analysis information can be generated. The degree of deviation can be calculated by calculating the probability. Even when the motion model is represented by a hidden Markov model, the probability that the divided analysis information can occur can be calculated, and the deviance can be calculated.

The analysis result may be a plurality of values obtained by calculation using a plurality of methods. For example, if the deviance is calculated by selectively using different parts of the human body such as the whole body, hands, feet, upper body, and lower body of the analysis information, the analysis results corresponding to each part can be obtained.

S316 is a process of accumulating the analysis result obtained in this manner in the log accumulating unit 138. As shown in FIG. 13, the data 1300 stored in the log storage unit is data in which a time stamp, a work ID, an individual ID, and an analysis result are associated with each other. Here, the time stamp is saved corresponding to the trigger 600 used in step S313. If the work start trigger is used, the time stamp (work start time stamp) is saved, and if the work end trigger is used, the time stamp (work end time stamp) is saved. If you are using both, save both.

S317 is a process in which the display unit 139 displays the time stamp, the work ID, the personal ID, and the analysis result on the production apparatus 100. Here, the time stamp is saved corresponding to the trigger 600 used in step S313. If the work start trigger is used, the time stamp (work start time stamp) is displayed, and if the work end trigger is used, the time stamp (work end time stamp) is displayed. If both are used, both are displayed.

FIG. 14 is a diagram showing an example 1400 of a screen displayed on the display screen 139. If any of the analysis results exceeds a predetermined threshold value, a message that a deviation motion has occurred is displayed as shown in FIG. 13, and the analysis result uses information for analysis of a specific part of the human body. If it is calculated by the above, the display can be made easier to understand by displaying the parts at the same time as shown in FIG. After that, the worker or supervisor who sees the display screen 139 can take measures such as discarding the product when the corresponding deviant action occurs or giving appropriate guidance to the worker who performed the corresponding deviant action. Become.

100: Production equipment 102: Analysis equipment 110: Equipment information transmission unit 111: Equipment information generation unit 112: Sensor 113: Control unit 114: Production unit 120: Equipment information reception unit 121: Trigger generation unit 123: Trigger generation table 131: Trigger Buffer 132: Analysis information division unit 133: Analysis information buffer 134: Camera ID table 135: Operation model selection unit 136: Operation model storage unit 137: Analysis unit 138: Log storage unit 139: Display unit 140: Camera 150: Individual Authentication device 151: Scheduler 400: Device information 500: Data stored in the trigger generation table 600: Trigger 700: Data in the analysis information buffer 701: Simulated diagram of analysis information 800: Data in the camera ID table 900: First example of division target 901: First example of timing indicated by start trigger 902: First example of timing indicated by end trigger 1000: Second example of division target 1001: Second example of timing indicated by start trigger 1002: Timing after a predetermined time from the second example of the timing indicated by the start trigger 1100: Third example of the division target 1101: Second example of the timing indicated by the end trigger 1102: Timing indicated by the end trigger Timing before a predetermined time from the second example 1200: Data stored in the operation model storage unit 1300: Data stored in the log storage unit 1400: Example of a screen to be displayed on the display device

Claims (4)

It is a work operation analysis system equipped with a production device and an analysis device.
The production device includes a sensor, a device information generation unit that generates device information including a plurality of time stamps that specify a time when the sensor issues a report, and a transmission unit that transmits the device information to the analysis device. Prepare,
The analysis device includes a receiving unit that receives the device information, an analysis information buffer that temporarily stores analysis information acquired from the imaging device, and an analysis unit.
The analysis device holds a motion model in which time-series information on a person's posture or its representative value is expressed as a probability distribution, in association with a work and an individual.
The analysis unit waits until the analysis information acquired at each of the times indicated by the plurality of time stamps included in the received device information is accumulated in the analysis information buffer, and sets the times indicated by the plurality of time stamps. With respect to the range specified by using, the work in the range and the motion model associated with the individual performing the work and the different parts of the human body from the analysis information are selectively used to deviate from the range. Calculate the degree ,
The plurality of time stamps include a first time stamp indicating the work start time point of the worker and a second time stamp indicating the work end time point of the worker.
A work motion analysis system, characterized in that the range is a range specified by the first time stamp and the second time stamp. In the work operation analysis system according to claim 1,
The sensor includes a contact sensor and a start switch.
The first time stamp is a time stamp that specifies the time when the contact sensor issues a report, and the second time stamp is a time stamp that specifies the time when the activation switch issues a report. A featured work motion analysis system. A procedure for generating device information including multiple time stamps that can identify the time when the sensor issued a report, and
The procedure for temporarily storing the analysis information acquired from the imaging device in the analysis information buffer, and
A procedure for holding a motion model in which time-series information on a person's posture or its representative value is expressed as a probability distribution in association with a work and an individual, and
A procedure of waiting until the analysis information acquired at each of the times indicated by the plurality of time stamps is accumulated in the analysis information buffer, and a procedure of waiting.
With respect to the range specified by using the times indicated by the plurality of time stamps, the operation model associated with the work in the range and the individual performing the work, and the different human body in the analysis information. possess a step of calculating a deviation degree by using a site-selectively, and
The plurality of time stamps include a first time stamp indicating the work start time point of the worker and a second time stamp indicating the work end time point of the worker.
A work motion analysis method, characterized in that the range is a range specified by the first time stamp and the second time stamp. A work motion analysis program that causes a computer to execute the work motion analysis method according to claim 3.

Images