WO2023058164A1

WO2023058164A1 - Behavior order abnormality detection device, method, and program

Info

Publication number: WO2023058164A1
Application number: PCT/JP2021/037017
Authority: WO
Inventors: 幸佑守脇
Original assignee: 日本電気株式会社
Priority date: 2021-10-06
Filing date: 2021-10-06
Publication date: 2023-04-13
Also published as: JPWO2023058164A1

Abstract

A video input means 81 is for inputting a work video in which a work process by a worker has been captured. A behavior identification means 82 identifies behaviors of the worker from the inputted work video. A behavior order identification means 83 traces the identified behaviors of the worker in a time series and, in a case where a change in behavior of the worker has been detected, adds a behavior before the change to a behavior order list. A behavior order comparison means 84 makes a comparison between the behavior order list and an ideal work procedure that defines the order of behaviors in accordance with the work, so as to detect abnormal work.

Description

Action order abnormality detection device, method and program

The present invention relates to an action sequence abnormality detection device, an action sequence abnormality detection method, and an action sequence abnormality detection program for detecting an abnormality in the order of actions by a worker.

In the manufacturing industry, improving productivity in factories and warehouses where products are assembled, sorted, and stored is an issue. As an effort to improve productivity, work procedures are confirmed based on decisions made by supervisors in real time or by visual confirmation of recorded work scenes by workers. However, judgment based on the supervisor's visual confirmation requires time and labor costs, and there is also the possibility of oversights. Therefore, there is a demand to automatically identify whether or not the work procedure by the worker is performed in the correct order.

As a technology related to such a demand, Patent Document 1 describes a work management device that outputs the status of work performed by a worker based on a video of the worker. The device described in Patent Literature 1 acquires a first image showing the worker at the start of work and a second image showing the worker at the end of work. Then, the device performs image analysis of the first image and the second image to measure the work required time, and recognizes the work item, thereby displaying information about the implementation status of each work item.

In addition, Non-Patent Document 1 describes MS-TCN (Multi-Stage Temporal Convolutional Network), which is an action segmentation technique, as a technique for identifying work content from video. For example, to classify action segments in long untrimmed videos, MS-TCN described in Non-Patent Document 1 generates frame-wise probabilities and feeds them into a high-level temporal model to determine the video frame classify directly.

JP 2015-225630 A

As described above, the device described in Patent Literature 1 measures the start time and end time of the work of the target worker using the video captured by the installed camera, and determines the work content. Identify. However, the device described in Patent Document 1 can automatically identify the work content and the required time of the worker from the video, and also has the function of determining whether the work procedure was performed in the correct order. do not have. Therefore, there is a problem that the quality of the work performed cannot be grasped.

In addition, the method described in Non-Patent Document 1 compares the result output by the machine learning model for each frame with the correct action content for that frame, and outputs the accuracy. Therefore, there is a problem that, for example, if the present technology is simply applied to images taken in a factory, it is not possible to evaluate the correctness of the above-described worker's work content.

Therefore, an object of the present invention is to apply an action sequence abnormality detection device, an action sequence abnormality detection method, and an action sequence abnormality detection program that can detect an abnormality in the action sequence of a worker.

An action sequence abnormality detection device according to the present invention includes: video input means for inputting a work video of a work process by a worker; and action identification means for identifying the action of the worker from the input work video. An action order identifying means for tracing the actions of a worker in chronological order and, when detecting a change in the action of the worker, adding the action before the change to an action order list, and prescribing the order of actions according to the work and action order comparison means for detecting an abnormal work by comparing the ideal work procedure and the action order list.

The action sequence abnormality detection method according to the present invention inputs a work video in which a worker's work process is shot, identifies the worker's actions from the input work video, and chronologically displays the identified worker's actions. If a change in the behavior of the worker is detected, the behavior before the change is added to the action order list, and the ideal work procedure that defines the order of actions according to the work is compared with the action order list. , to detect abnormal work.

The action sequence abnormality detection program according to the present invention includes video input processing for inputting work video of the work process by the worker into the computer, action identification processing for identifying the worker's action from the input work video, and identification. When a change in the behavior of the worker is detected, the action order identification process for adding the action before the change to the action order list, and the order of the action according to the work. is compared with an action order list defining an action order list to execute an action order comparison process for detecting an abnormal work.

According to the present invention, it is possible to detect anomalies in the action order of workers.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows the structural example of the action sequence abnormality detection system of 1st embodiment by this invention. It is a flowchart which shows the operation example of the action sequence abnormality detection system of 1st embodiment. FIG. 5 is an explanatory diagram showing an example of processing for generating an action order list from a work video; It is a block diagram which shows the structural example of the action sequence abnormality detection system of 2nd embodiment by this invention. It is a flowchart which shows the operation example of the action order abnormality detection system of 2nd embodiment. BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows the outline|summary of the action sequence abnormality detection apparatus by this invention. 1 is a schematic block diagram showing a configuration of a computer according to at least one embodiment; FIG.

Embodiments of the present invention will be described in detail below with reference to the drawings. In the following description, a series of actions by a worker is referred to as work, and individual actions included in the work are referred to as actions.

[First embodiment]
[Description of configuration]
FIG. 1 is a block diagram showing a configuration example of an action sequence abnormality detection system according to a first embodiment of the present invention. As shown in FIG. 1, an action sequence abnormality detection system 100 of the first embodiment of the present invention includes a video input means 1, an action sequence abnormality detection device 21, an ideal work procedure storage device 22, and an output device 3. and The action sequence abnormality detection device 21 and the ideal work procedure storage device 22 are connected so as to be able to communicate with each other.

The action order abnormality detection device 21 includes action identification means 211, action order identification means 212, and action order comparison means 213, as shown in FIG. The action order identifying means 212 is connected to the action identifying means 211 and the action order comparing means 213 respectively. Note that the unidirectional arrows shown in FIG. 1 simply indicate the direction of information flow and do not exclude bidirectionality.

The video input means 1 inputs a video of a worker's work process (hereinafter referred to as a work video). The content of the working video is arbitrary. In addition, in order to be able to detect an abnormality in a series of actions by the worker, it is preferable that the work video is a video in which the entire work is shot in chronological order. Note that the video input means 1 may be included in the action sequence abnormality detection device 21 .

The behavior identification means 211 identifies the behavior of the worker from the input work video. Specifically, the behavior identifying means 211 identifies the behavior of the worker in units of frames from the work video. Any method may be used by the action identifying means 211 to identify the action from the work video. For example, the method described in Patent Literature 1 may be used.

The action sequence identification means 212 receives the identification result from the action identification means 211, traces the identified worker's actions in chronological order, and detects a change in the worker's action. Add to order list. Specifically, the action sequence identifying means 212 follows the actions of the identified worker in chronological order from the first frame of the work video, and detects a change in action before and after the frame. to the action order list.

It should be noted that noise and erroneous identification results may be included when performing frame-by-frame identification. Therefore, the action order identification unit 212 may detect a change in action when the identification result of the worker's action continues with the same content for a predetermined period (for the number of frames). In this way, the identification result by the action identification unit 211 is data indicating the identification result for each frame, while the action order list can be said to be data obtained by compressing the identification result for each action.

By using the action order list generated in this way, it becomes possible for the action order comparison means 213, which will be described later, to easily perform comparison with the ideal work procedure.

The ideal work procedure storage device 22 is a device that records the ideal action sequence (hereinafter referred to as ideal work procedure) assumed in the work performed by the worker. The ideal work procedure is information that specifically defines the order of actions according to the type of work, and in the target assembly work and transportation work, the work is in the correct order as expected by the supervisor. It is an example of work when extra work is done without intervention. The ideal work procedure may be represented as frame-by-frame information, or may be represented in the same format as an action order list, which will be described later. The ideal work procedure storage device 22 stores ideal work procedures for each type of work. The ideal work procedure storage device 22 is implemented by, for example, a magnetic disk.

The action order comparison means 213 receives the action order list from the action order identification means 212, compares it with the ideal work procedures stored in the ideal work procedure storage device 22, and detects abnormal work. Furthermore, the action order comparison means 213 may output the result of detecting abnormal work to the output device 3 based on the edit distance between the action order list and the ideal work procedure. In addition, in this embodiment, the type of work is assumed to be known.

The evaluation index called edit distance will be explained below. In the evaluation using the edit distance, for each element (here, action) included in the identification result for the identification result of a series of actions, how many times the replacement, deletion, and insertion of the element content should be used to determine the correct action content A match is measured.

Therefore, the action order comparison means 213 outputs the result of detecting abnormal work by calculating the edit distance from the series of actions included in the action order list to the series of actions included in the ideal work procedure. may

Specifically, the action order comparison means 213 may output an edit distance indicating the number of times the action is replaced as the number of times the action is replaced. Also, the action order comparison unit 213 may output an edit distance indicating the number of times an action is added as the number of times an action is omitted. Further, the action order comparison unit 213 may output an edit distance indicating the number of actions deleted as the number of extra actions added.

Action identification means 211, action order identification means 212, and action order comparison means 213 are computer processors (e.g., CPU (Central Processing Unit), GPU (Graphics Processing Unit) that operate according to a program (action order abnormality detection program). )).

For example, the program is stored in a storage unit (not shown) provided in the action sequence abnormality detection device 21, the processor reads the program, and according to the program, the action identification means 211, the action order identification means 212, and the action order It may operate as the comparison means 213 . Also, the function of the action sequence abnormality detection device 21 may be provided in a SaaS (Software as a Service) format.

Also, the action identification means 211, the action order identification means 212, and the action order comparison means 213 may each be realized by dedicated hardware. Also, part or all of each component of each device may be implemented by general-purpose or dedicated circuitry, processors, etc., or combinations thereof. These may be composed of a single chip, or may be composed of multiple chips connected via a bus. A part or all of each component of each device may be implemented by a combination of the above-described circuits and the like and programs.

Further, when some or all of the constituent elements of the action sequence abnormality detection device 21 are implemented by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged. and may be distributed. For example, the information processing device, circuits, and the like may be implemented as a form in which each is connected via a communication network, such as a client-server system, a cloud computing system, or the like.

[Explanation of operation]
Next, the operation of the action sequence abnormality detection system 100 of this embodiment will be described. FIG. 2 is a flow chart showing an operation example of the action sequence abnormality detection system 100 of the first embodiment.

First, the video input means 1 inputs a work video (step S1). Next, the behavior identifying means 211 identifies the behavior of the worker from the input work video (step S2). Specifically, the action identifying means 211 identifies actions for each frame of the input work video.
Next, the action order identifying means 212 traces the identified actions of the worker in chronological order, and when detecting a change in the action of the worker, adds the action before the change to the action order list (step S3). . In other words, the action order identifying means 212 deletes the frame information from the action contents obtained for each frame and converts them into an action order list indicating which action was performed in which order.

FIG. 3 is an explanatory diagram showing an example of processing for generating an action order list from work videos. A specific example of the action order list generation method (conversion method) will be described below. First, the action sequence identifying means 212 compares the work content (here, action A) in the first frame of the work video with the action in the next frame. If the actions shown are the same, the action order identifying means 212 performs a process of comparing that action with the action shown by the next frame. Thereafter, this process is repeated until a frame showing a different action (here, action B) is reached.

When reaching a frame showing a different action, the action order identification means 212 adds the action before the change to the action order list. Similar processing is repeated for subsequent frames until a frame showing a different action is reached, and each time a frame showing a different action is reached, the action up to that point is added to the action order list. In the process of processing, when it is determined that the action indicated by the frame cannot be identified, the action identifying means 211 may ignore the frame and continue the process.

For example, the example shown in FIG. 3 indicates that the action order list L4 is generated from the identification result L2. Note that an action order list L3 representing an ideal work procedure may be generated in a similar manner from the identification result L1 of the image showing the ideal work.

Next, in FIG. 2, the action order comparison means 213 compares the ideal work procedure and the action order list to detect abnormal work (step S4). Specifically, the action order comparison means 213 compares the action order list generated in step S3 with the ideal work procedure stored in the ideal work procedure storage device 22 by calculating the edit distance. , to detect differences such as interchanges of procedures.

A specific example of the comparison process will be described below with reference to FIG. The edit distance is used to compare two action order lists, the action order list L4 generated from the working video and the ideal action order list L3. The edit distance is calculated by calculating how many times "element replacement", "element deletion", and "element insertion" are performed on the contents of the action order list L4 to obtain the same list as the action order list L3. obtained by

For example, the action order list L4 generated from the work video is [1: action A, 2: action B, 3: action C, 4: action B, 5: action D], and the ideal action order list L3 is Assume that [1: action A, 2: action B, 3: action D, 4: action C]. At this time, by deleting the fourth element, replacing the third element, and replacing the fifth element with respect to the action order list L4, an action order list that matches the action order list L3 is obtained. .

In this case, the edit distance is 3 because the two action order lists match after 3 operations. The action order comparison means 213 regards the number of times of "deleting an element" as the number of times of "adding an extra action", and regards the number of times of "inserting an element" as the number of times of "missing an action". ”, and the number of “replacement of elements” (every two cases) is regarded as the number of “swap of actions” (one case). Then, the action order comparison means 213 outputs the result of detecting the abnormal work, assuming that an abnormality of the considered content is detected.

For example, in the example shown in FIG. 3, deletion is performed once and replacement is performed twice. Therefore, the action order comparison unit 213 outputs a result that one case of "addition of extra action" and one case of "replacement of action" have occurred.

After that, in FIG. 2, the action order comparison means 213 outputs the result of detecting the abnormal work to the output device 3 (step S5).

As described above, in this embodiment, the video input means 1 inputs the work video, and the action identification means 211 identifies the action of the worker from the input work video. Then, when the action order identifying means 212 traces the actions of the identified worker in chronological order and detects a change in the action of the worker, the action before the change is added to the action order list, and the action order comparing means 213 compares the ideal work procedure with the action sequence list to detect abnormal work. Therefore, it is possible to detect an abnormality in the order of actions by the worker.

That is, by using the action sequence abnormality detection system 100 of the present embodiment, it is possible to identify whether the action of the worker in the work video is performed in the assumed correct procedure. The reason is that the action order identification means 212 generates an action order list indicating which action was performed in what order from the action identification results for each frame of the input work video.

For example, in the example shown in FIG. 3, the action order identification means 212 converts the identification result L2 of the work video identified by the action identification means 211 into the action order list L4. Similarly, the action order identification means 212 converts the identification result L1 into an action order list L3. As a result, the action sequence abnormality detection system 100 can detect the replacement of actions and the omission of actions by comparing the two action order lists L3 and L4 based on the edit distance.

[Second embodiment]
[Description of configuration]
Next, a second embodiment of the action order abnormality detection system according to the present invention will be described. FIG. 4 is a block diagram showing a configuration example of an action sequence abnormality detection system according to a second embodiment of the present invention. As shown in FIG. 4, an action sequence abnormality detection system 200 of the second embodiment of the present invention includes a video input means 1, an action sequence abnormality detection device 21, an ideal work procedure storage device 22, and a work type search. It comprises means 23 and an output device 3 .

In other words, the behavior sequence abnormality detection system 200 of the second embodiment differs from the behavior sequence abnormality detection system 100 of the first embodiment in that it further includes work type search means 23 . Other configurations are the same as those of the first embodiment. Note that the work type search means 23 may be included in the action sequence abnormality detection device 21 .

The work type search means 23 identifies the type of work performed by the worker based on the behavior of the worker identified from the work video. Then, the work type search means 23 searches and acquires the ideal work procedure corresponding to the identified work type from the ideal work procedure storage device 22 .

Any method may be used by the work type search means 23 to identify the type of work based on the behavior of the worker. The work type search means 23 may prepare, for example, a master of typical work types according to the behavior of the worker, and specify the work type according to the degree of matching with the master. Further, the work type search means 23 may identify the type of work using a model generated by learning the type of work based on the behavior of the worker.

In this embodiment, the behavior sequence abnormality detection system 200 includes the work type search means 23, so that behavior sequence abnormality detection can be performed with the same device for multiple types of work.

The operation of the work type search means 23 will be described below using a specific example. For example, it is assumed that there are three types of work: "assembly of products", "packaging of products", and "defect inspection of products". The first embodiment assumes that the type of work is known. In other words, it has been clarified in advance which type of work the work image is of the three types, and the ideal work procedure corresponding to the type of work has been acquired.

On the other hand, in the second embodiment, from the three types of ideal work procedures stored in the ideal work procedure storage device 22, the work type search means 23 acquires an appropriate ideal work procedure according to the type of work specified. do. As a result, it is possible to perform anomaly detection for a plurality of types of work using the same device.

The action identification means 211, the action order identification means 212, the action order comparison means 213, and the work type search means 23 are implemented by a computer processor that operates according to a program (action order abnormality detection program).

[Explanation of operation]
Next, the operation of the action sequence abnormality detection system 200 of this embodiment will be described. FIG. 5 is a flow chart showing an operation example of the action sequence abnormality detection system 200 of the second embodiment. The process from inputting the work video to identifying the action is the same as steps S1 to S2 illustrated in FIG.

Next, the work type search means 23 identifies the type of work performed by the worker based on the behavior of the worker identified from the work video (step S11). Then, the work type search means 23 searches and acquires the ideal work procedure corresponding to the identified work type from the ideal work procedure storage device 22 (step S12). Using this acquired ideal work procedure, comparison with the action order list is performed.

Thereafter, the processes from step S3 to step S5 illustrated in FIG. 2 are performed. Note that the process of step S3 for generating the action order list may be performed before the process of step S11 or step S12.

As described above, in this embodiment, the work type search means 23 identifies the work type based on the behavior of the worker identified from the work video, and determines the ideal work procedure corresponding to the identified work type. Acquired from the ideal work procedure storage device 22 . Therefore, in addition to the effects of the first embodiment, it is possible to perform abnormality detection for a plurality of types of work with the same device.

[Example]
Next, a specific example of a scene to which the action order abnormality detection system of each embodiment is applied will be described. The action order abnormality detection system of the above embodiment can be used in an actual factory, for example. When a fixed-point camera installed in the factory photographs the work process of the worker, the image input means 1 inputs the photographed work image.

Next, the action identification means 211 performs a predetermined action such as "taking a specific part" or "assembling a part" for all frames of the captured work video. Identify which action applies to you. Next, the action order identification means 212 converts the identification result into an action order list.

Finally, the action order comparison means 213 compares the action order list and the ideal work procedure, and determines how many actions are replaced, missing actions, or added extra work in the work to be identified. Output what happened.

Next, the outline of the present invention will be explained. FIG. 6 is a block diagram showing an outline of an action sequence abnormality detection device according to the present invention. An action order abnormality detection device 80 (for example, action order abnormality detection device 21) according to the present invention is input with video input means 81 (for example, video input means 1) for inputting a work video in which a work process by a worker is photographed. When the action identification means 82 (for example, the action identification means 211) that identifies the action of the worker from the work video obtained and the action of the identified worker are traced in chronological order, and a change in the action of the worker is detected , the action order identification means 83 (for example, the action order identification means 212) that adds the action before the change to the action order list, and the ideal work procedure that defines the action order according to the work and the action order list. and action order comparison means 84 (for example, action order comparison means 213) for detecting abnormal work.

With such a configuration, it is possible to detect anomalies in the action order of workers.

Specifically, the action identification means 82 identifies the actions of the worker on a frame-by-frame basis from the work video, and the action order identification means 83 chronologically identifies the actions of the identified worker from the first frame of the work video. If a change in behavior is detected before and after the frame, the behavior before the change may be added to the behavior order list.

Also, the action order comparison means 84 may output the result of detecting abnormal work based on the edit distance between the action order list and the ideal work procedure.

Specifically, the action order comparison means 84 calculates an edit distance from a series of actions included in the action order list to a series of actions included in the ideal work procedure. may be output.

At that time, the action order comparison means 84 outputs the edit distance indicating the number of times the action is replaced as the number of times the action is replaced, outputs the edit distance indicating the number of times the action is added as the number of times the action is omitted, and indicates the number of times the action is deleted. The edit distance may be output as the number of extra actions added.

In addition, the behavior sequence abnormality detection device 80 identifies the type of work performed by the worker based on the behavior of the worker identified from the work video, and determines the ideal work procedure corresponding to the identified type of work. from a storage device (eg ideal work procedure storage device 22).

FIG. 7 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. A computer 1000 comprises a processor 1001 , a main storage device 1002 , an auxiliary storage device 1003 and an interface 1004 .

Each of the above-described action order abnormality detection devices 80 is implemented in the computer 1000 . The operation of each processing unit described above is stored in the auxiliary storage device 1003 in the form of a program. The processor 1001 reads out the program from the auxiliary storage device 1003, develops it in the main storage device 1002, and executes the above processing according to the program.

Note that in at least one embodiment, the secondary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read-only memory), DVD-ROMs (Read-only memory), connected via interface 1004, A semiconductor memory etc. are mentioned. Further, when this program is distributed to the computer 1000 via a communication line, the computer 1000 receiving the distribution may develop the program in the main storage device 1002 and execute the above process.

In addition, the program may be for realizing part of the functions described above. Further, the program may be a so-called difference file (difference program) that implements the above-described functions in combination with another program already stored in the auxiliary storage device 1003 .

Some or all of the above embodiments can also be described as the following additional remarks, but are not limited to the following.

(Additional remark 1) video input means for inputting a work video in which a work process by a worker is filmed;
Action identification means for identifying actions of the worker from the input work video;
an action order identification means for tracing the identified actions of the worker in chronological order and, when a change in the action of the worker is detected, adding the action before the change to an action order list;
An action order abnormality detecting device, comprising action order comparison means for detecting an abnormal work by comparing an ideal work procedure defining an action order according to work with the action order list.

(Appendix 2) The action identification means identifies the action of the worker in units of frames from the work video,
The action sequence identifying means traces the identified actions of the worker in chronological order from the first frame of the work video, and when a change in action is detected before or after the frame, adds the action before the change to the action order list. The action order abnormality detection device according to appendix 1.

(Supplementary Note 3) The action order abnormality detection device according to Supplementary Note 1 or Supplementary Note 2, wherein the action order comparison means outputs a result of detecting an abnormal work based on an edit distance between the action order list and the ideal work procedure.

(Appendix 4) The action order comparison means outputs a result of detecting abnormal work by calculating an edit distance from a series of actions included in the action order list to a series of actions included in the ideal work procedure. The action sequence abnormality detection device according to appendix 3.

(Appendix 5) The action order comparison means outputs an edit distance indicating the number of times an action is replaced as the number of times an action is replaced, outputs an edit distance indicating the number of times an action is added as the number of times an action is omitted, and indicates the number of times an action is deleted. The action sequence abnormality detection device according to appendix 3 or appendix 4, wherein the edit distance is output as the number of additions of extra actions.

(Appendix 6) Based on the behavior of the worker identified from the work video, the type of work performed by the worker is identified, and the ideal work procedure corresponding to the identified type of work is acquired from the storage device. The action sequence abnormality detection device according to any one of appendices 1 to 5, comprising work type search means.

(Appendix 7) Input the work video of the work process by the worker,
Identifying the behavior of the worker from the input work video,
When the behavior of the identified worker is traced in chronological order and a change in the behavior of the worker is detected, the behavior before the change is added to the behavior order list,
An action order abnormality detection method, comprising: comparing an ideal work procedure in which an order of actions is defined according to work and the action order list to detect an abnormal work.

(Appendix 8) Identifying the action of the worker frame by frame from the work video,
The behavior of the identified worker is traced in time series from the first frame of the work video, and when a change in behavior is detected before and after the frame, the behavior before the change is added to the behavior order list. Behavior order anomaly detection method.

(Appendix 9) to the computer,
Video input processing for inputting work video that captures the work process by the worker,
Action identification processing for identifying actions of the worker from the input work video,
an action order identification process for tracing the identified actions of the worker in chronological order and, when detecting a change in the action of the worker, adding the action before the change to an action order list;
A program storage medium storing an action order abnormality detection program for executing an action order comparison process for detecting an abnormal work by comparing an ideal work procedure defining an action order according to work with the action order list.

(Appendix 10) to the computer,
In the action identification process, the action of the worker is identified frame by frame from the work video,
In the action order identification process, the actions of the identified worker are traced in chronological order from the first frame of the work video, and when a change in action is detected before or after the frame, the action before the change is added to the action order list. The program storage medium according to Supplementary Note 9, which stores an action order abnormality detection program for adding to.

(Appendix 11) to the computer,
Video input processing for inputting work video that captures the work process by the worker,
Action identification processing for identifying actions of the worker from the input work video,
an action order identification process for tracing the identified actions of the worker in chronological order and, when detecting a change in the action of the worker, adding the action before the change to an action order list;
An action order abnormality detection program for executing an action order comparison process for detecting an abnormal work by comparing an ideal work procedure in which an action order is defined according to work and the action order list.

(Appendix 12) to the computer,
In the action identification process, the action of the worker is identified frame by frame from the work video,
In the action order identification process, the actions of the identified worker are traced in time series from the first frame of the work video, and when a change in action is detected before or after the frame, the actions before the change are added to the action order list. Behavior sequence abnormality detection program according to appendix 11.

Although the present invention has been described with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

The present invention is suitably applied to an action sequence abnormality detection device that detects an abnormality in the order of actions by a worker. Specifically, the present invention can be used to analyze the work content of assembly work processes in the manufacturing industry, such as which work processes tend to take more time and which work processes are more likely to be mixed up. In addition, the present invention can also be applied to applications such as confirming whether or not important processes leading to product quality are being carried out without excess or deficiency in work management in the construction industry.

1 Video Input Means 3 Output Device 21 Action Order Abnormality Detection Device 22 Ideal Work

Procedure Storage Device

100, 200 Action Order Abnormality Detection System 211 Action Identification Means 212 Action Order Identification Means 213 Action Order Comparison Means 23 Work Type Search Means

Claims

a video input means for inputting a work video in which a work process by a worker is filmed;
Action identification means for identifying actions of the worker from the input work video;
an action order identification means for tracing the identified actions of the worker in chronological order and, when a change in the action of the worker is detected, adding the action before the change to an action order list;
An action order abnormality detection device, comprising action order comparison means for detecting an abnormal work by comparing an ideal work procedure defining an action order according to work with the action order list.
The action identification means identifies the action of the worker frame by frame from the work video,
The action sequence identifying means traces the identified actions of the worker in chronological order from the first frame of the work video, and when a change in action is detected before or after the frame, adds the action before the change to the action order list. The action sequence abnormality detection device according to claim 1.
3. The action order abnormality detection device according to claim 1, wherein the action order comparison means outputs a result of detecting an abnormal work based on an edit distance between the action order list and the ideal work procedure.
3. The action order comparison means outputs a result of detecting abnormal work by calculating an edit distance from a series of actions included in the action order list to a series of actions included in the ideal work procedure. Behavior sequence anomaly detection device as described.
The action order comparison means outputs an edit distance indicating the number of times the action is replaced as the number of times the action is replaced, outputs an edit distance indicating the number of times the action is added as the number of times the action is omitted, and outputs an edit distance indicating the number of times the action is deleted. 5. The work sequence abnormality detection device according to claim 3, wherein the number of additional actions is output.
Work type search means for identifying the type of work performed by the worker based on the behavior of the worker identified from the work video, and acquiring the ideal work procedure corresponding to the identified work type from the storage device. The action sequence abnormality detection device according to any one of claims 1 to 5, comprising:
Input a work video that captures the work process by the worker,
Identifying the behavior of the worker from the input work video,
When the behavior of the identified worker is traced in chronological order and a change in the behavior of the worker is detected, the behavior before the change is added to the behavior order list,
An action order abnormality detection method, comprising: comparing an ideal work procedure in which an order of actions is defined according to work and the action order list to detect an abnormal work.
Recognize the actions of the worker frame by frame from the work video,
8. The behavior of the identified worker is traced in time series from the first frame of the work video, and when a change in behavior is detected before and after the frame, the behavior before the change is added to the behavior order list. behavior sequence anomaly detection method.
to the computer,
Video input processing for inputting work video that captures the work process by the worker,
Action identification processing for identifying actions of the worker from the input work video,
an action order identification process for tracing the identified actions of the worker in chronological order and, when detecting a change in the action of the worker, adding the action before the change to an action order list;
A program storage medium storing an action order abnormality detection program for executing an action order comparison process for detecting an abnormal work by comparing an ideal work procedure defining an action order according to work with the action order list.
to the computer,
In the action identification process, the action of the worker is identified frame by frame from the work video,
In the action order identification process, the actions of the identified worker are traced in chronological order from the first frame of the work video, and when a change in action is detected before or after the frame, the action before the change is added to the action order list. 10. The program storage medium according to claim 9, which stores an action order abnormality detection program to be added to.